In this latest project spotlight, we'll look at SharpPulsar, a .NET Client written on top of Akka.NET to interact with the popular Apache Pulsar messaging platform. By the end of the post, you'll better understand what Apache Pulsar is, when to use it, and how to use SharpPulsar, the .NET client, to improve your existing applications. You'll also see an example you can run on your local development environment.
In the simplest terms, Apache Pulsar is a messaging platform, but that doesn't begin to scratch the surface of what it can do. For .NET developers, it is an alternative to RabbitMQ, Azure Service Bus, or even MSMQ. Here's how the Pulsar website describes Pulsar to potential users.
Apache Pulsar is an all-in-one messaging and streaming platform. Messages can be consumed and acknowledged individually or as streams with less than 10ms of latency. Its layered architecture allows rapid scaling across hundreds of nodes without data reshuffling.
Its features include multi-tenancy with resource separation and access control, geo-replication across regions, tiered storage, and support for five official client languages. It supports up to one million unique topics and is designed to simplify your application architecture.
Pulsar is a Top 10 Apache Software Foundation project with a vibrant and passionate community and user base spanning small companies and large enterprises.
Before adding a new dependency to your application's dependency graph, it's important to understand why you might want to add it. Here are the main advantages of Pulsar.
Producers publish messages to topics, and consumers wait for messages to be pushed to them by Brokers. Brokers orchestrate receiving messages by acknowledging and persisting them and then notifying subscribers of the messages. Conceptually, It's similar to other messaging systems such as RabbitMQ, MSMQ, or Azure Service Bus.
Messaging systems are typically used in all industries, but systems that operate at an enormous scale or deal with spikes in requests benefit the most from using Pulsar.
Industries that may benefit from Pulsar include, but are not limited to:
If you have bursts of activity in your system, using Pulsar might help you scale to meet the temporary increases in demand. Building a system on Pulsar might also help you deliver a more consistent user experience regardless of on or off-peak hours.
The most straightforward way to start using Pulsar is by running a container. To start, run the following command to pull the docker image and run it.
docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:3.0.0 bin/pulsar standalone
Afterward, you should have a Pulsar instance running in your development environment.
Before jumping into a code example, I mentioned that SharpPulsar was built on top of Akka.Net. Let's first understand what Akka.Net is and why it might be helpful to use it as a foundation for another OSS project.
Akka.NET is a toolkit and runtime for building highly concurrent, distributed, and fault-tolerant event-driven applications for .NET. The library is based on the Actor model, which separates part of your application logic into specialized message handler units called actors. These units communicate by passing messages back and forth through mailboxes. For the health and reliability of most systems, it's a good idea to persist these messages somewhere.
Given the communication models of both Akka.NET and Pulsar are similar, it's a natural choice to build a Pulsar client on top of an already well-trusted and reliable part of the .NET ecosystem. Developers have benchmarked Akka.NET performance at 50 million messages/second on a single machine with approximately 2.5 million actors (or Apache Pulsar Producers/Consumers) per gigabyte of heap. That's a fantastic amount of potential to scale any modern application.
OK, let's build a sample application.
Once you've created your Pulsar instance from the previous Docker command, paste the following C# code into an existing console application's Program.cs file.
using System.Text;
using SharpPulsar;
using SharpPulsar.Builder;
using SharpPulsar.Protocol.Extension;
var clientConfig = new PulsarClientConfigBuilder()
.ServiceUrl("pulsar://localhost:6650");
//pulsar actor system
var pulsarSystem = PulsarSystem.GetInstance(actorSystemName: nameof(Program));
var pulsarClient = await pulsarSystem.NewClient(clientConfig);
const string myTopic = nameof(myTopic);
var consumer = await pulsarClient
.NewConsumerAsync(new ConsumerConfigBuilder<byte[]>()
.Topic(myTopic)
.ForceTopicCreation(true)
.SubscriptionName($"sub-{Guid.NewGuid()}")
.IsAckReceiptEnabled(true));
var producer = await pulsarClient
.NewProducerAsync(new ProducerConfigBuilder<byte[]>()
.Topic(myTopic));
for (var i = 0; i < 10; i++)
{
var data = Encoding.UTF8.GetBytes($"tuts-{i}").ToByteArrays();
producer.NewMessage().Value(data).Send();
}
await Task.Delay(TimeSpan.FromSeconds(5));
for (var i = 0; i < 10; i++)
{
var message = (Message<byte[]>)await consumer.ReceiveAsync();
if (message != null)
{
await consumer.AcknowledgeAsync(message);
var res = Encoding.UTF8.GetString(message.Data).Trim();
Console.WriteLine($"message '{res}' from topic: {message.Topic}");
}
}
The example shows how you can create a producer and consumer of messages. In this case, the message is a string serialized into a byte[] for more efficient transport. The consumer converts the byte array into a string and writes the result to the console output.
message 'tuts-0' from topic: persistent://public/default/myTopic
message 'tuts-1' from topic: persistent://public/default/myTopic
message 'tuts-2' from topic: persistent://public/default/myTopic
message 'tuts-3' from topic: persistent://public/default/myTopic
message 'tuts-4' from topic: persistent://public/default/myTopic
message 'tuts-5' from topic: persistent://public/default/myTopic
message 'tuts-6' from topic: persistent://public/default/myTopic
message 'tuts-7' from topic: persistent://public/default/myTopic
message 'tuts-8' from topic: persistent://public/default/myTopic
message 'tuts-9' from topic: persistent://public/default/myTopic
Awesome! Also, be sure to check out SharpPulsar tutorials for more exciting scenarios.
If you're looking for a way to scale your solutions, SharpPulsar is worth your time and attention. It helps .NET developers leverage the strength of Pulsar, one of the more famous Apache projects. Additionally, the authors have chosen to build their solution on top of another popular .NET Foundation project, Akka.NET, which means you get the added benefit of two projects. Also, as a note, the authors of SharpPulsar are always looking for feedback to improve the user experience, so be sure to check out the SharpPulsar GitHub repository to see how you can get involved.