3 Messaging Strategies for Kafka

Cloud & Backend EP 11 - How to choose the right strategy?

Event Driven Systems rely on sending and receiving messages.

These messages are like little cars speeding across network highways. The better the highways, the more efficient the flow of traffic.

Hello 👋 and welcome to a new edition of the Cloud & Backend newsletter.

In this post, I will talk about 3 main strategies for sending messages using Kafka.

We’ve talked a lot about messages and event-driven systems recently. Check out the below post to catch up:

How to Keep Track of Duplicate Messages?

In this post, we go a bit more into the technical side of things considering Kafka as our message broker.

Why Kafka?

Because it’s extremely popular and widely used. Some of the things Kafka does is considered a pretty good standard for the overall ecosystem.

Here’s the full agenda for this post:

• 🏀 A Quick Look at the Kafka Producer

• 🚀 3 Kafka Message Sending Strategies

• 🪁When to Use Which Strategy?

🏀 How Does the Kafka Producer Work?

In the first step, we create a ProducerRecord.

The record contains two mandatory items:

• Kafka Topic (the place where we want to send the message)

• The Message (what we want to actually send)

Once we create a ProducerRecord in our application code, we call the send method. This is where the Producer kicks into action and executes a bunch of stuff:

• Serialize the message (key and value objects) to byte arrays to send the data over the network.

• Choose a partitioner in case the sender hasn’t specified one. This is usually based on the key of the message.

• Batch the records for a particular topic and partition.

• Send the batch to the right Kafka broker.

Here’s what it looks like:

How Does Kafka Producer Work?

So, what happens when the broker receives the message?

It sends back a response to the Producer. Think of it as an acknowledgment. More precisely, it’s the RecordMetadata object.

The object contains information such as topic, partition, and the offset of the record within the partition.

But that’s the happy path!

What happens if the broker fails to write the message for whatever reason?

In that case, the Producer gets an error and can retry sending the message a few more times before giving up and calling it a day.

So, what’s the deal with the 3 message-sending strategies?

Let’s find out.

🚀 The 3 Strategies to Send Messages

As mentioned earlier, there are 3 main strategies you can use to send messages using Kafka.

• Fire and Forget

• Synchronous Send

• Asynchronous Send

👉 Fire and Forget

In this strategy, we send a message to the Kafka broker and forget about it. We simply don’t care what happens to it.

Since Kafka is highly available, it will likely arrive on the other side successfully. In case of a minor issue, the Producer will retry sending the message automatically.

But yes, fire and forget means that messages can and will get lost. Also, the application won’t get any information or exceptions about these lost messages.

This is what it looks like in code:

ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "kafka trial message");

try {
     producer.send(record);
} catch(Exception e) {
     e.printStackTrace();
}

👉 Synchronous Send Approach

The strategy sounds dubious.

Why would someone want a messaging system to have synchronous behavior?

But Kafka lets you force a Producer to behave in a synchronous manner.

Here’s the code for the same:

ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "kafka trial message");

try {
     producer.send(record).get();
} catch(Exception e) {
     e.printStackTrace();
}

What’s the difference?

Nothing much!

We have simply turned the send method to a synchronous operation by calling producer.send(record).get().

Basically, the send() method returns a Future object. By using the get() method, we wait on the Future object to make sure if the call to send() was successful or not.

If not, the method throws an exception.

👉 Asynchronous Send

This is the third approach.

Here you call the send() method of the Producer with a callback function. The callback function gets triggered when it receives a response from the Kafka broker.

ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "new kafka async");

try {
		producer.send(record, new DemoProducerCallback());
} catch(Exception e) {
		e.printStackTrace();
}

The DemoProducerCallback is the callback class. The send() method takes an instance of that class.

🪁 When to Use Which Strategy?

Here are a few pointers:

👉 In my experience, Asynchronous Send is what you’d be using a majority of the time.

This strategy lets you send messages at high performance while also managing error situations and exceptions if any.

That’s what you want in a typical production system for any important domain requirement.

👉 Try to avoid Synchronous Send. It sounds safer, but you are essentially making a trade-off on performance.

Brokers in a typical Kafka cluster may take some time to respond to requests. With this strategy, you block the sending thread of your application. Not good for performance.

👉 What about Fire and Forget?

In my opinion, it has its uses. Since it is the best-performing in terms of throughput, you should use it for messages where delivery isn’t extremely critical.

Think of logs or sensor information where missing a few messages here and there isn’t a huge problem.

⚽ Over to you

• Are you using event-driven systems?

• If yes, which strategy would you prefer?

Write your replies or thoughts in the comments section.

---

And that’s all for this post.

If you found today’s post useful, give it a like and consider sharing it with friends and colleagues.

Wishing you a great week ahead! ☀️

See you later.