Event Driven Systems rely on sending and receiving messages.
These messages are like little cars speeding across network highways. The better the highways, the more efficient the flow of traffic.
Hello š and welcome to a new edition of theĀ Cloud & Backend newsletter.
In this post, I will talk about 3 main strategies for sending messages using Kafka.
Weāve talked a lot about messages and event-driven systems recently. Check out the below post to catch up:
In this post, we go a bit more into the technical side of things considering Kafka as our message broker.
Why Kafka?
Because itās extremely popular and widely used. Some of the things Kafka does is considered a pretty good standard for the overall ecosystem.
Hereās the full agenda for this post:
šĀ A Quick Look at the Kafka Producer
šĀ 3 Kafka Message Sending Strategies
šŖWhen to Use Which Strategy?
šĀ How Does the Kafka Producer Work?
In the first step, we create a ProducerRecord
.
The record contains two mandatory items:
Kafka Topic (the place where we want to send the message)
The Message (what we want to actually send)
Once we create a ProducerRecord
in our application code, we call the send method. This is where the Producer kicks into action and executes a bunch of stuff:
Serialize the message (key and value objects) to byte arrays to send the data over the network.
Choose a partitioner in case the sender hasnāt specified one. This is usually based on the key of the message.
Batch the records for a particular topic and partition.
Send the batch to the right Kafka broker.
Hereās what it looks like:
So, what happens when the broker receives the message?
It sends back a response to the Producer. Think of it as an acknowledgment. More precisely, itās the RecordMetadata
object.
The object contains information such as topic, partition, and the offset of the record within the partition.
But thatās the happy path!
What happens if the broker fails to write the message for whatever reason?
In that case, the Producer gets an error and can retry sending the message a few more times before giving up and calling it a day.
So, whatās the deal with the 3 message-sending strategies?
Letās find out.
šĀ The 3 Strategies to Send Messages
As mentioned earlier, there are 3 main strategies you can use to send messages using Kafka.
Fire and Forget
Synchronous Send
Asynchronous Send
šĀ Fire and Forget
In this strategy, we send a message to the Kafka broker and forget about it. We simply donāt care what happens to it.
Since Kafka is highly available, it will likely arrive on the other side successfully. In case of a minor issue, the Producer will retry sending the message automatically.
But yes, fire and forget means that messages can and will get lost. Also, the application wonāt get any information or exceptions about these lost messages.
This is what it looks like in code:
ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "kafka trial message");
try {
producer.send(record);
} catch(Exception e) {
e.printStackTrace();
}
šĀ Synchronous Send Approach
The strategy sounds dubious.
Why would someone want a messaging system to have synchronous behavior?
But Kafka lets you force a Producer to behave in a synchronous manner.
Hereās the code for the same:
ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "kafka trial message");
try {
producer.send(record).get();
} catch(Exception e) {
e.printStackTrace();
}
Whatās the difference?
Nothing much!
We have simply turned the send method to a synchronous operation by calling producer.send(record).get()
.
Basically, theĀ send()
method returns aĀ Future
object. By using theĀ get()
method, we wait on theĀ Future
object to make sure if the call toĀ send()
was successful or not.
If not, the method throws an exception.
šĀ Asynchronous Send
This is the third approach.
Here you call theĀ send()
method of theĀ Producer with a callback function. The callback function gets triggered when it receives a response from the Kafka broker.
ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "new kafka async");
try {
producer.send(record, new DemoProducerCallback());
} catch(Exception e) {
e.printStackTrace();
}
The DemoProducerCallback
is the callback class. The send()
method takes an instance of that class.
šŖĀ When to Use Which Strategy?
Here are a few pointers:
šĀ In my experience, Asynchronous Send is what youād be using a majority of the time.
This strategy lets you send messages at high performance while also managing error situations and exceptions if any.
Thatās what you want in a typical production system for any important domain requirement.
šĀ Try to avoid Synchronous Send. It sounds safer, but you are essentially making a trade-off on performance.
Brokers in a typical Kafka cluster may take some time to respond to requests. With this strategy, you block the sending thread of your application. Not good for performance.
šĀ What about Fire and Forget?
In my opinion, it has its uses. Since it is the best-performing in terms of throughput, you should use it for messages where delivery isnāt extremely critical.
Think of logs or sensor information where missing a few messages here and there isnāt a huge problem.
ā½ Over to you
Are you using event-driven systems?
If yes, which strategy would you prefer?
Write your replies or thoughts in the comments section.
And thatās all for this post.
If you found todayās post useful, give it a like and consider sharing it with friends and colleagues.
Wishing you a great week ahead! āļø
See you later.