Why and When Use Apache Kafka for Event-Driven Systems

In this post, we’ll explore why Apache Kafka is so integral to modern event-driven systems, explain the basics of event-driven architectures and pub/sub-models, discuss when to use a distributed event streaming platform (and when not to), illustrate event-driven microservices, and more.

What Is an Event-Driven System?

An event-driven system revolves around the production, detection, and reaction to events. An “event” can be anything from a user clicking a button on a website to a new reading from a temperature sensor in an IoT deployment.

Classic Event-Driven Examples

• User Interaction: Submitting a form on a website triggers an event that can notify other services (e.g., sending a verification email).
• IoT Sensor Data: A temperature sensor publishes data whenever it detects a change, triggering a workflow that checks thresholds or triggers alarms.
• Messaging Apps: Chat applications deliver new messages as soon as they occur, notifying all relevant users or channels in real-time.
• E-commerce Systems: When an order is placed, multiple downstream services are triggered, such as payment, fraud detection, inventory update, shipping, and user notifications.

Pub/Sub Architecture

A publish/subscribe (pub/sub) model is a foundational pattern in event-driven architectures. Publishers produce events, and subscribers consume them. A broker mediates the communication, ensuring events are reliably delivered to subscribers.

Apache Kafka Pub/Sub arhitecture. Souce: link

In Apache Kafka terminology:

1. Producers: Applications (or services) that publish (write) events to Kafka.
2. Brokers: Kafka servers that receive events from producers, store them in a fault-tolerant way and make them available to consumers.
3. Topics: Named streams of related events. You can think of a topic as a category or feed name to which events are published.
4. Consumers: Applications (or services) that subscribe (read) to events from Kafka topics.

Brokers ensure the continuous and decoupled transmission of messages

When to Use a Distributed Event Streaming Platform

A distributed event streaming platform, like Apache Kafka, is designed for high throughput, low latency, and fault tolerance. Such a platform excels in scenarios where:

1. High Data Volume: If you need to handle millions or billions of daily events.
2. Real-Time Processing: If your system must process and react to events in near real-time.
3. Decoupled Microservices: Multiple independent services need a robust way to publish and consume data across the organization.
4. Enterprise Data Pipelines: Large-scale data transformations or pipelines where data must flow reliably from sources (databases, logs, applications) to sinks (analytics systems, data warehouses).

When a Simpler Solution Might Suffice

Not every system needs the complexity of a distributed platform. If your application:

• It has relatively low event throughput,
• Operates in a monolithic design with minimal real-time constraints,
• Consumes or produces just a few messages daily or weekly,

Do not deploy Apache kafka in overly simple scenarios.

A simple in-memory queue or a lightweight messaging system might be sufficient. Deploying Kafka can be overkill if you’re dealing with a handful of events per day with no strict latency or availability requirements. Over-engineering can lead to unnecessary complexity, so consider your needs carefully.

Message pipeline

Brief explanation of Apache Kafka message pipeline. Source: link

Event-Driven Microservices

In a microservices architecture, each service handles a specific functional responsibility. Microservices can be tied together with synchronous APIs, but often, a better approach is to decouple them with an event-driven flow.

E-Commerce Use Case

Imagine an e-commerce scenario:

1. Order Service: Publishes an event “order placed” to Kafka.
2. Payment Service: Subscribes to “order placed” events, processes payments, and can then publish a “payment approved” event.
3. Fraud Detection Service: Subscribes to the same “order placed” event to quickly check for fraud, potentially publishing a “fraud detected” or “fraud cleared” event.
4. Inventory Service: Subscribes to “order placed” (or “payment approved”) events to update product stock.
5. Shipping Service: Subscribes to “order ready for shipping” events to manage delivery.

Services remain loosely coupled by using Kafka as a backbone for these events. You gain:

• Scalability: Each service can be scaled independently based on the event load.
• Resilience: If a service goes down, events are still stored in Kafka, so the service can resume processing upon recovery.
• Flexibility: Adding new services (e.g., a recommendation service) without modifying existing producer or consumer logic.

Compared to a classical “point-to-point” integration or synchronous HTTP-driven approach, an event-driven pipeline with Kafka reduces direct dependencies and increases fault tolerance.

Dealing with message streams and queues

Kafka topics can function similarly to queues or to message streams depending on how consumer groups are configured:

1. Topic as Broadcast Stream: If multiple consumers read from a topic with different consumer group IDs, each consumer will receive every message. This is akin to a “pub/sub” fan-out pattern.
2. Topic as a Queue: If multiple consumers share the same consumer group ID, Kafka load balances the messages among them. This is similar to a queue, where exactly one consumer consumes each message within the group.

This versatility allows you to design the data flow to match different use cases, whether you need to fan out events to many consumers or divide the workload among them.

Low Latency and High Availability

Apache Kafka is built for speed, scalability, and reliability:

• Speed: Kafka can handle high throughput (millions of messages per second) with latencies often in milliseconds, depending on configuration and infrastructure.
• Scalability: You can quickly scale out Kafka by adding more brokers and partitioning topics across them. Both producers and consumers can scale independently.
• Fault Tolerance: Kafka replicates data across multiple brokers. If one broker fails, another broker with the replicated partition takes over. This mechanism provides high availability and durability.

In some low-latency applications the message must be delivered in the shortest possible time

Durability

Kafka’s durability ensures that once a message is written to the broker and acknowledged, it is persisted on disk and replicated across brokers. This is especially important in scenarios where:

• Financial Transactions: You can’t afford to lose orders, payments, or billing events.
• Compliance: Regulatory requirements often demand an auditable log of all events.
• Critical System Logs: System behavior and error logs must be safely retained for diagnostics and analytics.

However, Kafka isn’t just a bottomless pit of data — retention policies govern how long messages are stored:

1. Time-Based Retention (e.g., 7 days): Messages stay on the broker for a set duration before they are removed.
2. Size-Based Retention: Messages are retained until log segments reach a certain size limit, after which older segments are discarded.
3. Log Compaction: An alternative or complementary strategy that keeps only the latest value per key, removing older versions. This is useful for scenarios where you need a “current state” for each key without retaining every historical event.

Integrations (Kafka Connectors)

Kafka Connect is a framework that simplifies integration with other systems. Common source connectors bring data into Kafka, while sink connectors write data from Kafka to external targets.

• Source Connectors: Databases (e.g., JDBC connectors for MySQL, PostgreSQL), messaging systems, filesystems, or IoT sensors.
• Sink Connectors: Data warehouses (e.g., Snowflake, BigQuery), NoSQL stores (e.g., MongoDB), search engines (e.g., Elasticsearch), and more.

You can rapidly build pipelines by leveraging ready-made connectors without writing and maintaining complicated custom code. This helps you quickly integrate disparate systems into a unified event-driven architecture.

Apache Kafka Community vs. Confluent Enterprise

Apache Kafka is open-source and managed by the Apache Software Foundation. You can download and run Kafka freely. However, Confluent — founded by Kafka’s original creators — offers an enterprise platform that includes:

• Enhanced Connectors: Expanded library of fully tested and supported connectors.
• Schema Registry: Simplifies schema evolution and ensures data compatibility between producers and consumers.
• Control Center: Centralized management and monitoring of Kafka clusters.
• Security & Governance: Additional security features, RBAC, and audit tools are especially important in enterprise environments.
• Fully Managed Services: Confluent Cloud for an “out-of-the-box” managed Kafka experience.

If you need strong SLAs, advanced features, and professional support, Confluent’s enterprise offerings might be the right choice. The Apache Kafka community edition might be enough if you’re starting or have a smaller-scale use case.

Apache Kafka vs. Alternatives

Although Kafka is a leading choice, there are other messaging and event streaming options:

• RabbitMQ: A traditional message queue system. It excels at complex routing scenarios (e.g., topic exchanges, direct exchanges) but typically isn’t as good at high-throughput event streaming as Kafka.
• Apache Pulsar: Another distributed messaging system offering both streaming and queue semantics. It uses a segmented architecture and sometimes offers simpler multi-tenancy and geo-replication features.
• AWS Kinesis: A fully managed service in Amazon Web Services that provides streaming functionality similar to Kafka with out-of-the-box hosting but is cloud-provider-specific.
• Azure Event Hubs: Microsoft Azure’s alternative for large-scale event ingestion and streaming is also fully managed on the Azure platform.

Conclusion

Modern applications require robust, scalable, and flexible ways to handle real-time events at scale — and Apache Kafka rises to that challenge. Whether you’re building microservices, processing large-scale data pipelines, or need a reliable pub/sub messaging system, Kafka’s speed, durability, and vast ecosystem of connectors make it a powerful and popular choice. By carefully considering your use cases, balancing complexity against requirements, and leveraging Kafka where it fits best, you can modernize your infrastructure and keep pace with today’s demanding applications.