Kafka in a Microservices Architecture

Most confusion about Kafka comes from reading it as a message queue with better throughput. It is not one, and the difference is not a detail — nearly every pattern it enables and every sharp edge it has follows from this single property.

It is a log, not a queue

In a queue, a message is delivered to one consumer and then it is gone. The queue's job is to hold work until somebody does it, and its natural state is empty.

Kafka is an append-only log. Records are written to the end, assigned an offset, and nothing is removed when it is read. Records leave because a retention policy expired them, not because a consumer took them. Each consumer tracks its own position in the log.

Three consequences, and they are the entire value proposition:

Many consumers, one stream. Billing, search indexing, analytics and notifications all read the same orders topic without knowing about each other, each at its own pace.
A slow consumer does not block anyone. It falls behind and catches up. In a queue with competing consumers, one stuck worker is everyone's problem.
New consumers can read history. A service written next quarter can start from the beginning of retention and build its state from what already happened. This is the capability nothing else in the stack gives you cheaply.

The primitives that matter

There are only a handful, and getting them wrong causes most production problems.

Topics are split into partitions, and ordering is per-partition only. There is no such thing as topic-wide ordering. If you need two events to be processed in order, they must land in the same partition.

The record key chooses the partition, by hash. So choosing your key is choosing your ordering domain, and it is the most consequential decision you will make:

await producer.send({
  topic: "orders",
  messages: [{
    key: order.id,          // all events for this order → same partition
    value: JSON.stringify(event),
  }],
});

Key by order_id and every event for one order is ordered relative to the others, while different orders process in parallel. Key by tenant_id instead and you get ordering across a whole tenant, at the cost of that tenant's throughput being limited to one partition — which is how a single large customer ends up creating a hot partition that lags while the rest of the cluster idles.

Consumer groups are how you scale out. Within a group, each partition is assigned to exactly one consumer. That means your parallelism ceiling is the partition count: ten partitions will never use more than ten consumers, and the eleventh sits idle. Partition count is a capacity decision made in advance.

Offsets are the consumer's position, and committing them is your call. Commit after the work succeeds and a crash means reprocessing — at-least-once. Commit before, and a crash means the record is silently skipped. Choose at-least-once and make your handlers idempotent; the alternative is losing data quietly, which is far worse than doing something twice.

Retention decides what the log is for. Time or size retention suits event streams. Log compaction — keeping only the latest record per key, forever — turns a topic into something quite different: a durable, replayable snapshot of current state per entity, which a new service can consume from the start to build its own local view.

What it actually fixes in a microservices architecture

The problem Kafka solves is not throughput for most teams. It is runtime coupling.

In a synchronous architecture, the checkout service calls inventory, which calls pricing, which calls the tax service. Every one of those is a chance to fail, they add up in latency, and the availability of checkout is the product of the availability of everything downstream. Add a service and you have added another way for checkout to break.

With a log in the middle, the producer writes an event and is finished. It does not know who consumes it, does not wait, and does not fail when a consumer is down. Consumers that were offline catch up from their last offset. Adding a capability becomes adding a consumer — with no change to the producing service at all, which is the property that actually makes independent deployment work.

The cost is honest and worth stating: you have traded synchronous failures you can see for asynchronous ones you have to look for. Nothing errors when a consumer silently stops processing. It just falls behind, and only lag monitoring will tell you.

The patterns worth knowing

Event notification. A thin event says something happened and carries identifiers. Consumers call back for details. Small messages, no stale data, but it reintroduces the synchronous call you were trying to avoid.

Event-carried state transfer. The event carries enough data for consumers to act without calling back, and they keep a local read model. This is what actually removes runtime coupling: the shipping service holds the customer address it needs and keeps serving when the customer service is down. The cost is duplicated data that is briefly stale, and a schema contract that is now genuinely public.

Compacted state topics. Publish the current state of each entity keyed by its id to a compacted topic. Any service can consume the topic from offset zero and materialise a complete local view of every entity, then keep it current from the same stream. This is how a new service bootstraps without a migration script or a bulk export.

Sagas for cross-service transactions. There is no distributed transaction here. A saga is a sequence of local transactions, each publishing an event that triggers the next, with a compensating action for each step if a later one fails: payment captured, inventory reserved, shipment created — and if the shipment fails, a refund event rather than a rollback. The discipline is that every step needs its compensation designed at the same time, not afterwards.

The outbox, to avoid the dual write. Writing to your database and publishing to Kafka are two systems and cannot be made atomic by hoping. Insert the event into an outbox table in the same transaction as the state change, and relay that table to Kafka. The event then exists if and only if the transaction committed.

The hard parts

Adding partitions breaks key ordering. Partition assignment is a hash over the partition count, so increasing it re-maps keys. Events for a key that used to be ordered can now be split across two partitions, with the old ones still queued behind. Size partitions for growth up front; changing later is a migration, not a config tweak.
Rebalancing stops the world for the group. When a consumer joins, leaves or is presumed dead, partitions are reassigned and consumption pauses. Handlers that take longer than the poll interval get the consumer evicted, which triggers a rebalance, which slows everyone, which evicts another — a rebalance storm from one slow handler. Keep processing short, or hand off to a worker and manage offsets deliberately.
There is no dead-letter queue. A record that always throws blocks its partition forever, because the offset never advances. You have to build the escape yourself: retry a bounded number of times, then produce to a topic.DLQ and commit past it. Without that, one bad message halts a partition indefinitely.
Schemas are a public contract. The moment a second team consumes your topic, the payload is an API. Use a schema registry with a compatibility mode set deliberately, and make additive changes only. Renaming a field is a new version, not an edit.
Consumer lag is the metric. Not broker CPU, not message rate — the number of records between a consumer's offset and the end of the log. It is the only number that tells you whether the system is keeping up, and it should page someone.

Exactly-once, honestly

Kafka advertises exactly-once semantics, and the claim is true within a narrow boundary that is easy to misread.

Idempotent producers prevent duplicates from retries. Transactions let you atomically consume from a topic, produce to another, and commit offsets — genuinely exactly-once, as long as every side effect stays inside Kafka. That covers stream processing well.

It stops the moment your consumer writes to Postgres, charges a card, or sends an email. That write is not in the Kafka transaction, so a crash between the write and the offset commit means the record is processed again. You are back to at-least-once, and the answer is the same as it always was: make the handler idempotent with a natural key or a processed-events table keyed by record id.

In practice, at-least-once plus idempotency is the design. Exactly-once is a Kafka-internal optimisation, not a property of your system.

When Kafka is the wrong tool

Task queues. If you need per-message acknowledgement, per-message retry with backoff, delayed delivery or priorities, you want SQS or RabbitMQ. Kafka's offset model makes “retry this one message later” awkward, because there is no per-message state — only a position.
Request/response. If the caller needs an answer to continue, call the service. Correlation ids over a pair of topics is a slow, hard-to-debug reimplementation of an HTTP request.
Three services and modest volume. A cluster is real operational weight — brokers, retention sizing, partition planning, rebalance tuning, consumer lag alerting, an on-call rotation that understands all of it. Below a certain size, an outbox table polled by a worker gives you most of the decoupling at a fraction of the cost, and it is not an embarrassing architecture.
As a database. Retention is not durability of record, replay is not a query, and reconstructing state by folding a topic on every read is not a substitute for a table you can index.

The short version

Kafka is a durable log that many services read independently, and that is what decouples them — not the throughput. Choose your record key carefully, because it decides both ordering and parallelism. Assume at-least-once and make every handler idempotent. Build the dead-letter path before you need it, treat the schema as a public API, and alert on consumer lag. If what you actually needed was a task queue, use a task queue.