PAPERS-05-006 · Paper — Apache Kafka

The paper

The landmark paper “Apache Kafka” (Kreps, Narkhede, and Rao, 2011, presented at the ACM SIGMOD conference) introduced a new way to handle massive data streams and quickly became a foundational work in distributed systems.

Before this paper

Before Kafka, large companies relied on a tangled web of point‑to‑point data feeds. Moving high‑volume event streams was done in an ad‑hoc, fragile manner, and scaling such pipelines was difficult. Systems could not reliably share streams among many consumers, leading to bottlenecks and data loss. The field lacked a durable, scalable abstraction for continuous data movement.

The key idea

The key insight was to treat data movement as a durable, distributed, append‑only log. This log lets many producers write concurrently while many consumers read at their own pace. By making the log a first‑class infrastructure primitive, the same stream can be replayed and shared across independent consumer groups.

How it works

Kafka stores messages in topics that are split into partitions. Each partition is replicated across multiple broker nodes for fault tolerance. Producers append messages to the end of a partition, and each message gets a sequential offset. Consumers keep track of their own offset, allowing them to read independently and restart from any point. Replication ensures durability, and the partitioned design provides high throughput and parallelism. Consumer groups can coordinate to balance load while still preserving each group’s independent view of the log.

See it — the distributed commit log

flowchart LR
  P["Producers"] --> L["Partitioned,<br/>replicated log"]
  L --> C1["Consumer group 1"]
  L --> C2["Consumer group 2"]

What they showed

The authors showed that Kafka could move LinkedIn’s huge event streams reliably, becoming the backbone for real‑time data pipelines across the industry. This demonstrated that a distributed log could handle production‑scale workloads with high throughput and durability.

What it changed

Kafka turned the distributed log into a first‑class infrastructure primitive, reshaping how systems are built for streaming data. It anchored the modern streaming ecosystem, enabling event‑driven architectures and real‑time analytics everywhere. By proving that a log‑based design could scale, it inspired a wave of new tools and platforms built on the same principles. Next: Apache Spark.

The one-line takeaway

Kafka proved that a durable, distributed log can be the universal plumbing for real‑time data streams.

Read it / meet it

Reading the original paper gives insight into the design choices that still power today’s streaming systems. Its legacy lives on in every modern event‑driven architecture and real‑time analytics platform you encounter.