Distributed Streaming Infrastructure
Apache Kafka
21 min read • Updated October 2026

Apache Kafka Architecture: Storage Internals, Exactly-Once & CDC

The definitive guide to Apache Kafka: Linux page cache zero-copy mechanics, In-Sync Replicas (ISR), transactional producers, cooperative sticky rebalancing, and Change Data Capture (CDC).


1. Kafka Broker & Partition Topology Blueprint

The following blueprint illustrates how Apache Kafka achieves high throughput across distributed partitions, in-sync replicas, and consumer group worker allocations:

+-----------------------------------------------------------------------------------------+
|                  APACHE KAFKA DISTRIBUTED CLUSTER & PARTITION TOPOLOGY                  |
+-----------------------------------------------------------------------------------------+

  Producers (acks=all, enable.idempotence=true)
  [ Producer 1 ]                    [ Producer 2 ]                    [ Producer 3 ]
         \                                |                                 /
          +-------------------------------+--------------------------------+
                                          |
                                          v (Murmur2 Hash on Message Key)
  +---------------------------------------------------------------------------------------+
  |                           KAFKA 3-BROKER CLUSTER (KRaft Mode)                         |
  |                                                                                       |
  |  +------------------------+  +------------------------+  +------------------------+   |
  |  | Broker 101             |  | Broker 102             |  | Broker 103             |   |
  |  | - Topic-A Partition 0  |  | - Topic-A Partition 1  |  | - Topic-A Partition 2  |   |
  |  |   (LEADER)             |  |   (LEADER)             |  |   (LEADER)             |   |
  |  | - Topic-A Partition 1  |  | - Topic-A Partition 2  |  | - Topic-A Partition 0  |   |
  |  |   (Follower - ISR)     |  |   (Follower - ISR)     |  |   (Follower - ISR)     |   |
  |  +------------------------+  +------------------------+  +------------------------+   |
  |               ^                           ^                           ^               |
  |               | (Zero-Copy sendfile)      | (Zero-Copy sendfile)      |               |
  +---------------|---------------------------|---------------------------|---------------+
                  |                           |                           |
                  v                           v                           v
  +---------------------------------------------------------------------------------------+
  |                   CONSUMER GROUP: "order-analytics" (CooperativeSticky)               |
  |                                                                                       |
  |       [ Consumer Instance A ]       [ Consumer Instance B ]       [ Consumer C ]      |
  |       Assigned: Partition 0         Assigned: Partition 1         Assigned: Part 2    |
  +---------------------------------------------------------------------------------------+

2. Storage Internals: Linux Page Cache & Zero-Copy Syscalls

Unlike traditional message brokers that manage complex in-memory queues within JVM heaps, Kafka delegates memory management entirely to the Linux OS Page Cache:

Traditional I/O vs. Kafka Zero-Copy sendfile():
Standard Java Network Transfer (4 context switches, 3 data copies):
1. Disk -> OS Page Cache (DMA Copy)
2. OS Page Cache -> JVM Application Memory (CPU Copy)
3. JVM Application Memory -> Socket Buffer (CPU Copy)
4. Socket Buffer -> NIC Buffer (DMA Copy)

Kafka Zero-Copy via sendfile() / TransferTo() (2 context switches, 0 CPU copies):
1. Disk -> OS Page Cache (DMA Copy)
2. OS Page Cache -> NIC Buffer (Direct DMA Copy via file descriptor pointer)
Result: Zero CPU cycles wasted copying bytes in user space; saturates 40Gbps NIC lines.

3. Producer Idempotency & Exactly-Once Semantics (EOS)

When a network partition or socket timeout occurs right after a broker writes a message but before sending the ACK, a retrying producer causes message duplication. Enabling idempotent producers eliminates this:

Exactly-Once Semantics — Idempotent Producer Invariants & Model
Algorithm Invariants & Class Model
Architecture Specification
1// =========================================================================
2// PROTOCOL SPECIFICATION & INVARIANTS: Exactly-Once Semantics (EOS)
3// Broker Invariants: Deduplication via PID (Producer ID) + Sequence Number
4// =========================================================================
5Properties props = new Properties();
6props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker-1:9092,broker-2:9092");
7props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
8props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());
9
10// Invariant 1: Mandatory for Exactly-Once Delivery
11props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
12props.put(ProducerConfig.ACKS_CONFIG, "all"); // Wait for all In-Sync Replicas (ISR)
13props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
14props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5); // Ordering preserved with idempotence
15
16// Invariant 2: High-Throughput Batching & Compression Strategy
17props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
18props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024); // 64KB batches
19props.put(ProducerConfig.LINGER_MS_CONFIG, 20); // Wait up to 20ms to pack batches

4. Kafka Delivery Guarantees Trade-Off Matrix

Systematic trade-offs between delivery guarantees and broker throughput:

Delivery GuaranteeConfigurationThroughput ImpactRisk / Edge CaseBest Suited Scenario
At-Most-Onceacks=0, auto-commit before processingHighest (Zero broker ACKs)Data lost on broker failure or consumer crashHigh-volume telemetry, metrics metrics, clickstream analytics
At-Least-Onceacks=all, manual commit after DB writeHighDuplicate messages upon consumer restart (requires downstream idempotency)General microservices, event-driven ordering, notifications
Exactly-Once (EOS)transactional.id, 2PC via coordinatorMedium (~15% to 25% throughput overhead)Zombie producers if transaction coordinator encounters clock skewFinancial ledgers, payment settlement, inventory count increments

Frequently Asked Questions (Apache Kafka Architecture)

How does Apache Kafka achieve millions of messages per second throughput?

Kafka achieves ultra-high throughput through three core architectural choices: 1) Sequential disk I/O on append-only segment files, which approaches the sequential write speed of physical disks/SSDs; 2) Zero-Copy data transfer using the Linux sendfile() system call, transferring data directly from the OS page cache to the network socket without copying into JVM user space; and 3) Aggressive batching and end-to-end compression (using LZ4, Snappy, or zstd) across producers, brokers, and consumers.

What is the difference between Eager Rebalance and Cooperative Sticky Rebalance in Kafka?

In the legacy Eager Rebalance protocol, whenever a consumer joins, leaves, or times out, all consumers in the group immediately revoke ALL their partition assignments and stop consuming (a 'stop-the-world' event) until new assignments are distributed. In Cooperative Sticky Rebalancing (Incremental Cooperative Rebalance), consumers only revoke partitions that are actively being migrated to another member, allowing continuous consumption on unchanged partitions and eliminating cluster-wide consumer freezes.

How does Kafka guarantee Exactly-Once Semantics (EOS)?

Kafka EOS combines Idempotent Producers (which assign an internal Producer ID and sequence numbers to every batch to prevent duplicate writes during network retries) with a Transaction Coordinator. The Transaction Coordinator manages a two-phase commit protocol across input offsets and output topic partitions via the __transaction_state internal topic, ensuring all reads, processing, and downstream writes commit or abort atomically.

Integrate Kafka with Distributed Microservices

Learn how to implement Transactional Outbox, Saga distributed orchestrations, and Dead Letter Queues using Spring Boot and Apache Kafka.