Apache Kafka Architecture: Storage Internals, Exactly-Once & CDC
The definitive guide to Apache Kafka: Linux page cache zero-copy mechanics, In-Sync Replicas (ISR), transactional producers, cooperative sticky rebalancing, and Change Data Capture (CDC).
1. Kafka Broker & Partition Topology Blueprint
The following blueprint illustrates how Apache Kafka achieves high throughput across distributed partitions, in-sync replicas, and consumer group worker allocations:
+-----------------------------------------------------------------------------------------+
| APACHE KAFKA DISTRIBUTED CLUSTER & PARTITION TOPOLOGY |
+-----------------------------------------------------------------------------------------+
Producers (acks=all, enable.idempotence=true)
[ Producer 1 ] [ Producer 2 ] [ Producer 3 ]
\ | /
+-------------------------------+--------------------------------+
|
v (Murmur2 Hash on Message Key)
+---------------------------------------------------------------------------------------+
| KAFKA 3-BROKER CLUSTER (KRaft Mode) |
| |
| +------------------------+ +------------------------+ +------------------------+ |
| | Broker 101 | | Broker 102 | | Broker 103 | |
| | - Topic-A Partition 0 | | - Topic-A Partition 1 | | - Topic-A Partition 2 | |
| | (LEADER) | | (LEADER) | | (LEADER) | |
| | - Topic-A Partition 1 | | - Topic-A Partition 2 | | - Topic-A Partition 0 | |
| | (Follower - ISR) | | (Follower - ISR) | | (Follower - ISR) | |
| +------------------------+ +------------------------+ +------------------------+ |
| ^ ^ ^ |
| | (Zero-Copy sendfile) | (Zero-Copy sendfile) | |
+---------------|---------------------------|---------------------------|---------------+
| | |
v v v
+---------------------------------------------------------------------------------------+
| CONSUMER GROUP: "order-analytics" (CooperativeSticky) |
| |
| [ Consumer Instance A ] [ Consumer Instance B ] [ Consumer C ] |
| Assigned: Partition 0 Assigned: Partition 1 Assigned: Part 2 |
+---------------------------------------------------------------------------------------+2. Storage Internals: Linux Page Cache & Zero-Copy Syscalls
Unlike traditional message brokers that manage complex in-memory queues within JVM heaps, Kafka delegates memory management entirely to the Linux OS Page Cache:
Traditional I/O vs. Kafka Zero-Copy sendfile():
Standard Java Network Transfer (4 context switches, 3 data copies): 1. Disk -> OS Page Cache (DMA Copy) 2. OS Page Cache -> JVM Application Memory (CPU Copy) 3. JVM Application Memory -> Socket Buffer (CPU Copy) 4. Socket Buffer -> NIC Buffer (DMA Copy) Kafka Zero-Copy via sendfile() / TransferTo() (2 context switches, 0 CPU copies): 1. Disk -> OS Page Cache (DMA Copy) 2. OS Page Cache -> NIC Buffer (Direct DMA Copy via file descriptor pointer) Result: Zero CPU cycles wasted copying bytes in user space; saturates 40Gbps NIC lines.
3. Producer Idempotency & Exactly-Once Semantics (EOS)
When a network partition or socket timeout occurs right after a broker writes a message but before sending the ACK, a retrying producer causes message duplication. Enabling idempotent producers eliminates this:
| 1 | // ========================================================================= |
| 2 | // PROTOCOL SPECIFICATION & INVARIANTS: Exactly-Once Semantics (EOS) |
| 3 | // Broker Invariants: Deduplication via PID (Producer ID) + Sequence Number |
| 4 | // ========================================================================= |
| 5 | Properties props = new Properties(); |
| 6 | props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker-1:9092,broker-2:9092"); |
| 7 | props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); |
| 8 | props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); |
| 9 | |
| 10 | // Invariant 1: Mandatory for Exactly-Once Delivery |
| 11 | props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); |
| 12 | props.put(ProducerConfig.ACKS_CONFIG, "all"); // Wait for all In-Sync Replicas (ISR) |
| 13 | props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE); |
| 14 | props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5); // Ordering preserved with idempotence |
| 15 | |
| 16 | // Invariant 2: High-Throughput Batching & Compression Strategy |
| 17 | props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"); |
| 18 | props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024); // 64KB batches |
| 19 | props.put(ProducerConfig.LINGER_MS_CONFIG, 20); // Wait up to 20ms to pack batches |
4. Kafka Delivery Guarantees Trade-Off Matrix
Systematic trade-offs between delivery guarantees and broker throughput:
| Delivery Guarantee | Configuration | Throughput Impact | Risk / Edge Case | Best Suited Scenario |
|---|---|---|---|---|
| At-Most-Once | acks=0, auto-commit before processing | Highest (Zero broker ACKs) | Data lost on broker failure or consumer crash | High-volume telemetry, metrics metrics, clickstream analytics |
| At-Least-Once | acks=all, manual commit after DB write | High | Duplicate messages upon consumer restart (requires downstream idempotency) | General microservices, event-driven ordering, notifications |
| Exactly-Once (EOS) | transactional.id, 2PC via coordinator | Medium (~15% to 25% throughput overhead) | Zombie producers if transaction coordinator encounters clock skew | Financial ledgers, payment settlement, inventory count increments |
Frequently Asked Questions (Apache Kafka Architecture)
How does Apache Kafka achieve millions of messages per second throughput?
Kafka achieves ultra-high throughput through three core architectural choices: 1) Sequential disk I/O on append-only segment files, which approaches the sequential write speed of physical disks/SSDs; 2) Zero-Copy data transfer using the Linux sendfile() system call, transferring data directly from the OS page cache to the network socket without copying into JVM user space; and 3) Aggressive batching and end-to-end compression (using LZ4, Snappy, or zstd) across producers, brokers, and consumers.
What is the difference between Eager Rebalance and Cooperative Sticky Rebalance in Kafka?
In the legacy Eager Rebalance protocol, whenever a consumer joins, leaves, or times out, all consumers in the group immediately revoke ALL their partition assignments and stop consuming (a 'stop-the-world' event) until new assignments are distributed. In Cooperative Sticky Rebalancing (Incremental Cooperative Rebalance), consumers only revoke partitions that are actively being migrated to another member, allowing continuous consumption on unchanged partitions and eliminating cluster-wide consumer freezes.
How does Kafka guarantee Exactly-Once Semantics (EOS)?
Kafka EOS combines Idempotent Producers (which assign an internal Producer ID and sequence numbers to every batch to prevent duplicate writes during network retries) with a Transaction Coordinator. The Transaction Coordinator manages a two-phase commit protocol across input offsets and output topic partitions via the __transaction_state internal topic, ensuring all reads, processing, and downstream writes commit or abort atomically.