Apache Kafka Internals Under the Hood: Partitions, Replication, and Exactly-Once Semantics
Kafka moves trillions of events per day at LinkedIn, Uber, Netflix, and Confluent. But its real power — and its real complexity — lives beneath the surface: in the append-only log segment files on disk, in the In-Sync Replica protocol that makes replication durable, and in the idempotent producer machinery that makes exactly-once delivery mathematically provable. This walkthrough takes you all the way down.
1. Why Kafka Exists: Log-Based Messaging vs Traditional Queues
1.1 The Problem With Traditional Message Queues
Traditional message queues — RabbitMQ, ActiveMQ, IBM MQ — operate on a message-as-a-task model. A producer enqueues a job; a consumer dequeues and processes it; the broker deletes the message on acknowledgment. This is excellent for task distribution (job queues, email dispatch), but fundamentally broken for three common streaming scenarios: multiple independent consumers reading the same stream, replaying history after a consumer failure, and backpressure handling when consumers are slower than producers.
When a traditional queue deletes messages after delivery, a second consumer application joining the system cannot access historical events — it only sees events from the moment it subscribes. If a downstream analytics system is added to process all user clickstream events retroactively, a traditional queue simply cannot serve it. The broker has already discarded the events.
Kafka's fundamental insight — borrowed from database write-ahead logs — is the durable, partitioned, replayable log. Messages are never deleted on consumption. Instead, they are appended to an ordered log and retained for a configurable period (or until a configurable size limit). Consumers track their own position (offset) in the log independently. This means any number of independent consumer applications can read the same Kafka topic at their own pace, from any historical position, without affecting each other or the broker's retention policy.
1.2 The Kafka Mental Model: A Distributed, Partitioned Commit Log
Think of a Kafka topic as a distributed version of a Unix log file. Records are appended to the end. Readers scan through it sequentially. The log is split into partitions for horizontal scalability — each partition is an independent, ordered, immutable sequence of records stored on a single broker. Partitions can be read in parallel by different consumers. Records within a partition have a monotonically increasing integer ID called an offset. Offsets are the consumer's bookmark — the consumer is responsible for storing and advancing its own offset, giving it complete control over replay semantics.
Developer Pitfall — Confusing Kafka With a Message Queue for Ephemeral Tasks:
Kafka is not a replacement for RabbitMQ in task queue scenarios (e.g., background job processing where each job should be processed by exactly one worker and then discarded). Kafka's log-based model means messages persist until retention expires — not until they are "consumed." For task queues where you want automatic requeue on failure, exactly-one-worker processing, and message deletion after processing, RabbitMQ or Amazon SQS is the right tool. Kafka excels at high-throughput event streaming, audit logs, event sourcing, stream processing, and any scenario requiring multiple consumers or historical replay.
2. Topics, Partitions, Offsets, and the Kafka Cluster Architecture
2.1 Topics and Partitions: The Unit of Parallelism
A topic is a named logical channel for events (e.g., user-clicks, payment-events). Topics are divided into one or more partitions. The number of partitions is the primary knob for controlling parallelism — more partitions means more consumers can read in parallel, but also more open file handles, more replication traffic, and higher Kafka controller overhead. LinkedIn's production Kafka clusters often have topics with thousands of partitions, but this requires careful broker sizing.
Each partition is stored on exactly one broker as its leader and replicated to one or more other brokers as followers. The replication factor (typically 3 for production) determines how many broker failures the topic can survive. The partition key determines which partition a producer's record goes to: if a key is specified, Kafka hashes it (using MurmurHash2) to pick a partition deterministically. If no key is specified, records are distributed round-robin across partitions using a sticky partitioner (batching multiple records to the same partition before switching — more on this in the producer section).
Diagram 1: Kafka Cluster Architecture. Each partition has one Leader (handles all reads/writes) and N-1 Followers (replicate from leader). Multiple independent Consumer Groups can read the same topic without interfering with each other.
Developer Pitfall — Setting Too Few Partitions at Topic Creation:
Kafka partition count can be increased but never decreased without deleting and recreating the topic. If you start with 3 partitions and your consumer group scales to 10 instances, 7 consumers will sit idle — you can't have more active consumers than partitions in a group. Plan for future throughput and add 20–50% more partitions than you think you'll need at creation time. The cost of extra partitions (file handles, replication overhead) is much lower than the operational pain of repartitioning a topic used by multiple downstream consumers in production.
3. Log Segment Internals: What Kafka Actually Stores on Disk
3.1 Segment Files, Index Files, and the Log Directory Structure
Every Kafka partition is stored on disk as a log directory containing multiple segment files. A segment is a pair of files: a .log file (the actual binary record data) and an .index file (a sparse offset-to-byte-position index for fast random access). There is also a .timeindex file (timestamp-to-offset index, used for time-based offset lookup). Each segment file is named after the first offset it contains:
Kafka writes to only one segment at a time — the active segment. When the active segment reaches a configurable size limit (log.segment.bytes, default 1GB) or age (log.roll.hours, default 168 hours), it is rolled: the current segment is closed and a new active segment is created. Closed segments are read-only and can be served to consumers and followers concurrently without any locking — a key reason for Kafka's extreme read throughput.
The .index file is a sparse index — it doesn't store the byte position for every offset, only for every Nth record (configurable via log.index.interval.bytes, default 4096 bytes). To find a specific offset, Kafka binary-searches the sparse index for the nearest entry below the target offset, then linearly scans the .log file from that byte position forward. This two-level lookup trades some scan time for dramatically smaller index files, and the linear scan is fast because records at a given offset are close together spatially on disk.
3.2 Zero-Copy: How Kafka Achieves 1M+ Messages/Second Throughput
One of Kafka's most important performance features is zero-copy data transfer using the sendfile() Linux syscall (or Java's FileChannel.transferTo()). When a consumer fetches records, conventional I/O would: read data from disk into kernel page cache, copy from page cache to a userspace buffer, copy from userspace buffer back to kernel socket buffer, then send over the network — 4 data copies total. With sendfile(), the OS copies data directly from the page cache to the socket buffer via DMA, completely bypassing userspace — just 1 copy. For read-heavy workloads (multiple consumer groups), this is the difference between a 100 MB/s throughput ceiling and a multi-GB/s ceiling on the same hardware.
Developer Pitfall — Enabling Compression on the Consumer Side Breaks Zero-Copy:
Zero-copy only works when the data passes through from disk to socket without modification. If you configure Kafka brokers to recompress data that was compressed by producers (e.g., producer sends snappy-compressed batches, but broker is configured to use gzip), the broker must decompress and recompress every batch — forcing a full userspace copy and eliminating the zero-copy benefit. Always configure compression.type=producer (the default) on the broker to pass producer-compressed batches through unchanged. Only override this if you have a specific reason to recompress at the broker, and be aware you're paying a significant CPU and latency cost.
4. Producer Internals: Batching, the RecordAccumulator, and acks
4.1 The RecordAccumulator: Why Kafka Producers Are Asynchronous by Default
A Kafka producer does not send each record immediately when you call producer.send(record). Instead, the record is placed into an in-memory buffer called the RecordAccumulator — a per-partition deque of ProducerBatch objects. The I/O thread (Sender thread) independently drains the accumulator in background, batching multiple records for the same partition into a single network request. This batching is the primary mechanism behind Kafka's extraordinary throughput — a single network round-trip carrying 10,000 records is dramatically more efficient than 10,000 individual round-trips.
Two producer configuration parameters control batch formation. batch.size (default 16KB) sets the maximum byte size of a batch — the Sender will not wait for a batch to fill before sending if the I/O thread is ready. linger.ms (default 0ms) adds an artificial delay: the Sender waits up to linger.ms milliseconds for the batch to accumulate more records before sending, even if the batch is not full. Setting linger.ms=5 with batch.size=65536 is the canonical production configuration for high-throughput producers, allowing batches to fill over a 5ms window before flushing.
4.2 The acks Setting: Durability vs Latency Tradeoff
The acks producer configuration controls when the broker acknowledges a produce request:
Most production systems use acks=all combined with min.insync.replicas=2 (a broker-level config that requires at least 2 replicas in the ISR for a write to succeed). This combination means: even if the leader crashes immediately after acknowledging, the write exists on at least one follower and will be elected as the new leader. The acks=1 setting is often mistakenly believed to be safe — it is not. It provides the same risk profile as a single-node system for any individual write.
Developer Pitfall — Using acks=1 and Claiming Kafka Is Durable:
With acks=1 and replication.factor=3, a common misconception is "I have 3 copies so I'm safe." But acks=1 only confirms the leader wrote to its log — the 2 follower replicas may lag by seconds. If the leader crashes in that window, the newly elected leader (the follower with the highest offset) will not have the unacknowledged records, and the producer receives no error — it believes the write succeeded. You silently lost data. Always use acks=all with min.insync.replicas=2 for any data you cannot afford to lose. Accept the ~10% latency increase as the cost of correctness.
5. Replication: ISR, High-Watermark, and Leader Epoch Fencing
5.1 The In-Sync Replica Set and the High-Watermark
The In-Sync Replica (ISR) set is the set of replicas (including the leader) that are fully caught up with the leader's log. A follower is considered in-sync if it has fetched up to the leader's log end offset within the last replica.lag.time.max.ms milliseconds (default 30 seconds). Followers that fall behind are removed from the ISR — this is called ISR shrinkage. ISR membership is tracked by the Kafka controller (or KRaft quorum) and stored in ZooKeeper (or the KRaft metadata log).
The high-watermark (HW) is the highest offset that has been replicated to all ISR members. Consumers can only read records up to the high-watermark — records above the HW are not yet guaranteed to survive a leader failure and are invisible to consumers until they are. When the leader receives an acks=all produce request, it appends to its log and then waits until all ISR followers fetch the new records. Once all ISR followers' fetch requests confirm they have the new records, the leader advances the high-watermark and acknowledges the producer.
5.2 Leader Epoch Fencing: Preventing Ghost Writers
Consider this scenario: Leader A writes records 98 and 99. Before followers replicate them, A is partitioned. B is elected leader (it has records 0–97). A's network partition heals. A believes it is still leader and tries to write more records. Without fencing, A and B could both be writing to the same partition — corrupting it.
Kafka solves this with leader epochs. Every time a new leader is elected, the cluster increments the leader epoch. Every record written by a leader is tagged with its epoch. When an old leader (with a stale epoch) tries to fetch or write, followers and the new leader reject its requests because their epoch is higher. The leader-epoch-checkpoint file on each broker persists the epoch history — allowing followers to correctly truncate their logs to match the new leader's history without using the high-watermark (which could be stale after a leader failover involving ISR shrinkage).
Developer Pitfall — ISR Shrinkage Under GC Pauses Causing Silent Durability Reduction:
A Kafka broker running a JVM application with a stop-the-world GC pause longer than replica.lag.time.max.ms (30s) will cause its follower replicas to be removed from ISR sets across all partitions it follows. During and after this period, writes that complete with acks=all are actually only replicated to the remaining ISR members — potentially just the leader if only one other broker is in ISR. Your perceived RF=3 durability has silently dropped to effective RF=1. Monitor UnderReplicatedPartitions JMX metric aggressively. A non-zero value means your actual durability is lower than your replication factor promises. Use G1GC or ZGC to keep pause times under 1 second for Kafka brokers.
6. Consumer Groups and Partition Assignment
6.1 How Consumer Groups Coordinate: The Group Coordinator
A consumer group is a set of consumer instances that collectively consume a topic — each partition in the topic is assigned to exactly one consumer in the group at any given time. This gives you parallel consumption with ordering guarantees within each partition. The coordination of partition assignment is managed by a broker designated as the Group Coordinator for that group, determined by hashing the group ID to a partition of the internal __consumer_offsets topic.
When consumers join or leave a group, the group coordinator initiates a rebalance. During a classic (eager) rebalance, all consumers in the group must: (1) stop consuming, (2) revoke all their current partition assignments, (3) rejoin the group, (4) wait for the Group Leader (the first consumer to join) to receive the full member list and compute a new assignment, (5) receive their new assignments, and (6) resume consuming. This stop-the-world pause can last seconds on large groups and causes consumer lag spikes. This is why Kafka 2.4 introduced Cooperative (Incremental) Rebalancing — only the partitions that need to move are revoked and reassigned, while other partitions continue being consumed uninterrupted.
6.2 The __consumer_offsets Topic: How Kafka Tracks Consumer Position
Kafka stores committed consumer offsets durably in an internal compacted topic named __consumer_offsets. When a consumer calls commitSync() or commitAsync(), it sends an OffsetCommit request to its Group Coordinator, which appends a key-value record to __consumer_offsets — key: (groupId, topic, partition), value: (committed_offset, metadata, timestamp). This topic uses log compaction, so the latest committed offset for each group-topic-partition key is always retained, regardless of retention period.
When a consumer group restarts or rebalances, each consumer fetches the latest committed offsets for its assigned partitions from __consumer_offsets and resumes from there. The auto.offset.reset config (latest or earliest) only applies when there is no committed offset yet — either the group is new, or the committed offset is older than the partition's retention period and the actual records have been deleted.
Developer Pitfall — Committing Offsets Before Processing Completes (At-Most-Once):
The auto-commit behavior (enabled by default with enable.auto.commit=true) commits the offset of the last fetched record every auto.commit.interval.ms (default 5 seconds) — regardless of whether you have finished processing those records. If your consumer crashes between the auto-commit and the completion of processing, those records are skipped on restart — at-most-once delivery. For at-least-once delivery, always disable auto-commit (enable.auto.commit=false) and call commitSync() explicitly only after successfully processing each batch. For exactly-once, use Kafka Transactions (Section 7).
7. Exactly-Once Semantics: Idempotent Producers and Transactions
7.1 The Delivery Semantics Landscape
Before exactly-once, Kafka producers faced a painful choice. Network retries (enabled by default) can cause duplicate records if the broker wrote the record and sent an ack that was lost in transit — the producer retries and the broker writes the record a second time. Disabling retries eliminates duplicates but accepts data loss on transient failures. This was the at-least-once vs at-most-once dilemma.
| Semantic | Producer Config | Consumer Behavior | Risk | Use Case |
|---|---|---|---|---|
| At-Most-Once | acks=0 retries=0 |
Auto-commit before processing | Data loss on failure | Metrics, telemetry (loss OK) |
| At-Least-Once | acks=all retries=MAX |
Manual commit after processing | Duplicates on retry | Most event streaming |
| Exactly-Once | enable.idempotence=true transactional.id=X |
isolation.level=read_committed | No loss, no duplicates | Financial events, EOS pipelines |
7.2 Idempotent Producers: PID and Sequence Numbers
The Idempotent Producer (enabled with enable.idempotence=true) solves the duplicate-on-retry problem. When an idempotent producer initializes, it receives a unique Producer ID (PID) from the broker. Every record batch sent by this producer carries the PID and a monotonically increasing sequence number per partition. The broker maintains the last 5 sequence numbers per (PID, partition) pair. When a batch arrives:
This deduplication is performed entirely within the broker using an in-memory (and checkpointed) sequence table. The producer PID is ephemeral — it resets on producer restart, which means idempotency is only guaranteed within a single producer session. It does not survive producer crashes. For true end-to-end exactly-once across consumer-process-produce pipelines, you need Kafka Transactions.
7.3 Kafka Transactions: Atomic Multi-Partition Writes
Kafka Transactions allow a producer to atomically write to multiple partitions — all records from a transaction either all become visible to read_committed consumers, or none do. This enables the Kafka Streams "consume-process-produce" pattern to be exactly-once end-to-end: consume from input topic, process, produce to output topic + commit input offsets — all as one atomic transaction.
The transactional producer is initialized with a stable transactional.id string (unlike PID, this persists across restarts). The first action of the transactional producer on startup is to find the Transaction Coordinator (a broker whose partition of the __transaction_state internal topic covers this transactional.id) and register with it. The coordinator assigns a Producer Epoch — a version counter that increments with every registration. Old producer instances with lower epochs are fenced (rejected) by the coordinator and all brokers — preventing zombie producers from completing stale transactions after a restart.
Developer Pitfall — Using Transactions Without read_committed on Consumers:
Kafka Transactions only provide end-to-end exactly-once if all consumers of the output topic use isolation.level=read_committed. A consumer with the default read_uncommitted isolation will see records from aborted transactions — data that was rolled back and should be invisible. This is a silent bug: the consumer processes records that the producer explicitly aborted, leading to incorrect state in the downstream system. Always set isolation.level=read_committed on any consumer that is part of an exactly-once pipeline, and audit your consumer configurations in code review to enforce this constraint.
8. Step-by-Step: A Complete Produce-to-Consume Trace
8.1 The Happy Path: One Record, One Partition, RF=3
Let's trace a single idempotent record through the full Kafka pipeline with concrete values. Topic: payment-events, partition 0, RF=3, acks=all, min.insync.replicas=2, ISR=[Broker1(leader), Broker2, Broker3].
Developer Pitfall — Consumer Lag From fetch.min.bytes and fetch.max.wait.ms Misconfiguration:
By default, Kafka consumers use fetch.min.bytes=1 (return as soon as any data is available) and fetch.max.wait.ms=500 (wait up to 500ms for data if broker has less than fetch.min.bytes). For low-throughput topics, this causes a 500ms artificial delay on every fetch when the broker has less than 1 byte ready — which is almost always for idle topics. Set fetch.min.bytes=1 with fetch.max.wait.ms=100 for latency-sensitive consumers. For high-throughput batch consumers, increase fetch.min.bytes=65536 and max.partition.fetch.bytes=10485760 to reduce the number of fetch round-trips.
9. KRaft Mode: Removing the ZooKeeper Dependency
9.1 Why ZooKeeper Was a Problem
Kafka historically used Apache ZooKeeper to store cluster metadata: broker registrations, topic configurations, partition leader assignments, ISR lists, and consumer group offsets (in older versions). This created a two-system dependency — operators had to maintain and monitor both a Kafka cluster and a separate ZooKeeper ensemble. ZooKeeper also imposed hard scalability limits: the Kafka controller (a single broker responsible for partition leadership management) serialized all metadata operations through ZooKeeper, and the ZooKeeper node storage model became a bottleneck for clusters with hundreds of thousands of partitions.
Kafka 2.8 introduced the KRaft (Kafka Raft) mode in preview, and Kafka 3.3 declared it production-ready. In KRaft mode, Kafka brokers run a built-in Raft consensus group — typically 3 or 5 designated controller nodes — that manages all cluster metadata in a dedicated internal topic called __cluster_metadata. ZooKeeper is completely eliminated. This reduces operational complexity to a single system and unlocks the ability to handle millions of partitions per cluster (benchmarks show 10x faster controlled shutdown and leader election with KRaft at scale).
9.2 The Metadata Quorum and Fast Controller Failover
In KRaft mode, the controller nodes run a Raft quorum over the __cluster_metadata log. All metadata changes (topic creation, partition reassignment, ISR changes) are proposed as records to this log. Only the active controller (Raft leader) processes metadata requests — follower controllers replicate the log and stand by for failover. When a controller failover occurs (leader crash or deliberate rollover), a new Raft leader is elected among the controller nodes and immediately has the complete, up-to-date metadata log — no need for a slow snapshot load from ZooKeeper. Broker nodes pull their metadata from the active controller via a fetch protocol similar to partition replication, ensuring they are always nearly up-to-date.
Developer Pitfall — Mixing ZooKeeper and KRaft Brokers in the Same Cluster:
Migration from ZooKeeper-based Kafka to KRaft is a staged process requiring a specific migration procedure (Kafka 3.5+). You cannot simply configure some brokers with process.roles=broker,controller (KRaft) while others remain in ZooKeeper mode and expect them to interoperate. Attempting to add KRaft brokers to a running ZooKeeper-based cluster without following the official migration procedure will result in the KRaft brokers failing to join the cluster and potentially causing controller split-brain. Follow Confluent's or Apache's documented migration guide step by step, including the bridging controller mode that maintains ZooKeeper compatibility during the transition window.
10. Performance Tuning: Producer Throughput vs Latency
10.1 The Batch Size and Linger Trade-off
The most impactful producer tuning knobs — batch.size and linger.ms — directly control the throughput-latency tradeoff. A larger batch.size allows more records to be sent in one network request, amortizing TCP and Kafka protocol overhead. linger.ms adds artificial delay to let batches fill up. The chart below shows representative producer throughput (records/second) at different configurations for a LAN-connected 3-broker Kafka cluster with snappy compression:
Chart 1: Producer throughput (records/sec) across batch.size and linger.ms combinations. Larger batches with a small linger (5–10ms) offer the best throughput without sacrificing excessive latency. Values are illustrative of typical production patterns.
10.2 Compression: CPU vs Network vs Throughput
Kafka supports four compression codecs: none, gzip, snappy, lz4, and zstd. The choice is a three-way tradeoff between compression ratio (network savings), compression speed (producer CPU), and decompression speed (consumer CPU). For most production workloads, lz4 is the default-safe choice — it offers 2–4x compression ratio with the lowest CPU overhead of all codecs. zstd provides better compression ratios (3–5x for typical JSON event payloads) with moderate CPU cost and is preferred for throughput-constrained network links (cross-region or expensive WAN). gzip has the highest compression ratio but is CPU-intensive — only worthwhile for archival/cold storage topics. snappy is the historical default but has been surpassed by lz4 in almost all metrics.
An important detail: compression is applied to batches, not individual records. A batch of 100 small JSON records compresses far better than a single record because string keys, field names, and repeated values appear across many records in the batch window. This means linger.ms > 0 not only increases throughput but also improves compression ratio, creating a beneficial compounding effect.
10.3 Consumer Tuning: Throughput and Partition Assignment
Consumer throughput is primarily limited by: (1) the number of partitions (max parallelism = partition count), (2) max.partition.fetch.bytes (default 1MB — the max bytes fetched per partition per request), and (3) processing latency per record. For high-throughput analytics consumers, increase max.partition.fetch.bytes to 10–50MB and process records in parallel threads (one thread per partition worker), keeping the Kafka consumer poll loop on a single thread as the Kafka client is not thread-safe for concurrent poll calls. For latency-sensitive consumers (payment fraud detection), keep max.poll.records=100 small and tune max.poll.interval.ms high enough that processing never exceeds it — otherwise the consumer is considered dead and triggers a rebalance.
Developer Pitfall — max.poll.interval.ms Timeout Causing Unexpected Rebalances:
If your consumer's business logic takes longer than max.poll.interval.ms (default 5 minutes) to process a batch of records before the next poll() call, the group coordinator assumes the consumer is dead and initiates a rebalance, revoking its partition assignments. The consumer then rejoins, fetches the same records again (from the last committed offset), and potentially processes them again — causing duplicate processing that defeats your at-least-once intent. The fix: either reduce batch size (max.poll.records), increase max.poll.interval.ms to exceed your worst-case processing time, or move heavy processing to a separate thread and call poll() frequently to send heartbeats even while processing continues in the background.
11. Log Compaction vs Log Retention: Choosing the Right Policy
11.1 Time-Based and Size-Based Retention
The default Kafka retention policy (cleanup.policy=delete) deletes log segments when they are older than log.retention.hours (default 168 hours / 7 days) or when the total partition size exceeds log.retention.bytes. This is the right policy for event streams where historical data has a time value — clickstream events, application logs, sensor telemetry. Older segments are simply deleted when they expire, and consumers that fall behind by more than the retention window will find their committed offset no longer exists in the log — they must reset to earliest or latest.
A critical operational consideration: retention is enforced at the segment granularity, not individual record granularity. A segment is only eligible for deletion when all records in it are older than the retention threshold. If your active segment is 1GB and you produce records slowly, the segment might stay active (and thus exempt from deletion) for weeks — effectively giving you much longer retention than configured. Keep log.roll.hours small (1–6 hours) to ensure segments roll over regularly and become deletion-eligible promptly.
11.2 Log Compaction: Keeping Only the Latest Value Per Key
Log compaction (cleanup.policy=compact) is Kafka's mechanism for topics where you care about the latest value for each key, not the full event history. Instead of deleting segments by age, the log cleaner periodically scans the log and removes records whose key appears again later in the log — keeping only the most recent record for each key. A record with a null value (tombstone) signals that the key should be deleted entirely. This is exactly the right model for change data capture (CDC) topics, configuration stores, and changelog topics in Kafka Streams applications.
Compaction runs in background threads and guarantees that the tail of the log (recent records) is never compacted — there is always a clean/dirty ratio where recent records are preserved as-is. The compaction guarantee is: any consumer that reads from offset 0 will see at least the latest record for every key that was ever written. This makes compacted topics suitable as changelog topics — a consumer that starts from the beginning will reconstruct the full current state of the key-value store, even if it was created months later.
Developer Pitfall — Using Log Compaction for Event Streams With Duplicate Keys:
Log compaction removes intermediate records for a key — it is not a history-preserving operation. If you use a compacted topic for an event stream where multiple events with the same key represent distinct, meaningful events (e.g., user sessions with the same user ID), compaction will silently delete all intermediate events, keeping only the latest. Use cleanup.policy=delete for event streams where every record is meaningful regardless of key. Only use cleanup.policy=compact for topics that are semantically key-value stores — where the value fully supersedes all previous values for the same key. You can also use cleanup.policy=compact,delete to apply both — compaction within the retention window, deletion after it — for slowly changing datasets with a finite history requirement.
12. Frequently Asked Questions
Q1: How does Kafka guarantee ordering, and when does it break?
Kafka guarantees ordering within a partition. Records produced to the same partition appear in the order they were appended, and consumers read them in that order. Ordering across partitions is not guaranteed — there is no global ordering across a topic's partitions. Ordering breaks in two scenarios: (1) a producer sends records for the same key to different partitions (which shouldn't happen with key-based routing unless you change the partition count mid-stream), and (2) with retries > 0 and max.in.flight.requests.per.connection > 1, a retry of batch N can arrive after batch N+1 has already been written, resulting in reordering. The fix: either use enable.idempotence=true (which automatically sets max.in.flight.requests.per.connection=5 safely via sequence number ordering enforcement) or set max.in.flight.requests.per.connection=1 (serialized, slower).
Q2: What is the difference between Kafka Streams and a Kafka Consumer with processing logic?
A raw Kafka Consumer with processing logic is a polling loop that reads records and does something with them — no state, no windowing, no join, no aggregation unless you implement it yourself. Kafka Streams is a JVM stream processing library that sits on top of Kafka consumers and producers to provide: stateful operations with embedded RocksDB state stores (backed by Kafka changelog topics for durability), time-windowed operations (tumbling, hopping, session windows), stream-table joins, and an exactly-once processing guarantee across the full consume-process-produce pipeline. Use a raw consumer when your processing is stateless and simple. Use Kafka Streams when you need aggregations, joins, windowed computations, or exactly-once semantics across a pipeline. For Python/Go environments, consider Faust (Python) or similar alternatives, as Kafka Streams is JVM-only.
Q3: How do I choose the right number of partitions for a topic?
The rule of thumb: number of partitions = max(desired consumer parallelism, target throughput / per-partition throughput). Per-partition throughput on modern hardware is roughly 10–50 MB/s depending on replication factor, compression, and disk type. If you need to sustain 500 MB/s with RF=3, you need at least 10–50 partitions. For consumer parallelism, partition count sets the maximum number of active consumers in a group — if you want 20 consumers processing in parallel, you need at least 20 partitions. Start with 3–6x your current consumer count to allow growth without repartitioning. For critical production topics at companies like Uber, partitions numbers in the hundreds are normal; LinkedIn has topics with thousands of partitions but with careful cluster sizing (more brokers to distribute the partition leader load).
Q4: What happens when an ISR goes below min.insync.replicas?
When the ISR for a partition falls below min.insync.replicas (e.g., ISR has only 1 broker but min.insync.replicas=2), the partition leader will refuse all produce requests with acks=all (-1) and return a NOT_ENOUGH_REPLICAS error. This is the correct behavior — it prevents data loss by refusing writes that cannot be durably replicated to the required minimum of replicas. The partition remains readable (consumers can still fetch up to the high-watermark), but no new writes are accepted until enough replicas rejoin the ISR. This is a cluster health emergency — the UnderMinIsrPartitionCount metric should trigger your highest-priority alerting. In practice, you must choose between write availability (allow writes with fewer replicas, risk data loss) and data durability (refuse writes, maintain durability guarantee) — min.insync.replicas is the knob that makes this tradeoff explicit.
Q5: How does Kafka handle large messages, and what are the limits?
Kafka is designed for small-to-medium messages (1 byte to ~1MB). The default maximum message size is 1MB, controlled by message.max.bytes on the broker and max.request.size on the producer. While you can increase these limits, messages over 10MB significantly degrade Kafka performance: they don't batch efficiently, consume disproportionate memory in the broker's page cache, slow replication (one large message can block the replication stream for many small messages), and exhaust consumer fetch buffers. For payloads over 1MB (images, video frames, large JSON blobs), the recommended pattern is to store the payload in object storage (S3, GCS) and publish a Kafka message containing only the reference URL/key — keeping the Kafka event small and the actual payload in a system designed for large objects.
Q6: What is the difference between Kafka's consumer group rebalance and partition reassignment?
Consumer group rebalance is a client-side protocol that redistributes topic partitions among the consumers in a group when membership changes (consumer joins, leaves, or crashes). It is entirely managed by the group coordinator broker and the consumer clients — no changes to the Kafka cluster configuration. Partition reassignment is a cluster-side administrative operation that moves partition replicas between brokers — used for rebalancing load across brokers after adding new brokers or decommissioning old ones. Partition reassignment is triggered by the Kafka admin tool (or the partition reassignment API) and involves copying large amounts of log data between brokers, which can significantly impact cluster I/O performance. Throttle partition reassignment with kafka-reassign-partitions.sh --throttle in production to avoid starving consumer fetch traffic.
Q7: How does Kafka's page cache interact with broker performance?
Kafka deliberately relies on the OS page cache rather than managing its own in-process cache. When a consumer reads records that were recently produced, the OS serves them from page cache memory — zero disk I/O. This works because producers and consumers are typically in temporal proximity: a record produced now is usually consumed within seconds or minutes. If the Kafka broker has sufficient RAM for the page cache to hold the active working set (recent records), read throughput is bounded by network speed, not disk speed. This is why Kafka brokers should have as much RAM as possible, and why running other memory-intensive processes on Kafka broker nodes (e.g., Elasticsearch or a database) dramatically degrades Kafka performance — they compete for page cache RAM and force Kafka reads to go to disk.
Q8: What monitoring metrics are most critical for a production Kafka cluster?
The five highest-priority Kafka metrics are: (1) UnderReplicatedPartitions — any non-zero value indicates durability risk; (2) UnderMinIsrPartitionCount — partitions refusing writes; (3) consumer group lag (records-lag-max per partition) — measures how far behind consumers are; (4) producer record error rate (record-error-rate) — confirms produce success; and (5) broker request handler idle ratio (RequestHandlerAvgIdlePercent) — below 30% indicates the broker is CPU-saturated and approaching overload. Secondary metrics include: ActiveControllerCount (must be exactly 1), network bytes in/out rates for capacity planning, and GC pause duration. Set up alerting on UnderReplicatedPartitions > 0 as your highest-priority page — it is the earliest warning of a degraded cluster that hasn't yet failed writes.
Q9: How does Kafka compare to Pulsar for high-scale streaming?
Apache Pulsar separates storage from compute more aggressively than Kafka: Pulsar brokers are stateless, and all data is stored in Apache BookKeeper (a distributed ledger). This means Pulsar can scale brokers independently from storage, and adding brokers doesn't require moving partition replicas — it's as simple as adding stateless compute. Kafka's tightly coupled broker-storage model requires partition reassignment (copying data) when adding brokers, which is operationally complex at scale. On the other hand, Kafka's co-location of compute and storage on the same machine enables the page cache optimization that gives Kafka extreme read throughput for recently produced data — Pulsar reads always go over the network to BookKeeper. For most use cases at moderate scale (<1 million partitions), Kafka is better supported with more tooling. Pulsar's architectural advantages become compelling at hyperscale multi-tenant deployments where storage/compute independent scaling is critical.
Q10: How do I implement a dead-letter queue pattern in Kafka?
Kafka doesn't have a built-in DLQ concept, but the pattern is straightforward to implement: when your consumer fails to process a record after N retries (due to deserialization error, downstream system unavailability, or business rule violation), instead of crashing or infinitely retrying, produce the failed record to a dedicated topic-name.DLT (dead-letter topic) with headers containing the original offset, partition, error message, and exception stack trace. The consumer then commits the original offset and continues. A separate DLQ consumer application reads from the dead-letter topic for monitoring, alerting, and manual or automated replay. Kafka's Spring library provides @RetryableTopic and DeadLetterPublishingRecoverer that implement this pattern automatically. Always keep the dead-letter topic's retention long (30+ days) to allow time for investigation and replay without data expiry.