Raft Consensus Algorithm Under the Hood: Leader Election, Log Replication, and Safety Guarantees
Raft is the consensus algorithm powering etcd, CockroachDB, TiKV, and Consul — the backbone of Kubernetes, distributed SQL, and modern service discovery. In this deep walkthrough, you'll trace every message in a real leader election, understand exactly why Raft's safety guarantee is mathematically unbreakable, and learn what production systems do differently from the textbook algorithm.
1. The Consensus Problem: Why Distributed Agreement Is Hard
1.1 What Consensus Actually Means
Imagine you're building a distributed database with three replicas. A client writes a value. Which replica is authoritative? What happens if one replica receives the write but crashes before confirming? What if the network partition isolates two replicas from the third — do both halves continue accepting writes, potentially diverging forever? Consensus is the problem of getting a collection of unreliable machines to agree on a single value despite these failures.
Formally, a consensus algorithm must satisfy three properties simultaneously. Safety: all non-faulty nodes agree on the same value — they never commit conflicting decisions. Liveness: the system eventually makes progress and does not get stuck forever. Fault Tolerance: the system continues operating correctly as long as a majority of nodes are healthy. For a cluster of $2f + 1$ nodes, the system tolerates up to $f$ simultaneous failures. A 3-node cluster tolerates 1 failure; a 5-node cluster tolerates 2.
The critical insight behind majority quorums: any two majorities in a cluster of $2f + 1$ nodes must overlap by at least one node. This single overlapping node is what prevents split-brain — it can only have voted for one leader, so two different candidates cannot simultaneously claim majority support with conflicting histories.
1.2 Why Existing Algorithms Were Inadequate
Multi-Paxos, the dominant consensus algorithm before Raft, has a well-earned reputation for being notoriously difficult to implement correctly. Lamport's original Paxos paper describes single-decree consensus (agreeing on one value); Multi-Paxos for replicated state machines requires significant extension that was never formally specified. Real implementations (Chubby, ZooKeeper's ZAB) made different design decisions and were not interoperable. A 2013 study by Diego Ongaro and John Ousterhout surveyed practicing distributed systems engineers and found that the majority found Paxos difficult to understand even after extended study — and even more had abandoned correct implementation attempts.
Raft was designed from scratch with understandability as a first-class requirement, deliberately decomposing the consensus problem into three relatively independent subproblems: leader election, log replication, and safety. This decomposition makes each part easier to reason about in isolation, and makes the complete algorithm significantly easier to implement correctly.
Developer Pitfall — Consensus ≠ Coordination ≠ Replication:
Consensus, coordination, and replication are often conflated. Consensus is about agreeing on a sequence of decisions — a totally ordered log. Coordination (like distributed locks) is a use case built on top of consensus. Replication (like MySQL semi-sync replication) copies data without necessarily providing strong ordering guarantees. Raft gives you a linearizable, totally ordered log. You build everything else — locks, leader election for your application, distributed transactions — as state machine commands applied to that log. Do not confuse Raft with a database replication protocol; it is far more general.
2. Server States and the Leader Election State Machine
2.1 The Three States Every Raft Node Lives In
At any point in time, every server in a Raft cluster is in exactly one of three states: Follower, Candidate, or Leader. Understanding the transitions between these states — and the precise conditions that trigger them — is the foundation of understanding Raft.
Followers are passive. They respond to RPCs from leaders and candidates, but never initiate communication themselves. When a follower receives valid AppendEntries RPCs from a leader (the heartbeat mechanism), it resets its election timer and stays a follower. If the election timer expires — meaning no heartbeat arrived within the timeout window — the follower concludes the leader has failed and transitions to Candidate.
Candidates are servers actively trying to become leader. On transitioning to Candidate, a server increments its current term, votes for itself, resets its election timer, and sends RequestVote RPCs to all other servers in parallel. If it receives votes from a majority (including its own), it becomes Leader. If it receives an AppendEntries RPC from a server with a term at least as large as its own, it recognizes a legitimate leader and reverts to Follower. If the election timer expires without a majority — meaning a split vote — it starts a new election by incrementing term again.
Leaders handle all client requests and replicate log entries to followers via AppendEntries RPCs. They send periodic heartbeats (empty AppendEntries) to prevent followers from timing out and starting unnecessary elections. A leader reverts to Follower immediately upon discovering a server with a higher term number — this is how stale leaders are safely deposed.
Diagram 1: Raft Server State Machine. Every node is always exactly one of Follower, Candidate, or Leader. Transitions are driven by term numbers, RPC receipt, and election timer expiry.
2.2 Term Numbers: Raft's Logical Clock
Raft uses term numbers as a monotonically increasing logical clock to detect stale information. Every server persists its currentTerm to stable storage before acting on it. When a server starts a new election, it increments its term. When any server receives a message with a higher term than its own, it immediately updates its term and reverts to Follower — regardless of what state it was in. This single rule ensures that stale leaders (partitioned leaders who missed elections while isolated) are immediately deposed the moment they reconnect to the cluster.
Terms serve as logical epochs. All valid communication within Raft includes the sender's current term, and any message with a stale (lower) term is immediately rejected. A RequestVote from term 3 is ignored if the receiver is already in term 5. This prevents old elections from interfering with new ones and ensures the cluster always converges to the latest legitimate leader.
Developer Pitfall — Forgetting to Persist currentTerm and votedFor Before Responding:
Raft's safety properties require that currentTerm and votedFor survive server crashes. If a server votes for Candidate A in term 5, crashes, restarts with votedFor = null, and votes for Candidate B in term 5, both A and B could gather a majority — creating two leaders in the same term, violating safety. You MUST flush currentTerm and votedFor to stable disk storage (fsync) before sending any RPC response. Implementations that use in-memory state only, or that batch writes without proper fsync ordering, will silently produce unsafe behavior under crash scenarios.
3. Leader Election: Randomized Timeouts and Vote Restriction
3.1 Randomized Election Timeouts: The Key to Liveness
If all followers had the same election timeout, they'd all become candidates simultaneously upon a leader failure, split the votes, and potentially cycle through elections indefinitely without electing a leader — a livelock. Raft breaks this symmetry with randomized election timeouts: each server chooses its timeout randomly from a range (typically 150–300ms in the original paper; production systems often use 1–10 seconds to reduce unnecessary elections over slower networks).
The server that times out first becomes a candidate before the others wake up. It sends RequestVote RPCs immediately. The other servers, which haven't timed out yet, receive these RPCs and — if they haven't voted in this term and the candidate's log is at least as up-to-date as theirs — grant their vote. The first candidate usually collects a majority and becomes leader before anyone else even starts an election. The leader then sends heartbeats that reset everyone's timers, preventing further elections.
The timeout range must be chosen carefully: the minimum timeout must be at least an order of magnitude larger than the typical network round-trip time (so that heartbeats reliably arrive before timeouts expire), and the maximum must be large enough that a single server has enough time to gather majority votes before anyone else times out and starts a competing election. In a typical LAN cluster with <1ms RTT, 150–300ms works well. In a cross-datacenter WAN cluster with 50ms RTT, 500ms–2s is more appropriate.
3.2 The Vote Restriction: Only Complete Logs Can Win
The most critical safety rule in Raft is the election restriction: a candidate only receives a vote if its log is at least as up-to-date as the voter's log. This single rule guarantees that any elected leader already has all committed log entries — it never needs to "catch up" on committed entries after winning, which would require complex reconciliation logic prone to bugs.
"At least as up-to-date" is defined precisely: compare the last log entries of the candidate and voter. If they have different terms, the server with the higher term is more up-to-date. If the last entries have the same term, the server with the longer log is more up-to-date. A RequestVote RPC includes the candidate's lastLogIndex and lastLogTerm. A voter rejects the request if its own last log entry is more up-to-date than the candidate's. This is enforced locally — no coordination needed.
Developer Pitfall — Not Resetting the Election Timer When Granting a Vote:
When you grant a vote to a candidate, you should reset your own election timer. This gives the candidate — who just received your vote and is close to winning the election — time to complete the election and send heartbeats before your timer fires and you start a competing election. Many Raft implementations miss this subtlety and produce unnecessarily frequent elections because followers time out and start new elections even while a valid election is nearly complete. The Raft paper explicitly states the timer should be reset on granting a vote, but this is easily missed when implementing from the pseudocode alone.
4. Log Replication: How the Leader Keeps Followers in Sync
4.1 The AppendEntries RPC: Raft's Replication Workhorse
Once a leader is elected, it handles all client writes. When a client sends a command (e.g., "SET x=42"), the leader: (1) appends the command to its own log as a new entry with the current term, (2) sends AppendEntries RPCs to all followers in parallel, (3) once a majority of servers (including itself) have written the entry to their logs, marks it as committed, (4) applies it to its state machine and returns the result to the client.
The genius of this design is that leaders never overwrite their logs. A leader only ever appends. Followers are forced to conform to the leader's log — if a follower has inconsistent entries (from a previous leader that partially replicated before crashing), the current leader overwrites them. This makes the leader's log the single source of truth for all committed entries.
Each AppendEntries RPC carries a consistency check: the leader includes the index and term of the log entry immediately preceding the new entries (prevLogIndex, prevLogTerm). The follower only appends the new entries if its own log at prevLogIndex has the same term. If not — a consistency mismatch — the follower rejects the RPC, and the leader decrements its nextIndex for that follower and retries. This process walks backward through the log until the leader finds the point of divergence, then overwrites the follower's conflicting entries forward from that point.
Diagram 2: Log Replication Sequence. The leader appends locally, replicates to followers in parallel, commits once a majority acknowledges, applies to state machine, then notifies followers of the new commitIndex via subsequent AppendEntries.
4.2 The nextIndex and matchIndex Arrays
The leader maintains two arrays — one entry per follower — to track replication state. nextIndex[i] is the index of the next log entry to send to server $i$ (initialized to leader's last log index + 1 after winning an election). matchIndex[i] is the highest log index known to be replicated on server $i$ (initialized to 0, updated when a follower confirms an AppendEntries). The leader advances commitIndex to the highest index $N$ such that a majority of servers have matchIndex[i] ≥ N and the entry at index $N$ was created in the current leader's term.
That last condition — the entry must be from the current term — is a subtle but critical safety rule. A leader cannot directly commit entries from previous terms by counting replicas. It can only commit them indirectly by committing a new entry in its own term that follows them in the log. This prevents the "leader commits an old entry but then crashes before a new leader can tell the difference" scenario described in Figure 8 of the Raft paper.
Developer Pitfall — Committing Entries from Previous Terms Directly:
This is the single most commonly missed safety rule in Raft implementations. Suppose a leader from term 2 replicated entry[4] to only 2 of 5 servers, then crashed. A new leader in term 3 is elected. It sees that 2 servers have entry[4] (term 2). It cannot advance commitIndex to 4 just because those 2 servers agree — the other 3 servers might have a different entry[4] from a different term-2 leader. The correct behavior: the term-3 leader appends a no-op entry (or the next client command) at index 5 with term 3. Once index 5 is committed (majority agrees), entry[4] is also safely committed by transitivity. Never commit an old-term entry by majority count alone.
5. Step-by-Step: A Complete Election Walkthrough with Real Values
5.1 Setup: 5-Node Cluster, Leader Fails
Let's trace a complete Raft leader election with concrete values. We have a 5-node cluster: S1 (current leader, term 3), S2, S3, S4, S5. S1 crashes. All followers have their election timeouts running. Election timeout range: 150–300ms. Suppose the following timeouts were chosen:
5.2 T=0ms: S1 Crashes — Followers Stop Receiving Heartbeats
S1's last heartbeat arrived at T=0ms. S1 then crashes. All four remaining servers are waiting for the next heartbeat (which would normally arrive ~every 50ms). Their election timers are counting down.
5.3 T=162ms: S2's Election Timer Fires
5.4 T=163ms: S3, S4, S5 Receive RequestVote from S2
5.5 T=164ms: S2 Receives Votes from S3 and S4 — Becomes Leader
Developer Pitfall — Processing Stale RPC Replies After State Changes:
A common Raft implementation bug: S2 sends RequestVote RPCs in term 4 and becomes leader in term 4. Later, a delayed reply arrives from term 4. The code processes it and increments vote count again — incorrectly. You MUST validate that the reply's term matches your current term and that you are still a Candidate when processing any RPC reply. Use the pattern: if rf.state != Candidate || rf.currentTerm != args.Term { return } at the top of every RPC reply handler. This check must happen inside the lock to prevent races with concurrent state changes.
6. Network Partition: What Happens to a Stale Leader
6.1 The Partition Scenario
One of Raft's most important properties is how it handles a network partition that isolates the leader. This is the scenario that makes or breaks a consensus algorithm's safety guarantees. Let's trace through it precisely.
Setup: 5-node cluster, S1 is leader in term 4. A network partition isolates S1 and S2 from S3, S4, and S5. The minority partition (S1+S2) still has a "leader" — S1. The majority partition (S3+S4+S5) has no leader and will elect one. What happens to clients that write to S1 during the partition?
6.2 Partition Heals: The Stale Leader Is Safely Deposed
When the partition heals and S1 can communicate with S3, S4, S5 again, Raft handles the reconciliation automatically. S1 sends an AppendEntries with term 4. S3 (now leader in term 5) receives it and replies with term 5. S1 sees a higher term in the response, immediately steps down to Follower, updates its term to 5, and discards its uncommitted entry[11]. S2 follows the same path when it receives messages from the term-5 cluster.
The critical safety property: entry[11] ("SET y=99") was never committed because S1 could never get a majority acknowledgment during the partition. Therefore, overwriting it on S1 and S2 is perfectly safe — no client was ever told this write succeeded. Any client that sent "SET y=99" to S1 during the partition received either a timeout or an error (because S1 correctly refused to respond until it had majority). This is the read-your-own-writes guarantee: Raft only acknowledges to clients what has been durably committed to a majority, so no committed data is ever lost.
Developer Pitfall — Returning Success to Clients Before Commit:
Never acknowledge a client write until commitIndex has advanced to include that entry. A common implementation error (especially in tutorial implementations) is to return success to the client after appending to the leader's log but before receiving majority acknowledgment. This violates Raft's linearizability guarantee: if the leader then crashes before committing, the write is lost, but the client believes it succeeded. The correct implementation: the client's request handler blocks (or registers a callback) until the entry's index has been applied to the state machine, which only happens after commitment. Use a per-log-index condition variable or channel for this.
7. Log Compaction: Snapshots and the InstallSnapshot RPC
7.1 Why Logs Cannot Grow Forever
A replicated log that only ever grows eventually consumes all available disk space and makes crash recovery prohibitively slow — replaying millions of log entries on startup would take minutes. Raft addresses this with snapshotting: periodically, each server serializes its current state machine state to a snapshot file and discards the log entries that led to that state. The snapshot file contains the complete state machine state plus the index and term of the last entry included in the snapshot (lastIncludedIndex, lastIncludedTerm).
Snapshotting is typically application-triggered. The application (e.g., a key-value store on top of Raft) decides when the log is large enough to warrant a snapshot, takes the snapshot of its current state, and tells the Raft library which log index the snapshot covers. The library then safely discards log entries up to and including that index. This is straightforward for an up-to-date server — it snapshots what it has already applied.
7.2 The InstallSnapshot RPC: Catching Up Lagging Followers
The harder case is a follower that has fallen so far behind the leader that the leader has already discarded the log entries the follower needs. This happens when a server is partitioned for a long time or is newly added to the cluster. The leader cannot send the missing AppendEntries because it no longer has them. Instead, it sends the entire snapshot via an InstallSnapshot RPC.
The follower receives the snapshot, discards its own log up to lastIncludedIndex, loads the snapshot's state into its state machine, and resumes normal AppendEntries from lastIncludedIndex + 1. For large state machines (hundreds of GB), this snapshot transfer can take significant time — production systems use chunked transfer and streaming to avoid RPC size limits and allow the follower to apply chunks incrementally rather than waiting for the entire snapshot before proceeding.
Developer Pitfall — Discarding the Snapshot's lastIncludedIndex Entry From the Log:
After installing a snapshot at lastIncludedIndex, many implementations incorrectly discard all log entries including index lastIncludedIndex. You must retain a dummy log entry at index lastIncludedIndex with term = lastIncludedTerm, even after the snapshot. This is because subsequent AppendEntries RPCs will arrive with prevLogIndex = lastIncludedIndex and the consistency check requires reading the term at that index. If the entry is gone, the consistency check will fail and the follower will reject all future AppendEntries — effectively becoming permanently stuck. Keep the sentinel entry; only discard the real command payload.
8. Cluster Membership Changes: The Joint Consensus Problem
8.1 Why You Can't Just Add Servers Naively
Changing the cluster membership (adding or removing servers) while the cluster is running is surprisingly dangerous. Consider adding a server to a 3-node cluster (S1, S2, S3) by directly switching to a 4-node configuration (S1, S2, S3, S4). During the brief window when some servers have switched to the new configuration and others haven't, you might have S1+S2 forming a majority of the old 3-node configuration simultaneously with S3+S4 forming a majority of the new 4-node configuration — two independent majorities, potentially electing two different leaders. This is precisely the split-brain scenario Raft is designed to prevent.
The Raft paper proposes two approaches. The simpler approach (preferred by most production implementations including etcd) is single-server membership changes: only add or remove one server at a time. A single addition or removal cannot create two disjoint majorities because any majority of the old configuration overlaps with any majority of the new configuration by at least one server — the overlapping server ensures at most one leader can be elected. Adding multiple servers is done as a sequence of single additions.
8.2 The Joint Consensus Approach for Arbitrary Changes
For cases where simultaneous multi-server changes are needed, Raft uses Joint Consensus: a two-phase protocol with a transitional joint configuration $C_{old,new}$ that requires majority agreement from both the old and the new configuration independently. During the joint phase, all decisions require both an old-config majority and a new-config majority. This ensures that neither the old nor the new configuration can independently elect a leader during the transition — eliminating the split-brain window. Once the joint configuration is committed, the cluster transitions to the final new configuration.
Developer Pitfall — Removing the Current Leader From the Cluster:
When you remove the current leader from the cluster configuration, a special edge case arises. The leader must continue operating (replicating the membership change log entry) until the new configuration is committed — but once it's committed, the leader is no longer part of the cluster. The correct behavior: the old leader steps down to Follower as soon as it commits the new configuration that excludes it. It then does not count its own heartbeats in the new election and allows the remaining cluster to elect a new leader. Some implementations miss this and keep the old leader running indefinitely, occasionally winning heartbeat races and confusing the new cluster.
9. Advanced: Production Optimizations Beyond the Paper
9.1 Pre-Vote: Eliminating Disruptive Reconnecting Servers
Consider a server that was partitioned from the cluster for an extended time. During its isolation, it kept timing out and incrementing its term — it might now have term 50 while the rest of the cluster is at term 5. When it reconnects, its first AppendEntries response will contain term 50, causing the current leader (at term 5) to immediately step down, even though the partitioned server has a stale log and cannot win an election. This causes unnecessary leader churn.
The Pre-Vote extension adds a pre-election phase: before incrementing its term and sending real RequestVotes, a candidate first sends a Pre-Vote request asking "would you vote for me if I started an election?" Servers respond yes only if they haven't heard from a leader recently and the candidate's log is up-to-date. Only if the candidate receives a pre-vote majority does it start a real election. Reconnecting partitioned servers cannot disrupt the cluster because their stale logs fail the pre-vote log check. etcd's Raft implementation has used Pre-Vote in production since 2017.
9.2 Leader Leases: Serving Reads Without Round-Trips
In basic Raft, a leader must confirm its leadership before serving linearizable reads (by either appending a no-op entry or doing a heartbeat round-trip). This adds latency to every read. Leader leases optimize this: after winning an election, a leader is guaranteed to be the only leader for at least one election timeout period (because followers won't elect a new leader until their timer expires). If the leader keeps track of when it last successfully sent heartbeats, it can serve reads locally for the duration of the lease without a round-trip — provided its clock drift relative to follower clocks is bounded and known.
Leader leases require careful clock handling. If a leader's clock runs slower than a follower's, the follower's election timer might expire before the leader's lease does, creating a brief window where two leaders coexist. Production implementations like CockroachDB use hybrid logical clocks (combining physical time with logical counters) to bound clock drift and ensure lease correctness. Never implement leader leases without understanding the clock assumptions your environment guarantees.
9.3 Pipelined AppendEntries: Saturating Bandwidth
The basic Raft protocol sends an AppendEntries RPC and waits for the response before sending the next batch — a stop-and-wait protocol that underutilizes network bandwidth. Pipelining allows the leader to send multiple in-flight AppendEntries RPCs to a follower without waiting for previous ones to complete. The leader maintains an in-flight window and advances nextIndex optimistically. If a follower rejects a batch (consistency failure), the leader falls back to the original sequential retry. TiKV's Raft implementation uses pipelining to achieve multi-Gbps replication throughput between replicas — compared to tens of Mbps with stop-and-wait.
Developer Pitfall — Heartbeat Interval vs Election Timeout Ratio:
The heartbeat interval must be significantly smaller than the election timeout minimum — the Raft paper recommends at least a 10:1 ratio. If heartbeats arrive at 100ms intervals and the minimum election timeout is 150ms, a single delayed heartbeat (due to network jitter, GC pause, or CPU scheduling) will trigger an unnecessary election, causing leader churn and a brief write unavailability window. Production systems set the heartbeat to 1/20th the election timeout minimum. In etcd, the default heartbeat is 100ms and the election timeout is 1000ms (10:1 ratio). If you're seeing frequent unexpected elections in your Raft cluster, the first thing to check is this ratio.
10. Raft vs Paxos vs Viewstamped Replication
Raft is not the only consensus algorithm. Understanding how it compares to its predecessors helps you appreciate its specific design tradeoffs and recognize when a different algorithm might be more appropriate.
| Dimension | Raft | Multi-Paxos | Viewstamped Replication |
|---|---|---|---|
| Primary design goal | Understandability & correctness | Theoretical elegance & generality | Practical state machine replication |
| Leader role | Strong leader — all writes go through leader | Weak leader (any server can propose independently) | Primary replica (similar to Raft leader) |
| Leader election | Randomized timeouts + log completeness vote restriction | Phase-1 prepare/promise (any proposer) | View change protocol with explicit timeout |
| Log conflict resolution | Leader's log always wins; followers overwritten | Any accepted value from highest-ballot proposal wins | Primary's log is authoritative |
| Membership changes | Joint consensus or single-server changes | Not specified in original paper | Reconfiguration protocol (view change) |
| Production adoption | etcd, CockroachDB, TiKV, Consul, RethinkDB | Chubby (Google), ZooKeeper (ZAB ≈ Paxos), Spanner | VoltDB, some academic systems |
| Implementation complexity | Moderate — decomposed subproblems | High — many subtle edge cases unspecified | Moderate — well-specified but less documented |
The practical reason Raft dominates new distributed systems development is not that it's theoretically superior — it isn't. Its quorum requirements, latency characteristics, and throughput limits are nearly identical to Multi-Paxos for typical workloads. The advantage is operational: Raft's stronger leadership model (one leader, all writes go through it) makes log reasoning simple, its paper is fully specified including membership changes and log compaction, and its reference implementation (the Go MIT 6.824 lab) has been studied, debugged, and extended by thousands of developers. A well-specified, commonly implemented algorithm produces fewer production surprises than a theoretically optimal but underspecified one.
11. Raft in Production: etcd, CockroachDB, and TiKV
11.1 etcd: Raft as Kubernetes' Backbone
etcd is a distributed key-value store that serves as Kubernetes' primary storage backend — every cluster state change (Pod scheduling, ConfigMap updates, Service endpoints) is written to etcd. etcd's Raft implementation adds several production enhancements: Pre-Vote (since v3.4), linearizable read index optimization (a lightweight leader confirmation that doesn't append a log entry), and watch streams (efficient long-polling for key changes via Raft log apply callbacks).
A critical etcd operational consideration: etcd stores all data in a BoltDB B-tree that holds the entire dataset in memory-mapped files. As the dataset grows beyond a configured threshold (default 2GB), etcd requires manual compaction (removing old MVCC revisions) and defragmentation (reclaiming freed BoltDB pages). Kubernetes clusters with high churn (frequent Pod scheduling) can hit this limit quickly. The Kubernetes recommendation is to compact and defrag etcd every 8 hours in production and to monitor etcd_mvcc_db_total_size_in_bytes.
11.2 CockroachDB: Multi-Raft for Distributed SQL
CockroachDB partitions its keyspace into ranges (typically 512MB chunks) and runs an independent Raft group for each range — thousands of concurrent Raft instances per node. This Multi-Raft architecture allows different key ranges to have different leaders on different nodes, distributing both read and write load across the cluster. It also means that a node failure affects only the Raft groups it was part of; the rest of the cluster continues uninterrupted.
CockroachDB's Raft implementation includes several optimizations for the SQL workload: transaction batching (multiple SQL DML operations are batched into a single Raft log entry to reduce consensus round-trips), closed timestamps (a gossip-based mechanism for serving consistent reads at a historical timestamp without Raft, dramatically reducing read latency for analytics queries), and raft log truncation policies that aggressively snapshot and compact to prevent any single range's log from growing unboundedly.
Developer Pitfall — Running Raft on Servers With Non-Monotonic Clocks:
Raft's election timeouts and heartbeat intervals rely on wall-clock time for liveness (not safety — safety is purely based on term numbers and log contents). NTP clock slews or settimeofday calls that jump the clock backward can cause election timers to fire prematurely or fail to fire, causing spurious leader elections or leader failures. Always use monotonic clocks (Go's time.Now().UnixNano() returns wall time; use time.Since() with a monotonic reference instead) for all timeout calculations in a Raft implementation. etcd explicitly uses monotonic clock reads for all timeout tracking.
12. Frequently Asked Questions
Q1: How long does a Raft leader election typically take in production?
A Raft leader election takes roughly one election timeout (the time from when a follower's timer fires to when a candidate receives majority votes). With a 150–300ms timeout range and <5ms network RTT (local datacenter), elections complete in 150–310ms. With the Pre-Vote extension, actual elections are rarer and the pre-vote phase adds at most one network round-trip. In etcd's default configuration (election timeout 1 second), clients experience 1–2 seconds of write unavailability during a leader election — this is why Kubernetes recommends using etcd in a 3+ node configuration to ensure fast re-election and minimizing planned maintenance windows to avoid forced leader elections.
Q2: Can Raft guarantee zero data loss on a leader crash?
Yes — for committed entries. Any entry that has been acknowledged to the client as committed (meaning a majority wrote it to their durable logs) will survive any combination of up to $f$ simultaneous failures in a $2f+1$ node cluster. The surviving majority always contains at least one server with the committed entry, and Raft's vote restriction ensures only a candidate with that entry can become leader. Uncommitted entries (those not yet acknowledged to the client because the leader crashed before getting majority) may be lost — but since the client never received a success response, this is not considered data loss. It is correct for the client to retry the operation on the new leader.
Q3: Why can't followers serve linearizable reads in basic Raft?
In basic Raft, followers may be arbitrarily behind the leader's commitIndex — they apply log entries at their own pace, and their state machine may be stale by seconds or more. A read served from a stale follower returns old data, violating linearizability (which requires reads to return the most recently committed value). To serve linearizable reads, either: route all reads through the leader (which confirms its leadership via a heartbeat round-trip or Read Index mechanism before answering), or use lease reads (if clock assumptions are met). Serving reads from followers is only safe for explicitly relaxed consistency models (e.g., "stale reads" or "bounded staleness"), which some systems (CockroachDB's follower reads, TiDB's stale read) offer as an optional lower-latency mode.
Q4: What is the Read Index optimization and how does it avoid a log append for reads?
A naive linearizable read appends a no-op entry to the log and waits for it to commit — expensive for read-heavy workloads. The Read Index optimization (used by etcd) works differently: when serving a read, the leader (1) records the current commitIndex as the read index, (2) sends a heartbeat to all followers and waits for majority acknowledgment (confirming it is still leader), (3) waits until its own applied index reaches the saved read index, then (4) serves the read from local state. This confirms leadership without appending to the log, saving a round of disk I/O. CockroachDB uses a similar mechanism with its "lease holder" read path, also skipping unnecessary log appends for local reads.
Q5: How does Raft handle write throughput at scale — doesn't the leader become a bottleneck?
The leader is inherently a write bottleneck in single-Raft deployments — all writes serialize through one node. The solutions used in production are: (1) Multi-Raft (CockroachDB, TiKV) — partition the keyspace into shards, each with an independent Raft group and a different leader node, distributing writes across all nodes; (2) Batching — aggregate multiple client commands into a single Raft log entry, amortizing the consensus round-trip cost across many operations; (3) Pipelining — send multiple AppendEntries in-flight without waiting for previous replies. With batching and pipelining, a single Raft group in TiKV can achieve 100k+ writes/second on NVMe storage. For truly global scale (millions of writes/second), Multi-Raft with hundreds of shard groups is the standard architecture.
Q6: What is the difference between Raft and ZooKeeper's ZAB protocol?
ZAB (ZooKeeper Atomic Broadcast) and Raft are both leader-based consensus algorithms that maintain a totally ordered, replicated log. The key differences are: ZAB uses explicit epoch numbers (called "epochs") while Raft uses terms; ZAB's leader recovery phase (Fast Leader Election) uses epoch and log offset for comparison rather than Raft's (term, index) pair; ZAB separates the broadcast phase (replication) more explicitly from the recovery phase. Practically, both algorithms provide the same safety and liveness guarantees with similar performance characteristics. ZooKeeper's ZAB implementation predates Raft and has decades of production hardening — for new systems, Raft is preferred due to better specification and more available implementations. For systems already using ZooKeeper, there's no compelling reason to migrate.
Q7: Can Raft tolerate Byzantine (malicious) failures?
No — Raft assumes crash-fault tolerance only (CFT), not Byzantine fault tolerance (BFT). A Byzantine node is one that behaves arbitrarily maliciously: sending conflicting messages to different peers, lying about its log, or colluding with other Byzantine nodes. Raft has no defense against this — a Byzantine candidate could claim false log completeness, a Byzantine leader could send different entries to different followers, and there is no cryptographic verification of message integrity. For Byzantine fault tolerance (required in blockchains and some financial systems), algorithms like PBFT, HotStuff, or Tendermint are necessary — but they require $3f+1$ nodes to tolerate $f$ failures (compared to Raft's $2f+1$) and have significantly higher message complexity. If your threat model includes untrusted nodes, Raft is the wrong algorithm.
Q8: How do you test a Raft implementation for correctness?
Testing distributed consensus implementations requires deliberate fault injection. The standard approach is: (1) deterministic simulation — run the entire cluster in a single process with a simulated network that can inject partitions, delays, and message reordering on demand; (2) Jepsen-style testing — deploy a real cluster, inject network partitions using iptables rules, crash servers randomly, and use a linearizability checker (like Knossos or Elle) to verify that the history of client operations is linearizable; (3) TLA+ model checking — the Raft paper comes with a TLA+ specification that can be model-checked for safety violations up to bounded state space sizes. The MIT 6.824 distributed systems course lab includes an aggressive test suite that injects concurrent failures, network delays, and unreliable message delivery — passing this suite is a strong signal of implementation correctness.
Q9: What happens if a Raft cluster loses a majority permanently?
If more than $f$ servers fail permanently in a $2f+1$ cluster (e.g., 2 of 3 servers fail), the cluster loses the ability to commit new entries — it becomes unavailable for writes. Reads from surviving servers may still be possible, but they will not reflect the latest committed state. Recovery from this scenario requires operator intervention: if you can recover the failed servers from backup and restore their state, the cluster can resume. If servers are truly gone, you may need to perform a "disaster recovery" by forcing a single surviving server to declare itself leader with reduced quorum — this is an explicitly unsafe operation that could violate safety if the failed servers come back online, and etcd requires explicit flags (--force-new-cluster) to prevent accidental use. Always design for failure scenarios before they happen by maintaining proper backups and monitoring.
Q10: What is the no-op log entry that a new Raft leader appends immediately after winning?
When a new Raft leader wins an election, it immediately appends a no-op entry to its log (an entry with no client command, just a term marker) and replicates it to followers. This is required because of the subtle safety rule discussed earlier: a leader cannot commit entries from previous terms by replication count alone — it can only commit them transitively by committing a current-term entry. By appending a no-op and committing it, the new leader simultaneously advances commitIndex to include all previously replicated-but-uncommitted entries from prior terms, bringing the cluster to a consistent state. Without this no-op, a newly elected leader would serve stale reads (its state machine is behind) and would not advance commitIndex for potentially committed prior entries until the next client write arrives — which could be never.