Quick Answer

KRaft (Kafka Raft) replaces Apache ZooKeeper by storing Kafka's cluster metadata as an internal Raft-replicated event log managed by a small quorum of controller nodes. The result: sub-2-second controller failover instead of tens of seconds, support for well over a million partitions per cluster, and one fewer distributed system to operate. Kafka 4.0 removed ZooKeeper support entirely, making KRaft mandatory for any new deployment.

For over a decade, Apache ZooKeeper was an unavoidable operational dependency for Apache Kafka. It stored broker metadata, partition leader assignments, and access control lists in an entirely separate distributed system that had to be sized, secured, and monitored independently from Kafka itself. With the introduction and production maturation of KRaft (Kafka Raft Metadata Mode) , metadata is now stored as an event log directly inside Kafka, dramatically reducing partition failover latency and operational surface area.

1. KRaft Architecture vs. ZooKeeper Architecture

The core architectural shift is where metadata lives and how it propagates. ZooKeeper treated Kafka brokers as clients of an external system; KRaft makes metadata management a native Kafka responsibility, handled by a subset of brokers running in controller mode.

LEGACY: ZOOKEEPER-BASED CLUSTER ZooKeeper Ensemble ZK-1 ZK-2 ZK-3 External metadata store Kafka Brokers (data plane only) Broker A Broker B Broker C Controller election: ~15–30s KRAFT: NATIVE CONSENSUS CLUSTER Controller Quorum (Raft) Ctrl-1 Ctrl-2 Ctrl-3 Metadata = internal event log Kafka Brokers (same process) Broker A Broker B Broker C Controller failover: < 2s ~10–15x faster failover with KRaft quorum
Fig. 1 — ZooKeeper externalizes metadata and elects a controller sequentially; KRaft replicates metadata via Raft consensus inside the same quorum that serves data.

2. KRaft vs. ZooKeeper: Side-by-Side Comparison

Dimension ZooKeeper Mode KRaft Mode
Metadata storage External ensemble, separate JVMs Internal Kafka topic ( __cluster_metadata )
Controller failover ~15–30 seconds, full state reload < 2 seconds, already replicated
Max practical partitions Tens of thousands per cluster 1M+ partitions demonstrated
Operational footprint Two systems to patch, secure, monitor One system, unified security model
Kafka version support Removed entirely in Kafka 4.0 Default since Kafka 3.3 (production-ready)

3. Migrating an Existing ZooKeeper Cluster to KRaft

Kafka ships a built-in, zero-downtime migration path rather than requiring a rebuild from scratch:

  1. Provision a KRaft controller quorum (3 or 5 nodes) running in migration mode alongside the existing ZooKeeper ensemble.
  2. Enable the migration flag ( zookeeper.metadata.migration.enable=true ) on the KRaft controllers so they begin shadowing ZooKeeper's metadata.
  3. Roll brokers one at a time into KRaft mode; each restarted broker registers with the new controller quorum instead of ZooKeeper.
  4. Verify metadata parity using kafka-metadata-quorum.sh to confirm the KRaft log matches the prior ZooKeeper state before cutting over.
  5. Decommission ZooKeeper once all brokers report KRaft mode and no client depends on direct ZooKeeper access (a common oversight with older admin tooling).

4. Configuring Spark Structured Streaming for Fault Tolerance

Pairing Kafka KRaft with Apache Spark 3.x requires careful tuning of offset commits and stateful watermarking to prevent data loss or duplicate execution across restarts:

val df = spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "broker-1:9092,broker-2:9092") .option("subscribe", "telemetry-events") .option("startingOffsets", "latest") .option("failOnDataLoss", "false") .load() val processed = df .withWatermark("timestamp", "10 minutes") .groupBy(window($"timestamp", "5 minutes", "1 minute"), $"device_id") .count() processed.writeStream .format("delta") .option("checkpointLocation", "/chk/telemetry-agg") .outputMode("update") .start("/lakehouse/telemetry_agg")

Exactly-once semantics across the pipeline

True end-to-end exactly-once delivery requires three cooperating guarantees, not just one Spark setting:

  • Kafka producer idempotence ( enable.idempotence=true ) prevents duplicate writes from producer-side retries.
  • Spark's checkpoint + write-ahead log ensures a failed micro-batch resumes from the exact last committed offset rather than reprocessing arbitrarily.
  • An idempotent sink — a Delta Lake MERGE keyed on event ID, or a JDBC upsert — absorbs any batch that is replayed after a restart without creating duplicate rows.

5. Common Migration Pitfalls

  • Skipping metadata parity verification. Cutting brokers over before confirming the KRaft log matches ZooKeeper state can silently drop topic configuration or ACLs.
  • Forgetting tools that talk to ZooKeeper directly. Older admin scripts, some GUI tools, and custom monitoring occasionally bypass the Kafka API and hit ZooKeeper directly — these break silently after decommission.
  • Under-provisioning the controller quorum. A 3-node quorum tolerates one failure; high-partition-count clusters (100k+) often warrant 5 controller nodes.
  • Not tuning watermarks for real event-time lateness. A too-tight watermark drops genuinely late IoT/mobile events; a too-loose one delays window finalization and inflates state size.
Key Takeaways
  • KRaft replaces ZooKeeper with Raft-based consensus stored inside Kafka itself, cutting controller failover from tens of seconds to under two.
  • Kafka 4.0 removed ZooKeeper entirely — KRaft is now mandatory for new clusters and the safe long-term target for existing ones.
  • Migration is zero-downtime via a shadow controller quorum, but requires verifying metadata parity before decommissioning ZooKeeper.
  • Exactly-once processing needs three layers working together: idempotent Kafka producers, Spark checkpointing, and an idempotent sink such as Delta Lake MERGE.
  • KRaft clusters scale to 1M+ partitions, roughly two orders of magnitude beyond practical ZooKeeper-based limits.

Frequently Asked Questions

What is KRaft mode in Apache Kafka?

KRaft (Kafka Raft) is Kafka's built-in consensus protocol that stores cluster metadata as an internal, Raft-replicated event log managed by a small quorum of controller nodes, removing the need for an external Apache ZooKeeper ensemble.

Is ZooKeeper still required for Apache Kafka?

No. KRaft has been production-ready since Kafka 3.3, and Kafka 4.0 removed ZooKeeper support entirely. Existing ZooKeeper clusters can migrate to KRaft with zero downtime using Kafka's built-in migration tooling.

How much faster is controller failover with KRaft?

KRaft clusters typically recover from a controller failure in under two seconds, versus tens of seconds in ZooKeeper-based clusters, because metadata is already replicated via Raft consensus rather than reloaded on election.

How do I achieve exactly-once processing with Kafka and Spark Structured Streaming?

Combine an idempotent Kafka producer, Spark Structured Streaming's checkpointing and write-ahead log, and an idempotent sink such as a Delta Lake MERGE keyed on event ID, so replayed micro-batches never create duplicate rows.

How many partitions can a KRaft cluster support?

KRaft-based clusters have been benchmarked well beyond a million partitions, roughly two orders of magnitude past the practical ceiling of ZooKeeper-based clusters, since metadata propagation no longer depends on synchronous ZooKeeper znode writes.

Summary

Deploying KRaft simplifies Kubernetes orchestration, removes external ZooKeeper pods, and ensures predictable sub-second streaming latencies. Read our Teaching Curriculum to explore hands-on Kafka streaming sandbox labs, or see how these pipelines feed downstream Lakehouse table formats .