kafka without zookeeper

A node is a machine that operates on the cluster. It's another set of metrics and another set of admin tools that you have to learn. The most exciting part of this project, of course, is the ability to run without ZooKeeper, in “KIP-500 mode.” When Kafka is run in this mode, we will use a Raft quorum to store our metadata rather than ZooKeeper. We expect that a single Kafka cluster will eventually be able to support a million partitions or more.Several administrative tools shipped as part of the Kafka release still allow direct communication with ZooKeeper. The nodes should be able to choose their own leader, and this is actually where Raft comes into the picture. That was obviously going through the broker. The broker partition hasn't really changed either. One example is Replacing ad hoc ZooKeeper APIs with well-documented and supported RPCs has many of the same benefits as removing client-side ZooKeeper access did. If you have a partition which relies on that other broker, that will not be replicated. Currently, when a topic is created or deleted, the controller must reload the full list of all topic names in the cluster from ZooKeeper. As the amount of cluster metadata grows, this process takes longer and longer.In contrast, in the post-KIP-500 world, there will be several standby controllers that are ready to take over whenever the active controller goes away. But in this chapter, we'll use ZooKeeper package that's available in Ubuntu's default repositories. I'm using kafka 0.10 without zookeeper. If we can't talk to the controller, or rather if the controller can't talk to us, we miss out on that stuff, but at the same time we're still regarded as part of the cluster because we keep our Zookeeper session alive.This is a very specific example of how the separation between the controller and Zookeeper actually creates awkward situations because they're two separate systems. This is a very important one.Let's talk a little bit about network partitions. There's a lot of different types of fencing in Kafka, and I couldn't possibly talk about all of them. This is the ideal model of what we would like to happen. For example, maybe they should be fenced if there's already a broker with that ID and stuff like that.Fencing is an important topic in general. It is planned to be programmable and simple to use. Rather than sending out snapshots, we'd actually like to treat it as a log. This is similar to how consumers that need the latest log entries only need to read the very end of the log, not the entire log. These standby controllers are simply the other nodes in the Raft quorum of the metadata partition. We have this concept of dynamic configuration, which is basically configurations that you can change over time in a managed way and have them changed at runtime.We also have this thing called the ISR, which is basically the in synch replica set, and this has to do with how we do replication. And it will finally prove that…Kafka needs no keeper.To learn about other work that is happening to make Kafka Colin McCabe currently works for Confluent to improve the performance, scalability, and general awesomeness of Apache Kafka. Then there's the case of just being partitioned from a broker or several brokers. This is necessary because while ZooKeeper notifies us when the set of topics in the cluster has changed, it doesn’t tell us exactly which topics were added or removed. ; Serf vs. ZooKeeper, doozerd, etcd — Hashicorp’s comparison of Serf to ZK and others. Information such as:It’s a tool that applications can use to recover from partial failures in your cluster. We have the benefits of better compatibility, and of course we just have better encapsulation. Finally, once you've rolled all the brokers, you can decommission the Zookeeper nodes.In summary, it is going to be possible to upgrade to a post Zookeeper release with zero downtime.Wrapping up here, Zookeeper has served us pretty well. It is used to establish co-ordination within a cluster of various nodes. Those things are all important here, I would say.I think this is a great example of how managing metadata as a log is much more effective because it eliminates this need to send full snapshots.

This is a familiar pattern if you've used Zookeeper before, this idea of having a leader, which is elected by Zookeeper, where you delegate certain responsibilities to that node. You can not use kafka without zookeeper. We would like to minimize the number of operations that require a time linearly proportional to the number of topics or partitions.One such operation is controller failover. This case is interesting because you will not be regarded as part of the cluster, and yet nodes can maybe continue talking to you, which creates a bit of an uncomfortable situation.