Apache Kafka cluster setup

This article will provide simple steps to set up a Kafka cluster with 3 nodes. Also, we will see how to set up a Kafka zookeeper cluster. So let’s get started.

From Apache’s site, download the Kafka binary and extract the zip file on each cluster node.

Steps to setup Zookeeper cluster configs

Go to the Kafka home directory and open zookeeper’s config file — config/zookeeper.properties and set the following fields.

1. dataDir — Directory where you want ZooKeeper to save its data.
2. clientPort — The port at which the clients will connect. Default — 2181
3. maxClientCnxns — This property limits the number of active connections from a host, specified by IP address, to a single ZooKeeper server. — Optional. Default — 60
4. tickTime — The number of milliseconds of each tick. — Optional
5. initLimit — The number of ticks that the initial synchronization phase can take. — Optional
6. syncLimit — The number of ticks that can pass between sending a request and getting an acknowledgment. — Optional
7. server.x=x.x.x.x:2888:3888 — For each zookeeper node there should be one config entry like this.

• x in server.x denotes the id of the zookeeper node. Each zookeeper server is assigned an id x by creating a file named myid with value x inside dataDir. It should be a unique id.
• The ports, 2888:3888 (Don’t change) that is at the end of the nodes. Zookeeper nodes will use these ports to connect the individual follower nodes to the leader nodes. The other port is used for leader election.
• x.x.x.x is each zookeeper node IP Address better to use private IP here. If you are trying to use public IP, current node IP should be replaced with 0.0.0.0 in each node.

Steps to setup Kafka broker cluster configs

Go to the Kafka home directory and open Kafka broker’s config file — config/server.properties and set the following fields.

1. broker.id — Id of the broker i.e. an integer — 1, 2 etc. Each broker in a cluster needs to have a unique id.
2. Uncomment listeners=PLAINTEXT://:9092 — For connection request, Kafka broker will be listening at this port.
3. log.dirs — Directory where you want Kafka to commit its message. Not to be confused with usual log files.
4. zookeeper.connect — Comma separate list of ZooKeeper nodes. E.g. zookeeper-node-1:port1,zookeeper-node-2:port2,zookeeper-node-3:port3.
5. advertised.host.name — Broker hostname/IP to publish to ZooKeeper for clients to use.
6. Set offsets.topic.replication.factor, transaction.state.log.replication.factor, and transaction.state.log.min.isr value to minimum 2 in a cluster of 1+ nodes. This will ensure the availabiltiy of the cluster.

Command to start the Zookeeper and Kafka brokers

• On each zookeeper’s node, run the command <Kafka Home Directory>/bin/zookeeper-server-start.sh -daemon config/zookeeper.properties to start the zookeeper as a daemon.
• On each Kafka node, run the command <Kafka Home Directory>/bin/kafka-server-start.sh -daemon config/server.properties to start the Kafka broker as a daemon.

Command to verify the Zookeeper and Kafka brokers

Run jps command and output must have following info -

• <PID> Kafka
• <PID> QuorumPeerMain