Kafka Cluster using Containers - p1

April 25, 2023
Category: Message Brokers
Tags: Certificates Kafka

Building a Kafka cluster using containers

Environment

Four servers running Debian 11

kfk231.domain.dom - 192.168.122.231 - zookeeper, broker, connect
kfk232.domain.dom - 192.168.122.232 - zookeeper, broker, connect
kfk233.domain.dom - 192.168.122.233 - zookeeper, broker, connect
kfk234.domain.dom - 192.168.122.234 - utilities

Docker 23.0.5

All four servers have a root drive and a second drive used for storing the data.
Docker's data-root points to the /data drive.
Also all the compose files will be run from the /data drive from their own directories.

Creating Certificates

Working on the fourth server, kfk234.

Installing openjdk-17-jre-headless:

apt install openjdk-17-jre-headless

which keytool
/usr/bin/keytool

Creating directories - on the second drive /data.

mkdir kafka
cd kafka
mkdir certificates
cd certificates/
mkdir cetificates-single
cd cetificates-single/

Creating an .env file to store the password for certificate.

.env file

SRVPASS=Password123

Config file for csr

openssl-config.cnf

[ v3_req ]
# Extensions to add to a certificate request


subjectAltName = @alt_names

[ alt_names ]
DNS.1 = kfk231
DNS.2 = kfk231-kafka
DNS.3 = kfk231-zk
DNS.4 = kfk231-cnt
DNS.5 = kfk231-reg
DNS.6 = kfk231.domain.dom
DNS.7 = kfk232
DNS.8 = kfk232-kafka
DNS.9 = kfk232-zk
DNS.10 = kfk232-cnt
DNS.11 = kfk232-reg
DNS.12 = kfk232.domain.dom
DNS.13 = kfk233
DNS.14 = kfk233-kafka
DNS.15 = kfk233-zk
DNS.16 = kfk233-cnt
DNS.17 = kfk233-reg
DNS.18 = kfk233.domain.dom
DNS.19 = kfk234
DNS.20 = kfk234.domain.dom
DNS.21 = IP:192.168.122.231
DNS.22 = IP:192.168.122.232
DNS.23 = IP:192.168.122.233
DNS.24 = IP:192.168.122.234

The certificate create script create_certificate.sh:

#!/bin/bash

set -o nounset \
    -o errexit \
    -o verbose \
    -o xtrace

# Get password from .env file
export $(grep -v '^#' .env | xargs)

# Generate CA key
openssl req -new \
    -newkey rsa:4096 \
    -x509 -keyout kfk-ca.key \
    -out kfk-ca.crt \
    -days 365 \
    -subj '/CN=kfk/OU=Test/O=domain.dom/L=Toronto/S=On/C=CA' \
    -passin pass:$SRVPASS \
    -passout pass:$SRVPASS

i=kfk-cluster

echo $i
# Create keystores
keytool -genkey -noprompt \
                -alias $i \
                -dname "CN=*.domain.dom" \
                -keystore kafka.$i.keystore.jks \
                -keyalg RSA \
                -validity 365 \
                -storetype pkcs12 \
                -storepass $SRVPASS \
                -keypass $SRVPASS

# Create CSR, sign the key and import back into keystore
keytool -noprompt -keystore kafka.$i.keystore.jks \
        -alias $i -certreq \
        -sigalg SHA256WITHRSA \
        -file $i.csr \
        -storepass $SRVPASS \
        -keypass $SRVPASS

openssl x509 -req -CA kfk-ca.crt \
        -CAkey kfk-ca.key \
        -in $i.csr \
        -out $i-ca-signed.crt \
        -days 365 \
        -CAcreateserial \
        -extfile openssl-config.cnf \
        -extensions v3_req \
        -passin pass:$SRVPASS

# import the public CA cert
keytool -noprompt -keystore kafka.$i.keystore.jks \
        -alias CARoot \
        -import \
        -file kfk-ca.crt \
        -storepass $SRVPASS \
        -keypass $SRVPASS

# import the signed cert
keytool -noprompt -keystore kafka.$i.keystore.jks \
        -alias $i -import \
        -file $i-ca-signed.crt  \
        -storepass $SRVPASS \
        -keypass $SRVPASS

# Create truststore and import the CA cert.
keytool -noprompt -keystore kafka.$i.truststore.jks \
        -alias CARoot \
        -import \
        -file kfk-ca.crt \
        -storepass $SRVPASS \
        -keypass $SRVPASS

echo "$SRVPASS" > ${i}_sslkey_creds
echo "$SRVPASS" > ${i}_keystore_creds
echo "$SRVPASS" > ${i}_truststore_creds

chmod +x create_certificate.sh

./create_single_certificate.sh

The certificates and keys will be copied in secrets directory with the folder from where the compose files are run. That secrets directory will be mapped inside the container.

Creating directories

These would be on the second drive (/data)

On each of the three servers (kfk231, kfk232, kfk233) which will have zookeeper, broker, and cnnect

cd /data
mkdir kafka
cd kafka

mkdir connectors
mkdir data
mkdir secrets

chmod -R 755 data

On the fourth server (kfk234)

cd /data
mkdir kafka
cd kafka
mkdir secrets

Zookeeper

Same on each the three cluster nodes (kfk231, kfk232, kfk233)

Working in /data/kafka directory

On kfk231 - kfk231-zookeeper-compose.yaml

version: '2.4'
name: kfk231-zk
services:
  kfk231-zk:
    image: confluentinc/cp-zookeeper:6.2.1
    hostname: kfk231.domain.dom
    container_name: kfk231-zk
    restart: "unless-stopped"
    mem_limit: 512m
    memswap_limit: 512m
    mem_swappiness: 0
    network_mode: "host"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: kfk231:2888:3888;kfk232:2888:3888;kfk233:2888:3888
    extra_hosts:
      - "kfk231:192.168.122.231"
      - "kfk231-kafka:192.168.122.231"
      - "kfk231-zk:192.168.122.231"
      - "kfk231-cnt:192.168.122.231"
      - "kfk231-reg:192.168.122.231"
      - "kfk231.domain.dom:192.168.122.231"
      - "kfk232:192.168.122.232"
      - "kfk232-kafka:192.168.122.232"
      - "kfk232-zk:192.168.122.232"
      - "kfk232-cnt:192.168.122.232"
      - "kfk232-reg:192.168.122.232"
      - "kfk232.domain.dom:192.168.122.232"
      - "kfk233:192.168.122.233"
      - "kfk233-kafka:192.168.122.233"
      - "kfk233-zk:192.168.122.233"
      - "kfk233-cnt:192.168.122.233"
      - "kfk233-reg:192.168.122.233"
      - "kfk233.domain.dom:192.168.122.233"
      - "kfk234:192.168.122.234"
      - "kfk234.domain.dom:192.168.122.234"

docker compose -f kfk231-zookeeper-compose.yaml up -d

On kfk232 - kfk232-zookeeper-compose.yaml

version: '2.4'
name: kfk232-zk
services:
  kfk232-zk:
    image: confluentinc/cp-zookeeper:6.2.1
    hostname: kfk232.domain.dom
    container_name: kfk232-zk
    restart: "unless-stopped"
    mem_limit: 512m
    memswap_limit: 512m
    mem_swappiness: 0
    network_mode: "host"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SERVER_ID: 2
      ZOOKEEPER_SERVERS: kfk231:2888:3888;kfk232:2888:3888;kfk233:2888:3888
    extra_hosts:
      - "kfk231:192.168.122.231"
      - "kfk231-kafka:192.168.122.231"
      - "kfk231-zk:192.168.122.231"
      - "kfk231-cnt:192.168.122.231"
      - "kfk231-reg:192.168.122.231"
      - "kfk231.domain.dom:192.168.122.231"
      - "kfk232:192.168.122.232"
      - "kfk232-kafka:192.168.122.232"
      - "kfk232-zk:192.168.122.232"
      - "kfk232-cnt:192.168.122.232"
      - "kfk232-reg:192.168.122.232"
      - "kfk232.domain.dom:192.168.122.232"
      - "kfk233:192.168.122.233"
      - "kfk233-kafka:192.168.122.233"
      - "kfk233-zk:192.168.122.233"
      - "kfk233-cnt:192.168.122.233"
      - "kfk233-reg:192.168.122.233"
      - "kfk233.domain.dom:192.168.122.233"
      - "kfk234:192.168.122.234"
      - "kfk234.domain.dom:192.168.122.234"

docker compose -f kfk232-zookeeper-compose.yaml up -d

On kfk233 - kfk233-zookeeper-compose.yaml

version: '2.4'
name: kfk233-zk
services:
  kfk233-zk:
    image: confluentinc/cp-zookeeper:6.2.1
    hostname: kfk233.domain.dom
    container_name: kfk233-zk
    restart: "unless-stopped"
    mem_limit: 512m
    memswap_limit: 512m
    mem_swappiness: 0
    network_mode: "host"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SERVER_ID: 3
      ZOOKEEPER_SERVERS: kfk231:2888:3888;kfk232:2888:3888;kfk233:2888:3888
    extra_hosts:
      - "kfk231:192.168.122.231"
      - "kfk231-kafka:192.168.122.231"
      - "kfk231-zk:192.168.122.231"
      - "kfk231-cnt:192.168.122.231"
      - "kfk231-reg:192.168.122.231"
      - "kfk231.domain.dom:192.168.122.231"
      - "kfk232:192.168.122.232"
      - "kfk232-kafka:192.168.122.232"
      - "kfk232-zk:192.168.122.232"
      - "kfk232-cnt:192.168.122.232"
      - "kfk232-reg:192.168.122.232"
      - "kfk232.domain.dom:192.168.122.232"
      - "kfk233:192.168.122.233"
      - "kfk233-kafka:192.168.122.233"
      - "kfk233-zk:192.168.122.233"
      - "kfk233-cnt:192.168.122.233"
      - "kfk233-reg:192.168.122.233"
      - "kfk233.domain.dom:192.168.122.233"
      - "kfk234:192.168.122.234"
      - "kfk234.domain.dom:192.168.122.234"

docker compose -f kfk233-zookeeper-compose.yaml up -d

Kafka Broker

Same on each the three cluster nodes (kfk231, kfk232, kfk233)

Working in /data/kafka directory

On kfk231 - kfk231-kafka-broker-compose.yaml

version: '2.4'
name: kfk231-kafka
services:
  kfk231-kafka:
    image: confluentinc/cp-kafka:6.2.1
    hostname: kfk231.domain.dom
    container_name: kfk231-kafka
    restart: "unless-stopped"
    # Increase open files limit
    ulimits:
      nofile:
        soft: 100000
        hard: 100000
    mem_limit: 1500m
    memswap_limit: 1500m
    # Disable RAM swap
    mem_swappiness: 0
    network_mode: "host"    
    volumes:
      - ./data/kafka:/data/kafka/
      - ./secrets:/etc/kafka/secrets
      - ./jolokia-jvm-1.7.1.jar:/usr/app/jolokia-jvm-1.7.1.jar
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.122.231:9092,SSL://192.168.122.231:9093
      KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL
      KAFKA_ZOOKEEPER_CONNECT: "kfk231:2181,kfk232:2181,kfk233:2181/kafka"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_NUM_PARTITIONS: 6
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_LOG_RETENTION_HOURS: 168
      KAFKA_OFFSETS_RETENTION_MINUTES: 4320
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: true 
      KAFKA_DELETE_TOPIC_ENABLE: true
      KAFKA_LOG_DIRS: /data/kafka
      KAFKA_SSL_KEYSTORE_FILENAME: kafka.kfk-cluster.keystore.jks
      KAFKA_SSL_KEYSTORE_CREDENTIALS: kfk-cluster_keystore_creds
      KAFKA_SSL_KEY_CREDENTIALS: kfk-cluster_sslkey_creds
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.kfk-cluster.truststore.jks
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kfk-cluster_truststore_creds
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: " "
      KAFKA_SSL_CLIENT_AUTH: required
      KAFKA_JMX_HOSTNAME: "kfk231"
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_OPTS: "-Djava.rmi.server.hostname=kfk231 
      -Dcom.sun.management.jmxremote.local.only=false 
      -Dcom.sun.management.jmxremote.rmi.port=9999 
      -Dcom.sun.management.jmxremote.port=9999 
      -Dcom.sun.management.jmxremote.authenticate=false 
      -Dcom.sun.management.jmxremote.ssl=false 
      -javaagent:/usr/app/jolokia-jvm-1.7.1.jar=host=*"
    extra_hosts:
      - "kfk231:192.168.122.231"
      - "kfk231-kafka:192.168.122.231"
      - "kfk231-zk:192.168.122.231"
      - "kfk231-cnt:192.168.122.231"
      - "kfk231-reg:192.168.122.231"
      - "kfk231.domain.dom:192.168.122.231"
      - "kfk232:192.168.122.232"
      - "kfk232-kafka:192.168.122.232"
      - "kfk232-zk:192.168.122.232"
      - "kfk232-cnt:192.168.122.232"
      - "kfk232-reg:192.168.122.232"
      - "kfk232.domain.dom:192.168.122.232"
      - "kfk233:192.168.122.233"
      - "kfk233-kafka:192.168.122.233"
      - "kfk233-zk:192.168.122.233"
      - "kfk233-cnt:192.168.122.233"
      - "kfk233-reg:192.168.122.233"
      - "kfk233.domain.dom:192.168.122.233"
      - "kfk234:192.168.122.234"
      - "kfk234.domain.dom:192.168.122.234"

docker compose -f kfk231-kafka-broker-compose.yaml up -d

On kfk232 - kfk232-kafka-broker-compose.yaml

version: '2.4'
name: kfk232-kafka
services:
  kfk232-kafka:
    image: confluentinc/cp-kafka:6.2.1
    hostname: kfk232.domain.dom
    container_name: kfk232-kafka
    restart: "unless-stopped"
    # Increase open files limit
    ulimits:
      nofile:
        soft: 100000
        hard: 100000
    mem_limit: 1500m
    memswap_limit: 1500m
    # Disable RAM swap
    mem_swappiness: 0
    network_mode: "host"    
    volumes:
      - ./data/kafka:/data/kafka/
      - ./secrets:/etc/kafka/secrets
      - ./jolokia-jvm-1.7.1.jar:/usr/app/jolokia-jvm-1.7.1.jar
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.122.232:9092,SSL://192.168.122.232:9093
      KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL
      KAFKA_ZOOKEEPER_CONNECT: "kfk231:2181,kfk232:2181,kfk233:2181/kafka"
      KAFKA_BROKER_ID: 2
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_NUM_PARTITIONS: 6
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_LOG_RETENTION_HOURS: 168
      KAFKA_OFFSETS_RETENTION_MINUTES: 4320
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: true 
      KAFKA_DELETE_TOPIC_ENABLE: true
      KAFKA_LOG_DIRS: /data/kafka
      KAFKA_SSL_KEYSTORE_FILENAME: kafka.kfk-cluster.keystore.jks
      KAFKA_SSL_KEYSTORE_CREDENTIALS: kfk-cluster_keystore_creds
      KAFKA_SSL_KEY_CREDENTIALS: kfk-cluster_sslkey_creds
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.kfk-cluster.truststore.jks
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kfk-cluster_truststore_creds
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: " "
      KAFKA_SSL_CLIENT_AUTH: required
      KAFKA_JMX_HOSTNAME: "kfk232"
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_OPTS: "-Djava.rmi.server.hostname=kfk232 
      -Dcom.sun.management.jmxremote.local.only=false 
      -Dcom.sun.management.jmxremote.rmi.port=9999 
      -Dcom.sun.management.jmxremote.port=9999 
      -Dcom.sun.management.jmxremote.authenticate=false 
      -Dcom.sun.management.jmxremote.ssl=false 
      -javaagent:/usr/app/jolokia-jvm-1.7.1.jar=host=*"
    extra_hosts:
      - "kfk231:192.168.122.231"
      - "kfk231-kafka:192.168.122.231"
      - "kfk231-zk:192.168.122.231"
      - "kfk231-cnt:192.168.122.231"
      - "kfk231-reg:192.168.122.231"
      - "kfk231.domain.dom:192.168.122.231"
      - "kfk232:192.168.122.232"
      - "kfk232-kafka:192.168.122.232"
      - "kfk232-zk:192.168.122.232"
      - "kfk232-cnt:192.168.122.232"
      - "kfk232-reg:192.168.122.232"
      - "kfk232.domain.dom:192.168.122.232"
      - "kfk233:192.168.122.233"
      - "kfk233-kafka:192.168.122.233"
      - "kfk233-zk:192.168.122.233"
      - "kfk233-cnt:192.168.122.233"
      - "kfk233-reg:192.168.122.233"
      - "kfk233.domain.dom:192.168.122.233"
      - "kfk234:192.168.122.234"
      - "kfk234.domain.dom:192.168.122.234"

docker compose -f kfk232-kafka-broker-compose.yaml up -d

On kfk233 - kfk233-kafka-broker-compose.yaml

version: '2.4'
name: kfk233-kafka
services:
  kfk233-kafka:
    image: confluentinc/cp-kafka:6.2.1
    hostname: kfk233.domain.dom
    container_name: kfk233-kafka
    restart: "unless-stopped"
    # Increase open files limit
    ulimits:
      nofile:
        soft: 100000
        hard: 100000
    mem_limit: 1500m
    memswap_limit: 1500m
    # Disable RAM swap
    mem_swappiness: 0
    network_mode: "host"    
    volumes:
      - ./data/kafka:/data/kafka/
      - ./secrets:/etc/kafka/secrets
      - ./jolokia-jvm-1.7.1.jar:/usr/app/jolokia-jvm-1.7.1.jar
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.122.233:9092,SSL://192.168.122.233:9093
      KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL
      KAFKA_ZOOKEEPER_CONNECT: "kfk231:2181,kfk232:2181,kfk233:2181/kafka"
      KAFKA_BROKER_ID: 3
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_NUM_PARTITIONS: 6
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_LOG_RETENTION_HOURS: 168
      KAFKA_OFFSETS_RETENTION_MINUTES: 4320
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: true 
      KAFKA_DELETE_TOPIC_ENABLE: true
      KAFKA_LOG_DIRS: /data/kafka
      KAFKA_SSL_KEYSTORE_FILENAME: kafka.kfk-cluster.keystore.jks
      KAFKA_SSL_KEYSTORE_CREDENTIALS: kfk-cluster_keystore_creds
      KAFKA_SSL_KEY_CREDENTIALS: kfk-cluster_sslkey_creds
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.kfk-cluster.truststore.jks
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kfk-cluster_truststore_creds
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: " "
      KAFKA_SSL_CLIENT_AUTH: required
      KAFKA_JMX_HOSTNAME: "kfk233"
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_OPTS: "-Djava.rmi.server.hostname=kfk233 
      -Dcom.sun.management.jmxremote.local.only=false 
      -Dcom.sun.management.jmxremote.rmi.port=9999 
      -Dcom.sun.management.jmxremote.port=9999 
      -Dcom.sun.management.jmxremote.authenticate=false 
      -Dcom.sun.management.jmxremote.ssl=false 
      -javaagent:/usr/app/jolokia-jvm-1.7.1.jar=host=*"
    extra_hosts:
      - "kfk231:192.168.122.231"
      - "kfk231-kafka:192.168.122.231"
      - "kfk231-zk:192.168.122.231"
      - "kfk231-cnt:192.168.122.231"
      - "kfk231-reg:192.168.122.231"
      - "kfk231.domain.dom:192.168.122.231"
      - "kfk232:192.168.122.232"
      - "kfk232-kafka:192.168.122.232"
      - "kfk232-zk:192.168.122.232"
      - "kfk232-cnt:192.168.122.232"
      - "kfk232-reg:192.168.122.232"
      - "kfk232.domain.dom:192.168.122.232"
      - "kfk233:192.168.122.233"
      - "kfk233-kafka:192.168.122.233"
      - "kfk233-zk:192.168.122.233"
      - "kfk233-cnt:192.168.122.233"
      - "kfk233-reg:192.168.122.233"
      - "kfk233.domain.dom:192.168.122.233"
      - "kfk234:192.168.122.234"
      - "kfk234.domain.dom:192.168.122.234"

docker compose -f kfk233-kafka-broker-compose.yaml up -d