kafkactl - Kafka Management Tool - wiki

kafkactl - Package and CLI Tool (written in Go) for mgmt of Apache Kafka and related components.

The kafkactl tool currently features the following:

• Search and Manage Groups and Topics
• Show Topic Partition Details - Offsets, Replicas, Leaders, etc.
• Show Group Details - Lag, Offsets, Members, etc.
• Display / Modify Topic Configs
• Create / Delete Topics
• Delete Consumer Groups
• Reset Partition Offsets for a Group
• Tail a Topic in realtime
• Launch a Consumer Group for a Topic
• Produce to a Specific Topic Partition or all Partitions at once
• Perform a Preferred Replica Election for a target topic or for all topics
• Increase / Decrease Replicas
• Perform Topic Partition Migrations between Brokers
• Query burrow details
• Monitor burrow lag via Terminal Graph
• Explore Zookeeper Paths
• Create and Delete Zookeeper Values
• Pass stdin to create Kafka messages or Zookeeper Values

ToDo:

• Add Security Features
• Add Metrics Testing

kafkactl is actively developed with new features being added and tested. Thus, ongoing optimization and re-factoring will occur so ensure you are aware of the latest releases.

Package

The package is mostly a wrapper around the excellent sarama library and for the kafkactl tool itself. The kafkactl tool utilizes a context style config, similar to the kubernetes tool - kubectl, grouping a set of Kafka clusters to corresponding Zookeeper and Burrow instances.

package example usage:

Get package:

go get -u github.com/jbvmio/kafkactl

List Topics:

package main

import (
	"fmt"
	"log"

	"github.com/jbvmio/kafkactl"
)

func main() {

	client, err := kafkactl.NewClient("localhost:9092")
	if err != nil {
		log.Fatalf("Error: %v\n", err)
	}

	topics, err := client.GetTopicMeta()
	if err != nil {
		log.Fatalf("Error: %v\n", err)
	}

	for _, t := range topics {
		fmt.Printf("TOPIC> %v  PARTITION> %v  LEADER> %v\n", t.Topic, t.Partition, t.Leader)
	}

}

kafkactl tool - Get Started

Manual Download

• Download the latest kafkactl tool and extract to a $PATH directory.
• Run "kafkactl config --sample" to generate a sample config at $HOME/.kafkactl.yaml
• Edit the config file and save.

• Alternatively, pass all arguments to the command when running.

# kafkactl --broker brokerhost01:9092 admin create --topic newtopic01 --partitions 15 --rfactor 3

Using Docker

Run with Docker using cmdline args:

docker run --rm jbvmio/kafkactl -b brokerAddress:9092
docker run --rm jbvmio/kafkactl -b brokerAddress:9092 topics
docker run --rm jbvmio/kafkactl -b brokerAddress:9092 topics mytopic -x -m
docker run --rm jbvmio/kafkactl -b brokerAddress:9092 groups
docker run --rm jbvmio/kafkactl -b brokerAddress:9092 groups mygroup --lag
docker run --rm jbvmio/kafkactl -b brokerAddress:9092 admin create --topic mytopic2 --partitions 5 --rfactor 3

Using a config file with Docker:

• Generate a sample config:

docker run --rm -v ${PWD}:/home/kafkactl jbvmio/kafkactl config --sample

• This will generate a sample config named .kafkactl.yaml, edit this file with your values.
• At minimum, specify at least one entry containing a Kafka broker.

# Minimum Config:
current: Cluster1
entries:
- name: Cluster1
  kafka:
  - broker01:9092

• Now mount your .kafkactl.yaml file whenever running the Docker image:

docker run --rm -v ${PWD}/.kafkactl.yaml:/home/kafkactl/.kafkactl.yaml jbvmio/kafkactl
docker run --rm -v ${PWD}/.kafkactl.yaml:/home/kafkactl/.kafkactl.yaml jbvmio/kafkactl topics
docker run --rm -v ${PWD}/.kafkactl.yaml:/home/kafkactl/.kafkactl.yaml jbvmio/kafkactl groups

kafkactl tool - Example Commands

example config commands

kafkactl config --sample
kafkactl config --show
kafkactl config --use testcluster1

example config file (~/.kafkactl.yaml)

current: testCluster1
entries:
- name: testCluster1
  kafka:
  - brokerHost1:9092
  - brokerHost2:9092
  burrow:
  - http://burrow1:3000
  - http://burrow2:3000
  zookeeper:
  - http://zk1:2181
  - http://zk2:2181
- name: testCluster2
  kafka:
  - brokerHost1:9092
  - brokerHost2:9092
  burrow:
  - http://burrow1:3000
  - http://burrow2:3000
  zookeeper:
  - http://zk1:2181
  - http://zk2:2181

Usage

kafkactl -h

Usage of kafkactl:
Usage:
  kafkactl [flags]
  kafkactl [command]

Available Commands:
  admin       Perform Various Kafka Administration Tasks
  burrow      Query Burrow
  config      Manage kafkactl Configuration
  consume     Consume from Topics using Consumer Groups.
  describe    Return Topic or Group details
  group       Search and Retrieve Group Info
  help        Help about any command
  message     Retreive Targeted Messages from a Kafka Topic
  meta        Return Metadata
  produce     Produce messages to a Kafka topic
  tail        Tail a Topic
  topic       Search and Retrieve Available Topics
  version     Print kafkactl version and exit
  zk          Perform Various Zookeeper Administration Tasks

Flags:
  -b, --broker string   Bootstrap Kafka Broker
  -g, --group string    Specify a Target Group
  -h, --help            help for kafkactl
      --port string     Port used for Bootstrap Kafka Broker (default "9092")
  -t, --topic string    Specify a Target Topic
  -v, --verbose         Display any additional info and error output.

kafkactl admin -h

  conf        Get and Set Available Kafka Related Configs (Topics Only for Now)
  create      Create Topics
  delete      Delete Topics or Groups
  offset      Manage Offsets
  pre         Perform Preferred Replica Election Tasks

kafkactl zk -h

Available Commands:
  ls          List or Get Values of a Given Path in Zookeeper

Flags:
  -c, --create string   Create a Zookeeper Path (Use with --value for setting a value)
  -d, --delete string   Delete a Zookeeper Path/Value
  -f, --force           Force Operation
  -h, --help            help for zk
  -s, --server string   Specify a targeted Zookeeper Server and Port (eg. localhost:2181
      --value string    Create a Zookeeper Value (Use with --create to specify the path for the value) Wins over StdIn

Example output:

kafka

# kafkactl

Brokers:  5
 Topics:  208
 Groups:  126

Cluster: (Kafka: v1.1)
  brokerhost01:9092/1
* brokerhost02:9092/2
  brokerhost03:9092/3
  brokerhost04:9092/4
  brokerhost05:9092/5

(*)Controller

# kafkactl group mygroup05

GROUPTYPE  GROUP      COORDINATOR
consumer   mygroup05  broker03:9092/3

kafkactl group mygroup05 --lag

GROUP      TOPIC      PART  MEMBER                    OFFSET  LAG
mygroup05  mytopic05  14    mygroup05-77885078-77abc  97      0
mygroup05  mytopic05  1     mygroup05-77885078-77abc  76      0
mygroup05  mytopic05  2     mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  9     mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  12    mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  3     mygroup05-77885078-77abc  135     0
mygroup05  mytopic05  8     mygroup05-77885078-77abc  90      0
mygroup05  mytopic05  7     mygroup05-77885078-77abc  83      0
mygroup05  mytopic05  4     mygroup05-77885078-77abc  111     0
mygroup05  mytopic05  6     mygroup05-77885078-77abc  114     0
mygroup05  mytopic05  0     mygroup05-77885078-77abc  2329    0
mygroup05  mytopic05  5     mygroup05-77885078-77abc  85      0
mygroup05  mytopic05  11    mygroup05-77885078-77abc  139     0
mygroup05  mytopic05  10    mygroup05-77885078-77abc  125     0
mygroup05  mytopic05  13    mygroup05-77885078-77abc  97      0

zookeeper

# kafkactl zk ls /cluster/id

VALUE: /cluster/id
 {"version":"1","id":"wwI7vYwOShKnEqVzDhhUYqi"}

# kafkactl zk ls /

PATH: /
 cluster
 controller
 brokers
 zookeeper
 admin
 isr_change_notification
 log_dir_event_notification
 controller_epoch
 consumers
 burrow
 latest_producer_id_block
 config

# kafkactl zk --create /testpath
Successfully Created: /testpath

# echo 'my testvalue from stdin' | kafkactl zk --create /testpath/testvalue
Successfully Created: /testpath/testvalue

# kafkactl zk ls /testpath/testvalue

VALUE: /testpath/testvalue
 my testvalue from stdin

burrow

kafkactl burrow mygroup07 -x

BURROW      GROUP      TOPIC      PARTITION  LAG  TOPICLAG  STATUS
dc01-kafka  mygroup07  mytopic07  0          0    0         OK
dc01-kafka  mygroup07  mytopic07  1          0    0         OK
dc01-kafka  mygroup07  mytopic07  2          0    0         OK
dc01-kafka  mygroup07  mytopic07  3          0    0         OK
dc01-kafka  mygroup07  mytopic07  4          0    0         OK
dc01-kafka  mygroup07  mytopic07  5          0    0         OK
dc01-kafka  mygroup07  mytopic07  6          0    0         OK
dc01-kafka  mygroup07  mytopic07  7          0    0         OK
dc01-kafka  mygroup07  mytopic07  8          0    0         OK
dc01-kafka  mygroup07  mytopic07  9          0    0         OK
dc01-kafka  mygroup07  mytopic07  10         0    0         OK
dc01-kafka  mygroup07  mytopic07  11         0    0         OK
dc01-kafka  mygroup07  mytopic07  12         0    0         OK
dc01-kafka  mygroup07  mytopic07  13         0    0         OK
dc01-kafka  mygroup07  mytopic07  14         0    0         OK