Examples

To run any example in this directory, use the run.sh script provided which will start the required infrastructure (kafka, etc.) using docker compose and then run the program, e.g.:

./run.sh kafkatokafka

Prerequisites: you'll need docker-compose, go 1.13+, and librdkafka (see: Developing) installed on your workstation to run these examples.

Kafka to Kafka

see examples/kafkatokafka

In this example, syslog log lines are read from a kafka topic, a few fields are parsed and formatted as JSON, and the result is written to a second kafka topic.

This uses one source and three nodes:

kafkaconsumer (built-in source) -> parser (custom node) -> jsonbuilder (custom node) -> kafkaproducer (built-in node)

Logging

see examples/logging

This example is a simple logging pipeline. Logs lines in syslog format are read from a kafka topic 'logs-raw', and bulk indexed to Elasticsearch.

This uses one source and two nodes:

kafkaconsumer (built-in source) -> parser (custom node) -> docbuilder (custom node) -> elasticsearch (built-in node)

If any errors occur during parsing, those records and the associated error are sent to the kafka topic 'logs-errors' via the errorkafkaproducer built-in node.