# README
kng
kng
is a kafka client wrapper lib for Go that is geared towards facilitating
high throughput message production and consumption.
It wraps both segmentio/kafka-go and Shopify/sarama libs for different functionality.
Special bits
The publisher does a lot of stuff under the hood which dramatically improves write performance.
- Automatically creates a dedicated publisher for the given topic IF a publisher does not already exist.
- Starts a background publisher in a goroutine that will clear its queue on
an interval defined by
PublishInterval
. - Publisher goroutine will be stopped if it is idle for longer than
WorkerIdleTimeout
. - To avoid kafka from rejecting a batch containing too many messages, the
publisher will automatically divide the batch into "sub-batches" (whose
size is defined by
DefaultSubBatchSize
.
Why multiple libs?
The segment lib works great for consuming and producing (and has internal batching mechanisms) but doesn't work as well when dealing with topic management and other administrative tasks.
We also prefer the kafka-go
interface but some things just work better with
the sarama
lib.
Name
kng
is "kafka next generation" which is mildly poking fun at the late 90's
software naming pattern that included "next generation" in its name (such as
syslog-ng
) :smiley: