clickhouse_sinker

clickhouse_sinker is a sinker program that consumes kafka message and import them to ClickHouse.

Features

• Easy to use and deploy, you don't need write any hard code, just care about the configuration file
• Support multiple sinker tasks, each runs on parallel.
• Support multiply kafka and ClickHouse clusters.
• Bulk insert (by config bufferSize and flushInterval).
• Loop write (when some node crashes, it will retry write the data to the other healthy node)
• Uses Native ClickHouse client-server TCP protocol, with higher performance than HTTP.

Install

• Install Golang

• Go Get

go get -u github.com/housepower/clickhouse_sinker

cd $GOPATH/src/github.com/housepower/clickhouse_sinker
go install github.com/kardianos/govendor

# may take a while
govendor sync

Run

go build -o sinker bin/main.go

## modify the config files, then run it
./sinker -conf conf

Support parsers

• Json

Supported data types

• UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64
• Float32, Float64
• String
• FixedString
• DateTime (not support, cause we use timestamp UInt32, that's enough)

Configuration

See config example