# README
Feeder
Http service which consumes Clickstream data and pushs it to Kafka in a proper Clickhouse format.
Features
- Support json requests with mixed events and attributes
- Determine device type, browser name, browser version, OS platform, OS name and version
- Provide Prometheus endpoint, input & output metrics
- Use hashes to pack events, atrributes and agents
Build
git clone https://github.com/click-stream/feeder.git
cd feeder/
go build
Example
Run Feeder
./feeder --http-listen ":1081" --kafka-brokers "kafka:9092" \
--log-template "{{.msg}}" --log-format stdout --log-level debug
Prepare v1.json
{
"origin": "https://www.somewebsite.com",
"agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X)",
"session": "sdfdfwdf33451345345",
"cookie": "PHPSESSID",
"referrer": "https://google.com/?text=some",
"lang": "en",
"fingerprint": "2",
"country": "US",
"property": "other",
"events": [
{"object":"player","event":"resume","url":"https://videohub.com/some_video.mp4","position":"30:41"},
{"event":"load","object":"document"}
],
"attributes": [
{"name":"attr1","value":"some","when":"01.01.2019"}
]
}
Push v1.json to Feeder
curl -H "Content-Type: application/json" -d @v1.json http://127.0.0.1:1081/v1
Output from Feeder
Body => { "origin": "https://www.somewebsite.com", "agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X)", "session": "sdfdfwdf33451345345", "cookie": "PHPSESSID", "referrer": "https://google.com/?text=some", "lang": "en", "fingerprint": "2", "country": "US", "property": "other", "events": [ {"object":"player","event":"resume","url":"https://videohub.com/some_video.mp4","position":"30:41"}, {"event":"load","object":"document"} ], "attributes": [ {"name":"attr1","value":"some","when":"01.01.2019"} ]}
Original event => {"version":"v1","timeMs":1581613922641,"method":"POST","object":{"origin":"https://www.somewebsite.com","ipv4":"127.0.0.1","agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X)","session":"sdfdfwdf33451345345","cookie":"PHPSESSID","referrer":"https://google.com/?text=some","country":"US","lang":"en","fingerprint":"2","property":"other","events":[{"event":"resume","object":"player","position":"30:41","url":"https://videohub.com/some_video.mp4"},{"event":"load","object":"document"}],"attributes":[{"name":"attr1","value":"some","when":"01.01.2019"}],"AgentObject":{"deviceType":"Phone","browserName":"Unknown","browserVersion":"0.0.0","osPlatform":"iPhone","osName":"iOS","osVersion":"11.4.1"}}}
Message to Kafka (topic: messages.v1) => {"timestamp":1581613922641,"origin":"https://www.somewebsite.com","ipv4":"127.0.0.1","agent":1420460528,"session":"sdfdfwdf33451345345","referrer":"https://google.com/?text=some","country":"US","lang":"en","fingerprint":"2","property":"other","events":[958374024,3984410905],"attributes":[3801366925]}
Message to Kafka (topic: agents.v1) => {"timestamp":1581613922641,"agent":1420460528,"property":"other","raw":"Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X)","json":"{\"deviceType\":\"Phone\",\"browserName\":\"Unknown\",\"browserVersion\":\"0.0.0\",\"osPlatform\":\"iPhone\",\"osName\":\"iOS\",\"osVersion\":\"11.4.1\"}"}
Message to Kafka (topic: events.v1) => {"timestamp":1581613922641,"event":3984410905,"property":"other","json":"{\"event\":\"load\",\"object\":\"document\"}","total":1}
Message to Kafka (topic: events.v1) => {"timestamp":1581613922641,"event":958374024,"property":"other","json":"{\"event\":\"resume\",\"object\":\"player\",\"position\":\"30:41\",\"url\":\"https://videohub.com/some_video.mp4\"}","total":1}
Message to Kafka (topic: attributes.v1) => {"timestamp":1581613922641,"attribute":3801366925,"property":"other","json":"{\"name\":\"attr1\",\"value\":\"some\",\"when\":\"01.01.2019\"}","total":1}
Usage
Feeder command
Usage:
feeder [flags]
feeder [command]
Available Commands:
help Help about any command
version Print the version number
Flags:
-h, --help help for feeder
--http-cert string Http cert file or content
--http-chain string Http CA chain file or content
--http-header-country string Http header country (default "X-Forwarded-Country")
--http-header-fingerprint string Http header fingerprint (default "X-Forwarded-Fingerprint")
--http-header-ipv4 string Http header ipv4 (default "X-Forwarded-IPv4")
--http-header-lang string Http header lang (default "X-Forwarded-Lang")
--http-header-origin string Http header origin (default "X-Forwarded-Origin")
--http-header-provider string Http header provider (default "X-Forwarded-Provider")
--http-header-referrer string Http header referrer (default "X-Forwarded-Referrer")
--http-header-session string Http header session (default "X-Forwarded-Session")
--http-header-user-agent string Http header user agent (default "X-Forwarded-UserAgent")
--http-key string Http key file or content
--http-listen string Http listen (default ":80")
--http-tls Http TLS
--http-url-v1 string Http url (default "/v1")
--http-cors Http CORS enabled true/false (default false)
--http-feeder-id-pattern Http feeder id pattern (default "{feeder_id:[a-z0-9]{8,8}}")
--kafka-brokers string Kafka brokers
--kafka-client-id string Kafka client id (default "feeder_kafka")
--kafka-flush-frequency int Kafka Producer flush frequency (default 1)
--kafka-flush-max-messages int Kafka Producer flush max messages (default 100)
--kafka-net-dial-timeout int Kafka Net dial timeout (default 30)
--kafka-net-max-open-requests int Kafka Net max open requests (default 5)
--kafka-net-read-timeout int Kafka Net read timeout (default 30)
--kafka-net-write-timeout int Kafka Net write timeout (default 30)
--kafka-topic-agents-v1 string Kafka agents v1 topic (default "agents.v1")
--kafka-topic-attributes-v1 string Kafka attributes v1 topic (default "attributes.v1")
--kafka-topic-default-part-v1 string Kafka messages default v1 topic part
--kafka-topic-events-v1 string Kafka events v1 topic (default "events.v1")
--kafka-topic-messages-v1 string Kafka messages v1 topic (default "messages.v1")
--log-format string Log format: json, text, stdout (default "text")
--log-level string Log level: info, warn, error, debug, panic (default "info")
--log-template string Log template (default "{{.func}} [{{.line}}]: {{.msg}}")
--prometheus-listen string Prometheus listen (default "127.0.0.1:8080")
--prometheus-url string Prometheus endpoint url (default "/metrics")
Environment variables
For containerization purpose all command switches have environment variables analogs.
- FEEDER_LOG_FORMAT
- FEEDER_LOG_LEVEL
- FEEDER_LOG_TEMPLATE
- FEEDER_PROMETHEUS_URL
- FEEDER_PROMETHEUS_LISTEN
- FEEDER_HTTP_LISTEN
- FEEDER_HTTP_TLS
- FEEDER_HTTP_CERT
- FEEDER_HTTP_KEY
- FEEDER_HTTP_CHAIN
- FEEDER_HTTP_URL_V1
- FEEDER_HTTP_CORS
- FEEDER_HTTP_HEADER_ORIGIN
- FEEDER_HTTP_HEADER_IPV4
- FEEDER_HTTP_HEADER_SESSION
- FEEDER_HTTP_HEADER_USER_AGENT
- FEEDER_HTTP_HEADER_REFERRER
- FEEDER_HTTP_HEADER_LANG
- FEEDER_HTTP_HEADER_FINGERPRINT
- FEEDER_HTTP_HEADER_COUNTRY
- FEEDER_HTTP_HEADER_PROVIDER
- FEEDER_HTTP_HEADER_PROPERTY
- FEEDER_KAFKA_BROKERS
- FEEDER_KAFKA_CLIENT_ID
- FEEDER_KAFKA_FLUSH_FREQUENCY
- FEEDER_KAFKA_FLUSH_MAX_MESSAGES
- FEEDER_KAFKA_NET_MAX_OPEN_REQUESTS
- FEEDER_KAFKA_NET_DIAL_TIMEOUT
- FEEDER_KAFKA_NET_READ_TIMEOUT
- FEEDER_KAFKA_NET_WRITE_TIMEOUT
- FEEDER_KAFKA_TOPIC_DEFAULT_PART_V1
- FEEDER_KAFKA_TOPIC_MESSAGES_V1
- FEEDER_KAFKA_TOPIC_AGENTS_V1
- FEEDER_KAFKA_TOPIC_EVENTS_V1
- FEEDER_KAFKA_TOPIC_ATTRIBUTES_V1