# README
scraper
The scraper is responsible for polling prometheus exporters and writing metrics onto the metric bus. Jobs are defined in a configuration store and a pool of scrapers share the load to fulfill these jobs. Scrapers react to other scrapers joining or leaving the pool. Scrapers react to jobs changing.
scraper architecture
╔═════════════════════════════════════════════════════════════════════════════╗
║ configuration store (zookeeper) ║
║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║
║ ┃ jobs ┃ ┃ alive scrapers ┃ ║
║ ┃┌─────────────────────────────────┐┃ ┃┌───────────────────────────────┐┃ ║
║ ┃│job_name: mysql │┃ ┃│scraper-ff55531 │┃ ║
║ ┃│static_configs: │┃ ┃│scraper-e85aae3 │┃ ║
║ ┃│ - targets: │┃ ┃│scraper-9c2ff7b │┃ ║
║ ┃│ - mysql1.example.com:9104 │┃ ┃└───────────────────────────────┘┃ ║
║ ┃│ - mysql2.example.com:9104 │┃ ┃ ┃ ║
║ ┃└─────────────────────────────────┘┃ ┃ ┃ ║
║ ┃┌─────────────────────────────────┐┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║
║ ┃│job_name: memcached │┃ △ ║
║ ┃│dns_sd_configs: │┃ │ ║
║ ┃│ - names: │┃ │ ║
║ ┃│ - _memcached._tcp.example.com│┃ │ ║
║ ┃└─────────────────────────────────┘┃ │ ║
║ ┃┌─────────────────────────────────┐┃ │ ║
║ ┃│job_name: nginx │┃ │ ║
║ ┃│consul_sd_configs: │┃ │ ║
║ ┃│ - server: 'consul:1234' │┃ │ ║
║ ┃│ services: ['nginx'] │┃ ┌─┘ ║
║ ┃└─────────────────────────────────┘┃ │ ║
║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ │ ║
║ △ │ ║
║ │ │ ║
╚════════════════════╬══════════════════════════════════╬═════════════════════╝
┌─┘ │
╔══════════════════╬════════════════════════════════════╬═════════════════════╗
║ │ each scraper node │ ║
║ │ │ ║
║ ▼ ▼ ║
║ ┌───────────────────────────────┐ ┌───────────────────────────────┐ ║
║ │ resolve targets │ │ consistent hash │ ║
║ │ │ ┌─▶│ │ ║
║ │┌─────────────────────────────┐│ │ └───────────────────────────────┘ ║
║ ││ ││ │ │ ║
║ ││ ││ │ ▼ ║
║ ││mysql1.example.com:9104 ││ │ ┌───────────────────────────────┐ ║
║ ││mysql2.example.com:9104 ││ │ │ my targets │ ║
║ ││memcached1.example.com:9106 ││ │ │ │ ║
║ ││memcached2.example.com:9106 ││─┘ │┌─────────────────────────────┐│ ║
║ ││memcached3.example.com:9106 ││ ││ ││ ║
║ ││nginx1.example.com:9113 ││ ││ ││ ║
║ ││nginx2.example.com:9113 ││ ││mysql1.example.com:9108 ││ ║
║ ││ ││ ││memcached3.example.com:9109 ││ ║
║ ││ ││ ││ ││ ║
║ │└─────────────────────────────┘│ ││ ││ ║
║ │ │ │└─────────────────────────────┘│ ║
║ └───────────────────────────────┘ └───────────────────────────────┘ ║
║ ║
╚═════════════════════════════════════════════════════════════════════════════╝
▲ │
│ ┌───────────────────────────┐ ┌───────────────┘
├───────▷│ mysql1.example.com:9108 │ │
│ └───────────────────────────┘ ▼
│ ┌───────────────────────────┐ ┌───────────────────────────────┐
└───────▷│memcached3.example.com:9109│ │ metric bus (kafka) │
└───────────────────────────┘ └───────────────────────────────┘
# Functions
NewConsistentHashTargeter returns a new instance of a ConsistentHashTargeter object.
NewHTTPTarget creates an instance of HTTPTarget.
NewScraper returns a new Scraper instance from the provided Config.
NewStaticJob returns a new instance of the StaticJob configuration.
NewWorker creates a new instance of a Worker.
# Structs
Config represents an instance of a scraper configuration.
ConsistentHashTargeter represents an object that orchestrates work between Zookeeper targets and available worker nodes in the pool.
ConsistentHashTargeterConfig represents an configuration for a ConsistentHashTargeter object.
HTTPTarget represents an instance of an HTTP scraper target.
HTTPTargetConfig represents the configuration of an HTTPTarget.
Scraper answers to a query on which targets are running and groups them by job.
StaticJob represents a vulcan job.
StaticJobConfig represents a StaticJob configuration.
Target represents a scrape target.
Worker represents an instance of a scraper worker.
WorkerConfig respresents an instance of a Worker's configuration.
# Interfaces
Job represents a discoverable targets to be be processed.
JobWatcher is an interface that wraps the Target method.
Pool is an interface that wraps the Scrapers method.
Targeter is an interface that wraps the methods of exporter/job target.
TargetWatcher is an interface that represents something that can return a slice of active Targeters.
Writer is an interface that wraps the Write method to a message bus.
# Type aliases
JobName is the name of an exporter/job.