Categorygithub.com/ONSdigital/dp-dimension-search-builder

# README

dp-dimension-search-builder

Handles inserting of dimension options into elasticsearch once a hierarchy for an instance becomes available; and creates an event by sending a message to the kafka $PRODUCER_TOPIC so services know when the data has successfully been inserted into elastic.

  1. Consumes from the $HIERARCHY_BUILT_TOPIC
  2. Retrieves the root node of the hierarchy via the hierarchy API, to get the root dimension option
  3. Creates elastic search index /<instance_id>_<dimension> and adds parent dimension option
  4. Retrieves all nodes in the tree below the root node and writing the data to the elasticsearch index
  5. Produces a message to the $SEARCH_BUILT_TOPIC

Requirements

In order to run the service locally you will need the following:

Refer to https://github.com/ONSdigital/dp-import#dp-import for infrastructure setup of service

Getting started

  • Clone the repo go get github.com/ONSdigital/dp-dimension-search-builder
  • Run zookeeper and then kafka
  • Run elasticsearch
  • Run the hierarchy API, see documentation here
  • Run the application make debug

Healthcheck

The endpoint /healthcheck checks the connection to the database and returns one of:

  • success (200, JSON "status": "OK")
  • failure (500, JSON "status": "error").

Kafka scripts

Scripts for updating and debugging Kafka can be found here(dp-data-tools)

Configuration

Environment variableDefaultDescription
AWS_REGIONeu-west-1The AWS region to use when signing requests with AWS SDK
AWS_SERVICE"es"The aws service that the AWS SDK signing mechanism needs to sign a request
BIND_ADDR:22900The host and port to bind to
CONSUMER_GROUPdp-dimension-search-builderThe name of the Kafka consumer group
ELASTIC_SEARCH_URLhttp://localhost:10200The host name for elasticsearch
EVENT_REPORTER_TOPICreport-eventsThe kafka topic to send errors to
GRACEFUL_SHUTDOWN_TIMEOUT5sThe graceful shutdown timeout
HEALTHCHECK_INTERVAL30sThe time between calling healthcheck endpoints for check subsystems
HEALTHCHECK_CRITICAL_TIMEOUT90sThe time taken for the health changes from warning state to critical due to subsystem check failures
HIERARCHY_API_URLhttp://localhost:22600The host name for the Hierarchy API
HIERARCHY_BUILT_TOPIChierarchy-builtThe name of the topic to consume messages from
PRODUCER_TOPICdimension-search-builtThe name of the topic to produces messages to
KAFKA_ADDRlocalhost:9092A list of Kafka host addresses
KAFKA_MAX_BYTES2000000The max message size for kafka producer
KAFKA_VERSION"1.0.2"The kafka version that this service expects to connect to
KAFKA_OFFSET_OLDESTtruesets kafka offset to oldest if true
KAFKA_SEC_PROTOunsetif set to TLS, kafka connections will use TLS [1]
KAFKA_SEC_CLIENT_KEYunsetPEM for the client key [1]
KAFKA_SEC_CLIENT_CERTunsetPEM for the client certificate [1]
KAFKA_SEC_CA_CERTSunsetCA cert chain for the server cert [1]
KAFKA_SEC_SKIP_VERIFYfalseignores server certificate issues if true [1]
REQUEST_MAX_RETRIES3The maximum number of request retries messages from
SEARCH_BUILDER_URLhttp://localhost:22900The host name for the service
SIGN_ELASTICSEARCH_REQUESTSfalseBoolean flag to identify whether elasticsearch requests via elastic API need to be signed if elasticsearch cluster is running in aws

Notes:

  1. For more info, see the kafka TLS examples documentation

Contributing

See CONTRIBUTING for details.

License

Copyright © 2016-2021, Office for National Statistics (https://www.ons.gov.uk)

Released under MIT license, see LICENSE for details

# Packages

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author

# Variables

BuildTime represents the time in which the service was built.
GitCommit represents the commit (SHA-1) hash of the service that is running.
Version represents the version of the service that is running.