Categorygithub.com/ONSdigital/dp-search-data-importer
modulepackage
1.1.0
Repository: https://github.com/onsdigital/dp-search-data-importer.git
Documentation: pkg.go.dev

# README

dp-search-data-importer

Service to store searchable content into elasticsearch. See search service architecture docs here

Getting started

  • Run make help to see full list of make targets
  • Run make debug

The service runs in the background consuming messages from Kafka. An example event can be created using the helper script, make produce.

Dependencies

  • Requires running…
    • go v1.19
    • ElasticSearch 7.10
    • kafka
  • No further dependencies other than those defined in go.mod

Configuration

Environment variableDefaultDescription
BIND_ADDRlocalhost:25900The host and port to bind to
GRACEFUL_SHUTDOWN_TIMEOUT5sThe graceful shutdown timeout in seconds (time.Duration format)
HEALTHCHECK_INTERVAL30sTime between self-healthchecks (time.Duration format)
HEALTHCHECK_CRITICAL_TIMEOUT90sTime to wait until an unhealthy dependent propagates its state to make this app unhealthy (time.Duration format)
KAFKA_ADDR"localhost:9092", "localhost:9093", "localhost:9094"The address of Kafka (accepts list)
KAFKA_OFFSET_OLDESTtrueStart processing Kafka messages in order from the oldest in the queue
KAFKA_NUM_WORKERS1The maximum number of parallel kafka consumers
PUBLISHED_CONTENT_GROUPdp-search-data-importerThe consumer group this application to consume Uploaded messages
PUBLISHED_CONTENT_TOPICsearch-data-importThe name of the topic to consume messages from
BATCH_SIZE500The default total number of messages that should be buffered (in batches) before writing to the search engine.
BATCH_WAIT_TIME5sThe default wait time for preparing the batch.
KAFKA_SEC_PROTOunsetif set to TLS, kafka connections will use TLS [1]
KAFKA_SEC_CA_CERTSunsetCA cert chain for the server cert [1]
KAFKA_SEC_CLIENT_KEYunsetPEM for the client key [1]
KAFKA_SEC_CLIENT_CERTunsetPEM for the client certificate [1]
KAFKA_SEC_SKIP_VERIFYfalseignores server certificate issues if true [1]
ELASTIC_SEARCH_URL"http://localhost:11200"The elastic search URL
AWS_REGION"eu-west-2"The default AWS region to be validated while connecting to elastic search
AWS_SERVICE"es"The default AWS service to be validated while connecting to elastic search
SIGN_ELASTICSEARCH_REQUESTSfalseThe default configuration for AWS authenticatioin while connecting to elastic search

Notes:

  1. For more info, see the kafka TLS examples documentation

Healthcheck

The /health endpoint returns the current status of the service. Dependent services are health checked on an interval defined by the HEALTHCHECK_INTERVAL environment variable.

On a development machine a request to the health check endpoint can be made by:

curl localhost:25900/health

Contributing

See CONTRIBUTING for details.

License

Copyright © 2024, Office for National Statistics

Released under MIT license, see LICENSE for details.

# Packages

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author

# Variables

BuildTime represents the time in which the service was built.
GitCommit represents the commit (SHA-1) hash of the service that is running.
Version represents the version of the service that is running.