Categorygithub.com/ONSdigital/dp-filter-api
modulepackage
2.32.0+incompatible
Repository: https://github.com/onsdigital/dp-filter-api.git
Documentation: pkg.go.dev

# README

dp-filter-api

Getting started

mongo

  • Run brew install mongodb
  • Run brew services start mongodb

Kafka

  • Run brew install zookeeper
  • Download kafka version 0.10.2.1
  • Run ./kafka-server-start.sh ../config/server.properties

Scripts for updating and debugging Kafka can be found here(dp-data-tools)

Configuration

  • Run brew install neo4j
  • Disable authentication in the config
  • Run brew services start neo4j
Environment variableDefaultDescription
BIND_ADDR:22100The host and port to bind to
HOSThttp://localhost:22100The host name used to build URLs
KAFKA_ADDRlocalhost:9092The kafka broker addresses (can be comma separated)
KAFKA_MAX_BYTES2000000The maximum permitted size of a message. Should be set equal to or smaller than the broker's message.max.bytes
KAFKA_SEC_PROTOunsetif set to TLS, kafka connections will use TLS [1]
KAFKA_SEC_CLIENT_KEYunsetPEM for the client key [1]
KAFKA_SEC_CLIENT_CERTunsetPEM for the client certificate [1]
KAFKA_SEC_CA_CERTSunsetCA cert chain for the server cert [1]
KAFKA_SEC_SKIP_VERIFYfalseignores server certificate issues if true [1]
FILTER_JOB_SUBMITTED_TOPICfilter-job-submittedThe kafka topic to write messages to
MONGODB_BIND_ADDRlocalhost:27017The MongoDB bind address
MONGODB_USERNAMEThe MongoDB Username
MONGODB_PASSWORDThe MongoDB Password
MONGODB_DATABASEfiltersThe MongoDB database
MONGODB_COLLECTIONSFiltersCollection:filters, OutputsCollection:filterOutputsThe MongoDB collections
MONGODB_REPLICA_SETThe name of the MongoDB replica set
MONGODB_ENABLE_READ_CONCERNfalseSwitch to use (or not) majority read concern
MONGODB_ENABLE_WRITE_CONCERNtrueSwitch to use (or not) majority write concern
MONGODB_CONNECT_TIMEOUT5sThe timeout when connecting to MongoDB (time.Duration format)
MONGODB_QUERY_TIMEOUT15sThe timeout for querying MongoDB (time.Duration format)
MONGODB_IS_SSLfalseSwitch to use (or not) TLS when connecting to mongodb
SHUTDOWN_TIMEOUT5sThe graceful shutdown timeout (time.Duration format)
DATASET_API_URLhttp://localhost:22000The URL of the Dataset API
HEALTHCHECK_INTERVAL30sTime between self-healthchecks (time.Duration format)
HEALTHCHECK_CRITICAL_TIMEOUT90sThe time taken for the health changes from warning state to critical due to subsystem check failures
OTEL_BATCH_TIMEOUT5sInterval between pushes to OT Collector
OTEL_EXPORTER_OTLP_ENDPOINThttp://localhost:4317URL for OpenTelemetry endpoint
OTEL_SERVICE_NAME"dp-filter-api"Service name to report to telemetry tools
SERVICE_AUTH_TOKENFD0108EA-825D-411C-9B1D-41EF7727F465The token used to identify this service when authenticating
ZEBEDEE_URL"http://localhost:8082"Zebedee URL
ENABLE_PRIVATE_ENDPOINTSfalsetrue if private endpoints should be enabled
DOWNLOAD_SERVICE_URLhttp://localhost:23600The URL of the download service
DOWNLOAD_SERVICE_SECRET_KEYQB0108EZ-825D-412C-9B1D-41EF7747F462The service token for the download service
ENABLE_URL_REWRITINGfalseFeature flag to enable URL rewriting

Notes:

  1. For more info, see the kafka TLS examples documentation

Healthchecking

Currently checked each $HEALTHCHECK_INTERVAL and reported on endpoint /healthcheck:

  • Neo4j
  • MongoDB
  • Dataset API

Acceptance tests

To run the acceptance tests for this project, use the following commands

  • Start Kafka and Mongodb
  • Run the dp-filter-api using make acceptance
  • Run the tests in dp-api-test

Contributing

See CONTRIBUTING for details.

License

Copyright © 2016-2022, Office for National Statistics (https://www.ons.gov.uk)

Released under MIT license, see LICENSE for details.

# Packages

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author