dp-geodata-api

⚠️ This repository will be archived in August 2024 as it is no longer in development. ⚠️

Census Atlas geodata API and database.

Further Docs

• Environment Variables

  Database connection details and feature flags are controlled with environment variables. Almost all processes depend on these variables being set correctly, so be careful to verify before running anything risky.

  • common
  • some extra API-related variables (below)

• Postgres/PostGIS

  The system is designed around Postgres with PostGIS extensions. The database may be run as a native local process, within a container, or as an RDS instance.

  • local container
  • local native instance (docs to be written)
  • RDS provisioning (see dp-setup repo)

• Data Ingest

  The "ingest" process takes source data files and loads them into the database. For best results, the ingest should be applied to a local native postgres instance. Running against RDS directly or against a containerised postgres instance is too slow.

  Source data files should be in place, and the postgres environment variables must be exported before running the ingest script (indigestion.sh).

  • source data files
    • S3 buckets
    • postcode data
    • ONS geo codes
    • nomis data
    • spatial data
  • running

• Export/Import

  The export/import processes copies a database. Our main application is to export a locally ingested database for import into RDS.

  • export/import procedure RDS
  • export/import local db

• API

  The API presents a specific interface for querying the database. It is mainly used by the front end application.

  The API may be run as a local process, within a container, or on some temporary EC2 instances we have in place for the moment. The plan is that instance of the API will run within the standard develop and prod environments.

  • building
  • deploying to EC2
  • running locally
  • sanity checking

• CLIs

  These clis are mostly for developers, and may not be fully caught up with features of the API.

  • geodata cli
  • cantabular cli
  • geobb (to generate static geoLookup.json file)

• Testing

  • api unit tests
  • component tests
  • integration tests

• Terraform

  • S3 and misc
  • RDS (see dp-setup repo)

• How-Tos

  • quick start running local API
  • end to end how to

Getting started

• Run make debug

To enable postgres and census queries, set ENABLE_DATABASE, and the postgres environment variables. To lookup the postgres password in Secrets Manager, set FI_PG_SECRET_ID instead of PGPASSWORD, and make sure to set AWS_REGION and any other AWS environment variables.

Auto generated code

swagger.yaml is used to generate code via make generate

Particularly api/api.go (and similar files) shouldn't be directly edited.

Configuration

Environment variable	Default	Description
BIND_ADDR	:25252	The host and port to bind to
GRACEFUL_SHUTDOWN_TIMEOUT	5s	The graceful shutdown timeout in seconds (time.Duration format)
HEALTHCHECK_INTERVAL	30s	Time between self-healthchecks (time.Duration format)
HEALTHCHECK_CRITICAL_TIMEOUT	90s	Time to wait until an unhealthy dependent propagates its state to make this app unhealthy (time.Duration format)
ENABLE_DATABASE	false	Enable postgres and census query functionality
AWS_REGION		used by AWS SDK when ENABLE_DATABASE is true and PGPASSWORD is empty
PGHOST		postgres host when ENABLE_DATABASE is true
PGPORT		postgres port when ENABLE_DATABASE is true
PGUSER		postgres user when ENABLE_DATABASE is true
PGPASSWORD		postgres password when ENABLE_DATABASE is true (also see FI_PG_SECRET_ID)
PGDATABASE		postgres database when ENABLE_DATABASE is true
FI_PG_SECRET_ID		ARN of key holding postgres password if PGPASSWORD is empty
DO_CORS	false	Add Access-Control-Allow-Origin: * to headers if true (not needed in develop / prod)

Contributing

See CONTRIBUTING for details.

License

Copyright © 2022, Office for National Statistics (https://www.ons.gov.uk)

Released under MIT license, see LICENSE for details.