Gohan - A Genomic Variants API

Prerequisites

Golang >= 1.15.5
- installation: https://golang.org/doc/install
UPX
- docs: https://github.com/upx/upx
Elasticsearch
- getting started: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
- overview tutorial: https://www.youtube.com/watch?v=C3tlMqaNSaI
Bash
- overview: https://en.wikipedia.org/wiki/Bash_%28Unix_shell%29
Make
- overview: https://en.wikipedia.org/wiki/Make_(software)
Docker
- getting started: https://www.docker.com/get-started
Docker-Compose
- getting started: https://docs.docker.com/compose/gettingstarted/
Visual Studio Code (recommended)
- getting started: https://code.visualstudio.com/docs
PERL (optional)
- installation: https://learn.perl.org/installing/unix_linux.html

Getting started

Environment :

First, from the project root, create a local file for environment variables with default settings by running

cp ./etc/example.env .env

and make any necessary changes, such as the Elasticsearch GOHAN_ES_USERNAME and GOHAN_ES_PASSWORD when in production.

Init

Run

make init

Elasticsearch & Kibana :

Run

make run-elasticsearch

and (optionally)

make run-kibana

The first startup may fail on an AccessDeniedException[/usr/share/elasticsearch/data/nodes]; and can be resolved by setting the data directory to have less strict permissions with

sudo chmod -R 777 data/

DRS :

Run

make build-drs
make run-drs

Data Access Authorization with OPA (more on this to come..) :

Run

make build-authz
make run-authz

Development

Gateway

To create and use development certs from the project root, run

mkdir -p gateway/certs/dev

openssl req -newkey rsa:2048 -nodes -keyout gateway/certs/dev/gohan_privkey1.key -x509 -days 365 -out gateway/certs/dev/gohan_fullchain1.crt

Note: Ensure your CN matches the hostname (gohan.local by default)

These will be incorporated into the Gateway service (using NGINX by default, see gateway/Dockerfile and gateway/nginx.conf for details). Be sure to update your local /etc/hosts (on Linux) or C:/System32/drivers/etc/hosts (on Windows) file with the name of your choice.

Next, run

make build-gateway
make run-gateway

API

From the project root, run

export GOHAN_API_INTERNAL_PORT=${GOHAN_API_INTERNAL_PORT}
export GOHAN_API_VCF_PATH=${GOHAN_API_VCF_PATH}

# Elasticsearch
export GOHAN_ES_URL=${GOHAN_PRIVATE_ES_URL}
export GOHAN_ES_USERNAME=${GOHAN_ES_USERNAME}
export GOHAN_ES_PASSWORD=${GOHAN_ES_PASSWORD}

# AuthX
export GOHAN_AUTHZ_ENABLED=${GOHAN_API_AUTHZ_ENABLED}
export GOHAN_PUBLIC_AUTHN_JWKS_URL=${GOHAN_PUBLIC_AUTHN_JWKS_URL}
export GOHAN_PRIVATE_AUTHZ_URL=${GOHAN_PRIVATE_AUTHZ_URL}
export GOHAN_AUTHZ_REQHEADS=${GOHAN_API_AUTHZ_REQHEADS}

# DRS
export GOHAN_DRS_URL=${GOHAN_PRIVATE_DRS_URL}
export GOHAN_DRS_BASIC_AUTH_USERNAME=${GOHAN_DRS_BASIC_AUTH_USERNAME}
export GOHAN_DRS_BASIC_AUTH_PASSWORD=${GOHAN_DRS_BASIC_AUTH_PASSWORD}

cd src/api

go run .

Endpoints :

/variants

Request

GET /variants/overview
params: none

Response

{
    "chromosomes": {
        "<CHROMOSOME>": `number`,
        ...
    },
    "sampleIDs": {
        "<SAMPLEID>": `number`,
        ...
    },
    "variantIDs": {
        "<VARIANTID>": `number`,
        ...
    }
}

Example :

{
    "chromosomes": {
        "21": 90548
    },
    "sampleIDs": {
        "hg00096": 33664,
        "hg00099": 31227,
        "hg00111": 25657
    },
    "variantIDs": {
        ".": 90548
    }
}

Requests

  GET /variants/get/by/variantId
   params:

chromosome : number

lowerBound : number

upperBound : number

reference : string an allele ( "A" | "C" | "G" | "T" or some combination thereof )

alternative : string an allele

ids : string (a comma-deliminated list of variant ID alphanumeric codes)

size : number (maximum number of results per id)

sortByPosition : string (<empty> | asc | desc)

includeSamplesInResultSet : boolean (true | false)

genotype : string ( "HOMOZYGOUS" | "HETEROZYGOUS_REFERENCE" | "HETEROZYGOUS_ALTERNATE" )

  GET /variants/count/by/variantId
   params:

chromosome : number

lowerBound : number

upperBound : number

reference : string an allele

alternative : string an allele

ids : string (a comma-deliminated list of variant ID alphanumeric codes)

genotype : string ( "HOMOZYGOUS" | "HETEROZYGOUS_REFERENCE" | "HETEROZYGOUS_ALTERNATE" )

  GET /variants/get/by/sampleId
   params:

chromosome : number

lowerBound : number

upperBound : number

reference : string an allele

alternative : string an allele

ids : string (comma-deliminated list of sample ID alphanumeric codes)

size : number (maximum number of results per id)

sortByPosition : string (<empty> | asc | desc)

includeSamplesInResultSet : boolean (true | false)

genotype : string ( "HOMOZYGOUS" | "HETEROZYGOUS_REFERENCE" | "HETEROZYGOUS_ALTERNATE" )

  GET /variants/count/by/sampleId
   params:

chromosome : number

lowerBound : number

upperBound : number

reference : string an allele

alternative : string an allele

ids : string (comma-deliminated list of sample ID alphanumeric codes)

genotype : string ( "HOMOZYGOUS" | "HETEROZYGOUS_REFERENCE" | "HETEROZYGOUS_ALTERNATE" )

Generalized Response Body Structure

{
    "status":  `number` (200 - 500),
    "message": `string` ("Success" | "Error"),
    "data": [
        {
            "variantId":  `string`,
            "sampleId":  `string`,
            "count":  `number`,
            "results": [
                {
                   "filter": `string`,
                   "pos": `number`,
                   "ref": `[]string`,  // list of alleles
                   "alt": `[]string`,  // list of alleles
                   "info": [
                       {
                           "id": `string`,
                           "value": `string`,
                       },
                       ...
                   ],
                   "format":`string`,
                   "qual": `number`,
                   "id": `string`,
                   "samples": [
                       {
                           "id": `string`,
                           "variation": {
                               "genotype": {
                                   "phased": `boolean`,
                                   "alleleLeft": `number`,
                                   "alleleRight": `number`,
                                   "zygosity": `number` (0 : "Unknown" | 1 : "Homozygous" | 2 : "Heterozygous")
                               },
                               "genotypeProbability": `[]float` | null,
                               "phredScaleLikelyhood": `[]float` | null,
                           }
                       },
                       ...
                   ],
                   "fileId": `string` (UUID),
                   "assemblyId": `string` ("GRCh38" | "GRCh37" | "NCBI36" | "Other"),
                },
                ...
            ]
        },
    ]
}

Examples :

http://localhost:5000/variants/get/by/sampleId?ids=HG00097&size=1000&sortByPosition=asc

Request

GET /variants/ingestion/run
params:

filename : string (required)

Response

{
    "state":  `number` ("Queuing" | "Running" | "Done" | "Error"),
    "id": `string`,
    "filename": `string`,
    "message": `string`,
}

Request

GET /variants/ingestion/requests
params: none

Response

[
  {
    "state":  `number` ("Queuing" | "Running" | "Done" | "Error"),
    "id": `string`,
    "filename": `string`,
    "message": `string`,
    "createdAt": `timestamp string`,
    "updatedAt": `timestamp string`
  },
  ...
]

Releases

API :

Local Release:

From the project root, run

make build-api-binaries

The binary can then be found at bin/api_${GOOS}_${GOARCH} and executed with

export GOHAN_API_INTERNAL_PORT=${GOHAN_API_INTERNAL_PORT}
export GOHAN_API_VCF_PATH=${GOHAN_API_VCF_PATH}

# Elasticsearch
export GOHAN_ES_URL=${GOHAN_PRIVATE_ES_URL}
export GOHAN_ES_USERNAME=${GOHAN_ES_USERNAME}
export GOHAN_ES_PASSWORD=${GOHAN_ES_PASSWORD}

# AuthX
export GOHAN_AUTHZ_ENABLED=${GOHAN_API_AUTHZ_ENABLED}
export GOHAN_PUBLIC_AUTHN_JWKS_URL=${GOHAN_PUBLIC_AUTHN_JWKS_URL}
export GOHAN_PRIVATE_AUTHZ_URL=${GOHAN_PRIVATE_AUTHZ_URL}
export GOHAN_AUTHZ_REQHEADS=${GOHAN_API_AUTHZ_REQHEADS}

# DRS
export GOHAN_DRS_URL=${GOHAN_PRIVATE_DRS_URL}
export GOHAN_DRS_BASIC_AUTH_USERNAME=${GOHAN_DRS_BASIC_AUTH_USERNAME}
export GOHAN_DRS_BASIC_AUTH_PASSWORD=${GOHAN_DRS_BASIC_AUTH_PASSWORD}

cd bin/

./api_${GOOS}_${GOARCH}

Containerized Alpine Release:

When ready, build the docker image and spawn the container by running

make build-api-container
make run-api

and the docker-compose.yaml file will handle the configuration.

Deployments :

All in all, run

make run-elasticsearch 
make run-drs
make build-gateway && make run-gateway 
make build-api && make run-api

# and optionally
make run-kibana

For other handy tools, see the Makefile. Among those already mentionned here, you'll find other build, run, stop and clean-up commands.

Tests :

Once elasticsearch, drs, the api, and the gateway are up, run

make test-api-dev

# README

Gohan - A Genomic Variants API

Prerequisites

Getting started

Environment :

Init

Elasticsearch & Kibana :

DRS :

Data Access Authorization with OPA (more on this to come..) :

Development

Gateway

API

Releases

API :

Deployments :

Tests :

# Packages