Categorygithub.com/RhythmicSound/sdsshared
modulepackage
0.1.0-alpha
Repository: https://github.com/rhythmicsound/sdsshared.git
Documentation: pkg.go.dev

# README

Simple Data Service - shared library

This package presents code to be used by each data resource in order to have a standard API used by every service.

Code Organisation

The following approach is used:

  1. Application domain types go in the root - api.go, server.go, util.go, etc
  2. Implementation of the application domain go in subpackages - badger, sqlite, etc
  3. Everything is tied together in the cmd subpackages - cmd/shared

There is no internal package here as there need be no hiding of code from implementation code.

Usage

Add to project

go get github.com/rhythmicsound/sdsshared

Then you can create a new service by picking your connector and running something like:

package main

import (
	"fmt"
	"log"

	sdsshared "github.com/RhythmicSound/sds-shared"
	badgerconnector "github.com/RhythmicSound/sds-shared/badgerConnector"
)

func main() {

	connector := badgerconnector.New(sdsshared.ResourceServiceName, sdsshared.DatasetURI)

	log.Fatalln(sdsshared.StartServer(connector, "", 0))
}

Using default type values for the arguments to StartServer allows service name and ports to be set using environment variables at runtime.

An example execute command is:

debug=false \
name="postcodeUK-Service" \
database_uri="working/databases/postcodeUKdb" \
dataset_uri="https://storage.cloud.google.com/simple-data-service/datasets/postcodesUK.zip" \ 
objectname="datasets/postcodesUK.zip" \
go run cmd/dummy.go

Settings

Settings for services created from this library can be hardcoded or set using environment variables

VariableExplanationDefault
debugWhether to print verbose output to log and load test data to database. Not for use in production"false"
database_uriThe path -URL or local path- to the database resource to connect to."working/databases/simpledataservice-default/" (N.B. this points at a directory as BadgerDB is the default db in use. This could be a URL or path to local file. In some instances, if no db exists in the path given, one could be created.)
dataset_uriThe path -URL or local path- to the dataset resource used to rebuild the database."working/datasets/data.zip"
bucketThe cloud bucket from which to find the dataset archive. (Required only if downloading the dataset from behind an authentication wall)"simple-data-service"
objectThe cloud object name found in DatasetBucketName that identifies the dataset archive for download. (Required only if downloading the dataset from behind an authentication wall)-
nameThe name of this service as visible to other services."Default Resource Name"
publicportPublicPort is the port from which this API can be accessed for data retrieval"8080"
downloaddirThe local path where download files will be saved to"working/downloads"

Writing new backend storage connectors

Implement DataResource interface

Example:

type impl struct{
	ResourceName string
	Database     *badger.DB
	versioner    sdsshared.VersionManager
}

func NewImpl(resourceName, datasetArchiveLocation string) *impl{
  return &impl{
    	ResourceName: resourceName,
		versioner: sdsshared.VersionManager{
			Repo:           sdsshared.DatasetURI,
			LastUpdated:    "",
			CurrentVersion: 0,
			DataSources:    make([]string, 0),
		},
  }
}

(im impl) Startup()error{
  ... //Startup scripts
}
vt := &VersionManager{
    Repo = repoURIAddress
}

(im impl) UpdateDataset(vt)(*VersionManager,error){
  ... // Logic to keep the database synced with a master versioned dataset archive somehwere

    //Passing a VersionManager as an arg should overwrite internal VM created in New. 
    //Must accept nil to use default
}
(im impl) Retrieve(someSearchString)(sdsshared.SimpleData,error){
   ... //Fetching data from the database based on given key value. 
    //Use the latest VersionManager to complete the Meta elements of SimpleData response object
}

(im impl) Shutdown()error{
  ... //Shutdown scripts
}

See the badgerConnector package for best practise

# Packages

No description provided by the author
No description provided by the author

# Functions

CreateKVStoreKey creates a standard key for use in kv storage dbs.
GCPDownload downloads assets from Google Cloud Storage with the given Object name and within the given Bucket.
GetEnvar get an environment variable and if it is black sets the given default.
NewHTTPClient is the default client to be used instead of default client.Ammended timeouts.
StartServer runs the server to interface with the system using the api methods of DataResource.

# Variables

DatasetBucketName is the cloud bucket from which to find the dataset archive.
DatasetObjectName is the cloud object name found in DatasetBucketName that identifies the dataset archive for download.
DBURI is the path (URL or local path) to the archive file forthe dataset used to rebuild the database.
DBURI is the path -URL or local path- to the database resource.
Whether verbose logging and test data should be used.
LocalDownloadDir is the local relative or absolute path to a downloads folder to use.
PublicPort is the port from which this API can be accessed for data retrieval.
ResourceServiceName is the name of this service as visible to other services.

# Structs

DataOutputis the SimpleData componant with the requested data payload.
DataResourceImplementorTemplate is a simple outline of the basic structure that can implement the full DataResource interface.
Meta is the SimpleData componant with meta data on the datasource being queried.
SimpleData is the standard response interface to the end user.
VersionManager is the struct that allows the instance to check its current used dataset version and compare it against the latest available to judge update need.

# Interfaces

DataResource is the interface each Resource service uses and is a central library unit used for a centralised server facility that handles JWT checking centrally.