# README
Dataflow-jobcontrol
This is a package for starting, monitoring and stopping GCP dataflow jobs. It uses a CloudSQL Postgres database to retain a list of jobs for progress tracking.
Main Files
| File | Purpose |
|---|---|
| schema/ | Postgres db creation scripts |
| jobdef/ | Example dataflow pipeline options json config file |
| dfmgr.go | Logic manager |
| dfmgr_test.go | Tests |
| pgmgr.go | Postgres logic manager |
| pgmgr_test.go | Tests |
| pgdatamgr.go | Data repo interface |
Ancillary Files
| File | Purpose |
|---|---|
| config.go | Boot package parameters, environment var collection |
| const.go | Package constants |
| entity.go | Package structs |
| env | Package environment variables for local/dev installation |
| gogets | Statements for go-getting required packages |
# Functions
NewGoogleCredentials returns a GCP/Google credential from a supplied file.
NewMgr returns a new manager.
NewPgMgr creates a new manager.
# Constants
CnstStateCancelled indicates that the job has been explicitly cancelled.
CnstStateCancelling indicates that the job has been explicitly cancelled and is in the process of stopping.
CnstStateDone indicates that the job has successfully completed.
CnstStateDrained indicates that the job has been drained.
CnstStateDraining indicates that the job is in the process of draining.
CnstStateFailed indicates that the job has failed.
CnstStatePending indicates that the job has been created but is not yet running.
CnstStateQueued indicates that the job has been created but is being delayed until launch.
CnstStateRunning indicates that the job is currently running.
CnstStateStopped indicates that the job has not yet started to run.
CnstStateUnknown The job's run state isn't specified.
CnstStateUpdated indicates that the job was successfully updated, meaning that this job was stopped and another job was started, inheriting state from this one.
# Variables
EnvDebugOn controls verbose logging.
ErrNoDataFound occurs if a json result returns null.
# Structs
DfMgr covers job management functionality.
DsJob covers the basic identifier data required to control a dataflow job.
JobRunParameter contains the full set of parameters to run a datflow job.
JobSimpleMeta contains the basic data.
PgMgr handles interactions with a postgres db store.