Categorygithub.com/wix/supraworker
modulepackage
0.4.5
Repository: https://github.com/wix/supraworker.git
Documentation: pkg.go.dev

# README

Supraworker

Supraworker

Pull Jobs from Anywhere

GitHub All Releases License Go Report Card

Supraworker is an abstraction layer around jobs, allowing one to pull any job from any API, callback your APIs, observe execution times and control concurrent execution.

TL;DR: Supraworker makes an API request for a job => API responds with a bash command => Supraworker executes a bash command. Easy.

It can pull any commands (jobs) from your APIs, run commands, and stream logs back to your API. It can also send state updates to a remote API.

Getting started

Prerequisites:

  1. An API service that can serve jobs:
  2. Supraworker configuration:
  3. A server to run Supraworker on

Installation

MacOs X

Binary install

Homebrew is a free and open-source package management system for Mac OS X.

 brew tap wix/tap

if you are using ssh then

brew tap wix/tap  [email protected]:wix/homebrew-brew.git
brew update
brew install wix/tap/supraworker

To update to the latest, run

brew upgrade wix/tap/supraworker

Installing from source

Find the version you wish to install on the GitHub Releases page and download either the darwin-amd64 binary for MacOS or the linux-amd64 binary for Linux. No pre-built binaries are provided for other operating systems or architectures at this time.

Running releases on MacOS:

You need to the download file, extract it, and remove attributes with the following command (where ~/Downloads/supraworker_darwin_amd64/supraworker is path to the file):

$ xattr -d com.apple.quarantine ~/Downloads/supraworker_darwin_amd64/supraworker
$ ~/Downloads/supraworker_darwin_amd64/supraworker

Linux

Download the latest release

curl --silent -L  "https://api.github.com/repos/wix/supraworker/releases/latest"  \
| jq --arg PLATFORM_ARCH "$(echo `uname -s`_amd| tr '[:upper:]' '[:lower:]')" -r '.assets[] | select(.name | contains($PLATFORM_ARCH)).browser_download_url' \
| xargs -I % curl -sSL  % \
| sudo tar --strip-components=1  -xzf  -

Installing from source

  1. install Go v1.13+

  2. clone into your $GOPATH:

    mkdir -p $GOPATH/src/github.com/wix
    git clone https://github.com/wix/supraworker $GOPATH/src/github.com/wix/supraworker
    cd $GOPATH/src/github.com/wix/supraworker
    
  3. install golangci-lint for linting + static analysis

    • Lint: docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.24.0 golangci-lint run -v

Configuration

Define the config file at $HOME/supraworker.yaml:


NOTE: Keys are not case-sensitive (https://github.com/spf13/viper/pull/635#issuecomment-580659676): Viper's default behaviour of lowercasing keys for key insensitivity is incompatible with these standards (like when using case-sensitive API credentials). For eg: MyApiKey=MySecret


# ClientId in case you need to identify the worker
clientId: "my_uniq_client_id"
# how often to call your API, in seconds
api_delay_sec: 10
# operations related to the jobs
jobs:
  get:
    url: "http://localhost:80/get/new/job"
    method: POST
    headers:
      "Content-type": "application/json"
      params:
        "clientid": "{{ .ClientId}}"

Use Cases

  • Airflow sends a task to be executed on AWS EMR
  • You're building your CI/CD system

Running tests

  • expires all test results
$ go clean -testcache
  • runs all tests
$ go test -bench= -test.v  ./...

# Packages

Package cmd defines command tools.
Package communicator provides transport implementations.
Package config provides configuration for `supraworker` application.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author