ObjInSync

Daemon to continuously synchronize a directory from remote object store to a local directory.

NOTE: it automatically ignores __pycache__ directory.

Usage

objinsync pull s3://bucket/keyprefix ./localdir

When running in daemon mode (without --once flag), a healthcheck endpoint is served at :8087/health and a prometheus metrics endponit is served at :8087/metrics. You can use --status-addr to override the binding address.

Objinsync also comes with builtin Sentry integraiton. To enable it, set the SENTRY_DSN environment variable.

Installation

Simply download the prebuilt binary from release page or use go get command:

go get github.com/scribd/objinsync

Development

Run tests

make test

Run from source

AWS_REGION=us-east-2 go run main.go pull s3://qph-test-airflow-airflow-code/airflow_home/dags ./dags