Open Science Data Federation (OSDF) Client

The OSDF Client downloads files from the Open Science Data Federation through a series of caches. The OSDF is used by multiple organizations to effeciently transfer input and output data.

The client consists of two parts: a command-line tool named StashCP, and an HTCondor file transfer plugin.

When configured, this plugin will allow the user to specify transfer_input_files with the stash:// protocol which will be downloaded through the OSDF caches. An example of a submit file:

...
transfer_input_files = stash:///osgconnect/public/dweitzel/blast/queries/query1
...

Note: This repo is the continuation of opensciencegrid/stashcp; visit that repo for old issues and releases.

Building

Building is performed with the goreleaser tool. To build a snapshot (not release):

$ goreleaser --clean --snapshot

The binaries will be located in ./dist directory.

Testing and Usage

Run this simple command to download a test file

$ ./stashcp /osgconnect/public/dweitzel/blast/queries/query1 ./

Configuration

stashcp is affected by the environment variables:

Environment Variable	Description
OSG_DISABLE_PROXY_FALLBACK	Do not disable using proxies. By default, stashcp will try to use an HTTP proxy when connecting to a cache. If this environment variable is set (no value necessary, only if it's set), then stashcp will not fallback to no proxy if the proxy download fails.
STASHCP_MINIMUM_DOWNLOAD_SPEED	The lower limit a download will be cancelled, in bytes per second
STASH_NAMESPACE_URL	The URL to download the namespace and cache information. Default: https://topology.opensciencegrid.org/stashcache/namespaces
OSDF_DIRECTOR_URL	The URL for an OSDF Director. If set, stashcp will get its list of geo-ordered caches from the Director. Default: Unset