modulepackage
6.13.0-rc1
Repository: https://github.com/htcondor/osdf-client.git
Documentation: pkg.go.dev
# README
Open Science Data Federation (OSDF) Client
The OSDF Client downloads files from the Open Science Data Federation through a series of caches. The OSDF is used by multiple organizations to effeciently transfer input and output data.
The client consists of two parts: a command-line tool named StashCP, and an HTCondor file transfer plugin.
When configured, this plugin will allow the user to specify transfer_input_files with the stash:// protocol which will be downloaded through the OSDF caches. An example of a submit file:
...
transfer_input_files = stash:///osgconnect/public/dweitzel/blast/queries/query1
...
Note: This repo is the continuation of opensciencegrid/stashcp; visit that repo for old issues and releases.
Building
Building is performed with the goreleaser tool. To build a snapshot (not release):
$ goreleaser --clean --snapshot
The binaries will be located in ./dist directory.
Testing and Usage
Run this simple command to download a test file
$ ./stashcp /osgconnect/public/dweitzel/blast/queries/query1 ./
Configuration
stashcp is affected by the environment variables:
| Environment Variable | Description |
|---|---|
OSG_DISABLE_PROXY_FALLBACK | Do not disable using proxies. By default, stashcp will try to use an HTTP proxy when connecting to a cache. If this environment variable is set (no value necessary, only if it's set), then stashcp will not fallback to no proxy if the proxy download fails. |
STASHCP_MINIMUM_DOWNLOAD_SPEED | The lower limit a download will be cancelled, in bytes per second |
STASH_NAMESPACE_URL | The URL to download the namespace and cache information. Default: https://topology.opensciencegrid.org/stashcache/namespaces |
OSDF_DIRECTOR_URL | The URL for an OSDF Director. If set, stashcp will get its list of geo-ordered caches from the Director. Default: Unset |
# Functions
Given a URL and a piece of the namespace, attempt to acquire a valid token for that URL.
AddError will add an accumulated error to the error stack.
No description provided by the author
Determine whether we are allowed to skip the proxy as a fallback.
No description provided by the author
Given the Director response, create the ordered list of caches and store it as namespace.SortedDirectorCaches.
Start the transfer, whether read or write back.
DownloadHTTP - Perform the actual download of the file.
ErrorsRetryable returns if the errors in the stack are retryable later.
Find takes a slice and looks for an element in it.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
GetNamespaces returns the list of namespaces.
HasPort test the host if it includes a port.
Simple parser to that takes a "values" string from a header and turns it into a map of key/value pairs.
No description provided by the author
Determines whether or not we can interact with the site HTTP proxy.
IsRetryable will return true if the error is retryable.
MatchNamespace matches the namespace passed in to the namespaces in the list.
NewTransferDetails creates the TransferDetails struct with the given cache.
NewTransferDetails creates the TransferDetails struct with the given cache.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
UploadFile Uploads a file using HTTP.
# Variables
CacheOverride.
No description provided by the author
Number of caches to attempt to use in any invocation.
Nearest cache.
List of caches, in order from closest to furthest.
No description provided by the author
No description provided by the author
# Structs
Cache.
ConnectionSetupError is an error that is returned when a connection to the remote server fails.
Credential generation information.
No description provided by the author
No description provided by the author
Namespace holds the structure of stash namespaces.
No description provided by the author
No description provided by the author
ProgressReader wraps the io.Reader to get progress Adapted from https://stackoverflow.com/questions/26050380/go-tracking-post-request-progress.
SlowTransferError is an error that is returned when a transfer takes longer than the configured timeout.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
# Interfaces
No description provided by the author