# README
#Storage Service
Storage service represents local or remote storage to provide unified storage operations. Remote storage could be any cloud storage i.e. google cloud, amazon s3, or simple SCP or HTTP.
Endly inline workflow
endly -r=copy
@copy.yaml
pipeline:
transfer:
action: storage:copy
source:
URL: s3://mybucket/dir
credentials: aws-west
dest:
URL: scp://dest/dir2
credential: dest
assets:
file1.txt:
file2.txt: renamedFile2
Endly workflow service action
Run the following command for storage service operation details:
endly -s=storage
endly -s=storage -a=copy
endly -s=storage -a=remove
endly -s=storage -a=upload
endly -s=storage -a=download
Service Id | Action | Description | Request | Response |
---|---|---|---|---|
storage | copy | copy one or more asset from the source to destination | CopyRequest | CopyResponse |
storage | remove | remove or more resources if exsit | RemoveRequest | RemoveResponse |
storage | upload | upload content pointed by context state key to target destination. | UploadRequest | UploadResponse |
storage | download | copy source content into context state key | DownloadRequest | DownloadResponse |
Storage service uses undelying Storage Service
Asset copy
Copy request provides flexible way of transferring assets from source to destination.
Make sure that when using cloud or other more specific URI scheme, the corresponding imports are in place.
Copy operation provides two way for asset content substitution:
- Replace - simple brute force key value pair replacement mechanism
- $ expression substitution from context.State() mechanism
import "github.com/viant/endly/storage"
import _ "github.com/viant/toolbox/storage/aws"
import "github.com/viant/endly"
import "log"
func copy() {
var manager = endly.New()
var context := manager.NewContext(nil)
var s3CredentialLocation = ""
var request = storage.NewCopyRequest(nil, NewTransfer(url.NewResource("s3://mybucket/asset1", s3CredentialLocation), url.NewResource("/tmp/asset1"), false, false, nil))
err := endly.Run(context, request, nil)
if err != nil {
log.Fatal(err)
}
}
Loading request from URL
CopyRequest can be loaded from URL pointing either to JSON or YAML resource.
copyReq1, err := storage.NewCopyRequestFromURL("copy.yaml")
copyReq2, err := storage.NewCopyRequestFromURL("copy.json")
Single asset transfer
@copy.yaml
source:
URL: mem://yaml1/dir
dest:
URL: mem://dest/dir2
Multi asset transfer with asset In this scenario source and dest are used to build full URL with assets
@copy.yaml
source:
URL: mem://yaml1/dir
dest:
URL: mem://dest/dir2
assets:
file1.txt:
file2.txt: renamedFile2
Multi asset transfer with transfers and compression
@copy.yaml
transfers:
- expand: true
source:
url: file1.txt
dest:
url: file101.txt
- source:
url: largefile.txt
dest:
url: file201.txt
compress: true
Single asset transfer with replace data substitution
@copy.json
{
"Source": {
"URL": "mem://yaml1/dir"
},
"Dest": {
"URL": "mem://dest/dir2"
},
"Compress":true,
"Expand": true,
"Replace": {
"k1": "1",
"k2": "2"
}
}
Single asset transfer with $expression data substitution
func copy() {
var manager = endly.New()
var context := manager.NewContext(nil)
var state = context.State()
state.PutValue("settings.port", "8080")
state.PutValue("settings.host", "abc.com")
var request = storage.NewCopyRequest(nil, NewTransfer(url.NewResource("myconfig.json"), url.NewResource("/app/config/"), false, true, nil))
err := endly.Run(context, request, nil)
if err != nil {
log.Fatal(err)
}
}
@myconfig.json
{
"Port":"$settings.port",
"Host":"$settings.host"
}