package
0.8.0
Repository: https://github.com/adrianwit/endly.git
Documentation: pkg.go.dev

# README

#Storage Service

Storage service represents local or remote storage to provide unified storage operations. Remote storage could be any cloud storage i.e. google cloud, amazon s3, or simple SCP or HTTP.

Endly inline workflow

endly -r=copy

@copy.yaml


pipeline:
  transfer:
    action: storage:copy  
    source:
      URL: s3://mybucket/dir
      credentials: aws-west
    dest:
      URL: scp://dest/dir2
      credential: dest
    assets:
      file1.txt:
      file2.txt: renamedFile2      

Endly workflow service action

Run the following command for storage service operation details:


endly -s=storage

endly -s=storage -a=copy
endly -s=storage -a=remove
endly -s=storage -a=upload
endly -s=storage -a=download

Service IdActionDescriptionRequestResponse
storagecopycopy one or more asset from the source to destinationCopyRequestCopyResponse
storageremoveremove or more resources if exsitRemoveRequestRemoveResponse
storageuploadupload content pointed by context state key to target destination.UploadRequestUploadResponse
storagedownloadcopy source content into context state keyDownloadRequestDownloadResponse

Storage service uses undelying Storage Service

Asset copy

Copy request provides flexible way of transferring assets from source to destination.

Make sure that when using cloud or other more specific URI scheme, the corresponding imports are in place.

Copy operation provides two way for asset content substitution:

  1. Replace - simple brute force key value pair replacement mechanism
  2. $ expression substitution from context.State() mechanism


    import   "github.com/viant/endly/storage"
    import _ "github.com/viant/toolbox/storage/aws"
    import   "github.com/viant/endly"
    import   "log"


    func copy() {
    	
    	var manager = endly.New()
    	var context := manager.NewContext(nil)
    	var s3CredentialLocation = ""
    	var request = storage.NewCopyRequest(nil, NewTransfer(url.NewResource("s3://mybucket/asset1", s3CredentialLocation), url.NewResource("/tmp/asset1"), false, false, nil))
    	err := endly.Run(context, request, nil)
    	if err != nil {
    		log.Fatal(err)
    	}
    }


Loading request from URL

CopyRequest can be loaded from URL pointing either to JSON or YAML resource.

    copyReq1, err := storage.NewCopyRequestFromURL("copy.yaml")
    copyReq2, err := storage.NewCopyRequestFromURL("copy.json")

Single asset transfer

@copy.yaml

source:
  URL: mem://yaml1/dir
dest:
  URL: mem://dest/dir2

Multi asset transfer with asset In this scenario source and dest are used to build full URL with assets

@copy.yaml

source:
  URL: mem://yaml1/dir
dest:
  URL: mem://dest/dir2
assets:
  file1.txt:
  file2.txt: renamedFile2  

Multi asset transfer with transfers and compression

@copy.yaml

transfers:
- expand: true
  source:
    url: file1.txt
  dest:
    url: file101.txt
- source:
    url: largefile.txt
  dest:
    url: file201.txt
  compress: true

Single asset transfer with replace data substitution

@copy.json

{
  "Source": {
    "URL": "mem://yaml1/dir"
  },
  "Dest": {
    "URL": "mem://dest/dir2"
  },
  "Compress":true,
  "Expand": true,
  "Replace": {
    "k1": "1",
    "k2": "2"
  }
}

Single asset transfer with $expression data substitution


  func copy() {
    	
    	var manager = endly.New()
    	  
    	var context := manager.NewContext(nil)
    	var state = context.State()
    	
    	state.PutValue("settings.port", "8080")
    	state.PutValue("settings.host", "abc.com")
    	
    	var request = storage.NewCopyRequest(nil, NewTransfer(url.NewResource("myconfig.json"), url.NewResource("/app/config/"), false, true, nil))
    	err := endly.Run(context, request, nil)
    	if err != nil {
    		log.Fatal(err)
    	}
    }


@myconfig.json

{
    "Port":"$settings.port",
    "Host":"$settings.host"
}

# Functions

Copy transfers data for provided transfer definition.
GetStorageService return toolbox storage service.
IsShellCompressable returns true if resource can be compress via shell command.
New creates a new storage service.
CopyRequest creates a new copy request.
NewCopyRequestFromuRL creates a new request from URL (JSON or YAML format are supported).
NewExpandedContentHandler return a new reader that can substitute content with state map, replacement data provided in replacement map.
NewRemoveRequest creates a new remove request.
NewTransfer creates a new transfer.
UseMemoryService sets flag on context to always use memory service (testing only).

# Constants

ServiceID represents transfer service id.

# Variables

CompressionTimeout compression/decompression timeout.
MaxContentSize represent max allowed expandable content size.

# Structs

CopyRequest represents a resources copy request.
CopyResponse represents a resources copy response.
DownloadRequest represents a resources download request, it downloads source into context.state target key.
DownloadResponse represents a download response.
RemoveRequest represents a resources Remove request.
RemoveResponse represents a resources Remove response, it returns url of all resource that have been removed.
Transfer represents copy instruction.
UploadRequest represents a resources upload request, it takes context state key to upload to target destination.
UploadResponse represents a upload response.

# Type aliases

AssetTransfer represents asset transfer.