Categorygithub.com/koblas/s3-cli
modulepackage
0.0.0-20231201181018-81861673e310
Repository: https://github.com/koblas/s3-cli.git
Documentation: pkg.go.dev

# README

s3-cli -- Go version of s3cmd

Command line utility frontend to the AWS Go SDK for S3. Inspired by s3cmd and attempts to be a drop-in replacement.

Features

  • Compatible with s3cmd's config file
  • Supports a subset of s3cmd's commands and parameters
    • including put, get, del, ls, sync, cp
    • commands are much smarter (get, put, cp - can move to and from S3)
  • When syncing directories, instead of uploading one file at a time, it uploads many files in parallel resulting in more bandwidth.
  • Uses multipart uploads for large files and uploads each part in parallel. This is accomplished using the s3manager that comes with the SDK
  • More efficent at using CPU and resources on your local machine

Install

go get github.com/koblas/s3-cli

Configuration

s3-cli is compatible with s3cmd's config file, so if you already have that configured, you're all set. Otherwise you can put this in ~/.s3cfg:

[default]
access_key = foo
secret_key = bar

You can also point it to another config file with e.g. $ s3-cli --config /path/to/s3cmd.conf ....

Documentation

In general the commands follow rsync as a guide for command options or the unix command line commands.

cp

Copy files to and from S3

Example:

s3-cli cp /path/to/file s3://bucket/key/on/s3
s3-cli cp s3://bucket/key/on/s3 /path/to/file
s3-cli cp s3://bucket/key/on/s3 s3://another-bucket/some/thing

get

Download a file from S3 -- really an alias for cp

put

Upload a file to S3 -- really an alias for cp

del

Deletes an object or a directory on S3.

Example:

s3-cli del [--recursive] s3://bucket/key/on/s3/

rm

Alias for del

s3-cli rm [--recursive] s3://bucket/key/on/s3/

sync

Sync a local directory to S3

s3-cli sync [--delete-removed] /path/to/folder/ s3://bucket/key/on/s3/

mv

Move an object which is already on S3.

Example:

s3-cli mv s3://sourcebucket/source/key s3://destbucket/dest/key

General Notes about s3cmd commpatability

DONE -

  • s3cmd mb s3://BUCKET
  • s3cmd rb s3://BUCKET
  • s3cmd ls [s3://BUCKET[/PREFIX]]
  • s3cmd la
  • s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
  • s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
  • s3cmd del s3://BUCKET/OBJECT
  • s3cmd rm s3://BUCKET/OBJECT
  • s3cmd du [s3://BUCKET[/PREFIX]]
  • s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
  • s3cmd modify s3://BUCKET1/OBJECT
  • s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
  • s3cmd info s3://BUCKET[/OBJECT]

TODO - for full compatibility (with s3cmd)

  • s3cmd restore s3://BUCKET/OBJECT

  • s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]

  • s3cmd setacl s3://BUCKET[/OBJECT]

  • s3cmd setpolicy FILE s3://BUCKET

  • s3cmd delpolicy s3://BUCKET

  • s3cmd setcors FILE s3://BUCKET

  • s3cmd delcors s3://BUCKET

  • s3cmd payer s3://BUCKET

  • s3cmd multipart s3://BUCKET [Id]

  • s3cmd abortmp s3://BUCKET/OBJECT Id

  • s3cmd listmp s3://BUCKET/OBJECT Id

  • s3cmd accesslog s3://BUCKET

  • s3cmd sign STRING-TO-SIGN

  • s3cmd signurl s3://BUCKET/OBJECT <expiry_epoch|+expiry_offset>

  • s3cmd fixbucket s3://BUCKET[/PREFIX]

  • s3cmd ws-create s3://BUCKET

  • s3cmd ws-delete s3://BUCKET

  • s3cmd ws-info s3://BUCKET

  • s3cmd expire s3://BUCKET

  • s3cmd setlifecycle FILE s3://BUCKET

  • s3cmd dellifecycle s3://BUCKET

  • s3cmd cflist

  • s3cmd cfinfo [cf://DIST_ID]

  • s3cmd cfcreate s3://BUCKET

  • s3cmd cfdelete cf://DIST_ID

  • s3cmd cfmodify cf://DIST_ID

  • s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]

# Functions

CamelToSnake converts a given string to snake case.
One command to do it all, since get/put/cp should be able to copy from anywhere to anywhere using standard "cp" command semantics .
No description provided by the author
One command to sync files/directories -- it's always recursive when directories are present Useful options --verbose --dry-run --no-check-md5 Notes: Sync is one of three forms file -> file file(s) -> directory directory -> directory -- If src ends in a '/' then a directory isn't created on the destination s3-cli sync foo/bar s3://bucket/data/ -- yields s3://bucket/data/bar/..
TODO: Handle --recusrive.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
Read the configuration file if found, otherwise return default configuration Precedence order (most important to least): - Command Line options - Environment Variables - Config File - Default Values.
No description provided by the author
SessionForBucket - For a given S3 bucket, create an approprate session that references the region that this bucket is located in.
SessionNew - Read the config for default credentials, if not provided use environment based variables.

# Constants

No description provided by the author
No description provided by the author
No description provided by the author
1M.

# Structs

Action for syncing.
This is the global configuration, it's loaded from .s3cfg (by default) then with added overrides from the command line Command lines are by default the snake case version of the the struct names with "-" instead of "_" .
No description provided by the author
No description provided by the author

# Type aliases

No description provided by the author