S3CP: Commands-line tools for Amazon S3 file manipulation
Just a few simple command-line utilities to list, copy, view S3 files, e.g., s3cp, s3ls, s3cat, s3rm, etc.
Installing
Make sure you have Rubygems installed on your system then run:
# gem install s3cp
Examples
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
s3ls s3://mybucket/path/to/some/files
s3dir s3://mybucket/path/to/some/files
s3cat s3://mybucket/path/to/some/file.txt
s3cp local_file.bin s3://mybucket/some/path
s3mod s3://mybucket/path/to/some/file.txt public-read
s3stat s3://mybucket/path/to/some/file.txt
s3du --depth 2 --unit mb s3://mybucket/some/path/
Use the -h option to learn about command-line options.
All commands support both s3://bucket/path/to/file and the legacy bucket:path/to/file syntax.
Commands are also TTY-aware; when run in an interactive shell, their behavior will change. For example, s3cat will launch your favorite PAGER or less (the default pager) whereas s3ls will display N items at a time, where N is the number of display lines on your terminal and pause between pages.
Security / Credentials
Starting with v1.1.0, S3CP uses the default credential provider from the aws-sdk which makes a best effort to locate your AWS credentials. It checks a variety of locations in the following order:
-
Static credentials from AWS.config (e.g. AWS.config.access_key_id, AWS.config.secret_access_key)
-
The environment (e.g. ENV['AWS_ACCESS_KEY_ID'] or ENV['AMAZON_ACCESS_KEY_ID'])
-
EC2 metadata service (checks for credentials provided by roles for instances).
Usage
$ s3cp
s3cp supports 4 copying use cases:
1. Copy from local machine to S3
2. Copy from S3 to local machine
3. Copy from S3 to S3
4. Copy from local machine to another path on local machine (for completeness)
Local to S3:
s3cp LOCAL_PATH S3_PATH
S3 to Local:
s3cp S3_PATH LOCAL_PATH
S3 to S3:
s3cp S3_PATH S3_PATH2
Local to Local:
s3cp LOCAL_PATH LOCAL_PATH2
-r, --recursive Recursive mode
--headers 'Header1: Header1Value','Header2: Header2Value'
Headers to set on the item in S3.
e.g.,
HTTP headers: 'Content-Type: image/jpg'
AMZ headers: 'x-amz-acl: public-read'
--verbose Verbose mode
--debug Debug mode
-h, --help Show this message
$ s3ls
s3ls [path]
-l Long listing format
--date-format FORMAT Date format (see http://strfti.me/)
--verbose Verbose mode
--rows ROWS Rows per page
-h, --help Show this message
$ s3cat
s3cat [path]
--debug Debug mode
--tty TTY mode
-h, --help Show this message
$ s3du [path] # Display disk usage
--unit UNIT Force unit to use for file size display: B, KB, MB, GB, TB, EB, ZB, YB, BB.
--precision PRECISION Precision used to display sizes, e.g. 3 => 0.123GB. (default 0)
--depth DEPTH Depth to report space usage (default 0).
--regex REGEX Regular expression to match keys.
-h, --help Show this message
$ s3mod
s3mod [path] [permission]
where [permission] is one of:
* private
* authenticated-read
* public-read
* public-read-write
-h, --help Show this message
$ s3rm
s3rm [path]
-r, --recursive Delete S3 keys matching provided prefix.
-i, --include REGEX Delete only S3 objects matching the following regular expression.
-x, --exclude REGEX Do not delete any S3 objects matching provided regular expression.
-F, --fail-if-not-exist Fail if no S3 object match provided key, prefix and/or regex
-t, --test Only display matching keys; do not actually delete anything.
--silent Do not display keys as they are deleted.
--verbose Verbose mode
-h, --help Show this message
Bash completion for S3 URLs
To install Bash completion for S3 URLs, add the following to ~/.bashrc:
for cmd in [ s3cat s3cp s3dir s3ls s3mod s3rm s3stat ]; do
complete -C s3cp_complete $cmd
done
Dependencies
- extensions '~> 0.6' (portability between Ruby 1.8.7/1.9.2)
- highline
>=1.5.1(console/terminal size guessing) - aws-sdk '~> 1.6.3' (underlying Amazon S3 API)
- progressbar '~> 0.10.0' (nice console progress output)
Target platform
- Ruby 1.8.7 / 1.9.2
Development
If you want to hack on s3cp and build the gem yourself, you will need Bundler (http://gembundler.com/) installed. Here is a typical development setup:
# git clone git@github.com:aboisvert/s3cp.git
# cd s3cp
# bundle install
(... hack on s3cp ...)
# bundle exec rake gem
# gem install s3cp-*.gem
License
S3CP is is licensed under the terms of the Apache Software License v2.0. http://www.apache.org/licenses/LICENSE-2.0.html