Smooth S3

http://github.com/nbrochu/smooth_s3

A user-friendly superset of the S3 gem geared towards file system backup operations.

Smooth S3 is a user-friendly superset of the S3 gem geared towards file system backup operations. It greatly simplifies regular file uploads to S3 by using Convention over Configuration™. The library also adds new features such as directory syncronization and timestamped uploads, which should come in real handy to anyone doing backup scripts on a regular basis. A decent amount of control is left to the developer: You can specify a prefix path to use with any upload and provide your own timestamp formats if desired.

The goal with Smooth S3 is to facilitate and simplify your S3 uploads. It is a library focused on the file system, so no integration with MySQL, third-party services or anything like that. Nothing prevents you from doing a mysqldump and uploading the results in the same script using Smooth S3 though ;)

Aren’t the AWS/S3 and S3 gems already good enough? Well yes, they are, but all they do is wrap the AWS S3 API. They do an awesome job at it, mind you, but if you’ve had to write more than a few apps and/or scripts where you need to use S3, you know that there is quite a bit of boilerplate code required even for the simplest upload operations.

There was an opportunity to make all of this way simpler (or, puts on sunglasses, smoother), and out of that opportunity Smooth S3 was born. For the people who like bullet points, here are a few reasons why you should probably use it:

• Smart defaults – No more boilerplate code. Once the service is initialized (a one-liner, of course), you only need to specify bucket and file(s) and they will be uploaded with a single method call on the service. The bucket doesn’t exist? No problem. As long as the name you provided is unique in the S3 global namespace, it will be created. Want to overwrite existing files in the bucket? Add a bang(!) to your method. You get the gist of it.

• Few methods, lots of power – On top of service initialization, the library currently only offers 4 methods (that could technically, and may eventually be reduced to 2) so it is pretty darn simple to use. Options are used to arm that simplicity with some power. There is a good chance that by combining options, you will get your uploads to behave the way you want with SmoothS3. If not, fork away or suggest one in the issues! :)

• No raising, unless absolutely necessary – The established S3 libraries are pretty trigger-happy when it comes to raising exceptions. We all love Fail Fast™ but there are times when it’s simply not the right approach, such as with backups. Sure, you will eventually have rescued everything but by then, your codebase will have tripled. Smooth S3 only raises when it absolutely has no choice to do so (ex: Wrong AWS credentials). Failed uploads are retried a few times, then skipped with a log message if they are still a no-go (permissions etc.). Bucket and object operations are also retried in case of a random HTTP failures and won’t throw exceptions if you look for a non-existant one.

• Dogfood is being eaten – Smooth S3 is being used in production to back up 100+ GBs of unique data daily over here at Needium. Most options that are added are derived from our direct needs, so you can be certain that the library will remain supported and new features will keep being added.

This library has been developed against and designed for MRI 1.9.2 in a UNIX environment. It should be compatible with 1.8.6+, as well as with other Ruby implementations, but no guarantees. Probably not compatible with Windows environments.

Using RubyGems - gem install smooth_s3
Using Bundler - gem "smooth_s3" in Gemfile, then bundle install

Before running upload operations using Smooth S3, you will need to initialize the service using your AWS credentials.

Once that is done you can use the following methods:

• upload() – Regular file upload. Can specify multiple files at once. Directory structure is not preserved, only the file name.

• directory_sync() – Uploads the entire content of a directory and its subdirectories. Preserves folder directory structure inside S3.

• timestamped_upload() – Like a regular upload. Files uploaded this way have a timestamp added in front of their names. Default timestamp format: YYYYmmddHHMMSS

• timestamped_directory_sync() – Like a regular directory sync. Provided directory has a timestamp added in front of its name. Default timestamp format: YYYYmmddHHMMS

Each of the 4 methods above also has a bang(!) version available. The difference between the regular and bang version is that the former won’t overwrite existing files in the selected bucket, while the latter will.

You can also specify a prefix as an option to the above methods. This will be inserted in front of what would have been the normal path on S3. For example, you upload ‘test.rb’ with the prefix ‘path/to’. It will show up on S3 as ‘path/to/test.rb’

That’s it! Code examples in the next section.

require 'rubygems' # if using 1.8
require 'smooth_s3'

# Initialize the service
# Params: :aws_key, :aws_secret(, :ssl => false)
 
@service = SmoothS3::Service.new(:aws_key => "MY_AWS_ACCESS_KEY", :aws_secret => "MY_AWS_SECRET_KEY")


# upload()
# Params: bucket_name, files(, :prefix)

@service.upload("my_test_bucket", "test.rb")
@service.upload!("my_test_bucket", ["here.rb", "../parent.rb"], :prefix => "prefix/to/files")


# directory_sync()
# Params: bucket_name, directory(, :prefix, :only, :except)

@service.directory_sync("my_test_bucket", "../reports")
@service.directory_sync!("my_test_bucket", "data", :prefix => "prefix/to/dir")
@service.directory_sync("my_test_bucket", "../reports", :only => /\.json/)
@service.directory_sync("my_test_bucket", "../reports", :except => [/\.xml/, /\.txt/])


# timestamped_upload()
# Params: bucket_name, files(, :prefix, :timestamp_type => :epoch/:strftime(default), :timestamp_format => "%Y%m%d%H%M%S" )

@service.timestamped_upload("my_test_bucket", "test.rb")
@service.timestamped_upload!("my_test_bucket", "test.rb", :timestamp_type => :epoch)
@service.timestamped_upload!("my_test_bucket", "test.rb", :timestamp_type => :strftime, :timestamp_format => "%Y%m%d")


# timestamped_directory_sync()
# Params: bucket_name, directory(, :prefix, :timestamp_type => :epoch/:strftime(default), :timestamp_format => "%Y%m%d%H%M%S", :only, :except )

@service.timestamped_directory_sync("my_test_bucket", "data")
@service.timestamped_directory_sync!("my_test_bucket", "data", :timestamp_type => :epoch)
@service.timestamped_directory_sync!("my_test_bucket", "data", :timestamp_type => :strftime, :timestamp_format => "%Y%m%d")

• Jakub Kuźma for his great S3 library, often overshadowed by AWS-S3.

Copyright © 2011 – Nicholas Brochu

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
‘Software’), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED ‘AS IS’, WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.