Nightcrawler Swift

Like the X-Men nightcrawler this gem teleports your assets to a OpenStack Swift bucket/container. It was designed to sync your assets with OpenStack Swift and allow some operations with your buckets/containers.

Compatibility

nightcrawler_swift version 1.x is compatible with Identity API v3. To use API v2, you should install version 0.x.

Installation

Add this line to your application's Gemfile:

gem 'nightcrawler_swift'

Or, if you want to use Identity API v2:

gem 'nightcrawler_swift', '< 1.0'

And then execute:

$ bundle

Or install it manually:

$ gem install nightcrawler_swift  # Install latest version

Or set a specific version:

$ gem install nightcrawler_swift -v 0.11.1

Usage

With Rails

1) Configure your swift credentials and options

In config/application.rb or config/environments/*.rb

config.nightcrawler_swift.bucket = "rogue"
config.nightcrawler_swift.tenant_name = "nightcrawler"
config.nightcrawler_swift.username = "my_username1"
config.nightcrawler_swift.password = "my_password1"
config.nightcrawler_swift.auth_url = "https://auth.url.com:123/v2.0/tokens"

The password option can be left blank and be defined by the env variable NSWIFT_PASSWORD. The env variable will take precedence if the option was set.

Optional configurations:

config.nightcrawler_swift.max_age = 3600 # default: nil
config.nightcrawler_swift.timeout = 10 # in seconds, default: nil

# default: uses the admin_url returned by authentication
config.nightcrawler_swift.admin_url = "https://api.host.com/v1/AUTH_1234"

# default: uses the public_url returned by authentication
config.nightcrawler_swift.public_url = "http://asset.host.com/v1/AUTH_1234"

# default: 5, to disable set it to false
config.nightcrawler_swift.retries = 3

# in seconds, default: 30
config.nightcrawler_swift.max_retry_time = 64

# default: false. You could use OpenSSL::SSL::VERIFY_PEER
config.nightcrawler_swift.verify_ssl = true

# default: nil
config.nightcrawler_swift.ssl_client_cert =
  OpenSSL::X509::Certificate.new(File.read("cert.pem"))

# default: nil
config.nightcrawler_swift.ssl_client_key =
  OpenSSL::PKey::RSA.new(File.read("key.pem"), "passphrase, if any")

# default: nil
config.nightcrawler_swift.ssl_ca_file = "ca_certificate.pem"

#default: nil
config.nightcrawler_swift.ssl_version = "SSLv23"

#default: {}
config.nightcrawler_swift.custom_headers = {custom_header: 'custom_value'}

By default it will use Rails.logger as logger, to change that use a different logger in configurations, like:

config.nightcrawler_swift.logger = Logger.new(STDOUT)

further explanation of configurations

max_age

Defines the Cache-Control:max-age= header.

expires

Defines the Expires header.

content_encoding

Defines the Content-Encoding: header

retries

The number of times to retry the request before failing. To disable this feature set it to false.

max_retry_time

Maximum delay in seconds between each retry. The delay will start with 1s and will double for each retry until this value.

custom_headers

Optional hash with header fields to be uploaded with the object.

2) Profit!

$ rake nightcrawler_swift:rails:asset_sync

It will invoke rake assets:precompile and will copy your public directory to swift bucket/container. To sync the public directory without the asset precompilation use the task: nightcrawler_swift:rails:sync

Programatically

1) Configure your swift credentials and options

NightcrawlerSwift.configure({
  bucket: "rogue",
  tenant_name: "nightcrawler",
  username: "my_username1",
  password: "my_password1",
  auth_url: "https://auth.url.com:123/v2.0/tokens"
})

The password option can be left blank and be defined by the env variable NSWIFT_PASSWORD. The env variable will take precedence if the option was set.

Optional configurations:

max_age: 3600,
timeout: 10, # in seconds

# default: uses the admin_url returned by authentication
admin_url: "https://api.host.com/v1/AUTH_1234",

# default: uses the public_url returned by authentication
public_url: "http://asset.host.com/v1/AUTH_1234",

# default: 5, to disable set it to false
retries: 3,

# in seconds, default: 30
max_retry_time: 64

# default: false. You could use OpenSSL::SSL::VERIFY_PEER
verify_ssl: true,

# default: nil
ssl_client_cert:
  OpenSSL::X509::Certificate.new(File.read("cert.pem")),

# default: nil
ssl_client_key:
  OpenSSL::PKey::RSA.new(File.read("key.pem"), "passphrase, if any"),

# default: nil
ssl_ca_file: "ca_certificate.pem"

# default: {}
custom_headers: {custom_header: 'custom_value'}

By default it will use Logger.new(STDOUT) as logger, to change that use:

NightcrawlerSwift.logger = Logger.new(StringIO.new)

2) Use method sync with the desired directory

NightcrawlerSwift.sync File.expand_path("./my-dir")

Command Line

The NightcrawlerSwift shell command (CLI) allows you to interact with your buckets/containers easily, it has the same commands of the gem. To see the help, use the cli without arguments or use the -h/--help switch.

$ nswift # or nswift -h

nswift will use the configurations stored at the file .nswiftrc located at your home directory. If you try to use any command without the file, it will create a sample configuration for you, but you can create your own.

The configuration is a json file, named .nswiftrc. You can include any configuration available to the gem (see the other usages example to know each option available). Follow the format:

{
  "bucket": "<bucket/container name>",
  "tenant_name": "<tenant name>",
  "username": "<username>",
  "password": "<password>",
  "auth_url": "<auth url, ex: https://auth.url.com:123/v2.0/tokens>"
}

The following commands are available through the cli:

$ nswift list

$ nswift upload <real_path> <swift_path> # nswift upload robots.txt assets/robots.txt

Upload also supports a custom max-age, to override the value defined in ".nswiftrc", example:

$ nswift upload <real_path> <swift_path> --max-age VALUE # nswift upload readme assets/readme.md --max-age 300

$ nswift download <swift_path> # nswift download assets/robots.txt > my-robots.txt

$ nswift delete <swift_path> # nswift delete assets/robots.txt

$ nswift url-for <swift_path> # nswift url-for assets/robots.txt

$ nswift metadata <swift_path>
# {
#   "date": "Tue, 31 Mar 2015 02:10:40 GMT",
#   "content_type": "text/css",
#   "content_length": "50013",
#   "last_modified": "Wed, 15 Oct 2014 13:38:56 GMT",
#   "etag": "e13839c545f32be23b942a41f3ea7724",
#   "x_timestamp": "1413380335.38118",
#   "cache_control": "public, max-age=604800",
#   "x_trans_id": "tx824d632ef9e54685a7563-005519f838"
# }

For any commands you could provide a different configuration file through the -c/--config switch, as:

$ nswift list -c /dir/my-nswift-rc

and a different bucket/container name through the -b/--bucket switch, as:

$ nwift list -b rogue

Commands

NightcrawlerSwift has some useful built-in commands. All commands require the configuration and will automatically connect/reconnect to keystone when necessary.

Upload

upload = NightcrawlerSwift::Upload.new
upload.execute "my_file_path.txt", File.open("../my_file_fullpath.txt", "r")
# true / false

This upload command was not designed to send very large files.

It will accept a custom max-age, overriding the configured value through NightcrawlerSwift.configure, example:

upload = NightcrawlerSwift::Upload.new
upload.execute "readme", File.open("readme.md", "r"), max_age: 300
# true / false

Download

download = NightcrawlerSwift::Download.new
download.execute "my_file_path.txt"
# File content

This download command was not designed to retrieve very large files

List

list = NightcrawlerSwift::List.new
list.execute
# [{"hash": "...", "name": "my_file_path.txt"}, {}, {}, ...]

This command supports the following parameters: limit, marker, end_marker, prefix, format, delimiter, path

Example:

list = NightcrawlerSwift::List.new
list.execute prefix: '/some/path'
# [{"hash": "...", "name": "/some/path/with/my_file_path.txt"}, {}, {}, ...]

Metadata

metadata = NightcrawlerSwift::Metadata.new
metadata.execute "my_file.css"
# {
#   :date=>"Tue, 31 Mar 2015 01:46:10 GMT",
#   :content_type=>"text/css",
#   :content_length=>"50013",
#   :last_modified=>"Wed, 15 Oct 2014 13:38:56 GMT",
#   :etag=>"e13839c545f32be23b942a41f3ea7724",
#   :x_timestamp=>"1413380335.38118",
#   :cache_control=>"public, max-age=604800",
#   :x_trans_id=>"tx92ca1becd1a74fa3ae1f2-005519fc62"
# }

# Consulting the bucket/container metadata
metadata.execute
# {
#   :date=>"Tue, 31 Mar 2015 01:41:25 GMT",
#   :content_type=>"application/xml; charset=utf-8",
#   :content_length=>"98171",
#   :x_container_object_count=>"374",
#   :x_storage_policy=>"default-3x",
#   :x_container_read=>".r:*",
#   :x_container_bytes_used=>"7171468",
#   :x_timestamp=>"1409750420.36261",
#   :x_trans_id=>"tx17d8c66ab746436b8b7b6-005519fb40"
# }

Delete

delete = NightcrawlerSwift::Delete.new
delete.execute "my_file_path.txt"
# true / false

Sync

sync = NightcrawlerSwift::Sync.new
sync.execute "/dir/to/synchronize"

Connection

To manually establish the connection with keystone, use:

NightcrawlerSwift.connection.connect!

To check if the connection is still valid, use:

NightcrawlerSwift.connection.connected?

To reconnect just use NightcrawlerSwift.connection.connect! again.

Options

After configure the NightcrawlerSwift you can access your configurations through the options method, like:

NightcrawlerSwift.options

The only difference is that you will access each configuration as a method instead of a hash style, like:

NightcrawlerSwift.configure tenant_name: "rogue"

# Can be used as:
NightcrawlerSwift.options.tenant_name # "rogue"

Contributors

Check it out!

https://github.com/tulios/nightcrawler_swift/graphs/contributors

License

See LICENSE for more details.

Contributing

Fork it ( https://github.com/tulios/nightcrawler_swift/fork )
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create a new Pull Request