MultiSync

❗ currently a functioning WIP thats not quite finished yet but its close! ❗

Flexible synchronisation for your assets.

MultiSync stands on the shoulders of giants. On one side is Celluloid allowing for the synchronisation of assets to be highly parallel. On the other is Fog::Storage allowing MulitSync to support various well known storage services.

What that means is when your configuring MultiSync your creating various pools of workers which then distrubute the work behind synchronising your assets. Meaning that when your site has thousands of files, you get alot more 💥 for your 💵 in less ⏰.

MultiSync tries to expose its asset synchronisation in a flexible way which should allow you to define how and where your assets live. Where possible though, MultiSync will try to provide support for various well known libraries.

Listed below are examples of how to get setup and started.

Installation

gem 'multi_sync', '~> 0.0.4'

require 'multi_sync'

MultiSync.configure do |config|
  # config.verbose = false  # turn on verbose logging (defaults to false)
  # config.force = false  # force syncing of outdated_files (defaults to false)
  # config.run_on_build = true # when within a framework which `builds` assets, whether to sync afterwards (defaults to true)
  # config.sync_outdated_files = true # when an outdated file is found whether to replace it (defaults to true)
  # config.delete_abandoned_files = true # when an abandoned file is found whether to remove it (defaults to true)
  # config.upload_missing_files = true # when a missing file is found whether to upload it (defaults to true)
  # config.target_pool_size = 8 # how many threads you would like to open for each target (defaults to the amount of CPU core's your machine has)
  # config.max_sync_attempts = 3 # how many times a file should be retried if there was an error during sync (defaults to 3)
end

Fog credentials support

MultiSync supports utilising Fog Credentials. Simply specify either a FOG_RC or .fog and we'll use it as the base for any :credentials used in a target.

Features / Usage Examples

MultiSync in its simplist form consists of three objects. sources, resources and targets. A source defines how and where a list of files (or resources) can be found. A resource represents a file from a source with additional properties (such as how to compare them). A target is destination which resources can be synchronised against.

Source

All sources takes one argument which is a Hash of configuration detailed below. There are currently two type's of sources which are

Source Types

local_source - Uses all files within the source_dir
manifest_source - Tries to find a Sprockets manifest.{yml,json} file within the source_dir

Key	Type	Default	Description
`source_dir`	`Pathname`, `String`	`nil`	The location this `source` should use
`resource_options`	`Hash`	`{}`	A hash of options for this `source`'s resources
`targets`	`Symbol`, `Array`	All targets	The `target`(s) this `source` should sync against
`include`	`String` (shell glob)	`*/`	A shell globe to use for inclusion
`exclude`	`String` (shell glob)	`nil`	A shell globe to use for exclusion

# A `local` `source` which will use all files within '../build'
local_source({
  source_dir: '../build'
})

# A `manifest` `source` which will use a Sprockets ':manifest' within '../public/assets'
manifest_source({
  source_dir: '../public/assets'
})

# A `local` `source` which will use all files
# within '../build' including only 'mp4, mpg, mov' files
local_source({
  source_dir: '../build',
  include: '*.{mp4,mpg,mov}'
})

# A `local` `source` which will use all files
# within '../build' excluding any 'jpg, gif, png' files
local_source({
  source_dir: '../build',
  exclude: '*.{jpg,gif,png}'
})

# A `manifest` `source` which will use use a Sprockets `manifest`
# within '../public/assets' including only 'jpg, gif, png' files
# which sets `cache_control` and `expires` headers
manifest_source({
  source_dir: '../public/assets',
  include: '*.{jpg,gif,png}',
  resource_options: {
    cache_control: 'public, max-age=31557600',
    expires: CGI.rfc1123_date(Time.now + 31557600)
  }
})

Target

All targets takes one argument which is a Hash of configuration detailed below. There is currently only one target type which is:

Target Types

aws_target - Synchronises to aws (S3)

Key	Type	Default	Description
`target_dir`	`Pathname`, `String`	`nil`	the name of the `target`'s directory (eg s3 bucket name)
`destination_dir`	`Pathname`, `String`	`nil`	the name of the `target` destination's directory (eg folder within target)
`credentials`	`Hash`	inherits Fog Credentials	credentionals needed by Fog

# An `aws` `target` which will sync to the root of a bucket named 's3-bucket-name'
# in region 'us-east-1', with access_key_id 'xxx', and secret_access_key 'xxx'
aws_target({
  target_dir: 's3-bucket-name'
  credentials: {
    region: 'us-east-1',
    aws_access_key_id: 'xxx',
    aws_secret_access_key: 'xxx'
  }
})

# An `aws` `target` which will sync to a bucket named 's3-bucket-name'
# using credentials sourced from Fog's credentials (.fog or FOG_RC)
aws_target({
  target_dir: 's3-bucket-name'
})

# An `aws` `target` which will sync to a bucket named 's3-bucket-name'
# within a directory named 'directory-within-s3'
# in region 'us-east-1', with access_key_id 'xxx', and secret_access_key 'xxx'
aws_target({
  target_dir: 's3-bucket-name'
  destination_dir: 'directory-within-s3'
  credentials: {
    region: 'us-east-1',
    aws_access_key_id: 'xxx',
    aws_secret_access_key: 'xxx'
  }
})

Supported Libraries

Rails
Sinatra (WIP)
Middleman (wip)
Jekyll (wip)
Nanoc (WIP)

Badges

Supported Storage Services

Behind the scenes we're using Fog::Storage which allows us to support the most popular storage providers

Supported Ruby Versions

This library aims to support and is tested against the following Ruby implementations:

Ruby 2.1.0
Ruby 2.0.0
Ruby 1.9.3
JRuby
Rubinius

multi_sync

Development

Runtime