Vendorificator

THIS PROGRAM IS STILL IN BETA, use at your own risk!

Vendor everything. Keep your own copies of upstream dependencies that your project needs somewhere close to your project, best in the same source control repository. But how to do it, and keep track of all the upstream modules, their origin and license, upgrades, local changes, and so on? It's what Vendorificator helps you with.

Vendorificator keeps the upstream dependencies in your Git repository, using Git itself to help you with tracking all the updates, upgrades, and changes. You specify how to get the dependency in a config file written in plain Ruby, called Vendorfile (you don't need to know much Ruby to be able to write it - but you can use all Ruby you know if you want to!). Then you run vendorify command, and everything is done by magic:

• All of defined dependencies are downloaded to specified directories of your repository;
• Every dependency has its own pristine branch, to make it possible to cleanly upgrade the third-party module even if you have introduced your local changes;
• After every download and change of the pristine branch, the resulting commit is annotated with the details: timestamp, origin (including version, download checksum, and/or Git SHA-1), and any comments you might need to keep;
• You can easily upgrade the dependencies, change their origin for subsequent updates (projects do move around), add or remove them, etc;
• At any moment, you can get a list of third-party modules you use with its origin, version, timestamp, and list of your own patches;
• When you upgrade the dependency, you can easily review differences introduced to pristine copy by yourself, difference between old and new version, and merge the new version as you'd merge any regular Git branch.

All kinds of external dependencies you need - be it Ruby, Python, JavaScript, shell scripts, CSS libraries, Chef cookbooks, or any other modules you may need, are specified in a single place, and managed with a single tool.

Add this line to your application's Gemfile:

gem 'vendorificator'

And then execute:

$ bundle

Or install it yourself as:

$ gem install vendorificator

Vendorificator is supported on Ruby 1.9.2 or newer. Rubinius and JRuby in 1.9 mode are also supported.

Vendorificator is a command-line tool. The command is called vendor (bundle exec vendor if you use Bundler). It accepts multiple subcommands.

Run vendor to see list of subcommands. Run vendor help _command_ to get detailed description of a command.

There is a lightning talk presentation/demo slide deck online at https://speakerdeck.com/mpasternacki/vendorificator

Most important commands are listed here; use vendor help for more detail.

• vendor sync will update all vendor modules that need updating
• vendor status will list all the modules and their status
• vendor pull will pull all vendor branches, tags, and notes from a Git remote
• vendor push will push all vendor branches, tags, and notes to a Git remote
• vendor diff will show a git diff of vendor module's files between vendor module's pristine branch and current work tree
• vendor log will show a git log of all changes made to a particular vendor module's files that are not in module's pristine branch

Vendorificator reads its configuration from a Ruby file named Vendorfile (or config/vendor.rb). If the file does not exist in the current directory, it looks for it in parent directories until it gets to the git repository's root. All files Vendorificator creates are rooted in directory containing the Vendorfile (or the directory containing config directory with vendor.rb file in it).

Vendorfile is a Ruby file containing configuration settings and description of upstream vendor modules. It may also contain any custom Ruby code if needed.

• basedir "subpath" -- directory below work root where Vendorificator will download modules. Defaults to "vendor".
• branch_prefix "prefix" -- prefix for name of Git branches. Defaults to "vendor", which means all branches created by Vendorificator start with "vendor/".
• remotes ['remote1', 'remote2', ...] -- list of Git remotes that vendor pull will use as default. Defaults to ['origin'].
• chef_cookbook_ignore_dependencies ['cookbook1', 'cookbook2', ...] -- default value for :ignore_dependencies argument of chef_cookbook modules.

A vendor module is a single piece of upstream code managed by Vendorificator. It is a basic building block. A vendor module is declared by calling one of a couple functions that define them.

Vendor is most general of those. You call it like this:

vendor 'name', :option => 'value' do
  # here in the block you code what needs to be done
  # to get ("conjure") the module. You're already in
  # the right directory, branch, etc, and what you add to the
  # current directory will be committed and tagged.
end

It takes following options:

• :version - the version of module. If the module's contents should change, increase the version, so that Vendorificator knows it needs to re-create the module.
• :group - module's group is subdirectory of the basedir where module's directory will be created. For example, vendor "foo" will go to vendor/foo by default, but vendor "foo", :group => :widgets will go to vendor/widgets/foo. It is also added in a similar way to module's branch name, tag names, etc.
• :path - lets you specify subdirectory in which the module will be downloaded

All other upstream modules take these options, and can be given a block to postprocess their content (e.g. if a tarball includes a .git file/directory that confuses Git, you can remove it in a block).

Example:

vendor 'generated', :version => '0.23' do |v|
  File.open('README', 'w') { |f| f.puts "Hello, World!" }
  File.open('VERSION', 'w') { |f| f.puts v.version }
end

Downloads a single file:

download 'socks.el', :url => 'http://cvs.savannah.gnu.org/viewvc/*checkout*/w3/lisp/socks.el?root=w3&revision=HEAD'
download 'http://mumble.net/~campbell/emacs/paredit.el'

Archive takes a tar.gz, tar.bz2, or zip file, downloads it, and unpacks it as contents of the module. It takes same options as vendor, plus:

• :url -- address from which to download the archive. If not given, module's name will be used as URL, and its basename as module's name (e.g. archive "http://ftp.gnu.org/gnu/hello/hello-2.8.tar.gz" will be named "hello-2.8" and downloaded from the URL).
• :filename -- a filename to download. Useful if URL doesn't specify this nwell.
• :type -- :targz, :tarbz2, or :zip
• :unpack -- a command that will be used to unpack the file
• :basename -- defaults to basename of filename, will be used as directory name
• :no_strip_root -- by default, if archive consists of a single directory, Vendorificator will strip it. Setting this to true disables this behaviour.
• :checksum -- if set to SHA256 checksum of the file, it will be checked on download.

Archive's :version defaults to file name.

Example:

    archive :hello,
      :url => 'http://ftp.gnu.org/gnu/hello/hello-2.8.tar.gz',
      :version => '2.8',
      :checksum => 'e6b77f81f7cf7daefad4a9f5b65de6cae9c3f13b8cfbaea8cb53bb5ea5460d73'

Downloads snapshot of a Git repository. Takes the same options as vendor, plus:

• :repository -- address of the repository. Defaults to name (and sets name to its basename then), just like :url for archive (e.g. git "git://github.com/github/testrepo.git" will be cloned from that repository, and named testrepo).
• :branch, :revision, :tag -- what to check out when repository is cloned.

Git module's :version defaults to the :tag if given, or the conjured revision otherwise.

Example:

git 'git://github.com/mpasternacki/nagios.git',
    :branch => 'COOK-1997',
    :group => :cookbooks,
    :version => '0.20130124.2'

Downloads an Opscode Chef cookbook from http://community.opscode.com/ website (same thing that knife cookbook site install does). It resolves dependencies -- all needed modules will be downloaded by default. Its group defaults to :cookbooks. It may take the same arguments as archive (but the name and possibly version is almost always enough), plus:

• :ignore_dependencies -- if true, ignore dependencies completely. If an array, don't download dependencies that are in the array. Default for that is chef_cookbook_ignore_dependencies setting.

Examples:

chef_cookbook 'apt'
chef_cookbook 'memcached'
chef_cookbook 'chef-server', :ignore_dependences => true

chef_cookbook_ignore_dependencies ['runit']
chef_cookbook 'memcached'

chef_cookbook 'memcached', ignore_dependencies => ['runit']

If you get Chef cookbooks from Git or anywhere else than Opscode's community website, you can still use dependency resolution by using a :hooks option to add it:

git 'git://github.com/user/cookbook.git',
  :group => :cookbooks,
  :hooks => 'ChefCookbookDependencies'
end

Runs a shell command to download dependencies, and then stores them in a pristine branch. You can use this to leverage platform-specific tools (Bundler for Ruby gems, pip for Python packages, Berkshelf for Chef cookbooks, etc), and keep track of downloaded modules with Vendorificator. Takes the same arguments as vendor, plus:

• :command -- command to run to download files
• :specs -- files specifying what to download; these will be kept on the vendor branch together with downloaded dependencies
• :extras -- files that are needed for the tool to work, but which won't be committed to the vendor branch together with specs and dependencies

Two convenience shortcuts are provided, rubygems_bundler, and chef_berkshelf. Their definitions are examples as well:

tool 'rubygems', # <- rubygems_bundler
     :path => 'cache',
     :specs => [ 'Gemfile', 'Gemfile.lock' ],
     :command => 'bundle package --all'

tool 'cookbooks', # <- chef_berkshelf
     :path => 'cookbooks',
     :specs => [ 'Berksfile', 'Berksfile.lock' ],
     :command => 'berks vendor vendor/cookbooks'

rubygems_bundler takes no arguments and it passes block to tool.

chef_berkshelf takes following arguments:

• :path (default: 'cookbooks') -- path to save cookbooks
• :specs (array, always present: Berksfile and Berksfile.lock) -- files treated as specs (you can specify metadata.rb files of your local cookbooks to include their dependencies automatically)
• :berks2 (default: false) -- if set to true, use command for Berkshelf 2 (default is Berkshelf 3)

Overlays multiple modules in the same directory, instead of each of them being conjured in its own.

overlay '/xyzzy' do
  vendor 'foo', :version => '0.23' do |v|
    File.open('README.foo', 'w') { |f| f.puts "Hello, foos!" }
  end
  vendor 'bar', :version => '0.42' do |v|
    File.open('README.bar', 'w') { |f| f.puts "Hello, bars!" }
  end
end

To develop a solution based on Vendorificator without having everything committed to a dozen of branches, tagged, and noted, set vendorificator.stub Git setting to true. Then, Vendorificator will work live in the main repo, and ignore the output directory in Git.

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request