The Vanagon Project

• What is Vanagon?
  • How is it pronounced?
• Runtime Requirements
  • Local Host:
  • Remote Build Target:
• Configuration and Usage
  • Overview
  • build usage
  • inspect usage
• Engines
  • Amazon Ec2
• Contributing
• License
• Maintainers

Vanagon is a tool to build a single package out of a project, which can itself contain one or more components. This tooling is being used to develop the puppet-agent package, which contains components such as openssl, ruby, and augeas among others. For a simple example, please see the project in the examples/ directory.

Vanagon builds up a Makefile and packaging files (specfile for RPM, control/rules/etc for DEB) and copies them to a remote host, where make can be invoked to build all of the components and make a package of the contents.

Vanagon (/ˈvænəgɪn/) sounds like "van again." It does not sound like "van wagon" or "van and gone."

Vanagon carries two sets of requirements: requirements for the local host where Vanagon is run, and for the remote target where compilation and packaging happens.

Also, Vanagon ships with a number of engines which may include additional optional dependencies if you wish to use them. These engines are currently considered experimental, and receive less attention than the Hardware or vmpooler engines do. If you find a bug in these engines, please open a ticket and let us know.

• Ruby (Ruby 2.3.x is the miniumum supported version)
• fustigit
• ruby-git
• docopt
• The command line tool ssh (homepage) available on the local ${PATH} (any modern version should suffice)
• The command line tool rsync (homepage) available on the local ${PATH} (At least rsync 2.6.x)
• The command line tool git (homepage) available on the local ${PATH} (Vanagon is tested against Git version 1.8.x but should work with any newer version)

• AWS SDK for Ruby, if you're using the EC2 engine
• Docker, if you're using the Docker engine

Note: package installation & builder configuration for the remote target can be customized in the Platform configuration that defines target provisioning instructions.

• GNU Make (homepage) (Vanagon specifically targets the feature set provided by GNU Make 3.81 but newer versions are known to work -- older versions are specifically known to not work!)
• Bash (homepage) is required by the Makefiles that Vanagon generates
• An ssh server (homepage) is required by most engines
• The command line tool rsync (homepage) (At least rsync 2.6.x)

Vanagon won't be much use without a project to build. Beyond that, you must define any platforms you want to build for. Vanagon ships with some simple binaries to use, but the one you probably care about is named 'build'.

Vanagon is broken down into three core ideas: the project, the component and the platform. The project contains one or more components and is built for a platform. As a quick example, if I had a ruby app and wanted to package it, the project would probably contain a component for ruby and a component for my app. If I wanted to build it for debian wheezy, I would define a platform called wheezy and build my project against it.

For more detailed examples of the DSLs available, please see the examples directory and the YARD documentation for Vanagon.

The vanagon command line tool contains these commands:

• build build a package given a project and platform
• build_host_info print information about build hosts
• build_requirements print external packages required to build project
• completion outputs path to tab completion script
• inspect a build dry-run, printing lots of information about the build
• list shows a list of available projects and platforms
• render create local versions of packaging artifacts for project
• sign sign a package
• ship upload a package to a distribution server
• help print this help

Commands are called with a git-like pattern of vanagon <subcommand>. For example: vanagon list

Prior to 0.16.0, the vanagon command line contained these commands

• build
• build_host_info
• build_requirements
• inspect
• render
• repo
• ship
• sign

With the exception of repo, which calls packaging methods, the remaining commands have been moved to a git-like pattern of vanagon <subcommand>. vangon build replaced build, vanagon ship replaced ship and so forth.

The older calling usage is deprecated.

The build command has positional arguments and position independent flags.

The name of the project to build; a file named <project_name>.rb must be present under configs/projects in the working directory.

The name of the target platform to build <project_name> against; a file named <platform_name>.rb must be present under configs/platforms in the working directory. This can also be a comma separated list of platforms such as platform1,platform2; note that there are no spaces after the comma.

Target host is an optional argument to override the host selection. Instead of using a random VM collected from the pooler (Vanagon's default build engine), the build will attempt connect to the target host over SSH as the root user.

If building on multiple platforms, multiple targets can also be specified using a comma separated list such as host1,host2 (note that there are no spaces after the comma). If less targets are specified than platforms, the default engine (pooler) will be used for platforms without a target. If more targets are specified than platforms, the extra platforms will be ignored.

Build machines should be cleaned between builds.

Note: command flags currently can be used anywhere in the command. Recommended usages is to use flagged arguments before the positional arguments.

Specifies a directory on the local host where the sources should be placed and builds performed. Defaults to a temporary directory created with Ruby's Dir.mktmpdir method.

Explicitly specify a directory on the remote target to place sources and perform builds. Components can then be rebuilt manually on the build host for faster iteration. Sources may not be correctly updated if this directory already exists. Defaults to a temporary directory created by running mktemp -d on the remote target.

Specifies where project configuration is found. Defaults to $pwd/configs.

Choose a different virtualization engine to use to select the build target. Currently supported engines are:

• base - Pure ssh backend; no teardown currently defined
• local - Build on the local machine; platform name must match the local machine
• docker - Builds in a docker container
• pooler - Selects a vm from Puppet's internal vmpooler to build on
• hardware - Build on a specific taget and lock it in redis
• ec2 - Build on a specific AWS instance.
• ABS - Selects a vm from Puppet's internal Always-be-scheduling service to build on

Note: command flags can be used anywhere in the command.

Indicates that the host used for building the project should be left intact after the build instead of destroyed. The host is usually destroyed after a successful build, or left after a failed build.

Display command-line help.

A full path on disk for a private ssh key to be used in ssh and rsync communications. This will be used instead of whatever defaults are configured in .ssh/config.

When set, Vanagon will forward the ssh authentication agent connection.

Used in conjunction with the pooler engine, this is a token to pass to the vmpooler to access the API. Without this token, the default lifetime of vms will be much shorter.

The name of the host where redis is running. Redis is used to handle a lock when using the hardware engine. It defaults to redis, with no domain.

Port of the system where redis is running. Defaults to 6379.

Controls whether component sources are downloaded directly from upstream URLs or from configured mirrors. Most Puppet projects using Vanagon default to fetching components from internal mirrors. Set this variable to n or false when building outside of the Puppet private network to download directly from upstream sources.

Some phases of compilation support retries. The default value is 1 but setting to any integer value greater than 1 will causes these components to retry operations on failure until the VANAGON_RETRY_COUNT limit is reached.

Some phases of compilation can take an indeterminate (but substantial) amount of time. The default value is 7200 seconds(120 minutes) but setting to any integer value these components to fail after the VANAGON_TIMEOUT count is reached. Note that this value is expected to be in seconds.

By default, Vanagon does not fail if extra files signing fails, it just logs an error and continues building the package. This is unwanted behavior in environments where we expect a hard failure when signing cannot proceed. To force Vanagon to fail if extra files signing fails, ensure this variable is set before starting a build.

vanagon build --preserve puppet-agent el-6-i386 will build the puppet-agent project on the el-6-i386 platform and leave the host intact afterward.

vanagon build --engine=docker puppet-agent el-6-i386 will build the puppet-agent project on the el-6-i386 platform using the docker engine (the platform must have a docker_image defined in its config).

The inspect command has positional arguments and position independent flags. It mirrors the build command, but exits with success after loading and interpolating all of the components in the given project. No attempt is made to actually build the given project; instead, a JSON formatted array of hashes is returned and printed to stdout. This JSON array can be further processed by external tooling, such as jq.

The name of the project to build, and a file named <project_name>.rb must be present in configs/projects in the working directory.

The name of the platform to build against, and a file named <platform_name>.rb must be present in configs/platforms in the working directory.

Platform can also be a comma separated list of platforms such as platform1,platform2.

Note: command flags currently can be used anywhere in the command. Recommended usages is to use flagged arguments before the positional arguments.

Specifies a directory where the sources should be placed and builds performed. Defaults to a temporary directory created with Ruby's Dir.mktmpdir.

Specifies where project configuration is found. Defaults to $pwd/configs.

Choose a different virtualization engine to use to select the build target. Engines are respected, but only insofar as components and projects are rendered -- the vanagon inspect command performs no compilation.

Supported engines are the same as the vanagon build command.

Display command-line help.

Environment variables are respected, but only insofar as components and projects are rendered -- the vanagon inspect command has no behavior to alter.

Supported environment variables are the same as the vanagon build command.

vanagon inspect puppet-agent el-6-i386 will load the puppet-agent project on the el-6-i386 platform and print the resulting list of dependencies, build-time configuration, environment variables, and expected artifacts.

Note: If you have the aws_ami setup Vanagon will default to the ec2 engine.

To use the ec2 engine you should have your credentials set either via your ~/.aws/credentials or environment variables. After this you can setup your configs/platforms/<platform>.rb to use your ami, instance type, and key_name to setup the instance.

A simple one looks like this

# configs/platforms/el-7-x86_64.rb
platform "el-7-x86_64" do |plat|
    plat.aws_ami "your-ami-id-here" # You must set this
    plat.aws_instance_type "t2.small" # Defaults to t1.micro
    plat.aws_key_name "vanagon" # this is the default but you can use whichever
    plat.aws_user_data <<-eos
#cloud-config
    runcmds:
        - echo #{my_ssh_key} > /root/.ssh/authorized_keys # Most amis block you from logging in as root.
    eos

### Rest of your code here

end

When using the ABS engine, there is a variety of ways you can specify your token:

• the environment variable ABS_TOKEN
• or vanagon token file ~/.vanagon-token (note this is the same file read by the pooler engine)
• or vmfloaty's config file ~/.vmfloaty.yml

In order to modify or track the VM via floaty or bit-bar you can optionally add the vmpooler token (different from the ABS token) via

• VMPOOLER_TOKEN
• or as a second line to the file ~/.vanagon-token
• or by using the existing mechanism in floaty using a vmpooler_fallback

We'd love to get contributions from you! Once you are up and running, take a look at the Contribution Documents to see how to get your changes merged in.

See LICENSE file.

See MAINTAINERS file.