KnifeSpork

KnifeSpork is a workflow plugin for Chef::Knife which helps multiple developers work on the same Chef Server and repository without treading on each other's toes. This plugin was designed around the workflow we have here at Etsy, where several people are working on the Chef repository and Chef Server simultaneously. It contains several functions, documented below:

Knife-Spork is not being particularly actively maintained at this time as the primary author does not currently use it day-to-day. Pull requests and issues might therefore take a while to be actioned - please feel free to reach out if you run into something you consider an urgent blocker.

knife-spork is available on rubygems. Add the following to your Gemfile:

gem 'knife-spork'

or install the gem manually:

gem install knife-spork

knife-spork is also included in the Chef Devlopment Kit.

Out of the box, knife spork will work with no configuration. However, you can optionally enable several features to enhance its functionality.

KnifeSpork will look for a configuration file in the following locations, in ascending order of precedence:

• config/spork-config.yml
• /etc/spork-config.yml
• ~/.chef/spork-config.yml
• .chef/spork-config.yml

Anything set in the configuration file in your home directory for example, will override options set in your Chef repository or /etc.

Below is a sample config file with all supported options and all shipped plugins enabled below, followed by an explanation of each section.

default_environments:
  - development
  - production
environment_groups:
  qa_group:
    - quality_assurance
    - staging
  test_group:
    - user_testing
    - acceptance_testing
version_change_threshold: 2
preserve_constraint_operators: true
environment_path: "/home/me/environments"
save_environment_locally_on_create: false
role_path: "/home/me/roles"
custom_plugin_path: "/home/me/spork-plugins"
always_promote_remote: true
skip_berkshelf: false
role_match_file_name: true
role_safe_delete: true
stdout_diffs
json_options:
  indent: "    "
plugins:
  campfire:
    account: myaccount
    token: a1b2c3d4...
  foodcritic:
    tags: ['any']
  hipchat:
    api_token: ABC123
    rooms:
      - General
      - Web Operations
    notify: true
    color: yellow
  jabber:
    username: YOURUSER
    password: YOURPASSWORD
    nickname: Chef Bot
    server_name: your.jabberserver.com
    server_port: 5222
    rooms:
      - engineering@your.conference.com/spork
      - systems@your.conference.com/spork
  git:
    enabled: true
    auto_push: true
    branch: some_branch
  irccat:
    server: irccat.mydomain.com
    port: 12345
    gist: "/usr/bin/gist"
    channel: ["chef-annoucements"]
  graphite:
    server: graphite.mydomain.com
    port: 2003
  influxdb:
    database: deployments
    username: deploy
    password: deploy
    series: deployments
    host: influx.example.com
    port: 8086
  eventinator:
    url: http://eventinator.mydomain.com/events/oneshot
  slack:
    webhook_url: https://hooks.slack.com/services/ABCD/efgh
    channel: "#sysops"
    username: knife
    icon_url: http://example.com/image.jpg
  rubocop:
    epic_fail: true
    show_name: false
    autocorrect: false
    out_file: <file>
    sev_level: <C|W|E>
    lint: false
    use_cookstyle: true

The default_environments directive allows you to specify a default list of environments you want to promote changes to. If this option is configured and you omit the environment parameter when promoting KnifeSpork will promote to all environments in this list.

The environment_groups directive allows you to specify a list of environments referenced by group names that you want to promote changes to.

The version_change_threshold directive allows you to customise the threshold used by a safety check in spork promote which will prompt for confirmation if you're promoting a cookbook by more than version_change_threshold versions. This defaults to 2 if not set, ie promoting a cookbook from v1.0.1 to v 1.0.2 will not trip this check, wheras promoting from v1.0.1 to v1.0.3 will.

The preserve_constraint_operators directive causes spork promote to preserve existing version constraint operators in your environment files, only updating the version number. This directive is disabled by default, which causes spork to always use the = constraint.

The always_promote_remote directive allows you to tell spork promote to always act as if the --remote option had been specified. This will also have the same effect on spork omni. This option should only be used if you're sure you want all changes to be uploaded to the server as soon as you run promote.

The skip_berkshelf directive is a temporary flag added in #138 to allow Berkshelf functionality to be optionally bypassed until Berkshelf 3 support has been added to knife-spork per #85. It simply removed the :Berkshelf constant from the namespace used by knife-spork.

The json_options directive allows you to tell spork to pass options to pretty_generate to control the format of the resulting json

The environment_path directive allows you to specify the path to where you store your chef environment json files. If this parameter is not specified, spork will default to using the first element of your cookbook_path, replacing the word "cookbooks" with "environments"

The save_environment_on_create directive allows you to have the knife spork environment create command save a copy of the new environment to your Chef repository. This will default to saving environment files in the location specified by the environment_path directive. If this parameter is not specified, spork will default to using the first element of your cookbook_path, replacing the word "cookbooks" with "environments"

The role_path allows you to specify the path to where you store your chef role json files. If this parameter is not specified, spork will default to using the first element of your cookbook_path, replacing the word "cookbooks" with "roles"

The role_safe_delete flag will, when you use the spork knife spork role delete command, check to see if any nodes currently have that role in their runlist and exit with an error message if any are found.

The role_match_file_name flag allows you to check whether the file name that is used to upload a role matches the role name as well. If the parameter is specified, or flag --match-filename is set, spork will not let you upload a role from a file unless the name matches the rolename.

The stdout_diffs flag will cause any diffs generated by knife-spork (for example on events such as data bag uploads, role edits etc) to be displayed on stdout in addition to any notifications generated by plugins.

The custom_plugin_path allows you to specify an additional directory from which to load knife-spork plugins. If this parameter is not specified or the path set does not exist, only the default plugins shipped with knife-spork will be loaded (if enabled in config)

Knife spork supports plugins to allow users to hook it into existing systems such as source control, monitoring and chat systems. Plugins are enabled / disabled by adding / removing their config block from the plugin section of the config file. Any of the default plugins shown above can be disabled by removing their section.

For more information on how to develop plugins for spork, please read the plugins/README.md file.

This function is designed to help you see which plugins you currently have loaded, and the current config Hash which knife spork is using.

knife spork info

$ knife spork info
Config Hash:
{"plugins"=>{"git"=>{"enabled"=>true}, "irccat"=>{"server"=>"irccat.mydomain.com", "port"=>12345, "gist"=>"usr/bin/gist", "channel"=>["#chef-announce"]}, "graphite"=>{"server"=>"graphite.mydomain.com", "port"=>2003}, "slack"=>{"api_token"=>"abc123", "channel"=>"random", "teamname"=>"myteam", "icon_url"=>"http://example.com/image.jpg"}, "eventinator"=>{"url"=>"http://eventinator.mydomain.com/events/oneshot"}}, "default_environments"=>["development", "production"], "version_change_threshold"=>2, "pplugins"=>{"foodcritic"=>{"fail_tags"=>["style,correctness,test"], "tags"=>["~portability"], "include_rules"=>["config/rules.rb"]}}}

Plugins:
KnifeSpork::Plugins::Campfire: disabled
KnifeSpork::Plugins::Eventinator: enabled
KnifeSpork::Plugins::Foodcritic: disabled
KnifeSpork::Plugins::Git: enabled
KnifeSpork::Plugins::Graphite: enabled
KnifeSpork::Plugins::HipChat: disabled
KnifeSpork::Plugins::Slack: enabled
KnifeSpork::Plugins::Irccat: enabled

This function is designed to help you avoid trampling on other people's cookbook versions, and to make sure that when you come to version your own work it's easy to see what version numbers have already been used and if the one you're using will overwrite anything.

knife spork check COOKBOOK [--all]

By default, spork check only shows the 5 most recent remote cookbook versions. Add the --all option if you want to see everything.

$ knife spork check apache2
Checking versions for cookbook apache2...

Local Version:
  1.1.49

Remote Versions: (* indicates frozen)
 *2.0.2
 *2.0.1
  1.1.49
 *1.1.14
 *1.1.13

ERROR: The version 1.1.49 exists on the server and is not frozen. Uploading will overwrite!

$ knife spork check apache2
Checking versions for cookbook apache2...

Local Version:
  2.0.2

Remote Versions: (* indicates frozen)
 *2.0.2
 *2.0.1
  1.1.49
 *1.1.14
 *1.1.13

WARNING: Your local version (2.0.2) is frozen on the remote server. You'll need to bump before you can upload.

$ knife spork check apache2
Checking versions for cookbook apache2...

Local Version:
  2.0.3

Remote Versions: (* indicates frozen)
 *2.0.2
 *2.0.1
  1.1.49
 *1.1.14
 *1.1.13

Everything looks good!

This function lets you easily version your cookbooks without having to manually edit the cookbook's metadata.rb file. You can either specify the version level you'd like to bump (major, minor, or patch), or you can manually specify a version number. This might be used if, for example, you want to jump several version numbers in one go and don't want to have to run knife bump once for each number. If no bump level is specified, a patch level bump will be performed.

knife spork bump COOKBOOK [major | minor | patch | manual x.x.x]

$ knife spork bump apache2
Successfully bumped apache2 to v2.0.4!

$ knife spork bump apache2 patch
Successfully bumped apache2 to v2.0.4!

$ knife spork bump apache2 manual 1.0.13
Successfully bumped apache2 to v1.0.13!

This function works mostly the same as normal knife cookbook upload COOKBOOK except that this automatically freezes cookbooks when you upload them.

knife spork upload COOKBOOK

$ knife spork upload apache2
Freezing apache2 at 1.0.13...
Successfully uploaded apache2@1.0.13!

This function lets you easily set a version constraint in an environment or group of environments for a particular cookbook. By default it will set the version constraint to whatever the local version of the specified cookbook is. Optionally, you can include a --version option which will set the version constraint for the specified cookbook to whatever version number you provide. You might want to use this if, for example, you pushed a version constraint for a cookbook version you don't want your nodes to use anymore, so you want to "roll back" the environment to a previous version. You can also specify the --remote option if you'd like to automatically upload your changed local environment file to the server. To promote to multiple environments, simply pass a list of them delimited with a comma.

If you don't specify an environment or environment group, the default_environments config directive will be used if set.

knife spork promote [ENVIRONMENT OR ENVIRONMENT GROUP NAME] COOKBOOK [--version, --remote]

$ knife spork promote my_environment apache2 --remote
Adding version constraint apache2 = 1.0.13
Saving changes to my_environment.json
Uploading my_environment to Chef Server
Promotion complete!

$ knife spork promote my_environment apache2 -v 2.0.2
Adding version constraint apache2 = 2.0.2
Saving changes to my_environment.json
Promotion complete. Don't forget to upload your changed my_environment to Chef Server

This function works mostly the same as normal knife cookbook delete COOKBOOK including options to delete all versions and delete without interactive confirmation.

knife spork delete COOKBOOK

$ knife spork delete apache2 -a
WARNING: Deleting cookbook apache2...
Do you really want to delete all versions of apache2? (Y/N) Y
Deleted cookbook[apache2][2.0.2]
Successfully deleted cookbook apache2 ALL versions from the Chef server
Cookbooks deleted from Chef server: apache2: ALL versions

Omni lets you combine one of the most common combinations of spork commands (bump, upload & promote or promote --remote) - into one handy shortcut.

As omni is designed for use only in those cases where you want to perform all three of bump, upload and promote at the same time it supports a limited subset of the command line options supported by the individual bump, upload and promote commands.

If you run omni with no extra options, it will default to performing a patch level bump, and promote locally to the environments listed in the default_environments variable in your spork configuration file.

Alternatively, you can specify any of the following options:

--cookbook-path PATH:PATH: A colon-separated path to look for cookbooks in

--include-dependencies: Also upload cookbook dependencies during the upload step

--bump-level [major|minor|patch]: Version level to bump the cookbook (defaults to patch)

--environment ENVIRONMENT: Environment to promote the cookbook to',

--remote: Make omni perform a promote --remote instead of a local promote',

knife spork omni COOKBOOK [--bump-level, --cookbook-path, --include-dependencies, --environment, --remote]

$ knife spork omni apache2
OMNI: Bumping apache2
Successfully bumped apache2 to v0.3.99!

OMNI: Uploading apache2
Freezing apache2 at 0.3.99...
Successfully uploaded apache2@0.3.99!

OMNI: Promoting apache2
Adding version constraint apache2 = 0.3.99
Saving changes to development.json
Promotion complete. Don't forget to upload your changed development.json to Chef Server
Adding version constraint apache2 = 0.3.99
Saving changes to production.json
Promotion complete. Don't forget to upload your changed production.json to Chef Server

$ knife spork omni apache2 --remote
OMNI: Bumping apache2
Successfully bumped apache2 to v0.3.99!

OMNI: Uploading apache2
Freezing apache2 at 0.3.99...
Successfully uploaded apache2@0.3.99!

OMNI: Promoting apache2
Adding version constraint apache2 = 0.3.99
Saving changes to development.json
Uploading development.json to Chef Server
Promotion complete at 2013-08-08 11:43:12 +0100!
Adding version constraint apache2 = 0.3.99
Saving changes to production.json
Uploading production.json to Chef Server
Promotion complete at 2013-08-08 11:43:12 +0100!

$ knife spork omni apache2 -l minor -e development
OMNI: Bumping apache2
Successfully bumped apache2 to v0.4.0!

OMNI: Uploading apache2
Freezing apache2 at 0.4.0...
Successfully uploaded apache2@0.4.0!

OMNI: Promoting apache2
Adding version constraint apache2 = 0.4.0
Saving changes to development.json
Promotion complete. Don't forget to upload your changed development.json to Chef Server

Environment check provides the ability to validate a local chef environment file is locked to versions that actually exist on the chef server. Running this check prior to uploading your environment files is recommended as it can prevent your chef server from becoming unresponsive due to a version constraint that will never be valid.

If you run environment check with no options it will collect all invalid cookbook versions before reporting a failure.

Alternatively, you can specify any of the following options:

-f, --fatal: Quit on first invalid constraint located

knife spork environment check ENVIRONMENT (options)

$ knife spork environment check production

Checking constraints for environment: production
Environment production looks good

$ knife spork environment check production

Checking constraints for environment: production
ERROR: mysql@10.2.0 does not exist on Chef Server! Upload the cookbook first by running:

	knife spork upload mysql

ERROR: rbenv@10.7.1 does not exist on Chef Server! Upload the cookbook first by running:

	knife spork upload rbenv

FATAL: Environment production has constraints that point to non existent cookbook versions.

$ knife spork environment check production

Checking constraints for environment: production
FATAL: mysql@10.2.0 does not exist on Chef Server! Upload the cookbook first by running:

	knife spork upload mysql

Knife spork contains "wrappers" around several stock knife commands for changing nodes, roles, environments and databags. These commands work exactly as in the original knife, with the addition of being wrapped in Spork's plugin API. This means that for example, when you upload a role using spork you will see an IRC message containing details, and a gist of the role change.

The following data bag commands are provided in knife-spork:

knife spork data bag create
knife spork data bag delete
knife spork data bag edit
knife spork data bag from file

The following node commands are provided in knife-spork:

knife spork node create
knife spork node delete
knife spork node edit
knife spork node from file
knife spork node run_list add
knife spork node run_list remove
knife spork node run_list set

The following role commands are provided in knife-spork:

knife spork role create
knife spork role delete
knife spork role edit
knife spork role from file

The following environment commands are provided in knife-spork:

knife spork environment create
knife spork environment delete
knife spork environment edit
knife spork environment from file

If you get an error when running knife spork <command> and the message shown when running with the -VV flag contains:

undefined method `gsub' for #<Pathname:0x00000002d3a6b0> (NoMethodError)

...then you are probably using Librarian::Chef.install_path() in your knife.rb file.

To fix this you need to call .to_s on the install path, i.e. ``Librarian::Chef.install_path().to_s`.