avrolution

Support for the evolution of Avro schemas stored in a schema registry.

This gem provides utilities to help with the management of Avro JSON schemas in a schema registry. The compatibility of Avro JSON schema files can be checked against a registry. Expected compatibility breaks can also be declared.

Add this gem to your application's Gemfile, typically in a dev/test group:

group :development, :test do
  gem 'avrolution'
end

And then execute:

$ bundle install

Or install it yourself as:

$ gem install avrolution

Within a Rails project, create an avro_compatibility_breaks.txt file at Rails.root by running:

$ rails generate avrolution:install

The gem supports the following configuration:

• root - The directory to search for Avro JSON schemas (.avsc). This is also the default location for the compatibility breaks file. In a Rails application, Avrolution.root defaults to Rails.root.
• compatibility_breaks_file - The path to the compability breaks file. Defaults to #{Avrolution}.root/avro_compatibility_breaks.txt.
• compatibility_schema_registry_url - The URL for the schema registry to use for compatibility checking, or a Proc to determine the value. ENV['COMPATIBILITY_SCHEMA_REGISTRY_URL'] overrides this value if set.
• deployment_schema_registry_url - The URL for the schema registry to use when registering new schema version, or a Proc to determine the value. ENV['DEPLOYMENT_SCHEMA_REGISTRY_URL'] overrides this value if set.
• logger - A logger used by the rake tasks in this gem. This does NOT default to Rails.logger in Rails applications.

There is a rake task to check the compatibility of all Avro JSON schemas under Avrolution.root against a schema registry.

For Rails applications, the avro:check_compatibility task is automatically defined via a Railtie.

This task does not require any arguments. It checks the compatibility of all unregistered Avro JSON schemas found recursively under Avrolution.root against the schema registry ENV['COMPATIBILITY_SCHEMA_REGISTRY_URL'] or Avroluion.compatibility_schema_registry_url.

rake avro:check_compatibility

If a schema is incompatible, then Avrolution.compatibility_breaks_file is also consulted. If the schema is still incompatible with the last registered version then the differences are displayed and the command to add a compatibility break is printed.

For non-Rails projects, tasks can be defined as:

require 'avrolution/rake/check_compatibility_task'
Avrolution::Rake::CheckCompatibilityTask.define

There is a rake task to register new schemas.

For Rails applications, the avro:register_schemas task is automatically defined via a Railtie.

This rake task requires a comma-separated list of files for the schemas to register.

rake avro:register_schemas schemas=/app/avro/schemas/one.avsc,/app/avro/schema/two.avsc

Schemas are registered against the schema registry ENV['DEPLOYMENT_SCHEMA_REGISTRY_URL'] or Avroluion.deployment_schema_registry_url.

The Avrolution.compatibility_breaks_file is consulted prior to registering the schema, and if an entry is found then the specified compatibility settings are used.

For non-Rails projects, tasks can be defined as:

require 'avroluation/rake/register_schemas_task'
Avrolution::Rake::RegisterSchemasTask.define

This rake task allows you to register all schemas discovered under Avrolution.root.

Similarly to the task avro:register_schemas, it will register them against the configured registry. Additionally, this task will be auto included for Rails applications.

rake avro:register_all_schemas

For non-Rails projects, tasks can be defined as:

require 'avroluation/rake/register_all_schemas_task'
Avrolution::Rake::RegisterAllSchemasTask.define

There is a rake task add an entry to the Avrolution.compatibility_breaks_file.

This rake task accepts the following arguments:

• name - The full name of the Avro schema.
• fingerprint - The Resolution fingerprint as a hex string.
• with_compatibility - Optional compatibility level to use for the check and during registration.
• after_compatibility - Optional compatibility level to set after registration.

rake avro:add_compatibility_break name=com.salsify.alerts.example_value \
  fingerprint=36a2035c15c1bbbfe895494697d1f760171d00ab4fd39d0616261bf6854374f9 \
  with_compatibility=BACKWARD after_compatibility=FULL

For Rails applications, the avro:add_compatibility_break task is automatically defined via a Railtie.

For non-Rails projects tasks can be defined as:

require 'avrolution/rake/add_compatibility_break_task'
Avrolution::Rake::AddCompatibilityBreakTask.define

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.

To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Bug reports and pull requests are welcome on GitHub at https://github.com/salsify/avrolution.

The gem is available as open source under the terms of the MIT License.