Etcher

Etcher allows you to take raw settings and/or user input and etch them into a concrete and valid configuration for use within your application. As quoted from Wikipedia, to etch is to:

[Use] strong acid or mordant to cut into the unprotected parts of a metal surface to create a design in intaglio (incised) in the metal.

By using Etcher, you have a reliable way to load default configurations (i.e. Hash, Environment, JSON, YAML) which can be validated and etched into frozen records (i.e. Hash, Data, Struct) for consumption within your application which doesn’t violate the Law of Demeter. This comes complete with loaders, transformations, and validations all via a simple Object API that pairs well with the XDG, Runcom, and Sod gems.

Table of Contents

Features
Requirements
Setup
Usage
- Steps
- Registry
- Contracts
- Types
  - Pathnames
  - Versions
- Models
- Loaders
  - Environment
  - Hash
  - JSON
  - YAML
  - Custom
- Transformers
  - Basename
  - Root
  - Format
  - Time
- Overrides
- Resolver
Development
- Architecture
Tests
License
Security
Code of Conduct
Contributions
Developer Certificate of Origin
Versions
Community
Credits

Features

Supports contracts which respond to #call to validate a Hash before building the final record. Pairs well with the Dry Schema and Dry Validation gems.
Supports models which respond to .[] for consuming a splatted Hash to instantiate new records. Pairs well with primitives such as: Hash, Data, and Struct.
Supports loading of default configurations from a Hash, the Environment, a JSON configuration, a YAML configuration, or anything that can answer a hash.
Supports multiple transformations which can process loaded configuration hashes and answer a transformed hash.
Supports Hash overrides as a final customization which is handy for Command Line Interfaces (CLIs), as aided by Sod, or anything that might require user input at runtime.

Requirements

Ruby.

Setup

To install with security, run:

# 💡 Skip this line if you already have the public certificate installed.
gem cert --add <(curl --compressed --location https://alchemists.io/gems.pem)
gem install etcher --trust-policy HighSecurity

To install without security, run:

gem install etcher

You can also add the gem directly to your project:

bundle add etcher

Once the gem is installed, you only need to require it:

require "etcher"

Usage

Basic usage is to new up an instance:

etcher = Etcher.new
etcher.call({one: 1, two: 2})

# Success({:one=>1, :two=>2})

Notice you get a monad — either a Success or Failure — as provided by the Dry Monads gem. This allows you to create more sophisticated pipelines as found with the Pipeable gem or any kind of failsafe workflow you might need.

By default, any attributes you message the instance with will only pass through what you gave it and always answer a Success. This is nice for initial experimentation but true power comes with full customization of the instance. Here’s an advanced configuration showing all features:

require "dry/monads"
require "dry/schema"

Dry::Schema.load_extensions :monads

contract = Dry::Schema.Params do
  required(:user).filled :string
  required(:home).filled :string
end

model = Data.define :user, :home

transformer = lambda do |attributes, key = :user|
  Dry::Monads::Success attributes.merge! key => attributes[key].upcase
end

Etcher::Registry.new(contract:, model:, transformers: [transformer])
                .add_loader(:environment, %w[USER HOME])
                .then { |registry| Etcher.new(registry).call }

# Success(#<data user="DEMO", home="/Users/demo">)

The above can be broken down into a series of steps:

A Dry Schema contract — loaded with Dry Monads extensions — is created to verify untrusted attributes.
A model is created with attributes: user and home.
A registry is constructed with a custom contract, model, loader, and transformer.
Finally, we see a successfully built configuration for further use within your application.

While this is a more advanced use case, you’ll usually only need to register a contract and model. The loaders and transformers provide additional firepower in situations where you need to do more with your data. We’ll look at each of these components in greater detail next.

ℹ️ All keys are converted to symbols before being processed. This is done to ensure consistency and improve debugablity when dealing with raw input that might be a mix of strings and/or symbols.

Steps

As hinted at above, the complete sequence of steps are performed in the order listed:

Load: Each loader, if any, is called and merged with the previous loader to build initial attributes.
Transform: Each transformer, if any, is called to transform and manipulate the attributes.
Override: Overrides, if any, are merged with the result of the last transformer so you can fine tune the data as desired.
Validate: The contract is called to validate the attributes as previously loaded, overwritten, and transformed.
Model: The model consumes the attributes of the validated contract and creates a new record for you to use as needed.

Each step mutates the attributes of the previous step in order to produce a record (success) or error (failure). You can use the above steps as a reference when using this gem. Each step is explained in greater below.

Registry

The registry provides a way to register any/all behavior for before creating a new Etcher instance. Here’s what you get by default:

Etcher::Registry.new
# #<data Etcher::Registry contract=#<Proc:0x000000010e393550 contract.rb:7 (lambda)>, model=Hash, loaders=[], transformers=[]>

Since the registry is Data, you can initialize with everything you need:

Etcher::Registry[
  contract: MyContract,
  model: MyModel,
  loaders: [MyLoader.new],
  transformers: [MyTransformer]
]

You can also add additional loaders and/or transformers after the fact:

registry = Etcher::Registry.new
                           .add_loader(MyLoader.new)
                           .add_transformer(MyTransformer)

💡 Order matters so ensure you list your loaders and transformers in the order you want them processed.

Contracts

Contracts are a critical piece of this workflow as they provide a way to validate incoming data, remove unwanted data, and create a sanitized record for use in your application. Any contract that has the following behavior will work:

#call: Must be able to consume a Hash and answer an object which can respond to #to_monad.

Both Dry Schema and Dry Validation respond to the #to_monad message. Ensure the Dry Monads extensions are loaded too, as briefly shown earlier, so the result will respond to the #to_monad message. Here’s how to enable monad support if using both gems:

Dry::Schema.load_extensions :monads
Dry::Validation.load_extensions :monads

Using Dry Schema syntax, we could create a contract for verifying email addresses and use it to build a new Etcher instance. Example:

require "dry/schema"

Dry::Schema.load_extensions :monads

contract = Dry::Schema.Params do
  required(:from).filled :string
  required(:to).filled :string
end

etcher = Etcher::Registry[contract:].then { |registry| Etcher.new registry }
etcher.call

# Failure({:step=>:validate, :payload=>{:from=>["is missing"], :to=>["is missing"]}})

etcher.call from: "Mork", to: "Mindy"
# Success({:from=>"Mork", :to=>"Mindy"})

Here you can see the power of using a contract to validate your data both as a failure and a success. Unfortunately, with the success, we only get a Hash as a record but it would be better to have a data structure which will be explained shortly.

Types

To support contracts further, there are a couple custom types which might be of interest. Each custom type, as described below, is made possible via Dry Types.

Pathnames

Etcher::Types::Pathname

The above allows you to use pathname types in your contracts to validate and cast as pathnames:

contract = Dry::Schema.Params do
  required(:path).filled Etcher::Types::Pathname
end

contract.call(path: "a/path").to_monad
# Success(#<Dry::Schema::Result{:path=>#<Pathname:a/path>} errors={} path=[]>)

Versions

Etcher::Types::Version

The above allows you to validate and cast versions within your contracts — via the Versionaire gem — as follows:

contract = Dry::Schema.Params do
  required(:version).filled Etcher::Types::Version
end

contract.call(version: "1.2.3").to_monad
# Success(#<Dry::Schema::Result{:version=>"1.2.3"} errors={} path=[]>)

Models

A model is any object which responds to .[] and can accept a splatted hash. Example: Model[**attributes]. These primitives are excellent choices: Hash, Data, and Struct.

ℹ️ Keep in mind that using a Hash is the default model and will only result in a pass through situation. You’ll want to reach for the more robust Data or Struct objects instead.

The model is used in the last step of the etching process to create a frozen record for further use by your application. Here’s an example where a Data model is used:

model = Data.define :from, :to
etcher = Etcher::Registry[model:].then { |registry| Etcher.new registry }

etcher.call
# Failure({:step=>:model, :payload=>"Missing keywords: :from, :to."})

etcher.call from: "Mork", to: "Mindy"
# Success(#<data Model from="Mork", to="Mindy">)

Notice we get an failure if all attributes are not provided but if we supply the required attributes we get a success.

ℹ️ Keep in mind the default contract is always a pass through so no validation is being done when only using a Hash. Generally you want to supply both a custom contract and model at a minimum.

Loaders

Loaders are a great way to load a default configuration for your application which can be in multiple formats. Loaders can either be defined when creating a new registry instance or added after the fact. Here are a couple examples:

# Initializer
registry = Etcher::Registry[loaders: [MyLoader.new]]

# Method
registry = Etcher::Registry.new.add_loader MyLoader.new

You can also remove a previously added loader by index:

registry = Etcher::Registry.new

# Application
registry.add_loader MyLoader.new

# RSpec
registry.remove_loader 0

The ability to remove a loader is especially handy in a testing environment where you might need to temporarily remove a loader or don’t need a specific loader for testing purposes.

There are a few guidelines to using loaders:

All loaders must respond to #call with no arguments.
All loaders must answer either a success with attributes (i.e. Success attributes) or a failure with details about the failure (i.e. Failure step: :load, constant: MyLoader, payload: "My error message.)
All keys are symbolized after the loader is called which helps streamline merging and overriding values from the same keys across multiple configurations.
All nested keys will be flattened after being loaded. This means a key structure of {demo: {one: "test"}} will be flattened to demo_one: "test" which adheres to the Law of Demeter when a new recored is etched for you.
The order in which you define your loaders matters. This means the first loader defined will be processed first, then the second, and so forth. Loaders defined last take precedence over previously defined loaders when overriding the same keys.

For convenience, all loaders — only packaged with this gem — can be registered by symbol instead of constant/instance. Example:

registry = Etcher::Registry.new

# Environment
registry.add_loader :environment

# JSON
registry.add_loader :json, "path/to/configuration.json"

# YAML
registry.add_loader :yaml, "path/to/configuration.yml"

Any positional or keyword arguments will be passed to the loader’s constructor. This only works when using Registry#add_loader, though.

The next sections will help you learn about the supported loaders and how to build your own custom loader.

Environment

Use :environment or Etcher::Loaders::Environment to load configuration information from your Environment. By default, this object wraps ENV, uses an empty array for included keys, and answers a filtered hash where all keys are downcased. If you don’t specify keys to include, then an empty hash is answered back. Here’s a few examples:

# Default behavior.
loader = Etcher::Loaders::Environment.new
loader.call
# Success({})

# With specific includes.
loader = Etcher::Loaders::Environment.new %w[RACK_ENV DATABASE_URL]
loader.call
# Success({"rack_env" => "test", "database_url" => "postgres://localhost/demo_test"})

# With a custom environment and specific include.
loader = Etcher::Loaders::Environment.new "USER", source: {"USER" => "Jack"}
loader.call
# Success({"user"=>"Jack"})

This loader is great for pulling from environment variables as a fallback configuration for your application.

Hash

Use :hash or Etcher::Loaders::Hash to load in-memory attributes. By default, this loader will answer an empty hash if not supplied with any attributes. Here’s a few examples:

# Default behavior.
loader = Etcher::Loaders::Hash.new
loader.call
# Success({})

# With custom attributes
loader = Etcher::Loaders::Hash.new one: 1, two: 2
loader.call
# Success({:one=>1, :two=>2})

This loader is great for adding custom attributes, overriding/adjusting attributes from a previous loader, or customizing attributes for testing purposes within a test suite.

JSON

Use Etcher::Loaders::JSON to load configuration information from a JSON file. Here’s how to use this loader (using a file that doesn’t exist):

# Default behavior (a custom path is required).
loader = Etcher::Loaders::JSON.new "your/path/to/configuration.json"
loader.call  # Success({})

You can also customize the fallback and logger used. Here are the defaults:

loader = Etcher::Loaders::JSON.new "your/path/to/configuration.json",
                                   fallback: {},
                                   logger: Logger.new(STDOUT)
loader.call  # Success({})

If the file exists with valid content, you’ll get a Hash wrapped as a Success. In situations in which the file doesn’t exist, you’ll get a Success with an empty hash and debug information logged instead. Any failures will be provided with step, constant, and payload details. Example:

Failure step: :load, constant: Etcher::Loaders::JSON, payload: "Danger!"

YAML

Use Etcher::Loaders::YAML to load configuration information from a YAML file. Here’s how to use this loader (using a file that doesn’t exist):

# Default behavior (a custom path is required).
loader = Etcher::Loaders::YAML.new "your/path/to/configuration.yml"
loader.call  # Success({})

You can also customize the fallback and logger used. Here are the defaults:

loader = Etcher::Loaders::YAML.new "your/path/to/configuration.yml",
                                   fallback: {},
                                   logger: Logger.new(STDOUT)
loader.call  # Success({})

Failure step: :load, constant: Etcher::Loaders::YAML, payload: "Danger!"

Custom

You can always create your own loader if you don’t need or want any of the default loaders provided for you. The only requirement is your loader must respond to #call and answer a monad with a Hash for content which means you can use a class, method, lambda, or proc. Here’s an example of creating a custom loader, registering, and using it:

require "dry/monads"

class Demo
  include Dry::Monads[:result]

  def initialize processor: Processor.new
    @processor = processor
  end

  def call
    Success processor.call
  rescue ProcessorError => error
    Failure step: :load, constant: self.class, payload: error.message
  end

  private

  attr_reader :processor
end

registry = Etcher::Registry[loaders: [Demo.new]]

Etcher.new(registry).call

While the above assumes you have some kind of Processor for loading attributes, you can see there is little effort required to implement and customize as desired.

Transformers

Transformers are great for mutating specific keys and values. They give you fine grained customization over your configuration. Transformers can either be defined when creating a new registry instance or added after the fact. Here are a couple examples:

# Initializer
registry = Etcher::Registry[transformers: [MyTransformer]]

# Method
registry = Etcher::Registry.new.add_transformer MyTransformer

You can also remove a previously added transformer by index:

registry = Etcher::Registry.new

# Application
registry.add_transformer MyTransformer

# RSpec
registry.remove_transformer 0

The ability to remove a transformer is especially handy in a testing environment where you might need to temporarily remove a transformer or don’t need a specific transformer for testing purposes.

The guidelines for using transformers are:

They can be initialized with whatever requirements you need.
They must respond to #call which takes a required attributes positional argument and answers a modified version of these attributes (Hash) wrapped as a monad.
They must answer either a success with attributes (i.e. Success attributes) or a failure with details about the failure (i.e. Failure step: :transform, constant: MyTransformer, payload: "My error message.)
When using a proc/lambda, the first, required, parameter should be the attributes parameter followed by a second positional key parameter.
When using a class, the key should be your first positional parameter. Additional parameters can be supplied after if desired.
The attributes passed to your transformer will have symbolized keys so you don’t need to transform them further.

For example, the following capitalizes all values (which may or may not be good depending on your data structure):

require "dry/monads"

Capitalize = -> attributes { Dry::Monads::Success attributes.transform_values!(&:capitalize) }
Capitalize.call(name: "test")

# Success({:name=>"Test"})

The following obtains the current Git user’s email address from the global Git configuration using the Gitt gem:

require "dry/monads"
require "gitt"

class GitEmail
  def initialize key = :author_email, git: Gitt::Repository.new
    @key = key
    @git = git
  end

  def call(attributes) = git.get("user.email").fmap { |value| attributes[key] = value }

  private

  attr_reader :key, :git
end

GitEmail.new.call({})

# Success("demo@alchemists.io")

To use all of the above, you’d only need to register and use them:

registry = Etcher::Registry[transformers: [Capitalize, GitEmail.new]]
etcher = Etcher.new(registry)
etcher.call

For convenience, all transformers — only packaged with this gem — can be registered by symbol instead of constant/instance. Example:

registry = Etcher::Registry.new

# Format
registry.add_transformer :format, :project_uri

# Time
registry.add_transformer :time

Any positional or keyword arguments will be passed to the transformers’s constructor. This only works when using Registry#add_transformer, though. The following sections provide more details on each.

Basename

Use Etcher::Transformers::Basename to dynamically obtain the name of the current directory as a value for a key. This is handy for scripting or CLI purposes when needing to know the name of the current project you are working in. Example:

transformer = Etcher::Transformers::Basename.new :demo
transformer.call({})
# Success({:demo=>"scratch"})

transformer = Etcher::Transformers::Basename.new :demo, fallback: "undefined"
transformer.call({})
# Success({:demo=>"undefined"})

transformer = Etcher::Transformers::Basename.new :demo
transformer.call({demo: "defined"})
# Success({:demo=>"defined"})

Root

Use Etcher::Transformers::Root to dynamically obtain the current path as a value for a key. This is handy for obtaining the absolute path to a new or existing directory. Example:

transformer = Etcher::Transformers::Root.new :demo
transformer.call({})
# Success({:demo=>#<Pathname:/Users/demo/Engineering/OSS/scratch>})

transformer = Etcher::Transformers::Root.new :demo, fallback: "undefined"
transformer.call({})
# Success({:demo=>#<Pathname:/Users/demo/Engineering/undefined>})

transformer = Etcher::Transformers::Root.new :demo
transformer.call({demo: "defined"})
# Success({:demo=>#<Pathname:/Users/demo/Engineering/defined>})

Format

Use Etcher::Transformers::Format to transform any key’s value by using the configuration’s existing attributes to format the value of a specific key using the String Formats Specification. To start, we’ll use the same attributes for all examples:

attributes = {
  organization_uri: "https://acme.io",
  project_name: "test",
  project_uri: "%<organization_uri>s/projects/%<project_name>s"
}

Using the above attributes, you’ll get a Success when all required keys exist:

Etcher::Transformers::Format.new(:project_uri).call attributes
# Success(
  {
    organization_uri: "https://acme.io",
    project_name: "test",
    project_uri: "https://acme.io/projects/test"
  }
)

When some required keys are missing, you’ll get a Failure:

attributes.delete :project_name
Etcher::Transformers::Format.new(:project_uri).call attributes

# Failure(
#   {
#     step: :transform,
#     constant: Etcher::Transformers::Format,
#     payload: "Unable to transform :project_uri, missing specifier: \"<project_name>\"."
#   }
# )

You can partially transform a value using retainers and/or mappings for situations where you need to format a value while preserving and/or remapping string specifiers for delayed formatting. Here’s an example using a retainer which preserves the :project_name.

Etcher::Transformers::Format.new(:project_uri, :project_name).call attributes

# Success(
#  {
#      organization_uri: "https://acme.io",
#      project_name: "test",
#      project_uri: "https://acme.io/projects/%<project_name>s"
#   }
# )

Notice the organization_uri was formatted in the project_uri while the project_name was preserved. This allows you to format the project_name when you can supply the value later. Similarly, you can remap a string specifier. Example:

Etcher::Transformers::Format.new(:project_uri, project_name: "%<id>s").call attributes

# Success(
#  {
#      organization_uri: "https://acme.io",
#      project_name: "test",
#      project_uri: "https://acme.io/projects/%<id>s"
#   }
# )

Notice the organization_uri was formatted in the project_uri (same as before) while the project_name was remapped as %<id>s. As shown mentioned earlier, this allows you to delay supplying the id when you might not have a value for it yet.

You can also, safely, transform a value which doesn’t have string specifiers:

Etcher::Transformers::Format.new(:version).call(version: "1.2.3")
# Success({:version=>"1.2.3"})

Normally, you’d get a "too many arguments for format string" warning but this transformer detects and immediately skips formatting when no string specifiers are detected. This is handy for situations where your configuration supports values which may or may not need formatting.

Time

Use Etcher::Transformers::Time to transform the any key in your configuration when you want to know the current time at which the configuration was loaded. Handy for situations where you need to calculate relative time or format time based on when your configuration was loaded.

You must supply a key and Time.now.utc is the default fallback. You can customize as desired. Example:

transformer = Etcher::Transformers::Time.new :now
transformer.call({})
# Success({:now=>2024-06-15 22:43:29.178488 UTC})

transformer = Etcher::Transformers::Time.new :now, fallback: Time.utc(2000, 1, 1)
transformer.call({})
# Success({:now=>2000-01-01 00:00:00 UTC})

transformer = Etcher::Transformers::Time.new :now
transformer.call({now: Time.utc(2000, 1, 1)})
# Success({:now=>2000-01-01 00:00:00 UTC})

Overrides

Overrides are what you pass to the Etcher instance when called. They allow you to override any values that were loaded and/or transformed. Example:

etcher = Etcher.new

# With symbol keys.
etcher.call name: "test", label: "Test"
# Success({:name=>"test", :label=>"Test"})

# With string keys.
etcher.call "name" => "test", "label" => "Test"
# Success({:name=>"test", :label=>"Test"})

Overrides are applied after any transforms and before validations. They are a nice way to deal with user input during runtime or provide additional attributes not supplied by the loading and/or transforming of your default configuration while ensuring they are validated properly. Any string keys will be transformed to symbol keys to ensure consistency and reduce issues when merged.

Resolver

In situations where you’d like Etcher to handle the complete load, transform, override, validate, and model steps for you, then you can use the resolver. This is provided for use cases where you’d like Etcher to handle everything for you and abort if otherwise. Example:

Etcher.call name: "demo"
# {:name=>"demo"}

When called — and there are no issues — you’ll get the fully formed record as a result (in this case a Hash which is the default model). You’ll never a get a monad when using Etcher.call because this is meant to resolve the monadic pipeline for you. If any failure is encountered, then Etcher will abort with a fatal log message. Here’s a variation of earlier examples which demonstrates fatals:

require "dry/monads"
require "dry/schema"

Dry::Schema.load_extensions :monads

contract = Dry::Schema.Params do
  required(:to).filled :string
  required(:from).filled :string
end

model = Data.define :to, :from
registry = Etcher::Registry.new(contract:, model:)

Etcher.call registry

# 🛑 Etcher validate failure (Etcher::Builder). Unable to load configuration:
#   - to is missing
#   - from is missing

Etcher.call registry, to: "Mindy"

# 🛑 Etcher validate failure (Etcher::Builder). Unable to load configuration:
#   - from is missing

registry = Etcher::Registry.new(model: Data.define(:name, :label))
Etcher.call registry, to: "Mindy"

# 🛑 Etcher model failure (Etcher::Builder). Missing keywords: :name, :label.

💡 When using a custom registry, make sure it’s the first argument. Additional arguments can be supplied afterwards and they can be any number of key/value overrides which is similar to how Etcher.new works.

Development

To contribute, run:

git clone https://github.com/bkuhlmann/etcher
cd etcher
bin/setup

You can also use the IRB console for direct access to all objects:

bin/console

Architecture

The following illustrates the full sequences of events when etching new records:

Tests

To test, run:

bin/rake

Credits

Built with Gemsmith.
Engineered by Brooke Kuhlmann.

Runtime