JSON Map Transform

See Documentation

When building data pipelines, it is often useful to extract and transfrom data from an input JSON and output it in a different format. The standard process for doing this in Ruby is to write a series of if-else logic coupled with for-loops. This code ends up being largely redundant, confusing, and difficult to maintain or change. This Gem provides an easy and extensible solution to this problem by allowing you to define your mapping in YAML and apply it to any JSON object in a single line of code.

The general format of the transform mapping looks as follows:

---
conditions:
  condition_name:
    class: (required)
    predicate: (optional)
objects:
- name: (required)
  path: (optional)
  default: (optional)
  attributes: (optional)
  transform: (optional)
  conditions: (optional)
  - name: (required)
    output: (optional)
    field: (optional)

Add the gem to the Gemfile

gem 'json-mapping-transform'

Require the mapping in your code

require 'json_mapping'

Define your schema in a YAML file.

# schema.yaml
---
objects:
  - name: food
    path: "/fruit"
  - name: vehicle
    path: "/car/honda"

Instantiate a new JsonMapping class with the schema path.

schema_path = 'path/to/schema.yaml'

mapping = JsonMapping.new(schema_path)

Call the #apply method on source data.

source_data = {
  'fruit' => 'banana',
  'car' => {
    'honda' => 'Civic'
  }
}

output = mapping.apply(source_data)
# output => {"food"=>"banana", "vehicle"=>"Civic"}

Objects are the output keys of the mapping. JsonMapper#apply will output a single Ruby Hash when applied to an input. The following rules apply to objects:

• Each object has a name that translates to its key in the output JSON
• The path specifies the input key in the source JSON that the object corresponds to
  • Paths are defined from the top level of the JSON: /
  • When * is included in the path, the result will be an array
  • When a path is not found (or not provided), the object evaluates to nil
• Objects can have a default value which is returned if the path evaluates to nil
• Objects can have attributes which are a list of more objects (nested JSON objects)
  • Note: Paths in nested objects are relative to the path of the top-level object

Conditions are if statements performed on an extracted value. They are defined as a hash in the mapping file.

• By default, conditions are evaluated against the object path
  • If field is specified, the condition is evaluated against the path relative to the object path
• If the extracted value satisfies the condition, the output will be set to output (to the extracted value if output is not specified)
• If the extracted value does not satisfy the condition, the output will be set to the object's default
• If the extracted value is nil, conditions are not evaluated
• Conditions are referenced by name in the object definition
• If multiple conditions are defined and satisfied, the output will be an array

There are several built-in condition types which can be used for the class field of the condition.

• InCondition: Check if an object/Array is in/intersects with the predicate, an array
• RegexCondition: Check if a string matches the predicate, a regular expression
• AnyCondition: Check if an object/Array is/contains a truthy value
• LessThanCondition: Check if a Numeric is less than the predicate, another Numeric
• GreaterThanCondition: Check if a Numeric is less than the predicate, another Numeric
• AndCondition: Check if an object satisfies all conditions provided in the predicate
• OrCondition: Check if an object satisfies at least one condition provided in the predicate
• NotCondition: Check if an object does not satisfy the condition provided as the predicate

Developers can create their own custom conditions by extending BaseCondition inside of the Conditions module

• Transforms are arbitrary blocks of code which act on the extracted value for an object
• Transforms are applied after conditions (i.e they will only be applied if at least one condition is satisfied)
• If the extracted value is nil (or all conditions fail), then transforms are not evaluated
• Transforms are referenced by name in YAML. You must pass in a hash of them to the JsonMapper during initialization

The mapping will gracefully fail (fall back on default) when

1. Encountering a null object in the original object
2. Encountering non-existent paths
3. Indexing an array out of bounds

The mapping will raise an exception when

1. The YAML map is not formatted properly (JsonMapper::FormatError)
2. A condition is referenced but not defined (Conditions::ConditionError)
3. Unknown condition type (NameError)
4. A condition is defined with an incorrect predicate (Conditions::ConditionError)
5. A condition is given a value it can't compare to the predicate (Conditions::ConditionError)
6. A provided transform is not callable (JsonMapper::TransformError)
7. The * operator is used on a non-array (JsonMapper::PathError)
8. An exception is encountered while applying a transform (StandardError)

For all the examples provided below, this is the input JSON that is being mapped:

{
  "name": "Trader Joe's",
  "location": "Berkeley, California",
  "weeklyVisitors": 5000,
  "storeId": 1234,
  "employees": [
    { "name": "Jim Shoes" },
    { "name": "Kay Oss" }
  ],
  "inventory": [
    { "itemName": "Apples", "price": 0.5, "unit": "lb" },
    { "itemName": "Oranges", "price": 2, "unit": "lb" },
    { "itemName": "Bag of Carrots", "price": 1.5, "unit": "count" }
  ]
}

An simple example which just converts between two objects

---
objects:
- name: name
  path: "/name"
- name: profits
  default: 0
- name: location
  path: "/location"
- name: weekly_visitors
  path: "/weeklyVisitors"
- name: store_id
  path: "/storeId"
- name: employees
  path: "/employees/*/name"
- name: inventory
  path: "/inventory/*"
  attributes:
  - name: item_name
    path: /itemName
  - name: price
    path: /price
  - name: unit
    path: /unit

{
  "name": "Trader Joe\'s",
  "profits": 0,
  "location": "Berkeley, California",
  "weekly_visitors": 5000,
  "store_id": 1234,
  "employees": ["Jim Shoes", "Kay Oss"],
  "inventory": [
    { "item_name": "Apples", "price": 0.5, "unit": "lb" },
    { "item_name": "Oranges", "price": 2, "unit": "lb" },
    { "item_name": "Bag of Carrots", "price": 1.5, "unit": "count" }
  ]
}

An example of a custom transformation

---
objects:
- name: name
  path: "/name"
- name: inventory
  path: "/inventory/*/"
  transform: listing_transform

transforms = {
  'listing_transform' => ->(list) { list.map { |x| "#{x['itemName']} at $#{x['price']}/#{x['unit']}" } }
}

output = JsonMapping.new(schema_path, transforms).apply(store_fixture)

{
  "name": "Trader Joe\'s",
  "inventory": ["Apples at $0.5/lb", "Oranges at $2/lb", "Bag of Carrots at $1.5/count"]
}

An example using conditions

---
conditions:
  apple_condition:
    class: AppleCondition
  high_performance_condition:
    class: AndCondition
    predicate:
    - class: LessThanCondition
      predicate: 10000
    - class: GreaterThanCondition
      predicate: 1000
  
objects:
- name: performance
  path: "/weeklyVisitors"
  conditions:
  - name: high_performance_condition
    output: high
- name: apple
  path: "/inventory"
  conditions:
  - name: apple_condition

module Conditions
  class AppleCondition < BaseCondition
    def apply(value)
      puts value
      value.is_a?(Hash) && value['itemName'] == 'Apples'
    end
  end
end

output = JsonMapping.new(schema_path).apply(store_fixture)

{
  "performance": "high",
  "apple": [{ "itemName": "Apples", "price": 0.5, "unit": "lb" }]
}