Knuckles (Because Sonic was Taken)

Knuckles is a performance focused data serialization pipeline. More simply, it tries to serialize models into large JSON payloads as quickly as possible.

• Emphasis on caching as a composable operation
• Reduced object instantiation
• Complete instrumentation of every discrete operation
• Entirely agnostic, can be dropped into any project and integrated over time
• Minimal runtime dependencies
• Explicit serializer view API with as little overhead and no DSL

Knuckles is absolutely faster and has a lower memory overhead than uncached or cached usage of ActiveModelSerializers, and significantly faster than cached use of ActiveModelSerializers with the perforated gem.

Here are performance and memory comparisons for an endpoint that has been cached with Perforated and with Knuckles. All measurments were done with production settings over the local network.

These are measurements for a sizable payload with hundreds of associated records.

Add this line to your application's Gemfile:

gem "knuckles"

There isn't a hard dependency on Oj or Readthis, but you'll find they are drastically more performant.

require "activesupport"
require "oj"
require "readthis"

Knuckles.configure do |config|
  config.cache = Readthis::Cache.new(
    marshal: Oj,
    compress: true,
    driver: :hiredis
  )

  config.keygen = Readthis::Expanders
  config.serializer = Oj
end

With the top level module configured it is simple to jump right into rendering, but we'll look at configuring the pipeline first.

Knuckles renders and serializes data through a series of stages composed into a pipeline. Stages can easily be added or removed to control how data is transformed. Here is a breakdown of the default stages and what their role is within the pipeline.

The fetcher is responsible for bulk retrieval of data from the cache. Fetching is done using a single read_multi operation, which is multiplexed in caches like Redis or MemCached.

pipeline = Knuckles::Pipeline.new

pipeline.call(posts)

Models that couldn't be retrieved from the cache will then be hydrated, a process where the stripped down model that was given for fetching is replaced with a full model with preloaded associations. The behavior of the hydrator stage is entirely controlled by passing a Proc as the hydrate option. If the hydrate proc is omitted hydration will be skipped. Skipping hydration is useful if you want a simplified pipeline where full models and their associations are preloaded before starting serialization.

See Knuckles::Active::Hydrator for an alternative ActiveRecord specific hydrator. If you are using Knuckles within a Rais app, this is probably the hydration stage you want to use.

# Using the standard hydrator
pipeline.call(posts, hydrator: -> (model) { model.fetch })

# Using active hydrator with a relation that has a `prepared` scope
pipeline.call(posts, relation: posts.prepared)

After un-cached models have been hydrated they can be rendered. Rendering is synonymous with converting a model to a hash, like calling as_json on an ActiveRecord model. Knuckles provides a minimal (but fast) view module that can be used with the rendering step. Alternatively, if you're migrating from ActiveModelSerializers you can pass in an AMS class instead.

# Using Knuckles::View
pipeline.call(models, view: PostView)

# Using ActiveModelSerializer
pipeline.call(models, view: PostSerializer)

After un-cached models have been serialized they are ready to be cached for future retrieval. Each fully serialized model is written to the cache in a single write_multi operation if available (using Readthis, for example). Only previously un-cached data will be written to the cache, making the writer a no-op when all of the data was cached initially.

The enhancer modifies rendered data using proc passed through options. The enhancer stage is critical to customizing the final output. For example, if staff should have confidential data that regular users can't see you can enhance the final values. Another use of enhancers is personalizing an otherwise generic response.

# Removing staff only content from the rendered data
pipeline.call(posts,
  scope: current_user,
  enhancer: lambda do |result, options|
    scope = options[:scope]

    unless scope.staff?
      result.delete_if { |key, _| key == "confidential" }
    end

    result
  end
)

The combiner stage merges all of the individually rendered results into a single hash. The output of this stage is a single object, ready to be serialized.

The dumping process combines de-duplication and actual serialization. For every top level key that is an array all of the children will have uniqueness enforced. For example, if you had rendered a collection of posts that shared the same author, you will only have a single author object serialized. Be aware that the uniqueness check relies on the presence of an id key rather than full object comparisons.

Dumping is the final stage of the pipeline. At this point you have a single serialized payload in the format of your choice (JSON by default), ready to send back as a response.

Pipelines stages can be removed, swapped out or otherwise tuned. An array of stages can be passed when building a new pipeline. Here is an example of creating a customized pipeline without any caching, hydration, or enhancing:

Knuckles::Pipeline.new(stages: [
  Knuckles::Stages::Renderer,
  Knuckles::Stages::Combiner,
  Knuckles::Stages::Dumper
])

Or, perhaps you want to use the active hydrator instead:

Knuckles::Pipeline.new(stages: [
  Knuckles::Stages::Fetcher,
  Knuckles::Active::Hydrator,
  Knuckles::Stages::Renderer,
  Knuckles::Stages::Writer,
  Knuckles::Stages::Enhancer,
  Knuckles::Stages::Combiner,
  Knuckles::Stages::Dumper
])

Note that once the pipeline is initialized the stages are frozen to prevent modification.

While you can use Knuckles with other serializers, you can also use the provided view layer. Knuckles views are simple templates that let you build up data and relations. They look like this:

module ScoutView
  extend Knuckles::View

  def self.root
    :scouts
  end

  def self.data(object, options)
    {id: object.id, email: object.email, name: object.name}
  end

  def self.relations(object, options)
    {things: has_many(object.things, ThingView)}
  end
end

See Knuckles::View for more usage details.

One driving factor of Knuckles is that code should be explicit. As a result there isn't a default Railtie that will integrate Knuckles into the ActiveController rendering process for you. Luckily there isn't much to setting up a new pipeline for rendering. Add this to your ApplicationController or an API specific controller:

def knuckles_render(relation, options)
  Knuckles::Pipeline.new.call(relation, options)
end

Now you can easily render responses:

def index
  posts = posts.published.paginate(pagination_params)

  render json: knuckles_render(
    posts.select(:id, :updated_at),
    relation: posts.prepared,
    view: PostView,
    scope: current_user,
  )
end

1. Fork it ( https://github.com/sorentwo/knuckles/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request