AmsLazyRelationships

Eliminates N+1 queries problem in Active Model Serializers gem thanks to batch loading provided by a great BatchLoader gem.

The gem provides a module which defines a set of methods useful for eliminating N+1 query problem during the serialization. Serializers will first prepare a tree of "promises" for every nested lazy relationship. The relationship promises will be evaluated only when they're requested. E.g. when including blog_posts.user: instead of loading a user for each blog post separately it'll gather the blog posts and load all their users at once when including the users in the response.

In many cases it's fine to use includes method provided by Rails. There are a few problems with includes approach though:

• It loads all the records provided in the arguments hash. Often you may not need all the nested records to serialize the data you want. AmsLazyRelationships will load only the data you need thanks to lazy evaluation.
• When the app gets bigger and bigger you'd need to update all the includes statements across your app to prevent the N+1 queries problem which quickly becomes impossible.
• It lets you remove N+1s even when not all relationships are ActiveRecord models (e.g. some records are stored in a MySQL DB and other models are stored in Cassandra)

1. Add this line to your application's Gemfile:

gem "ams_lazy_relationships"

2. Execute:

$ bundle

3. Include AmsLazyRelationships::Core module in your base serializer

class BaseSerializer < ActiveModel::Serializer
  include AmsLazyRelationships::Core
end

4. Important: This gem uses BatchLoader heavily. I highly recommend to clear the batch loader's cache between HTTP requests. To do so add a following middleware: config.middleware.use BatchLoader::Middleware to your app's application.rb.

For more info about the middleware check out BatchLoader gem docs: https://github.com/exAspArk/batch-loader#caching

Adding the AmsLazyRelationships::Core module lets you define lazy relationships in your serializers:

class UserSerializer < BaseSerializer
  # Short version - preloads a specified ActiveRecord relationship by default
  lazy_has_many :blog_posts
  
  # Works same as the previous one, but the loader option is specified explicitly
  lazy_has_many :blog_posts,
                serializer: BlogPostSerializer,
                loader: AmsLazyRelationships::Loaders::Association.new("User", :blog_posts)
  
  # The previous one is a shorthand for the following lines:
  lazy_relationship :blog_posts, loader: AmsLazyRelationships::Loaders::Association.new("User", :blog_posts)
  has_many :blog_posts, serializer: BlogPostSerializer do |serializer|
    # non-proc custom finder will work as well, but it can produce redundant sql
    # queries, please see [Example 2: Modifying the relationship before rendering](#example-2-modifying-the-relationship-before-rendering)
    -> { serializer.lazy_blog_posts }
  end
   
  lazy_has_one :poro_model, loader: AmsLazyRelationships::Loaders::Direct.new(:poro_model) { |object| PoroModel.new(object) }
  
  lazy_belongs_to :account, loader: AmsLazyRelationships::Loaders::SimpleBelongsTo.new("Account")
  
  lazy_has_many :comment, loader: AmsLazyRelationships::Loaders::SimpleHasMany.new("Comment", foreign_key: :user_id)
end

As you may have already noticed the gem makes use of various loader classes.

I've implemented the following ones for you:

• AmsLazyRelationships::Loaders::Association - Batch loads a ActiveRecord association (has_one/has_many/has_many-through/belongs_to). This is a deafult loader in case you don't specify a loader option in your serializer's lazy relationship. E.g. in order to lazy load user's blog posts use a following loader: AmsLazyRelationships::Loaders::Association.new("User", :blog_posts).

• AmsLazyRelationships::Loaders::SimpleBelongsTo - Batch loads ActiveRecord models using a foreign key method called on a serialized object. E.g. AmsLazyRelationships::Loaders::SimpleBelongsTo.new("Account") called on users will gather their account_ids and fire one query to get all accounts at once instead of loading an account per user separately. This loader can be useful e.g. when the serialized object is not an ActiveRecord model.

• AmsLazyRelationships::Loaders::SimpleHasMany - Batch loads ActiveRecord records belonging to given record by foreign key. E.g. AmsLazyRelationships::Loaders::SimpleHasMany.new("BlogPosts", foreign_key: :user_id) called on users will and fire one query to gather all blog posts for the users at once instead of loading an the blog posts per user separately. This loader can be useful e.g. when the serialized object is not an ActiveRecord model.

• AmsLazyRelationships::Loaders::Direct - Lazy loads data in a "dumb" way - just executes the provided block when needed. Useful e.g. when the relationship is just a PORO which then in its own serializer needs to lazy load some relationships. You can use it like this: AmsLazyRelationships::Loaders::Direct.new(:poro_model) { |object| PoroModel.new(object).

The abovementioned loaders are mostly useful when using ActiveRecord, but there should be no problem building a new loader for different frameworks. If you're missing a loader you can create an issue or create your own loader taking the existing ones as an example.

Here are a few use cases for the lazy relationships. Hopefully they'll let you understand a bit more how the gem works.

If the relationships in your serializers are plain old ActiveRecord relationships you're lucky, because ams_lazy_relationships by default assumes that the relationship is an ActiveRecord relationship, so you can use the simplest syntax. Imagine you have an endpoint that renders a list of blog posts and includes their comments. The N+1 prone way of defining the serializer would be:

class BlogPostSerializer < BaseSerializer
  has_many :comments
end

To prevent loading comments using a separate DB query for each post just change it to:

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments
end

Sometimes it may happen that you need to process the relationship before rendering, e.g. decorate the records. In this case the gem provides a special method (in our case lazy_comments) for each defined relationship. Check out the example - we'll decorate every comment before serializing:

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments do |serializer|
    -> { serializer.lazy_comments.map(&:decorate) }
   end
end

Despite the fact that non-block custom finder such as

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments do |serializer|
    serializer.lazy_comments.map(&:decorate)
   end
end

will work still, it's better to implement it in a form of lambda, in order to avoid redundant SQL queries when include_data AMS setting appears to be false:

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments do |serializer|
    include_data :if_sideloaded
    -> { serializer.lazy_comments.map(&:decorate) }
   end
end

Feel free to skip custom lazy finder for association if your goal is just to define include_data setting and/or to specify some links and metas:

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments do
    include_data :if_sideloaded
    link :self, 'a link'
    meta name: 'Dan Brown'
   end
end

Under the hood ams_lazy_relationships uses special loader classes to batch load the relationships. By default the gem uses serializer class names and relationship names to instantiate correct loaders, but it may happen that e.g. your serializer's class name doesn't match the model name (e.g. your model's name is BlogPost but the serializer's name is PostSerializer).

In this case you can define the lazy relationship by passing a correct loader param:

class PostSerializer < BaseSerializer
  lazy_has_many :comments, serializer: CommentSerializer,
    loader: AmsLazyRelationships::Loaders::Association.new(
              "BlogPost", :comments
            )
end

This one is interesting. It may happen that the root record is not an ActiveRecord model (e.g. a Cequel model), however its relationship is an AR model. Imagine that BlogPost is not an AR model and Comment is a standard AR model. The lazy relationship would look like this:

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments, 
    loader: AmsLazyRelationships::Loaders::SimpleHasMany.new(
      "Comment", foreign_key: :blog_post_id
    )
end

Sometimes you may just want to make use of lazy relationship without rendering the whole nested record.  For example imagine that your BlogPost serializer is supposed to render author_name attribute. You can define the lazy relationship and just use it in other attribute evaluator:

class BlogPostSerializer < BaseSerializer
  lazy_relationship :author
  
  attribute :author_name do
    lazy_author.name
  end
end

In additional to previous example you may want to make use of nested lazy relationship without rendering of any nested record. There is an lazy_dig method to be used for that:

class AuthorSerializer < BaseSerializer
  lazy_relationship :address
end

class BlogPostSerializer < BaseSerializer
  lazy_relationship :author

  attribute :author_address do
    lazy_dig(:author, :address)&.full_address
  end
end

In general the bigger and more complex your serialized records hierarchy is and the more latency you have in your DB the more you'll benefit from using this gem. Example results for average size records tree (10 blog posts -> 10 comments each -> 1 user per comment, performed on local in-memory SQLite DB) are:

# With lazy relationships:    0.860000   0.010000   0.870000 (  0.870297)
# Vanilla AMS:                1.050000   0.000000   1.050000 (  1.059801)

This means your serializers should get ~13% speed boost by introducing lazy relationships.

# With lazy relationships:
#                         46.283M memsize (     0.000  retained)
#                        506.696k objects (     0.000  retained)
#                         50.000  strings (     0.000  retained)
# Vanilla AMS:            42.738M memsize (     0.000  retained)
#                        545.266k objects (     0.000  retained)
#                         50.000  strings (     0.000  retained)

This means that serialization may consume ~5% more memory.

Detailed benchmark script & results can be found here.

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Bug reports and pull requests are welcome on GitHub at https://github.com/Bajena/ams_lazy_relationships.

The gem is available as open source under the terms of the MIT License.