Esse ActiveRecord Plugin

This gem is a esse plugin for the ActiveRecord ORM. It provides a set of methods to simplify implementation of ActiveRecord models as datasource of esse indexes.

Installation

Add this line to your application's Gemfile:

gem 'esse-active_record'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install esse-active_record

Usage

Add the :active_record plugin and configure the collection with the ActiveRecord model you want to use.

class UsersIndex < Esse::Index
  plugin :active_record

  repository :user do
    collection ::User
    document # ...
  end
end

Using multiple repositories is also possible:

class UsersIndex < Esse::Index
  plugin :active_record

  repository :account do
    collection ::Account
    document # ...
  end

  repository :admin do
    collection ::User.where(admin: true)
    document # ...
  end
end

Collection Scope

It's also possible to specify custom scopes to the repository collection to be used to import data to the index:

class UsersIndex < Esse::Index
  plugin :active_record

  repository :user do
    collection ::User do
      scope :active, -> { where(active: true) }
      scope :role, ->(role) { where(role: role) }
    end
    document # ...
  end
end

# Import data using the scopes
#   > UsersIndex.import(context: { active: true, role: 'admin' })
#
# Streaming data using the scopes
#   > UsersIndex.documents(active: true, role: 'admin').first

Collection Batch Context

Assume that you have a collection of orders and you want to also include the customer data that lives in a external system. To avoid making a request for each order, you can use the batch_context to fetch the data in batches and make it available in the document context.

class OrdersIndex < Esse::Index
  plugin :active_record

  repository :order do
    collection ::Order do
      batch_context :customers do |orders, **_existing_context|
        # The return value will be available in the document context
        # { customers: <value returned from this block> }
        ExternalSystem::Customer.find_all_by_ids(orders.map(&:customer_id)).index_by(&:id) # => { 1 => <Customer>, 2 => <Customer> }
      end
    end
    document do |order, customers: {}, **_|
      customer = customers[order.customer_id]
      {
        id: order.id,
        customer: {
          id: customer&.id,
          name: customer&.name
        }
      }
    end
  end
end

For active record associations, you can define the repository collection by eager loading the associations as usual:

class OrdersIndex < Esse::Index
  plugin :active_record

  repository :order do
    collection ::Order.includes(:customer)
    document do |order, **_|
      {
        id: order.id,
        customer: {
          id: order.customer&.id,
          name: order.customer&.name
        }
      }
    end
  end
end

Data Streaming Options

As default the active record support 3 streaming options:

batch_size: the number of documents to be streamed in each batch. Default is 1000;
start: the primary key value to start from, inclusive of the value;
finish: the primary key value to end at, inclusive of the value;

This is useful when you want to import simultaneous data. You can make one process import all records between 1 and 10,000, and another from 10,000 and beyond

UsersIndex.import(context: { start: 1, finish: 10000, batch_size: 500 })

The default valueof batch_size can be also defined in the collection configuration:

class UsersIndex < Esse::Index
  plugin :active_record

  repository :user do
    collection ::User, batch_size: 500
    document # ...
  end
end

Indexing Callbacks

The index_callback callback can be used to automaitcally index or delete documents after commit on create/update/destroy events.

class UsersIndex < Esse::Index
  plugin :active_record

  repository :user, const: true do
    collection ::User
    document # ...
  end

end

class User < ApplicationRecord
  belongs_to :organization

  # Using a index and repository as argument. Note that the index name is used instead of the
  # of the constant name. it's so because index and model depends on each other should result in
  # circular dependencies issues.
  index_callback 'users_index:user'
  # Using a block to direct a different object to be indexed
  index_callback('organizations') { user.organization } # The `_index` suffix and repo name  is optional on the index name
end

Callbacks can also be disabled/enabled globally:

Esse::ActiveRecord::Hooks.disable!
Esse::ActiveRecord::Hooks.enable!
Esse::ActiveRecord::Hooks.without_indexing do
  10.times { User.create! }
end

or by some specific list of index or index's repository

Esse::ActiveRecord::Hooks.disable!(UsersIndex.repo)
Esse::ActiveRecord::Hooks.enable!(UsersIndex.repo)
Esse::ActiveRecord::Hooks.without_indexing(AccountsIndex, UsersIndex.repo) do
  10.times { User.create! }
end

or by the model that the callback is configured

User.without_indexing do
  10.times { User.create! }
end
User.without_indexing(AccountsIndex) do
  10.times { User.create! }
end

Asynchronous Indexing

If you are using a background job processor like Sidekiq or Faktory, you may be interested in indexing documents asynchronously. For this, you can use the esse-async_indexing gem.

Add the esse-async_indexing gem to your Gemfile and require the esse/async_indexing/active_record file in your application initialization. Make sure to setup the gem configurationg according to the esse-async_indexing documentation.

require 'esse/async_indexing/active_record'

Then, you can use the async_index_callback or async_update_lazy_attribute_callback methods to push the indexing job to the background job processor.

class City < ApplicationRecord
  include Esse::ActiveRecord::Model
- include Esse::ActiveRecord::Model
+ include Esse::AsyncIndexing::ActiveRecord::Model

  belongs_to :state, optional: true


  async_indexing_callback('geos_index:city') { id }
- index_callback('geos_index:city') { id }
- update_lazy_attribute_callback('geos_index:state', 'cities_count', if: :state_id?) { state_id }
+ async_index_callback('geos_index:city', service_name: :sidekiq) { id }
+ async_update_lazy_attribute_callback('geos_index:state', 'cities_count', if: :state_id?, service_name: :sidekiq) { state_id }
end

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake none to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/marcosgz/esse-active_record.

License

The gem is available as open source under the terms of the MIT License.