elastics

Simple ElasticSearch client. Everything for deployment & maintaince included.

• Basic API only
• Transparent aliases management & zero-downtime migrations
• Capistrano integration
• Auto refresh in tests
• Instrumentation

Fast and thread-safe httpclient is under the hood.

# Gemfile
gem 'elastics', '~> 0.3' # use version from the badge above
# or
gem 'elastics', github: 'printercu/elastics-rb'

# initialize client with
client = Elastics::Client.new(options)
# options is hash with
#   :host   - hostname with port or array with hosts (default 127.0.0.1:9200)
#   :index  - (default index)
#   :type   - (default type)

# basic request
client.request(params)
# params is hash with
#   :method - default :get
#   :body   - post body
#   :query  - query string params
#   :index, :type, :id - query path params to override defaults

# method shortcuts for #put, #post #delete
client.delete(params)

# getter/setter shortcuts
client.set(id, data)
client.get(id)
client.get(params) # as usual

# other shortcuts (set method & id)
client.put_mapping(index: index, type: type, body: mapping)
client.search(params)
client.index(params) # PUT if :id is set, otherwise POST

# utils
client.index_exists?(name)

# bulk
client.bulk(params) do |bulk|
  # if first param is not a Hash it's converted to {_id: param}
  bulk.index override_params, data
  bulk.create id, data
  bulk.update id, script
  bulk.update_doc id, fields
  bulk.delete id
end

Most of ActiveRecord integration functionality is available for plain ruby.

class User
  include Elastics::Model

  # set connection config, check 'configure section' for available options
  self.elastics_config = {host: 'hostname:port'}
  # set index base and version manager will manage aliases
  self.elastics_index_base = 'user'
  self.elastics_type_name = 'user'
end

Check out available HelperMethods.

class User < ActiveRecord::Base
  indexed_with_elastics
  # it'll set after_commit callbacks and add helper methods
  # optionally pass :index, :type

  # optionally override to export only selected fields
  def to_elastics
    serializable_hash(only: [:id, :first_name, :last_name])
  end
end

User.elastics # Elastics::Client instance
User.elastics_params # hash with index & type values for the model
User.request_elastics(params) # performs request merging params with elastics_params

search = User.search_elastics(data)
# Returns Elastics::ActiveRecord::SearchResult object with some useful methods
search.relation # User.where(id: found_ids)
search.collection # Array of users sorted as in elastics response

# Set scope applied to relation:
search = Project.search_elastics(data, scope: ->(s) { s.includes(:user) })
search.collection.first.user # eagger-loaded

# Indexing on create/update can be skipped with skip_elastics
User.skip_elastics { users.each { |x| x.update_attributes(smth: 'not indexed') } }
user.skip_elastics { user.update_attributes(smth: 'not indexed') }

Check out Model::HelperMethods and AR::HelperMethods for more information.

# database.yml
development:
  elastics:
    # use single index (app_dev/users, app_dev/documents)
    index: app_dev

    # use index per type (app_dev_users/users, app_dev_documents/documents)
    index_prefix: app_dev_

production:
  elastics:
    host: 10.0.0.1:1234
    # or
    host:
      - 10.0.0.1:1234
      - 10.0.0.2:1234

    index: app
    # or
    index_prefix: app_

YAML is passed througn ERB before parsing.

$ rake elastics:migrate elastics:reindex

Mappings & index settings .yml files are placed in db/elastics/mappings & db/elastics/indices. For now this files are not related to models and only used by rake tasks.

When index is created elastics transparently manages aliases for it. Instead of creating index1 it creates index1-v0 and create index1 alias for it. When you perform normal migration, mappings are applied to the current version. Later when you perform full migration index1-v1 is created, after reindexing aliases are changed and index-v0 is droped.

Versions of indices are stored in ElasticSearch in .elastics index.

All rake tasks except purge accepts list of indices to process (rake elastics:create[index1,index2]). Also you can specify index version like this rake elastics:migrate version=next. Version can be set to next or current (default).

Rake tasks are just frontend for Elastics::Tasks's methods. For complex migrations, when you need partially reindex data, you may want to write custom scripts using this methods.

• rake elastics:create (.create_indices) creates index with settings for each file from indices folder.

• rake elastics:migrate (.migrate) puts mappings from mappings folder.

• rake elastics:migrate! (.migrate!) performs full migration.

• rake elastics:reindex (.reindex) reindexes data.

You need to setup Elastics::Tasks yourself. This can be done in environment or db:load_config rake tasks.

task :environment do
  Elastics::Tasks.base_paths = '/path/to/your/elastics/folder'
  Elastics::Tasks.config = your_configuration
end

Also you need to install active_support to be able to run tasks.

Add Elastics::AutoRefresh.enable! to your test helper, this will run POST /:index/_refresh request after each modifying request. You can also use it for a block or skip auto refresh after it was enabled:

# enable test mode in rspec's around filter
around { |ex| Elastics::AutoRefresh.enable! { ex.run } }

# disable auto refresh for block & perform single refresh
# assume test mode is enabled here
Elastics::AutoRefresh.disable! { Model.reindex_elastics }
Model.refresh_elastics

Instrumentation works out of box in rails. For pure ruby there is only ActiveSupport version (remember to install gem).

# Activate instrumentation (no need for rails)
Elastics::Instrumentation::ActiveSupport.install

# Activate body prettifier (off by default)
Elastics::Instrumentation.body_prettifier = true
# can be
#   true - JSON.pretty_generate
#   :ap  - awesome_print
#   :pp  - pretty_print
#   Proc - ->(str) { your_prettifier(str) }

When you pass array in :host option to initializer, client will work in cluster mode. There are some options for this mode:

• :connect_timeout - timeout to mark the host as dead in cluster-mode (default 10)
• :resurrect_timeout - timeout to mark dead host as alive in cluster-mode (default 60)
• :discover - enable nodes discovering

In plain ruby you should also install thread_safe gem.

Client will perform requests to discover nodes with enabled http module. After discovering hosts will be overwritten with discovered ones.

discover: true will keep you from editing config too frequently, but keep in mind that you should set enough hosts explicitly, so that discover will be able to continue if some of hosts are down. Also this is performed automaticaly only once, when client is initialized. It will not track nodes that go online after client was instantiated. Anyway you still can call .discover_cluster whenever you want, or just restart app when you add more nodes.

Elastics designed to be thread-safe. It should be ok to have single client instance for the whole application.

Add following lines to your deploy.rb and all rake tasks will be available in cap.

role :elastics, '%HOSTNAME%', primary: true

require 'elastics/capistrano'

Indices & rake options can be passed like this:

cap --dry-run elastics:migrate INDICES=index1,index2 ES_OPTIONS='full=true no_drop=true'

elastics for node.js

MIT