Elasticsearch::Index::Transfer

Ruby gem for transfering elasticsearch index data from one source to another. Currently this gem can transfer elasticsearch index data between

elasticsearch to elasticsearch
elasticsearch to s3(AWS S3)
s3 to elasticsearch

This gem is using scroll API provided by elasticsearch for backing up the elasticsearch index data.

Installation

Add this line to your application's Gemfile:

gem 'elasticsearch-index-transfer'

And then execute:

$ bundle

Or install it yourself as:

$ gem install elasticsearch-index-transfer

Usage

If you are using irb console

require 'elasticsearch-index-transfer'

From one elasticsearch host to another elasticsearch host

options = {
            "source": {
              "elasticsearch": {
                "host": * source-host-ip *,
                "port": * source-host-port *,
                "index": * elasticsearch-index-name *
              }
            },
            "target": {
              "elasticsearch": {
                "host": * target-host-ip *,
                "port": * target-host-port *,
                "index": * elasticsearch-index-name *
              }
            }
          }
Elasticsearch::Index::Transfer.execute(options)

From elasticsearch host to s3(AWS S3)

options = {
            "source": {
              "elasticsearch": {
                "host": * source-host-ip *,
                "port": * source-host-port *,
                "index": * elasticsearch-index-name *
              }
            },
            "target": {
              "s3": {
                "region": * S3-region-name *,
                "access_key_id": * S3-access-key-id *,
                "secret_access_key": * S3-secret-access-key *,
                "bucket": * S3-bucket-name *,
                "prefix": * S3-folder/prefix * # optional 
              }
            }
          }
Elasticsearch::Index::Transfer.execute(options)

From s3(AWS S3) to elasticsearch host

This gem can only transfer data from AWS S3 to elasticsearch host only if backup on S3 is made by this gem only.

options = {
            "source": {
              "s3": {
                "region": * S3-region-name *,
                "access_key_id": * S3-access-key-id *,
                "secret_access_key": * S3-secret-access-key *,
                "bucket": * S3-bucket-name *,
                "prefix": * S3-folder/prefix * # optional 
              }
            },
            "target": {
              "elasticsearch": {
                "host": * targer-host-ip *,
                "port": * target-host-port *,
                "index": * elasticsearch-index-name * # if index name not given it will use index name of backed up index.
              }
            },
            
          }
Elasticsearch::Index::Transfer.execute(options)

Test

rspec spec/elasticsearch-index-transfer.rb

Contribute

Issue Tracker: https://github.com/imran3180/elasticsearch-index-transfer/issues

Pull Request: https://github.com/imran3180/elasticsearch-index-transfer/pulls

License

The gem is available as open source under the terms of the MIT License.