elastic-backup-2s3 – Snapshot, Backup, and Restore Elasicsearch Indicies to and from S3

When you have an itch to scratch…

I simply want to be able to control the backup and restoration of the Elasticsearch cluster to S3 without any fuss or having to dilly around with curl -XPUTS and friends.

I have finally added Shared Filesystem Support! Please see the docs below on this.

Overview

Elastic Backup simply allows you to specifiy some or all of the indicies, some target bucket, and run the backup / snapshot.

It hides all the sordid details about “registration” and the like, and will optionally clean up behind itself and remove the registration if you did not specify a name for the registration.

Breaking Changes

v1.0.0

The New 1.x version is not backards-compatable with the 0.x versions. Especially for the File System-style backups, the repo name is not appended to the directory path, to allow for multiple repo backups on the file system. Please be aware of this moving forward.

Installation

To install from command line:

gem install elastic-backup-2s3

For a Gemfile entry:

gem  'elastic-backup-2s3'

Documentation

Documentation is comming. Basically, just running the command without any parameters will give you some basic usage.

S3URL format

We adopt a convention to make it easy for you to specify the S3 bucket, path, and snapshot name stored in the bucket. The S3 URL format is:

s3://AWS_S3_BUCKET/path/in/bucket:SNAPSHOT

In some cases, :SNAPSHOT is optional, but usually expected to be there.

Support for Shared Filesystem Snapshots

To do S3 backups requires no additional configurations outside of being in the AWS clould (or at least being connected to one.) Shared File System (FS) backups require that you have each and every node configured to permit the backup path on all your nodes in the cluster, and thus not as simple as merely writing it through the JSON REST interface.

So presumably you either have this setup in place, and in any rate, it would be a bit messy to have to specify the “local” path in the repo here.

For this reason we will not (over)write the repo settings for FS backups by default. We provide a new option that will allow you to write out the FS repo, specifying all the parameters you need to at that time.

The default repository name should be configurable in any case. we could do a configuration file, but for this I think it’s overkill. so we will use environment variables instead.

Environment variable	Description
ESB_DEFAULT_REPO	Changes the default repository from ‘elastic-backup’ to what is specified here.
ESB_SHARED_VOLUME	For Shared Volume backups, default local path to use. Note that this path must be configured on all nodes.
ESB_TIMEOUT	Default for –timeout option
ESB_SNAPSHOT_MAX_BYTES_SEC	Shared Volume max_snapshot_bytes_per_sec
ESB_RESTORE_MAX_BYTES_SEC	Shared Volume max_restore_bytes_per_sec

Examples

To backup/snapshot your ES to S3 and wait until the backup is completed:

es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w

The default repository of ‘elastic-backup’ is used. To specify your own repository name:

es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w --repo your-repo-name

To backup/snapshot two indices index1 and index2:

es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w --indices index1 index2

To restore your snapshot

es-snapshot restore s3://your-backup-bucket/your-backup-path:snap_1 localhost -v2 -w

To restore your snapshot using your-repo-name:

es-snapshot restore s3://your-backup-bucket/your-backup-path:snap_1 localhost -v2 -w --repo your-repo-name

You may specify –dryrun (-u) to see the JSON that will be sent without actually executing the commands. Also -v2 will show you the same while allowing your commands to execute.

To see the status of your snapshots:

es-snapshot list status localhost

To see a list of your indices:

es-snapshot list indices localhost

Release Notes

Date	Version	Description
2016-07-21	v1.0.0	Breaking changes in how the subdirectory paths are created. Please see Breaking Changes
2016-07-12	v0.1.5	Fixed single index restore bug.
2016-06-10	v0.1.3	Added a new –nv flag to switch off shared FS verification.
2016-05-02	v0.1.2	Fixed a couple of minor issues with the restore from shared volume.
2016-04-22	v0.1.0	Support for FS volume backups added
2016-04-17	v0.0.3	Added a delete snapshot option to allow deletion of a running snapshot.

Known Issues

Date	Version	Description
2061-04-17	v0.0.3	‘delete’ options not fully implemented.
		Error return codes not fully fleshed out
		‘delete snapshot’ requires the full S3URL, but only needs the snapshot part.
2061-04-21	v0.0.4	Some features remain unimplemented, though there are options for them. Please bear with me.
		What shall we do with configurations? I am thinking we save them in ES, but also allow for config files. The ES settings will override the config file settings. Config files in Yaml format.

Scratchpad

My own personal notes on this project. These are not gauranteed to be useful to you nor up-to-date.

Contributing to elastic-backup-2s3

Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet.
Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it.
Fork the project.
Start a feature/bugfix branch.
Commit and push until you are happy with your contribution.
Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.
Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.