webtranslateit-safe

Simple database and filesystem backups with Amazon S3 support (with optional encryption).

This is a fork of now abandonned astrails-safe that we’ve been using since 2010. It is now unmaintained and not compatible with ruby 3.2, so we forked it.

We’ve added:

Support for ruby 3.2
Standardized code with rubocop
Added support for SCP transfers — On some conditions with servers with higher latency, transfering large files (> 18GB) with SFTP can take a very long time.
Removed svndump feature
Removed FTP transfer feature

Installation

gem install webtranslateit-safe

Reporting problems

Please report problems at the Issues tracker

Usage

Usage:
   webtranslateit-safe [OPTIONS] CONFIG_FILE
Options:
  -h, --help           This help screen
  -v, --verbose        be verbose, duh!
  -n, --dry-run        just pretend, don't do anything.
  -L, --local          skip remote storage, only do local backups

Note: CONFIG_FILE will be created from template if missing

Encryption

If you want to encrypt your backups you have 2 options:

use simple password encryption
use GPG public key encryption

IMPORTANT: some gpg installations automatically set 'use-agent' option in the default configuration file that is created when you run gpg for the first time. This will cause gpg to fail on the 2nd run if you don't have the agent running. The result is that 'webtranslateit-safe' will work ONCE when you manually test it and then fail on any subsequent run. The solution is to remove the 'use-agent' from the config file (usually /root/.gnupg/gpg.conf) To mitigate this problem for the gpg 1.x series '--no-use-agent' option is added by defaults to the autogenerated config file, but for gpg2 is doesn't work. as the manpage says it: "This is dummy option. gpg2 always requires the agent." :(

For simple password, just add password entry in gpg section. For public key encryption you will need to create a public/secret keypair.

We recommend to create your GPG keys only on your local machine and then transfer your public key to the server that will do the backups.

This way the server will only know how to encrypt the backups but only you will be able to decrypt them using the secret key you have locally. Of course you MUST backup your backup encryption key :) We recommend also pringing the hard paper copy of your GPG key 'just in case'.

The procedure to create and transfer the key is as follows:

run gpg --gen-key on your local machine and follow onscreen instructions to create the key (you can accept all the defaults).
extract your public key into a file (assuming you used test@example.com as your key email): gpg -a --export test@example.com > test@example.com.pub
transfer public key to the server scp test@example.com.pub root@example.com:
import public key on the remote system:

$ gpg --import test@example.com.pub
gpg: key 45CA9403: public key "Test Backup <test@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1

since we don't keep the secret part of the key on the remote server, gpg has no way to know its yours and can be trusted. To fix that we can sign it with other trusted key, or just directly modify its trust level in gpg (use level 5):

$ gpg --edit-key test@example.com
...
Command> trust
...
1 = I don't know or won't say
2 = I do NOT trust
3 = I trust marginally
4 = I trust fully
5 = I trust ultimately
m = back to the main menu

Your decision? 5
...
Command> quit

export your secret key for backup (we recommend to print it on paper and burn to a CD/DVD and store in a safe place):

$ gpg -a --export-secret-key test@example.com > test@example.com.key

Example configuration

safe do
  verbose true

  local path: "/backup/:kind/:id"

  s3 do
    key "...................."
    secret "........................................"
    bucket "backup.astrails.com"
    path "servers/alpha/:kind/:id"
  end

  sftp do
    host "sftp.astrails.com"
    user "astrails"
    # port 8023
    password "ssh password for sftp"
    use_scp: true # use SCP if possible
  end

  gpg do
    command "/usr/local/bin/gpg"
    options  "--no-use-agent"
    # symmetric encryption key
    # password "qwe"

    # public GPG key (must be known to GPG, i.e. be on the keyring)
    key "backup@astrails.com"
  end

  keep do
    local 20
    s3 100
    sftp 100
  end

  mysqldump do
    options "-ceKq --single-transaction --create-options"

    user "root"
    password "............"
    socket "/var/run/mysqld/mysqld.sock"

    database :blog
    database :servershape
    database :astrails_com
    database :secret_project_com do
      skip_tables "foo"
      skip_tables ["bar", "baz"]
    end

  end

  pgdump do
    options "-i -x -O"   # -i => ignore version, -x => do not dump privileges (grant/revoke), -O => skip restoration of object ownership in plain text format

    user "username"
    password "............"  # shouldn't be used, instead setup ident.  Current functionality exports a password env to the shell which pg_dump uses - untested!

    database :blog
    database :stateofflux_com
  end

  tar do
    options "-h" # dereference symlinks
    archive "git-repositories", files: "/home/git/repositories"
    archive "dot-configs",      files: "/home/*/.[^.]*"
    archive "etc",              files: "/etc", exclude: "/etc/puppet/other"

    archive "blog-astrails-com" do
      files "/var/www/blog.astrails.com/"
      exclude "/var/www/blog.astrails.com/log"
      exclude "/var/www/blog.astrails.com/tmp"
    end

    archive "astrails-com" do
      files "/var/www/astrails.com/"
      exclude ["/var/www/astrails.com/log", "/var/www/astrails.com/tmp"]
    end
  end
end