SidekiqRunner

This gem allows to run and monitor multiple instances of Sidekiq. Configuration of the individual instances can either be done in code or in a configuration file.

SidekiqRunner uses God for monitoring Sidekiq instances. By default, God keeps all instances running and starts them whenever they fail.

As of today, Sidekiq does not provide any means of starting/stopping the process automatically given a set of configuration options. Often, custom Rake tasks are written that encapsulate the Sidekiq command line parameters. Configuration is either stored directly in such Rake tasks or scripts or is placed in a YAML-formatted configuration file, in Rails environments often called config/sidekiq.yml.

For one of our products, we wanted to be able to both specify configuration options in (versioned) code and in a configuration file that could be customized per installation site. The user should be able to adapt settings such as the path to Sidekiq's logfile, whereas basic options such as the name and weight of the queues to be processed should be set within the code and should not be modifiable by the user.

Additionally, we wanted an easy way to monitor and automatically restart failed Sidekiq instances which is why God was introduced as a lightweight supervisor.

Add SidekiqRunner to your Gemfile:

# Gemfile

gem 'sidekiq-runner'

Add configuration code to some arbitrary file (e.g., lib/sidekiq.rb):

# lib/sidekiq.rb

require 'sidekiq-runner'

SidekiqRunner.configure do |config|
  # See below.
end

Add Rake tasks to your application, making sure the configuration is loaded, too:

# Rakefile

require 'sidekiq-runner/tasks'
require_relative './lib/sidekiq.rb' # Or any other way...

The configuration code can be put into an initializer:

# config/initializer/sidekiq.rb

require 'sidekiq-runner'

SidekiqRunner.configure do |config|
  # See below.
end

SidekiqRunner will automatically load the Rails environment as a prerequisite of its own Rake tasks, so this time there is no need to require the configuration on its own:

# Rakefile

require 'sidekiq-runner/tasks'

Example of a factory configuration of 2 Sidekiq processes:

SidekiqRunner.configure do |config|
  config.config_file = '/some/other/path/sidekiq.yml'

  config.add_instance('bigbang') do |instance|
    instance.verbose = true

    # Add a queue 'sheldon' with weight 2.
    instance.add_queue 'sheldon', 2
    instance.add_queue 'penny', 4

    # Default weight is 1.
    instance.add_queue 'raj'
  end

  config.add_instance('southpark') do |instance|
    instance.concurrency = 30
    instance.add_queue 'cartman'
    instance.logfile = '/path/to/the/log-file.log'

    # Add custom god worker configuration options
    instance.god_config do |w|
      w.restart_if do |restart|
        restart.condition(:my_custom_restart_condition) do |c|
          c.interval = 60
        end
      end
    end
  end
end

SidekiqRunner.configure_god do |god_config|
  god_config.interval = 30
  god_config.maximum_memory_usage = 4000 # 4 GB.
end

Until now, there is only one global SidekiqRunner option which has to be set outside of any instance blocks:

Options for Sidekiq instances may either be set inside an instance block, in which case they apply only to the current Sidekiq instance, or outside of it (calling them on config), in which case they apply to all previously defined Sidekiq instances (see below). Some of them are overwritable by the user-provided configuration file, while others are not. Queues (set with add_queue method) are never overwritable.

God configuration options, also some of them overwritable by the config file. For more information, please see the God documentation.

You may apply instance configuration options to all instances by specifying them after the instances have been defined.

SidekiqRunner.configure do |config|
  config.add_instance('1') do ... end

  config.verbose = true # Applies to '1'

  config.add_instance('2') do ... end

  config.concurrency = 40 # Applies to '1' and '2'
end

Please note that when no instance has been defined, a default instance called sidekiq_default will be created. It is therefore possible to configure SidekiqRunner without defining any instances:

SidekiqRunner.configure do |config|
  config.add_queue 'local'
end

Optionally you can add some callbacks which are executed after various events:

SidekiqRunner.configure do |config|
  config.on_start_success do
    puts 'Yaaay, processes were started successfully.'
  end
  config.on_start_error do ... end
  config.on_stop_success do ... end
  config.on_stop_error do ... end
end

SidekiqRunner.configure_god do |config|
  config.before_start do
    require_relative '../../lib/my_custom_restart_condition'
  end
  config.before_stop do ... end
end

For convenience (and because we found some of our Sidekiq processes gaining a lot of memory weight over time) there's a rbtrace wrapper included. After you set the :rbtrace configuration option to your instance, you'll be able to gather all kinds of process information. For a start, try:

rbtrace -p <PID of sidekiq process> --firehose

Start SidekiqRunner from the root of your application to start all defined Sidekiq instances:

$ [RAILS_ENV=production] bundle exec rake sidekiqrunner:start

$ [RAILS_ENV=production] bundle exec rake sidekiqrunner:restart

$ [RAILS_ENV=production] bundle exec rake sidekiqrunner:stop

The SidekiqRunner gem is licensed under the MIT license. Please see the LICENSE file for more details.