Delayed Job Worker Pool

Delayed Job's built-in worker pooling daemonizes all worker processes. This is great for certain environments but not so great for environments like Heroku that really want your processes to run in the foreground. Delayed Job Worker Pool runs a pool of Delayed Job workers without daemonizing them.

Salsify is currently using Delayed Job Worker Pool to run multiple Delayed Job workers on a single Heroku PX dyno. Read more about our experience using this gem on our blog.

This gem only works with MRI on Linux/MacOS X.

Add this line to your application's Gemfile:

gem 'delayed_job_worker_pool'

And then execute:

$ bundle

Or install it yourself as:

$ gem install delayed_job_worker_pool

From your Rails root directory run:

delayed_job_worker_pool <config file>

The config file is a Ruby DSL inspired by the Puma configuration DSL. Here's an example:

worker_group do |g|
  g.workers = Integer(ENV['NUM_WORKERS'] || 1)
  g.queues = (ENV['QUEUES'] || ENV['QUEUE'] || '').split(',')
  g.sleep_delay = ENV['WORKER_SLEEP_DELAY']
end

preload_app

# This runs in the master process after it preloads the app
after_preload_app do
  puts "Master #{Process.pid} preloaded app"
  
  # Don't hang on to database connections from the master after we've 
  # completed initialization
  ActiveRecord::Base.connection_pool.disconnect!
end

# This runs in the worker processes after it has been forked
on_worker_boot do |worker_info|
  puts "Worker #{Process.pid} started"
  
  # Reconnect to the database
  ActiveRecord::Base.establish_connection
end

# This runs in the master process after a worker starts
after_worker_boot do |worker_info|
  puts "Master #{Process.pid} booted worker #{worker_info.name} with " \
        "process id #{worker_info.process_id}"
end

# This runs in the master process after a worker shuts down
after_worker_shutdown do |worker_info|
  puts "Master #{Process.pid} detected dead worker #{worker_info.name} " \
        "with process id #{worker_info.process_id}"
end

You can configure multiple worker groups, i.e.:

worker_group(:default) do |g|
  g.workers = 1
  g.queues = ['default']
end

worker_group(:mails) do |g|
  g.workers = 1
  g.queues = ['mail']
end

Here's more information on each setting:

• worker_group - You need at least one worker group. Group settings can be set as illustrated above. Worker group settings:
  • workers - The number of Delayed Job worker processes to fork. The master process will relaunch workers that fail.
  • Delayed Job worker settings (queues, min_priority, max_priority, sleep_delay, read_ahead) - These are passed through to the Delayed Job worker.
• preload_app - This forces the master process to load Rails before forking worker processes causing the memory consumed by the code to be shared between workers. If you use this setting make sure you re-establish any necessary connections in the on_worker_boot callback.
• after_preload_app - A callback that runs in the master process after preloading the app but before forking any workers.
• on_worker_boot - A callback that runs in the worker process after it has been forked.
• after_worker_boot - A callback that runs in the master process after a worker has been forked.
• after_worker_shutdown - A callback that runs in the master process after a worker has been shutdown.

All settings are optional and nil values are ignored.

• Convert your worker settings to a single worker group (see Usage)
• Please note the delayed job worker names changed to include group: <group_name>, e.g. if you are monitoring them by their name

Bug reports and pull requests are welcome on GitHub at https://github.com/salsify/delayed_job_worker_pool.

The gem is available as open source under the terms of the MIT License.