DelayedJobMaster

A simple delayed_job master process to control multiple workers.

• Support concurrent workers with multiprocess and multithread.
• Preload application and fork worker processes on demand.
• Check memory usage after processing a job to kill workers consuming large memory.
• Support signals for restarting master process / reopening log files.
• Check new jobs by polling multiple databases having delayed_jobs table.
• Detect new jobs quickly by using listen/notify feature. (only postgresql)

• ruby 2.7+
• activesupport 6.0+
• delayed_job 4.1
• delayed_job_active_record 4.1 (execlude sqlite)

Add this line to your application's Gemfile:

gem 'delayed_job_master'

And then execute:

$ bundle

Generate bin/delayed_job_master and config/delayed_job_master.rb:

$ rails generate delayed_job_master:config

Edit config/delayed_job_master.rb:

# working directory
working_directory Dir.pwd

# monitor interval for events (in seconds)
monitor_interval 5

# polling interval for new jobs (in seconds)
polling_interval 30

# path to pid file
pid_file "#{Dir.pwd}/tmp/pids/delayed_job_master.pid"

# path to log file
log_file "#{Dir.pwd}/log/delayed_job_master.log"

# log level
log_level :info

# databases for checking new jobs in case multiple databases
# databases [:primary, :secondary]

# worker1
add_worker do |worker|
  # queue name for the worker
  worker.queues %w(queue1)

  # max process count
  worker.max_processes 1

  # max thread count for each worker
  worker.max_threads 1

  # max memory in MB
  worker.max_memory 300

  # configs below are same as delayed_job, see https://github.com/collectiveidea/delayed_job
  # worker.sleep_delay 5
  # worker.read_ahead 5
  # worker.max_attempts 25
  # worker.max_run_time 4.hours
  # worker.min_priority 1
  # worker.max_priority 10
  # worker.destroy_failed_jobs true
end

# worker2
add_worker do |worker|
  worker.queues %w(queue2)
  worker.max_processes 2
  worker.max_threads 2
end

before_fork do |master, worker|
  ActiveRecord::Base.connection.disconnect!
end

after_fork do |master, worker|
  ActiveRecord::Base.establish_connection
end

Listen/notify feature is enabled by default if pg gem is detected. To disable this feature, put config/initializers/delayed_job_master.rb as follows:

DelayedJobMaster.configure do |config|
  config.listener = nil
end

Start master:

$ RAILS_ENV=production bin/delayed_job_master -c config/delayed_job_master.rb -D

Command line options:

• -c, --config: Specify configuration file.
• -D, --daemon: Start master as a daemon.

Stop master immediately, stop workers gracefully:

$ kill -TERM `cat tmp/pids/delayed_job_master.pid`

Stop master and workers gracefully:

$ kill -WINCH `cat tmp/pids/delayed_job_master.pid`

Stop master and workers immediately:

$ kill -QUIT `cat tmp/pids/delayed_job_master.pid`

Reopen log files:

$ kill -USR1 `cat tmp/pids/delayed_job_master.pid`

Restart master immediately, stop workers gracefully:

$ kill -USR2 `cat tmp/pids/delayed_job_master.pid`

Workers handle each signal as follows:

• TERM/WINCH/USR2: Workers stop after finishing current jobs.
• QUIT: Workers are killed immediately.
• USR1: Workers reopen log files.

ps command shows worker status as follows:

$ ps aux
... delayed_job: worker[0] (queue1) [BUSY]  # BUSY process is currently proceeding some jobs

After graceful restart, you may find OLD process.

$ ps aux
... delayed_job: worker[0] (queue1) [BUSY] [OLD]  # OLD process will stop after finishing current jobs.

Bug reports and pull requests are welcome on GitHub at https://github.com/kanety/delayed_job_master.

The gem is available as open source under the terms of the MIT License.