Resque Director¶ ↑

Resque Director is a plugin for the Resque queueing system (github.com/defunkt/resque) that “directs” workers on a queue by automatically adding or removing workers from a queue based on how backed up a queue becomes, or how long it takes for a job to get pulled off the queue.

Resque-director is useful for when you are managing a large number of workers and don’t want to waste resources keeping all of them waiting when they are not being used. For jobs that are high priority or time sensitive you can have it autoscale workers based on how long it takes to go through the queue, or how big the queue becomes. In queues where the influx of jobs can change dramatically from time to time, resque director automatically scales more workers during the times when the queue is filling up more quickly, and less in the opposite scenario. Different queues can be given different directions as well.

When creating your jobs you should extend Resque::Plugins::Director and add direction options.

For Example:

class Job
  extend Resque::Plugins::Director
  direct :min_workers => 2, :max_workers => 4, :max_time => 60, :max_queue => 10, :wait_time => 30
  @queue = :test

  #rest of your Job class here
end

min_workers

specifies the minimum number of workers running at any point in time. If there are no workers running or less than the minimum running it will start as many workers necessary to get it to the minimum. The default is 1.

max_workers

specifies the maximum number of workers running at any point in time. It will never start more than the maximum number of workers. If anything less than or equal to zero is specified as the maximum it will be treated as if there is no maximum, and theoretically an infinite number of workers could be started. The default is 0.

max_time

the maximum time in seconds that a job takes to go through the queue, if it takes longer than this time then a worker is started. If anything less than or equal to zero is specified as the maximum time, this field will be ignored. The default is 0.

max_queue

the maximum jobs that can build up in a queue, if more than this number of jobs build up then a worker is started. If anything less than or equal to zero is specified as the maximum queue, this field will be ignored. The default is 0.

wait_time

the time in seconds that it will wait after adding or removing a worker before being allowed to add or remove workers again. The default is 60 seconds.

A worker will be started if the queue length is greater than max_queue or if the time it takes a job to go through the queue is greater than max_time. Also a worker will only be started if the time since the last scaling is greater than wait_time. Workers will not be started if there are already the max_workers number of workers. By default resque-director allows you to have zero workers running, when a job is enqueued then workers will be scaled within the max/min requirements you set. If there is not a single worker running, then the minimum number of workers will be scaled up (one worker will be scaled up if the minimum is zero).

A worker will be removed if the jobs in the queue fall below half of the max_queue, and if the time it takes for a job to be pulled off of a queue falls below half of the max_time. Workers will be scaled down to the minimum if there are no jobs on the queue. Workers will not be stopped if there are only min_workers number of workers. Also a worker will only be stopped if the time since the last scaling is greater than wait_time.

• If a max_worker is less than min_worker then the default for max_worker will be used (there will be no maximum).

• If a min_workers is set to anything less than 1 then it will be treated as 0.

The above options are plenty to handle the basic scaling of workers. The options below allow the user much more control over the way in which workers are scaled, the information tracked, and the times when a worker is scaled.

start_override

You can set this option if you want to override the way to start a worker. This option takes a lambda closure that accepts the queue as an argument, the block you pass in will be responsible for starting a SINGLE worker allowing you to fully customize the starting of a worker. If your block returns false then that signifies that a worker was not started and the last scaled time will not be set. The default way a worker is started is with the system command: “QUEUE=queue_name rake resque:work &” where “queue_name” is the queue on which the job is running.

stop_override

You can set this option if you want to override the way to stop a worker. This option takes a lambda closure that accepts the queue as an argument, the block you pass in will be responsible for stopping a SINGLE worker allowing you to fully customize the stopping of a worker. If your block returns false then that signifies that a worker was not stopped and the last scaled time will not be set. The default way a worker is stopped is that a QUIT signal is sent to a worker process.

class Job
  extend Resque::Plugins::Director

  start_block = lambda{|queue| ... }
  stop_block = lambda{|queue| ... }
  direct :start_override => start_block, :stop_override => stop_block
  @queue = :test

  #rest of your Job class here
end

To add director logging you can pass a logger as an option. It will prepend all log messages with “DIRECTORS LOG:” and will log when scaling up or down a worker. It will also log when a worker wants to scale up or down but is unable to do so due to the fact that the maximum or minimum number of workers has been reached for that queue.

class Job
  logger = Logger.new('logfile.log')
  extend Resque::Plugins::Director

  direct :logger => logger, :log_level => :info
  @queue = :test

  #rest of your Job class here
end

logger

This will set the logger that will be used. It takes a Logger from the Ruby Standard Library. If this is not set the director will not write to any logs.

log_level

This sets the level to log at as a symbol. The default level is :debug.

no_enqueue_scale

When a job is enqueued workers will be scaled within the max/min requirements you set, if there is not a single worker running then the minimum number of workers will be scaled up (one worker will be scaled up if the minimum is zero). The worker itself handles the rest of the scaling. If you do not want scaling on enqueue and want to have the workers perform all the scaling then you can set this option to true, however if you do this you must be responsible for making sure that at least one worker is running on the queue. The default is false.

queue

The queue name(s) that you will be scaling and using to calculate the number of workers working. The default is the queue name of the Job.

class Job
  direct :queue = [:test, :test2]
  @queue = :test

  #rest of your Job class here
end

resque-director will still look at the job’s queue to determine when to scale. However if you set the queue option it will scale a worker working on that queue, or a worker working on multiple queues if you pass it an array of queues.

• resque-director requires resque ~> 1.10.

• resque-director may be incompatible with gems that modify resque’s push/pop functionality.

• Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet

• Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it

• Fork the project

• Start a feature/bugfix branch

• Commit and push until you are happy with your contribution

• Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.

• Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.

Copyright © 2011 Nolan Frausto. See LICENSE.txt for further details.