ResumableJob
Make any ActiveJob
resumable.
Use exception flow to make jobs exceptionally resumable, whilst retaining other state, with automatic exponential
backoff handling. ActiveJob
is not a dependency, so this could be used with "anything". Adds a module
to include
somewhere that adds a method which yields a block. During this block, you can throw a ResumableJob::ResumeLater
to
call the following:
self.class
.set(wait_until: resume_at || ResumableJob::Backoff.to_time(attempt))
.perform_later(pause(state).merge(attempt: attempt + 1))
State is passed through pause
and when pause
is not overridden will be all the arguments you passed to your job plus
an attempt
argument that is steadily increased in order to to exponential backoff.
Installation
Add this line to your application's Gemfile:
gem 'resumable_job'
And then execute:
$ bundle
Or install it yourself as:
$ gem install resumable_job
Usage
Make a job resumable
Simple example to implement pagination that resumes later if you receive a "Rate Limit Exceeded".
class FetchDataJob < ApplicationJob
include ResumableJob::Resumable
def perform(state)
page = state.fetch(:page) { 1 }
resumable(state) do
loop do
result = DataFetcher.call(page: page)
raise ResumableJob::ResumeLater(state: state.merge(page: page)) if result.status == 429
break unless result.next_page?
page = result.next_page
end
end
end
end
Turn inner exception into resumable
When the exception has more information (for example a "rate limit reset" value), it can be turned into a resume later.
Additionally, the state
of the resume later exception will me merged into the original state, and then into the pause
state.
class FetchDataJob < ApplicationJob
include ResumableJob::Resumable
def perform(state)
resumable(state) do
fetch_data(state)
end
end
private
def fetch_data(state)
RateLimitableFetcher.call(state)
rescue RateLimitableFetcher::RateLimited => ex
raise ResumableJob::ResumeLater.new(state: state, utc: ex.retry_at, message: ex.message)
end
end
Filter out keys from the state
Some state is not serializable. You may be calling your job with perform_now
, but when it resumes later, some
arguments can not be serialized. Use the pause
override to include state not originally present, modify state that is
not passed by your exception (ResumeLater exception state), or remove state.
class FetchDataJob < ApplicationJob
include ResumableJob::Resumable
def pause(state)
state.slice(:attempt, :page, :token)
end
end
Development
After checking out the repo, run bin/setup
to install dependencies. Then, run rake test
to run the tests. You can
also run bin/console
for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install
. To release a new version, update the
version number in version.rb
, and then run bundle exec rake release
, which will create a git tag for the version,
push git commits and tags, and push the .gem
file to rubygems.org.
Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/SleeplessByte/resumable_job.