CheckerJobs
This gems provides a small DSL to check your data for inconsistencies.
Introduction
To ensure database integrity, DBMS provides some tools: foreign keys, triggers, strored procedures, ... Those tools aren't the easiests to maintain unless your project is based on them. Also, you may want to avoid to duplicate your business rules from your application to another language and add operational complexity around deployment.
This gem doesn't aim to replace those tools but provides something else that could serve a close purpose: ensure that you work with the data you expect.
This gem helps you schedule some verifications on your data and get alerts when something is unexpected. You declare checks that could contain any business code you like and then those checks are run by your application, in background jobs.
A small DSL is provided to helps you express your predicates:
-
ensure_no
will check that the result of a given block iszero?
,empty?
orfalse
-
ensure_more
will check that the result of a given block is>=
than a given number -
ensure_fewer
will check that the result of a given block is<=
than a given number
and an easy way to configure notifications.
For instance, at Drivy we don't expect users to get a negative credit amount. It isn't easy to get all the validation right because many rules are in play here. Because of those rules the credit isn't just a column in our database yet but needs to be computed based on various parameters. What we would like to ensure is that no one ends up with a negative credit. We could write something like:
class UsersChecker
include CheckerJobs::Base
options sidekiq: { queue: :slow }
notify :email, to: "oss@drivy.com"
ensure_no :negative_rental_credit do
# The following code is an over-simplification
# Real code is more performance oriented...
user_ids_with_negative_rental_credit = []
User.find_each do |user|
if user.credit_amount < 0
user_ids_with_negative_rental_credit << user.id
end
end
user_ids_with_negative_rental_credit
end
end
Then when something's wrong, you'll get alerted.
You'll find more use cases and tips in the wiki.
Installation
Add this line to your application's Gemfile:
gem 'checker_jobs'
Usage
Have a look at the examples directory of the repository to get a clearer idea about how to use and the gem is offering.
Configure
require "checker_jobs"
CheckerJobs.configure do |c|
c.jobs_processor = :sidekiq
c.notifier :bugsnag do |options|
options[:formatter_class] = CheckerJobs::Notifiers::BugsnagDefaultFormatter
end
c.notifier :email do |options|
options[:formatter_class] = CheckerJobs::Notifiers::EmailDefaultFormatter
options[:email_options] = {
from: "oss@drivy.com",
reply_to: "no-reply@drivy.com",
}
end
c.notifier :logger do |options|
options[:logdev] = STDOUT
options[:level] = Logger::INFO
end
c.repository_url = { github: "drivy/checker_jobs" }
end
This piece of code usually goes into the config/initializers/checker_jobs.rb
file in a rails application. It relies on the fact that ActionMailer and sidekiq
are already configured.
If you're on a different stack and you'll like to add a new job processor or notification backend in this gem, drop us a line.
Job Processor
At the moment, only Sidekiq is supported as a job processor to asynchronously check for data inconsitencies. The gem is designed to supports more processor. PRs are appreciated 🙏
Notifiers
We support different kind of notifiers, as of today we have the following:
-
:bugsnag
: usesBugsnag
to send notifications. It takes the global Bugsnag configuration. -
:email
: usesActionMailer
to send emails. You can pass it anyActionMailer
options. -
:logger
: UsesLogger
to output inconsitencies in the log. Takes the following params:-
logdev
: The log device. This is a filename (String) or IO object (typically STDOUT, STDERR, or an open file). -
level
: Logging severity threshold (e.g. Logger::INFO)
-
Write checkers
A checker is a class that inherits CheckerJobs::Base
and uses the
DSL to declare checks.
class UserChecker
include CheckerJobs::Base
options sidekiq: { queue: :fast }
notify :email, to: "oss@drivy.com"
ensure_no :user_without_email do
UserRepository.missing_email.size
end
end
The UserChecker
will have the same interface as your usual jobs. In this
example, UserChecker
will be a Sidekiq::Worker
. Its #perform
method will
run the check named :user_without_email
and if
UserRepository.missing_email.size
is greater than 0 then an email will be
fired through ActionMailer to oss@drivy.com
.
Schedule checks
Once you have checker jobs, you'll need to run them. There are many task schedulers out there and it isn't really relevant what you'll be using.
You have to enqueue your job as often as you like and that's it.
UserChecker.perform_async
Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/drivy/checker_jobs.
You'll find out that the CI is setup to run test coverage and linting.
License
The gem is available as open source under the terms of the MIT License.