Interface to pragmatically setup a queue job for Q4M!

Q4M is a open-sourced MySQL Engine. that attempts to take away some logic from your application by encapsulating it inside of mysql.

The documentation is kinda bad, but the magic that it does in the background is sweeeeeet!

You basically store all of your background jobs inside a MySQL table. that table’s engine should be a queue (q4m) engine!

Exampe:

drop table if exists shell_jobs, ruby_jobs;
create table shell_jobs (command text) engine=queue;
create table ruby_jobs (command text) engine=queue;

That engine will work under some kind of transaction. When you connect to the database the engine will give you a list of all the pending jobs. When you start a transaction you’ll receive ONE job to work with. In the mean time the whole table gets locked.. And if you do a ‘select *’ all results will return emtpy!

Don’t panic!

This is great, you can never access two jobs at the same time. if another worker/machine connects they’ll get a list of jobs except the one is wrapped in the transaction!

So that means that the locking only happens on the connection that is perfoming the transaction but any other workers will get the full list of jobs except the ones that are being processed by other workers.

When the first transaction gets terminated the engine will automatically unlock the table and delete the record.

This is great, because you can have any number of workers working concurrently without making a mess!

Meaning that a job can never be called in twice by anyone, and that you can grow horizontally, the limit is the sky!

What’s next?

The job is done here!

Go do something more interesting like clustering! ;)

Seek & destroy, Seek & destroy!

Gives you an interface to quickly setup how specificly a job is supposed to run!

Basically, you define a class create an method called execute, which receives a job argument. truly a MySQL record in the form of an array, what you do from there is up to you!

if something goes wrong just call the queue_abort method and everything gets rolled back!

kazu@utopia:~$ gem install q4m

require 'rubygems'
require 'q4m'

# Specify how shell jobs will run

class ShellJob < ::Q4m::Queue::Job

  def execute job
    queue_abort unless system(job.first)
  end

end

host = 'localhost'
user = 'root'
pass = 'secret'
database = 'my_db'

# Instantiate and run a job!
mysql = Mysql.new host, user, pass, database
job = ShellJob.new mysql, Logger.new('q4m.log')

job.run

Note that: The ShellJob class will try to connect to the ‘shell_jobs’ q4m table by default, if you want to use a different table override the table_name method and specify one of your liking!

class ShellJob < ::Q4m::Queue::Job

 def execute job
    queue_abort unless system(job.first)
 end

  def table_name
    'my_cool_table'
  end

end

Basically any machine running rails with this code, can pull jobs and do some hard work for you! even if they are not part of your cluster!

The only requirement is that they can access the mysql-database.

and no, you don’t need to install that engine locally!

Go farther just create a new job class and run whatever you like!

This is how I installed it.

wget http://q4m.kazuhooku.com/dist/old/mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar.gz
unzip *.gz
tar -xvf mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar
cd q4m-0.8.9-linux-x86_64
sudo cp libqueue_engine.so /usr/lib/mysql/plugin
sudo chmod 777 /usr/lib/mysql/plugin/libqueue_engine.so
mysql -u root -p -f mysql < support-files/install.sql

You might wanna go to:

http://q4m.github.com/

And see tha latest on how to get it installed.

just thought of posting it, so you’ll get an idea.

Yeah, but what’s your distro?

it’s right here!

deploy@staging> cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04 LTS"