qwirk

http://github.com/ClarityServices/qwirk

Library for performing background tasks as well as asynchronous and parallel RPC calls Alpha version. API still subject to change.

gem install qwirk

TODO: This stuff is all obsolete modern_times documentation. Look at the project http://github.com/ClarityServices/qwirk_examples for usage information.

Create config/qwirk.yml which might look as follows:

development_server: &defaults
  :factory: org.apache.activemq.ActiveMQConnectionFactory
  :broker_url: tcp://127.0.0.1:61616
  :require_jars:
    - <%= Rails.root %>/lib/activemq/activemq-all.jar

development_vm:
  <<: *defaults
  :broker_url: vm://127.0.0.1
  :object_message_serialization_defered: true

staging:
  <<: *defaults
  :broker_url: tcp://stage2:61616
  :username: myuser
  :password: mypassword

production:
  <<: *defaults
  :broker_url: failover://(tcp://msg1:61616,tcp://msg2:61616)?randomize=false&initialReconnectDelay=100&useExponentialBackOff=true&maxCacheSize=524288&trackMessages=true
  :username: myuser
  :password: mypassword

In development and test mode, you will notice that there is no configuration defined. In this case, published messages will cause synchronous calls to the Worker's perform method which matches the destination queue or topic. This will allow your coworkers to use the functionality of the queueing system without having to startup a JMS server. If you wanted to start up in an actual server-type mode, you might set the QWIRK_ENV environment variable to "development_server" to override the Rails.env. This will allow you to test the queueing system without having to make temporary changes to the config file which could accidentally get checked in. For staging and production modes, you will need to have a JMS server running. Note that this library has only been tested with ActiveMQ.

Create config/workers.yml which might look as follows:

development:
  Analytics:
    :count: 1
  Dashboard:
    :count: 1

stage1:
  Analytics:
    :count: 2
  Dashboard:
    :count: 2

app1: &default_setup
  Analytics:
    :count: 2
  Dashboard:
    :count: 2

app2:
  <<: *default_setup

app3:
  <<: *default_setup

In this file, the count represents the number of threads dedicated per worker. The worker first looks for a key that matches the Rails.env. If it doesn't find one, it will look for a key matching the non-qualified hostname of the machine. (TODO: Show how to add options that get passed to the constructor and a single worker class that operates on 2 different queues). This file is optional and workers can be configured ad-hoc instead.

If you don't want to explicitly define your workers in a config file, you can create them ad-hoc instead. Configure your workers by starting jconsole and connecting to the manager process. Go to the MBeans tab and open the tree to Qwirk => Manager => Operations => start_worker

Start/stop/increase/decrease workers as needed. The state is stored in the log directory (by default) so you can stop and start the manager and not have to reconfigure your workers.

Create config/initializers/qwirk.rb which might look as follows (TODO: Maybe add or refer to examples for registering marshal strategies):

Qwirk.init_rails
# Publishers can be defined wherever appropriate, probably as class variables within the class that uses it
$foo_publisher = Qwirk::Adapter::JMS::Publisher.new('Foo')

When creating publishers, you will probably want to store the value in a class variable. Publishers internally make use of a session pool for communicating with the JMS server so you wouldn't want to create a new connection every time you published an object.

In your code, queue foo objects:

$foo_publisher.publish(my_foo_object)

In app/workers, create a FooWorker class:

class FooWorker
  include Qwirk::Adapter::JMS::Worker
  def perform(my_foo_object)
    # Operate on my_foo_object
  end
end

For the staging and production environment, you will need to startup a Manager process on each machine that handles messages. You might create script/worker_manager as follows (assumes Rails.root/script is in your PATH):

#!/usr/bin/env runner

manager = Qwirk.create_rails_manager
manager.join

TODO: Refer to example jsvc daemon script

By default, a worker operates on the queue with the same name as the class minus the Worker postfix. You can override this by explicitily by specifying a queue or a virtual topic instead. A virtual_topic (ActiveMQ only) allows you to publish to one destination and allow for multiple workers to subscribe. (TODO: need to completely remove the use of topics as every thread for every worker receives all messages instead of a group of workers (threads) collectively receiving all messages. Virtual topics get around this problem). For instance, suppose you have the following workers:

class FooWorker
  include Qwirk::Adapter::JMS::Worker
  virtual_topic 'inquiry'

  def perform(my_inquiry)
    # Operate on my_inquiry
  end
end

class BarWorker
  include Qwirk::Adapter::JMS::Worker
  virtual_topic 'inquiry'

  def perform(my_inquiry)
    # Operate on my_inquiry
  end
end

Then you can create a publisher where messages are delivered to both workers:

@@publisher = Qwirk::Adapter::JMS::Publisher.new(:virtual_topic_name => 'inquiry')
...
@@publisher.publish(my_inquiry)

TODO: See examples/requestor

TODO: See examples/advanced_requestor

Brad Pardee

Copyright (c) 2012 Clarity Services. See LICENSE for details.