Pipeline¶ ↑

Pipeline is a Rails plugin/gem to run asynchronous processes in a configurable pipeline.

rdoc.info/projects/dtsato/pipeline

• Execution of sequential user-defined stages in an asynchronous pipeline

• Persistence of pipeline instances and stages

• Error recovery strategies:

  • Irrecoverable errors fail the entire pipeline

  • Recoverable errors are automatically retried (using dj’s exponential retry strategy)

  • Recoverable errors that require user input pause the pipeline for further retry

• Cancelling/Resuming of a paused pipeline

• Callbacks before and after executing stages and pipeline

pipeline can be installed as either a RubyGem (recommended) or as a plugin.

To install pipeline as a RubyGem, add the following lines to your config/environment.rb file:

config.gem "pipeline"
config.gem "delayed_job"

And execute:

rake gems:install
rake gems:unpack # Optional, if you want to vendor the gem

To install it as a plugin, run:

script/plugin install git://github.com/dtsato/pipeline.git

In order to persist your pipelines and stages, execute the following:

script/generate pipeline # To generate the migration scripts that will store pipelines
script/generate delayed_job # To generate the migration scripts for delayed_job
rake db:migrate

You will also need to run your Delayed Job workers that will process the pipeline jobs in the background:

rake jobs:work

• Rails (ActiveRecord)

• Delayed job (github.com/collectiveidea/delayed_job)

Check examples for more examples (including error-recovery and cancelling)

class Step1 < Pipeline::Stage::Base
  def perform
    puts("Started step 1")
    sleep 2
    puts("Finished step 1")
  end
end

class Step2 < Pipeline::Stage::Base
  def perform
    puts("Started step 2")
    sleep 3
    puts("Finished step 2")
  end
end

class TwoStepPipeline < Pipeline::Base
  define_stages Step1 >> Step2
end

Pipeline.start(TwoStepPipeline.new)

Copyright © 2009 Danilo Sato. See LICENSE for details.