Project

hsdq

0.0
No commit activity in last 3 years
No release in over 3 years
Hsdq: Light weight and distributed, Hsdq allow message passing distributed applications to exchange requests and data at high speed, work in parallel and scale horizontaly.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

Runtime

~> 2.0
~> 3.3
 Project Readme

Hsdq

###High Speed Distributed Queue: #####A messaging layer allowing distributed applications/scripts to communicate easily and exchange data at high speed. Also allow to offload deferred tasks to outside applications/scripts.

Features

  • Connect with ease applications and/or scripts together.
  • Easy to scale horizontally by adding or removing new application on line.
  • Easy to balance the load, no round-robbing, the first listening application listening will get the message, then the next one will get the next message.
  • hot swap, hot deployment. If the app message API is compatible with the previous API version, just deploy the new version and stop the old one.
  • No need to have the target application on line to send a message, they will be stored in the bus and the target app will begin processing as soon as back online. (HTTP API need to be online to get the request).

Dependencies

  • HSDQ rely on Redis for the transport layer

Recommended

  • Ruby 2.3.1, Minimum ruby 2
  • Jruby 9.1.2 Minimum 9.0.3

How to use HSDQ

####Step 0: Getting a bit accustomed

Play with some quick and dirty scripts to get familiar, check the examples folder.

####Step 1: create the connected class

  • Create a class in your application or script (for a Rails app in lib or accustomed in models)
  • Prefix this class name with Hsdq, ie: HsdqMyClass
  • Extend Hsdq into your class

The listening channel will then be created based on your class name ie: my_class

####Step 2: to get the events

Into your class override the 5 following methods:

Method Comment
hsdq_request Called when a request is received
hsdq_ack Called when an acknowledgment your request has been received by the receiving app
hsdq_callback Called when the other other app respond to your request
hsdq_feedback Called when the other app is sending intermediate feedback
hsdq_error Called when an error occur

You must implement these class methods into your class. They will receive the event. All methods are receiving 2 parameters: A Hash for the message and a Hash or nil for the context.

def self.hsdq_request(message, context)  
  self.new.hsdq_request message, context  
end  
  
def self.hsdq_request(message, context)  
  # Start your processing here  
end  

Important: Unless options[:threaded] is set to false, each event received are totally decoupled and independent. So you need to keep track of the context if needed.

####Step 3: sending messages

Four methods are used to send and respond to messages.
They are implemented internally in Hsdq and need a Hash as parameter.

Method Comment
hsdq_send_request(message) To send a request
hsdq_send_calback(message) To send the final response to a request
hsdq_send_feedback(message) To send intermediate data
hsdq_send_error(message) To send an error message in case of error

####Step 4: Set the authorized topics and tasks in your class:
hsdq_authorized_topics :beer, :wine, :cheese, :dishes
hsdq_authorized_tasks :drink, :eat, :clean
Note you can in simple cases like in scripts only use topic or task but it is recommended to use both.

####Step 5: Setup the hsdq config file:

In the config folder create a file name hsdq_my_class.yml There ia a sample file in hsdq config folder.

  • Each class you connect to HSDQ must have it's specific yml file. This allow you to shard the bus for heavy traffic application.
  • Each subsection can point to a different Redis instance/host/db.
    For heavy traffic sites, you can shard messages and have for example a unique host for admin.
  • Session can be split to different host if different applications do not share the sessions data
development:
  redis:
    message:
      host: 127.0.0.1
      port: 6379
      db:   0
    admin:
      host: 127.0.0.1
      port: 6379
      db:   0
    session:
      host: 127.0.0.1
      port: 6379
      db:   0

Message specifications

#####Five type of message are running into the bus:

Type Description
1 Request Initial message sent
2 Ack Acknowledgment of reception
3 Feedback Intermediate response (progress)
4 Callback Final response with the data
5 Error Final response when an error occur

#####A message event is composed of 2 parts:

1 - The spark:

  • An ephemeral tiny part that include the minimum but sufficient information pointing to the 2nd part, the message itself.
  • The spark is what the listener will be getting from the list queue and this will ignite the process.
  • The spark is pushed to a redis list and popped by the listener. Once popped it will not be available in the redis layer anymore.

2 - The Burst:

  • An event in the life of the message, the burst is stored as one value in a Redis hash and can be retrieved using the spark data.

#####The complete message

  • It will contain all the operations executed during the life cycle of the message.
  • It is stored into a hash and each element of this hash is one message event.
  • The message element is stored in a Redis Hash with a default expiration of 72 hours (can be adjusted)

A message is composed of multiple events. At minimum there are 3 events:

  1. Request
  2. Ack
  3. Callback or Error

Multiple feedback can be sent before the callback/error and multiple events can be present in the case of chained events.

#####Structure of a message: The burst:

Key Type Description
sent_to M String Name channel you publish
topic O Symbol Topic to be processed (ex: :beer, :wine, :cheese etc...).
task O Symbol Task to be processed (ex: :drink, eat, clean ,etc...).
params M Hash parameters matching the receiver message API (can be empty hash)
data C Hash Mandatory in the responses as this is the response
type I Symbol Type of the message ie: :request, :callback, etc
sender I String the listening channel of the sender. Used to reply
uid I UUID Unique identifier for the message container (redis hash)
spark_uid I UUID Unique identifier for the 'spark' and the 'burst'
tstamp I UTC Timestamp for the event UTC
context I Hash Data from the previous request. keys: :reply_to, spark_uid
previous_sender I String The previous sender of a request. used in chained queries
hsdq_session O String Key to load session data related to the context

M Mandatory, O Optional (but recommended for pre-filtering), I Hsdq internal

  • Topic and task values must be present respectively in the topics or task white lists when used.
  • Redis return symbols as strings in the responses

The spark:

The spark has the same structure and values as the burst except it do not carry the payload. The params and data as well as eventual custom keys keys which can contain heavy payloads are not included.