Klunk

Create and setup publishers (topics) and subscribers (queues) for Rails applications.

"Stop the pigeon! Stop the pigeon!"

Dick Dastardly and Muttley, the villains from Wacky Races, are now flying aces and members of the Vulture Squadron, a crew of aviators on a mission to stop a homing pigeon named Yankee Doodle Pigeon from delivering messages to the other side. Each story features variations on the same plot elements: the Vulture Squadron tries to trap Yankee Doodle Pigeon using one or more planes equipped with Klunk's latest contraptions, but one or more of the Squadron messes up and the planes either crash, collide or explode (or all of the above).

In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead characterize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.

Publish–Subscribe is a sibling of the message queue paradigm, and is typically one part of a larger message-oriented middleware system. This pattern provides greater network scalability and a more dynamic network topology, with a resulting decreased flexibility to modify the publisher and the structure of the published data. In other words, publish-subscribe is a pattern used to communicate messages between different system components without these components knowing anything about each other’s identity.

This design pattern is not new, but it’s not commonly used by Rails developers. But it can bring major advantages for a Rails application.

• Reduce model/controller bloat

It can help decomposing fat models or controllers.

• Fewer callbacks

Enable models to work independently with minimum knowledge about each other, ensuring loose coupling. Expanding the behavior to additional actions is just a matter of hooking to the desired event.

• Single Responsibility Principle (SRP)

Models should handle persistence, associations and not much else. Controllers should handle user requests and be a wrapper around the business logic (Service Objects). Service Objects should encapsulate one of the responsibilities of the business logic, provide an entry point for external services, or act as an alternative to model concerns. Thanks to its power to reduce coupling, the publish-subscribe design pattern can be combined with single responsibility service objects (SRSOs) to help encapsulate the business logic, and forbid the business logic from creeping into either the models or the controllers. This keeps the code base clean, readable, maintainable and scalable.

• Testing

By decomposing the fat models and controllers, and having a lot of SRSOs, testing of the code base becomes a much, much easier process. This is particularly the case when it comes to integration testing and inter-module communication. Testing should simply ensure that events are published and received correctly.

• Loose coupling

The greatest of the publish-subscribe pattern’s strength may be also it’s greatest weakness. The structure of the data published (the event payload) must be well defined, and quickly becomes rather inflexible. In order to modify data structure of the published payload, it is necessary to know about all the subscribers, and either modify them also, or ensure the modifications are compatible with older versions. This makes refactoring of publisher code much more difficult. If you want to avoid this you have to be extra cautious when defining the payload of the publishers. Of course, if you have a great test suite, that tests the payload as well as mentioned previously, you don’t have to worry much about the system going down after you change the publisher’s payload or event name.

• Messaging Bus stability

Publishers have no knowledge of the status of the subscriber and vice versa. Using simple publish-subscribe tools, it might not be possible to ensure the stability of the messaging bus itself, and to ensure that all the published messages are correctly queued and delivered.

• Visibility

Eventually, messages can cause receiver processes (workers) to crash. Specify maximum retry limits and message hospital (dead letter queues), where messages got sent if they failed. And, in order to avoid a catastrophic failover, having an UI to monitor those messages and retry them if needed is strongly encouraged.

• Infinite event loops

When the system is completely driven by events, you should be extra cautious not to have event loops. These loops are just like the infinite loops that can happen in code. However, they are harder to detect ahead of time, and they can bring your system to a standstill. They can exist without your notice when there are many events published and subscribed across the system.

• Setup

When you split the monolith and move towards fine grained systems, the setup and deployment of publishers and subscribers can be painful and error prone. This is why we created Klunk.

Add it to your Gemfile:

gem 'klunk'

Run the following command to install it:

bundle install

Run the generator:

rails generate klunk:install

Check your config/initializers/klunk.rb file for basic and default values.

• prefix: organize names by using a prefix, usually the system name. Defaults to current Rails application name.
• deadletter_suffix: suffix for dead letter queue names. Default value is failures.
• retries_limit: the number of times that a message can be received before being sent to a dead letter queue. Must be between 1 and 1000. Default is 8.
• message_retention_period: The number of seconds for which the queue should retain a message. An integer representing seconds, from 60 (1 minute) to 1209600 (14 days). The default is 345600 (4 days).
• deadletter_message_retention_period: Same as above, for the corresponding dead letter queue. It's a good practice to keep the failed messages as long as you can to be able to retry them, so the default value is 14 days.

• Queue names and queue URLs are case-sensitive

Use the file config/queues.yml to define your application queues.

• name: the queue name is mandatory. This value will be used together with the application and environment names to build the final queue name. Example: klunk_development_queue and klunk_development_queue_failures.
• retries_limit: optional, if set, this value will override the default maximum retries.
• subscribes: the topics this queue subscribes to (optional).
  • name: the topic name (required). The topic name will be built like queue names (based on prefix and environment values unless prefixis set to false, as described below.
  • prefix: boolean flag to decide if the topic name should be built like the queues naming convention. If set to false, the full topic name must be provided. Optional, defaults to true.
  • system: optional, used as prefix for external system name. Defaults to the current global prefix name.

-
  :name: queue_one
-
  :name: queue_two
  :retries_limit: 3
-
  :name: queue_three
  :subscribes:
    -
      :system: service_one
      :name: topic_one
    -
      :system: service_one
      :name: topic_two
    -
      :prefix: false
      :name: full_topic_name

TODO: Explain how to setup - configuration files, values and so on...

TODO: Write usage instructions here - explain the Rake tasks...

So far, Klunk only works with Amazon SNS and SQS services. But you shouldn't be using anything else, right?! 😉

• Marco Antonio Gonzalez Junior (https://github.com/kayaman)
• Wagner Vaz (https://github.com/0xdco)

Bug reports and pull requests are welcome on GitHub at https://github.com/kayaman/klunk.

Klunk is an Open Source project licensed under the terms of the LGPLv3 license. Please see http://www.gnu.org/licenses/lgpl-3.0.html for license text.