Wolox on Rails - Batchifier
Gem that allows you to easily divide processing or requests that work with a lot of information into several batches with smaller chunks of data, taking care of the result from each one and providing it joint together based on different strategies for parsing said response.
Installation
Add this line to your application's Gemfile:
gem 'wor-batchifier'
And then execute:
$ bundle
Or install it yourself as:
$ gem install wor-batchifier
Usage
Basic use:
The first step is to include the Wor::Batchifier in the Class or Module you intend to use it:
class MyClass
include Wor::Batchifier
end
If you're going to use the gem in your controllers, a good practice would be to define a parent controller from which all other controllers will have to extend to have access to the batchifier's methods. So, let's do that in our ApplicationController.rb
:
class ApplicationController < ActionController::Base
include Wor::Batchifier
end
You could also include the batchifier just in the controllers you intend to use it in.
The final step, is to find any request or process you wish to perform with smaller chunks of data and utilize the batchifier's methods to divide it into smaller tasks.
For example, let's pretend we have an endpoint called bulk_request that communicates with a third API and sends a lot of information to be utilized.
def bulk_request
ThirdAPI.bulk_request(params[:information])
end
Now we will partition that request into chunks using the batchifier as to not overburden the ThirdAPI:
def bulk_request
execute_in_batches(params[:information], batch_size: 100, strategy: :add) do |chunks|
ThirdAPI.bulk_request(chunks)
end
end
The batchifier will take three parameters, the first one being the information that needs to be partitioned, then the batch_size we wish to utilize and finally the symbol of the strategy that will be implemented on the response of each batch.
Available strategies
- Add: For each request, it joins together each response no matter the result.
- Maintain-Unique: It will only add the results that are not present already in the response.
- No-Response: It will not provide any response whatsoever.
Adding new strategies
Should you desire to add new strategies, it's as simple as creating a new class and defining a method called merge_strategy
which will hold the logic that will be implemented to parse and merge the results from each batch. Let's look at an example:
module Wor
module Batchifier
class MaintainUnique < Strategy
def merge_strategy(response,memo)
return response.merge(memo) { |_, v1, _| v1 }
end
end
end
end
The merge_strategy
will receive two parameters, the first being "response" which is the total response which will be returned from execute_in_batches
, and "memo" is the recursive response from each batch that will be added to response in each iteration. If you want to merge or do something else entirely, you have the option to do so.
All strategies have a base_case
which by default is an empty hash {}
but if you wish to override it, you can define your own in your strategy by simply adding a method called base_case
which should return the value you desire for your own personal needs.
def base_case
# An initial step or value where the responses will be merged to.
end
The new class that will hold the method merge_strategy
should inherit the class Strategy
. If the strategy doesn't define the method, an exception will be raised when trying to utilize it
warning that it does not respect the contract set by the Strategy
Interface.
You can also define a merge strategy via Proc
, without the need of creating a new class. The Proc
should receive two parameters: the first being "response" and the second one being "memo", both of which work the same way as they do when you create a class and define its merge strategy
. All Procs
have {}
as their base case which cannot be changed. Let's look at an example:
merge_strategy = Proc.new do |response, memo|
memo = [] if memo.empty?
memo + response
end
execute_in_batches(collection, batch_size: 10, strategy: merge_strategy) do |chunk|
...
end
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Run rubocop lint (
rubocop -R --format simple
) - Run rspec tests (
bundle exec rspec
) - Push your branch (
git push origin my-new-feature
) - Create a new Pull Request
About
This project was developed by Pedro Jara along with Diego Raffo at Wolox.
Maintainers: Pedro Jara, Federico Volonnino
Contributors: Pedro Jara, Federico Volonnino
License
wor-batchifier is available under the MIT license.
Copyright (c) 2017 Wolox
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.