SlowEnumeratorTools

SlowEnumeratorTools provides tools for transforming Ruby enumerators that produce data slowly and unpredictably (e.g. from a network source):

• SlowEnumeratorTools.merge: given a collection of enumerables, creates a new enumerator that yields elements from any of these enumerables as soon as they become available.

• SlowEnumeratorTools.batch: given an enumerable, creates a new enumerable that yields batches containing all elements currently available.

• SlowEnumeratorTools.buffer: given an enumerable and a number, will create a buffer of that number of elements and try to fill it up with as many elements from that enumerable, so that they can be yielded immediately.

Add this line to your application's Gemfile:

gem 'slow_enumerator_tools'

And then execute:

$ bundle

Or install it yourself as:

$ gem install slow_enumerator_tools

Given a collection of enumerables, creates a new enumerator that yields elements from any of these enumerables as soon as they become available.

This is useful for combining multiple event streams into a single one.

# Generate some slow enums
enums = []
enums << 5.times.lazy.map { |i| sleep(0.1 + rand * 0.2); [:a, i] }
enums << 5.times.lazy.map { |i| sleep(0.1 + rand * 0.2); [:b, i] }
enums << 5.times.lazy.map { |i| sleep(0.1 + rand * 0.2); [:c, i] }

# Merge and print
merged_enum = SlowEnumeratorTools.merge(enums)
merged_enum.each { |e| p e }

Example output:

[:b, 0]
[:a, 0]
[:b, 1]
[:c, 0]
[:a, 1]
[:b, 2]
[:c, 1]
[:c, 2]
[:a, 2]
[:b, 3]
[:c, 3]
[:b, 4]
[:a, 3]
[:c, 4]
[:a, 4]

Given an enumerable, creates a new enumerable that yields batches containing all elements currently available.

This is useful for fetching all outstanding events on an event stream, without blocking.

# Generate a slow enum
enum = 4.times.lazy.map { |i| sleep(0.1); i }

# Batch
batch_enum = SlowEnumeratorTools.batch(enum)

# Wait until first batch is available
# … prints [0]
p batch_enum.next

# Give it enough time for the second batch to have accumulated more elements,
# … prints [1, 2]
sleep 0.25
p batch_enum.next

# Wait until final batch is available
# … prints [3]
p batch_enum.next

Given an enumerable and a number, will create a buffer of that number of elements and try to fill it up with as many elements from that enumerable.

This is particularly useful when reading from a slow source and writing to a slow sink, because the two will be able to work concurrently.

# Create (fake) articles enumerator
articles =
  Enumerator.new do |y|
    5.times do |i|
      sleep 1
      y << "Article #{i}"
    end
  end

# Buffer
articles = SlowEnumeratorTools.buffer(articles, 5)

# Print each article
# This takes 6 seconds, rather than 10!
articles.each do |a|
  sleep 1
end

Install dependencies:

$ bundle

Run tests:

$ bundle exec rake

Bug reports and pull requests are welcome on GitHub at https://github.com/ddfreyne/slow_enumerator_tools. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

The gem is available as open source under the terms of the MIT License.

Everyone interacting in the SlowEnumeratorTools project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.