FastFind
FastFind
is a multi-threaded alternative to the standard Find
module,
offering increased performance on Rubies which can run Dir#entries
and
File#lstat
calls concurrently (i.e. JRuby).
While it can operate as a drop-in replacement for Find
, it's best used with
a two-argument block which also yields the File::Stat
object associated with
each yielded path.
Installation
Add this line to your application's Gemfile:
gem 'fast_find'
And then execute:
$ bundle
Usage
Traditional Find-style (not recommended):
FastFind.find(dir) { |entry| frob(entry) }
FastFind.find(dir, ignore_errors: false) { .. } # => explodes in your face
FastFind.find(dir) # => Enumerator
Extended style using the second argument to get a File::Stat
, or Exception
object (if ignore_errors
is false, this will be raised after the block).
FastFind.find(dir) { |entry, stat| frob(entry, stat) }
FastFind
uses a concurrent-ruby executor to run, which can be customised
by passing it as a named argument:
executor = Concurrent::FixedThreadPool.new(16, idletime: 90)
FastFind.find(dir, executor: executor)
Or while no concurrent find
operations are in progress, to the module itself:
FastFind.default_executor = Concurrent::FixedThreadPool.new(16, idletime: 90)
Due to the use of a bounded result queue it is not recommended to use an executor with a bounded queue or which runs in the same thread as this may result in deadlocks or dropped results.
As with Find
, FastFind#prune
will avoid recursing into a directory.
Performance
Scanning a cached copy of the NetBSD CVS repository with default settings:
jruby 9.2.14.0 (2.5.7) 2020-12-08 ebe64bafb9 OpenJDK 64-Bit Server VM 15.0.2+7-1 on 15.0.2+7-1 +jit:
user system total real
FastFind 20.492188 41.898438 62.390625 ( 7.558871)
Find 14.742188 27.804688 42.546875 ( 42.355761)
FastFind as Find 33.648438 44.757812 78.406250 ( 26.047635)
These results highlight the importance of the two-argument version.
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-freebsd12.2]:
user system total real
FastFind 27.181013 26.598176 53.779189 ( 38.016831)
Find 10.672928 20.198666 30.871594 ( 30.878547)
FastFind as Find 27.008096 36.266163 63.274259 ( 38.832825)
Sadly the current implementation is a significant pessimisation on MRI, likely due to thread overhead.
Development
After checking out the repo, run bin/setup
to install dependencies. Then, run
bin/console
for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install
. To
release a new version, update the version number in version.rb
, and then run
bundle exec rake release
to create a git tag for the version, push git commits
and tags, and push the .gem
file to rubygems.org.