fast_find 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 9c5cd7c541e439cb4bd46acd6c28838e7867a369
-  data.tar.gz: 827db02601353683fd17828a779f71d5a282f80c
+SHA256:
+  metadata.gz: 84526cadc7a5aaedce296eafbef1df65c837fba29f0eca88a8510eb89d92a105
+  data.tar.gz: 74d015c6eea35d56091f2ad5a0c34bf68d59cb893e0eee2272228d83bcd2ce6b
 SHA512:
-  metadata.gz: dd2cde72e12d19e762e9a6830f73c12024bab29cecdb7f439501627f0a4c306aafc7130bd2e3ac6344ea7e0b20b4c57c22510f2859cdfd5a2c37dc7935d93913
-  data.tar.gz: af278abe2c42f33cb905e227c389a4a8c0d73aa5c4b30eb7f08b479159a21049e59ba1353776888c72ac1a7dbef1135931b7439060adc07b804f4fd5c8fbadec
+  metadata.gz: '0985c016e8df0f5cd0d1c1a4a7f8e0601c8f573345bbf5c10b1cf649affdc24b20a2b8ea045e4417f5e94b42c45245847b35b93e77b4d85bc15711d626c51931'
+  data.tar.gz: 5c866bc2cd95b4ad053a1c17069d67fe48e1895a8141724f56e1787e9ea632075150e43db26e4ed5c74ec56e172e596bb6cbcaa569e29816a8a4c2bf04b408a4

data/.github/workflows/ci.yml

@@ -0,0 +1,17 @@
+name: CI
+on: [push, pull_request]
+jobs:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest]
+        ruby: ['2.5', '2.6', '3.0', head, jruby, jruby-head, truffleruby]
+    runs-on: ${{ matrix.os }}
+    steps:
+    - uses: actions/checkout@v2
+    - uses: ruby/setup-ruby@v1
+      with:
+        ruby-version: ${{ matrix.ruby }}
+        bundler-cache: true # runs 'bundle install' and caches installed gems automatically
+    - run: bundle exec rake test

data/README.md

@@ -1,80 +1,90 @@
 # FastFind
 
-FastFind is a performance-oriented multi-threaded alternative to the standard
-`Find` module that ships with Ruby.  It should generally be a drop-in
-replacement.
+`FastFind` is a multi-threaded alternative to the standard `Find` module,
+offering increased performance on Rubies which can run `Dir#entries` and
+`File#lstat` calls concurrently (i.e. JRuby).
 
-FastFind is expected to be marginally slower on MRI/YARV, since multithreaded
-`File#lstat` calls there appear to serialize.  However, using the FastFind-
-specific second argument to pass in the `File::Stat` object for each file may
-still prove a win.
-
-This code is considered experimental.  Beware of dog.
+While it can operate as a drop-in replacement for `Find`, it's best used with
+a two-argument block which also yields the `File::Stat` object associated with
+each yielded path.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
-    gem 'fast_find', git: 'https://github.com/Freaky/fast_find.git'
+```shell
+gem 'fast_find'
+```
 
 And then execute:
 
-    $ bundle
+```shell
+$ bundle
+```
 
 ## Usage
 
-Traditional Find-style:
+Traditional Find-style (not recommended):
 
-    FastFind.find(dir) {|entry| frob(entry) }
-    FastFind.find(dir, ignore_errors: false) { .. } # => explodes in your face
-    FastFind.find(dir) # => Enumerator
+```ruby
+FastFind.find(dir) { |entry| frob(entry) }
+FastFind.find(dir, ignore_errors: false) { .. } # => explodes in your face
+FastFind.find(dir) # => Enumerator
+```
 
 Extended style using the second argument to get a `File::Stat`, or `Exception`
-object (if `ignore_errors` is false, this will be raised instead).
-
-    FastFind.find(dir) {|entry, stat| frob(entry, stat) }
+object (if `ignore_errors` is false, this will be raised after the block).
 
-For increased performance and better scaling behaviour, it is recommended to use
-a single shared FastFind object.  Multiple concurrent calls to
-`FastFind::Finder#find` are safe and will share a persistant work pool.
+```ruby
+FastFind.find(dir) { |entry, stat| frob(entry, stat) }
+```
 
-    Finder = FastFind::Finder.new
-    Finder.find(dir) { .. }
+`FastFind` uses a concurrent-ruby executor to run, which can be customised
+by passing it as a named argument:
 
-You can call `Finder#shutdown` to close the work pool if you're done with the
-instance for the time being.  Ensure no other calls to its `#find` are in flight
-beforehand.  The pool is restarted the next time `#find` is called.
+```ruby
+executor = Concurrent::FixedThreadPool.new(16, idletime: 90)
+FastFind.find(dir, executor: executor)
+```
 
-Use the `concurrency` named argument to change the number of worker threads:
+Or while no concurrent `find` operations are in progress, to the module itself:
 
-    FastFind.find(dir, concurrency: 4)
-    FastFind::Finder.new(concurrency: 4)
+```ruby
+FastFind.default_executor = Concurrent::FixedThreadPool.new(16, idletime: 90)
+```
 
-Defaults are `8` for Rubinius and JRuby, `1` for anything else.
+Due to the use of a bounded result queue it is *not* recommended to use an
+executor with a bounded queue or which runs in the same thread as this may
+result in deadlocks or dropped results.
 
-Note the yielded blocks are all executed in the parent thread, *not* in workers.
-
-`FastFind#prune` works.  So does `Find#prune`.
+As with `Find`, `FastFind#prune` will avoid recursing into a directory.
 
 ## Performance
 
-Scanning a cached copy of the NetBSD CVS repository:
+Scanning a cached copy of the NetBSD CVS repository wtih default settings:
+
+jruby 9.2.14.0 (2.5.7) 2020-12-08 ebe64bafb9 OpenJDK 64-Bit Server VM 15.0.2+7-1 on 15.0.2+7-1 +jit:
 
-jruby 9.0.1.0-SNAPSHOT (2.2.2) 2015-07-23 e88911e OpenJDK 64-Bit Server VM
-25.51-b03 on 1.8.0_51-b16 +jit [FreeBSD-amd64]:
+```
+                       user     system      total        real
+FastFind          22.296875  40.851562  63.148438 (  8.014612)
+Find              15.179688  28.031250  43.210938 ( 43.036654)
+FastFind as Find  37.679688  44.375000  82.054688 ( 29.502235)
+```
 
-                   user     system      total        real
-    Find      32.890625  27.742188  60.632813 ( 47.518944)
-    FastFind  35.273438  41.742188  77.015625 (  8.140893)
+These results highlight the importance of the two-argument version.
 
-ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-freebsd10.1]:
+ruby 3.0.0p0 (2020-12-25 revision 95aff21468) \[x86_64-freebsd12.2]:
 
-                   user     system      total        real
-    Find      10.187500  22.351562  32.539062 ( 32.545201)
-    FastFind   9.039062  14.226562  23.265625 ( 23.277589)
+```
+                       user     system      total        real
+FastFind          30.436830  29.265892  59.702722 ( 41.432262)
+Find              10.346662  20.317705  30.664367 ( 30.666812)
+FastFind as Find  28.300448  35.654871  63.955319 ( 39.212032)
+```
 
-On MRI `Find` here is penalised because both `Find` and the benchmark code is
-performing a `File#lstat`.
+Sadly the current implementation is a significant pessimisation on MRI, likely
+due to thread overhead.
 
 ## Development
 

data/bin/benchmark

@@ -1,28 +1,49 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'find'
 require 'benchmark'
 
-require "bundler/setup"
+require 'bundler/setup'
 require 'fast_find'
 
-FastFinder = FastFind::Finder.new
-
 test_dirs = ARGV
-abort("Usage: #{$0} [dir1 [dir2[ ..]]]") if test_dirs.empty?
+abort("Usage: #{$PROGRAM_NAME} [dir1 [dir2[ ..]]]") if test_dirs.empty?
 
 Benchmark.bmbm do |b|
-	b.report("Find") do
-		files = Set.new
-		Find.find(*test_dirs) do |f|
-			files << [f, File.lstat(f)]
-		end
-	end
+  b.report('FastFind') do
+    files = directories = 0
+    FastFind.find(*test_dirs) do |f, stat|
+      if stat.directory?
+        directories += 1
+      else
+        files += 1
+      end
+      # files << [f, stat]
+    end
+  end
+
+  b.report('Find') do
+    files = directories = 0
+    Find.find(*test_dirs) do |f|
+      if File.directory?(f)
+        directories += 1
+      else
+        files += 1
+      end
+      # files << [f, File.lstat(f)]
+    end
+  end
 
-	b.report("FastFind") do
-		files =  Set.new
-		FastFinder.find(*test_dirs) do |f, stat|
-			files << [f, stat]
-		end
-	end
+  b.report('FastFind as Find') do
+    files = directories = 0
+    FastFind.find(*test_dirs) do |f|
+      if File.directory?(f)
+        directories += 1
+      else
+        files += 1
+      end
+      # files << [f, File.lstat(f)]
+    end
+  end
 end

data/fast_find.gemspec

@@ -9,10 +9,10 @@ Gem::Specification.new do |spec|
   spec.authors       = ["Thomas Hurst"]
   spec.email         = ["tom@hur.st"]
 
-  spec.summary       = %q{High performance 'find' alternative.}
-  spec.description   = %q{FastFind is a find workalike which optionally passes
-                       in the File::Stat to the block, and can use multile
-                       threads to walk directories concurrently.}
+  spec.summary       = %q{Multi-threaded Find alternative.}
+  spec.description   = %q{FastFind is a high-performance Find alternative for
+                          Ruby implementations with effective thread
+                          concurrency, particularly JRuby.}
   spec.homepage      = "https://github.com/Freaky/fast_find"
   spec.license       = "MIT"
 
@@ -24,14 +24,16 @@ Gem::Specification.new do |spec|
     raise "RubyGems 2.0 or newer is required to protect against public gem pushes."
   end
 
-  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features|\.git)/}) }
   spec.bindir        = "exe"
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.required_ruby_version = ">= 2.0"
+  spec.required_ruby_version = ">= 2.5"
 
-  spec.add_development_dependency "bundler", "~> 1.9"
-  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_dependency "concurrent-ruby", "~> 1.1"
+
+  spec.add_development_dependency "bundler", "~> 2.0"
+  spec.add_development_dependency "rake", "~> 13.0"
   spec.add_development_dependency "minitest"
 end

data/lib/fast_find.rb

@@ -1,158 +1,112 @@
-#
-# fast_find.rb: A Find workalike optimized for performance.
-#
+# frozen_string_literal: true
 
 require 'set'
-require 'thread'
+
+require 'concurrent'
+
 require 'fast_find/version'
 
+# A Find workalike optimized for multithreaded operation on supporting Rubies
 module FastFind
-	DEFAULT_CONCURRENCY = %w(jruby rbx).include?(RUBY_ENGINE) ? 8 : 1
-
-	def self.find(*paths, concurrency: DEFAULT_CONCURRENCY, ignore_error: true,
-	              &block)
-		Finder.new(concurrency: concurrency, one_shot: true)
-			.find(*paths, ignore_error: ignore_error, &block)
-	end
-
-	def self.prune
-		throw :prune
-	end
-
-	class Finder
-		def initialize(concurrency: DEFAULT_CONCURRENCY, one_shot: false)
-			@mutex       = Mutex.new
-			@queue       = Queue.new
-			@one_shot    = one_shot
-			@concurrency = concurrency
-			@walkers     = nil
-		end
-
-		def startup
-			@mutex.synchronize do
-				return if @walkers
-
-				@walkers = @concurrency.times.map { Walker.new.spawn(@queue) }
-			end
-		end
-
-		def shutdown
-			@mutex.synchronize do
-				return unless @walkers
-
-				@queue.clear
-				@walkers.each { @queue << nil }
-				@walkers.each(&:join)
-
-				@walkers = nil
-			end
-		end
-
-		def find(*paths, ignore_error: true, &block)
-			block or return enum_for(__method__, *paths, ignore_error: ignore_error)
-
-			results = Queue.new
-			pending = Set.new
-
-			startup
-
-			paths.map!(&:dup).each do |path|
-				path = path.to_path if path.respond_to? :to_path
-				results << [path, Util.safe_stat(path)]
-			end
-			results << [:initial, :finished]
-			pending << [:initial, :initial.encoding]
-
-			while result = results.deq
-				path, stat = result
-
-				if stat == :finished
-					pending.delete([path, path.encoding])
-
-					if pending.empty?
-						break
-					else
-						next
-					end
-				end
-
-				catch(:prune) do
-					yield_entry(result, block) if path.is_a? String
-
-					case stat
-					when Exception then raise stat unless ignore_error
-					when File::Stat
-						if stat.directory? and !pending.include?(pe = [path, path.encoding])
-							pending << pe
-							@queue << [path, results]
-						end
-					end
-				end
-			end
-		ensure
-			if one_shot?
-				@queue.clear
-				shutdown
-			end
-		end
-
-		private
-
-		def one_shot?() !!@one_shot end
-
-		def yield_entry(entry, block)
-			if block.arity == 2
-				block.call(entry[0].dup.taint, entry[1])
-			else
-				block.call entry[0].dup.taint
-			end
-		end
-	end
-
-	class Walker
-		def spawn(queue)
-			Thread.new do
-				@encoding = Encoding.find("filesystem")
-				while job = queue.deq
-					walk(job[0], job[1])
-				end
-			end
-		end
-
-		def walk(path, results)
-			enc = path.encoding == Encoding::US_ASCII ? @encoding : path.encoding
-
-			Dir.entries(path, encoding: enc).each do |entry|
-				next if entry == '.' or entry == '..'
-
-				stat(File.join(path, entry), results)
-			end
-		rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP,
-		       Errno::ENAMETOOLONG => e
-			error(e, results)
-		ensure
-			finish(path, results)
-		end
-
-		def stat(entry, results)
-			results << [entry, Util.safe_stat(entry)]
-		end
-
-		def finish(path, results)
-			results << [path, :finished]
-		end
-
-		def error(e, results)
-			results << [:exception, e]
-		end
-	end
-
-	module Util
-		def self.safe_stat(path)
-			File.lstat(path)
-		rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP,
-		       Errno::ENAMETOOLONG => e
-			e
-		end
-	end
+  class << self
+    attr_accessor :default_executor
+
+    def prune
+      throw :prune
+    end
+
+    def find(*paths, ignore_error: true, executor: default_executor, &block)
+      block or return enum_for(__method__, *paths, ignore_error: ignore_error, executor: executor)
+
+      results = SizedQueue.new(1024)
+      pending = Set.new
+
+      paths.map!(&:dup).each do |path|
+        path = path.to_path if path.respond_to? :to_path
+        results << [path, safe_stat(path)]
+      end
+      results << %i[initial finished]
+      pending << path_signature(:initial)
+
+      while (result = results.deq)
+        path, stat = result
+
+        if stat == :finished
+          break if pending.delete(path_signature(path)).empty?
+
+          next
+        end
+
+        catch(:prune) do
+          yield_entry(result, block) if path.is_a? String
+
+          if stat.is_a?(File::Stat) && stat.directory? && pending.add?(path_signature(path))
+            executor.post(path, results) do |path_, results_|
+              walk(path_, results_)
+            end
+          end
+        end
+
+        raise stat if stat.is_a?(Exception) && !ignore_error
+      end
+    end
+
+    private
+
+    FS_ENCODING = Encoding.find('filesystem')
+
+    def walk(path, results)
+      enc = path.encoding == Encoding::US_ASCII ? FS_ENCODING : path.encoding
+
+      # This benchmarks as about 10% faster than Dirs.foreach
+      Dir.entries(path, encoding: enc).each do |entry|
+        next if (entry == '.') || (entry == '..')
+
+        results << stat(File.join(path, entry))
+      end
+    rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP,
+           Errno::ENAMETOOLONG => e
+      results << error(e)
+    ensure
+      results << finish(path)
+    end
+
+    def stat(entry)
+      [entry, safe_stat(entry)]
+    end
+
+    def finish(path)
+      [path, :finished]
+    end
+
+    def error(e)
+      [:exception, e]
+    end
+
+    def path_signature(path)
+      [path, path.encoding]
+    end
+
+    def yield_entry(entry, block)
+      if block.arity == 2
+        block.call(entry[0].dup, entry[1])
+      else
+        block.call entry[0].dup
+      end
+    end
+
+    def safe_stat(path)
+      File.lstat(path)
+    rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP,
+           Errno::ENAMETOOLONG => e
+      e
+    end
+  end
+
+  self.default_executor = case RUBY_ENGINE
+                          when 'jruby', 'rbx'
+                            Concurrent::FixedThreadPool.new(8, idletime: 60)
+                          else
+                            Concurrent::FixedThreadPool.new(1, idletime: 60)
+                          end
 end

data/lib/fast_find/version.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 module FastFind
-  VERSION = "0.1.1"
+  VERSION = '0.2.0'
 end

metadata

@@ -1,43 +1,57 @@
 --- !ruby/object:Gem::Specification
 name: fast_find
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - Thomas Hurst
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2015-08-01 00:00:00.000000000 Z
+date: 2021-02-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.9'
-  name: bundler
+        version: '1.1'
+  name: concurrent-ruby
+  type: :runtime
   prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  name: bundler
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.9'
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '13.0'
   name: rake
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '13.0'
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
@@ -45,25 +59,25 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
   name: minitest
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 description: |-
-  FastFind is a find workalike which optionally passes
-                         in the File::Stat to the block, and can use multile
-                         threads to walk directories concurrently.
+  FastFind is a high-performance Find alternative for
+                            Ruby implementations with effective thread
+                            concurrency, particularly JRuby.
 email:
 - tom@hur.st
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- ".github/workflows/ci.yml"
 - ".gitignore"
-- ".travis.yml"
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -87,16 +101,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.0'
+      version: '2.5'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project:
-rubygems_version: 2.4.8
+rubygems_version: 3.0.6
 signing_key:
 specification_version: 4
-summary: High performance 'find' alternative.
+summary: Multi-threaded Find alternative.
 test_files: []

data/.travis.yml

@@ -1,3 +0,0 @@
-language: ruby
-rvm:
-  - 2.2.2