fast_ignore 0.13.0 → 0.14.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ee7e9eb231ff6441ca703c935e9c270e4cb6c2f8f3216d993d4fc891807e77f6
-  data.tar.gz: 68f1ace7ab6420ce140a43a747a969f29401eba718dd9d1e94b9538774b0ac6d
+  metadata.gz: 05f6139deefedc14e0c58acd66909aa25af5240a83e67a6463eacedd6cec9eb3
+  data.tar.gz: c6becf91ea8bc9ccf0524309c5cad1e01485c4e2324222f231ade33855c65528
 SHA512:
-  metadata.gz: 83609a93cf79d3d4063e69d213450d85f81c4b9bb7c1200468e46fa7fce01602c751c4afd6dbaef652d1761c0f180fa5806ff29d0913ca184a5e41850875a8d0
-  data.tar.gz: dbf3e341e671fa1a51dce4c885c04e1eadf0717d78ffd94f1746c63b1024f78bd9d4b6915e7b892ca3af21d27ef2566df980be41101573c3dd586384b6f6d7c6
+  metadata.gz: cca040e6401d0c412498dff47b460ba7a19d5c9cb810cc205151a56b9db959a88ea3c4137610537bda43c5520611b1711abc8124b0ab98dd6bda261de9638588
+  data.tar.gz: a3fe257c77f8350c2dde1bcfcde01293b5af87b5d6b627b4e44555b3b6a45154c3d567d1e68f8f03520d21e28e78eda439e92b7a8b5b2d0b43b6f2a23a2f5ae4

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+# v0.14.0
+- significant performance improvements ~50% faster
+- add `FastIgnore#to_proc` for no good reason
+
 # v0.13.0
 - Attempt to improve documentation structure
 - Remove `gitignore: true` raising `Errno::ENOENT` if root:/.gitignore didn't exist. I can't think of a use. Now `gitignore: true` is just the default behaviour.

data/README.md

@@ -23,7 +23,7 @@ FastIgnore.new(relative: true).sort == `git ls-files`.split("\n").sort
 - supports [matching by shebang](#shebang_rules) rather than filename for extensionless files: `#!:`
 - reads .gitignore in all subdirectories
 - reads .git/info/excludes
-- reads the ignore file mentioned in your git config
+- reads the global gitignore file mentioned in your git config
 
 ## Installation
 
@@ -141,7 +141,7 @@ FastIgnore.new(root: '../relative/path/to/root').to_a
 
 A relative root will be found relative to the current working directory when the FastIgnore instance is initialized, and that will be the last time the current working directory is relevant.
 
-**Note: Changes to the current working directory (e.g. with `Dir.chdir`), after initialising a FastIgnore instance, will _not_ affect the FastIgnore instance. `root:` will always be what it was when the instance was initialized.**
+**Note: Changes to the current working directory (e.g. with `Dir.chdir`), after initialising a FastIgnore instance, will _not_ affect the FastIgnore instance. `root:` will always be what it was when the instance was initialized, even as a default value.**
 
 ### `gitignore:`
 
@@ -302,34 +302,37 @@ FastIgnore.new.allowed?('relative/path', directory: false, content: "#!/usr/bin/
 ```
 This is not required, and if FastIgnore does have to go to the filesystem for this information it's well optimised to only read what is necessary.
 
-
-## Known issues
+## Limitations
 - Doesn't know what to do if you change the current working directory inside the [`FastIgnore#each`](#each_map_etc) block.
   So don't do that.
 
-  (It does handle changing the current working directory between [`FastIgnore#allowed?`](#allowed) calls) (changing directories doesn't affect the [`root:`](#root) directory, that's frozen at FastIgnore.new (this is a design decision, not an issue)).
+  (It does handle changing the current working directory between [`FastIgnore#allowed?`](#allowed) calls)
 - FastIgnore always matches patterns case-insensitively. (git varies by filesystem).
-
-## Development
-
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake` to run the tests and linters.
-
-You can run `bin/console` for an interactive prompt that will allow you to experiment.
-`bin/ls [argv_rules]` will return something equivalent to `git ls-files` and `bin/time [argv_rules]` will give you the average time for 30 runs.
-This repo is too small to stress bin/time more than 0.01s, switch to a large repo and find the average time before and after changes.
-
-To install this gem onto your local machine, run `bundle exec rake install`.
-
-### Goals
-
-1. Match `git ls-files` behaviour quirk for quirk.
-2. Provide a convenient interface for allowlist/denylist files in ruby.
-3. Be fast.
+- FastIgnore always outputs paths as literal UTF-8 characters. (git depends on your core.quotepath setting but by default outputs non ascii paths with octal escapes surrounded by quotes).
+- Because git looks at its own index objects and FastIgnore looks at the file system there may be some differences between FastIgnore and `git ls-files`
+  - Tracked files that were committed before the matching ignore rule was committed will be returned by `git ls-files`, but not by FastIgnore.
+  - Untracked files will be returned by FastIgnore, but not by `git ls-files`
+  - Deleted files whose deletions haven't been committed will be returned by `git ls-files`, but not by FastIgnore
+  - On a case insensitive file system, with files that differ only by case, `git ls-files` will include all case variations, while FastIgnore will only include whichever variation git placed in the file system.
 
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/robotdana/fast_ignore.
 
+Some tools that may help:
+
+- `bin/setup`: install development dependencies
+- `bundle exec rspec`: run all tests
+- `bundle exec rake`: run all tests and linters
+- `bin/console`: open a `pry` console with everything required for experimenting
+- `bin/ls [argv_rules]`: the equivalent of `git ls-files`
+- `bin/prof/ls [argv_rules]`: ruby-prof report for `bin/ls`
+- `bin/prof/parse [argv_rules]`: ruby-prof report for parsing root and global gitignore files and any arguments.
+- `bin/time [argv_rules]`: the average time for 30 runs of `bin/ls`<br>
+  This repo is too small to stress bin/time more than 0.01s, switch to a large repo and find the average time before and after changes.
+- `bin/compare`: compare the speed and output of FastIgnore and `git ls-files`.
+  (suppressing differences that are because of known [limitations](#limitations))
+
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/fast_ignore.rb

@@ -3,12 +3,15 @@
 require_relative './fast_ignore/backports'
 
 require 'set'
-require_relative './fast_ignore/rule_set_builder'
-require_relative './fast_ignore/rule_builder'
+require 'strscan'
+require_relative './fast_ignore/rule_sets'
 require_relative './fast_ignore/rule_set'
+require_relative './fast_ignore/global_gitignore'
+require_relative './fast_ignore/rule_builder'
+require_relative './fast_ignore/gitignore_rule_builder'
 require_relative './fast_ignore/rule'
+require_relative './fast_ignore/unmatchable_rule'
 require_relative './fast_ignore/shebang_rule'
-require_relative './fast_ignore/fn_match_to_re'
 
 class FastIgnore
   class Error < StandardError; end
@@ -26,7 +29,7 @@ class FastIgnore
     @gitignore_enabled = gitignore
     @loaded_gitignore_files = ::Set[''] if gitignore
     @root = "#{::File.expand_path(root.to_s, Dir.pwd)}/"
-    @rule_sets = ::FastIgnore::RuleSetBuilder.build(root: @root, gitignore: gitignore, **rule_set_builder_args)
+    @rule_sets = ::FastIgnore::RuleSets.new(root: @root, gitignore: gitignore, **rule_set_builder_args)
 
     freeze
   end
@@ -34,7 +37,7 @@ class FastIgnore
   def each(&block)
     return enum_for(:each) unless block_given?
 
-    dir_pwd = Dir.pwd
+    dir_pwd = ::Dir.pwd
     root_from_pwd = @root.start_with?(dir_pwd) ? ".#{@root.delete_prefix(dir_pwd)}" : @root
 
     each_recursive(root_from_pwd, '', &block)
@@ -50,12 +53,16 @@ class FastIgnore
 
     filename = ::File.basename(relative_path)
 
-    @rule_sets.all? { |r| r.allowed_recursive?(relative_path, false, full_path, filename, content) }
+    @rule_sets.allowed_recursive?(relative_path, full_path, filename, content)
   rescue ::Errno::ENOENT, ::Errno::EACCES, ::Errno::ENOTDIR, ::Errno::ELOOP, ::Errno::ENAMETOOLONG
     false
   end
   alias_method :===, :allowed?
 
+  def to_proc
+    method(:allowed?).to_proc
+  end
+
   private
 
   def load_gitignore_recursive(path)
@@ -67,19 +74,17 @@ class FastIgnore
     paths.reverse_each(&method(:load_gitignore))
   end
 
-  def load_gitignore(parent_path, soft: true)
+  def load_gitignore(parent_path, check_exists: true)
     return if @loaded_gitignore_files.include?(parent_path)
 
-    ::FastIgnore::RuleSetBuilder.append_gitignore(
-      @rule_sets, project_root: @root, relative_path: parent_path + '.gitignore', soft: soft
-    )
+    @rule_sets.append_subdir_gitignore(relative_path: parent_path + '.gitignore', check_exists: check_exists)
+
     @loaded_gitignore_files << parent_path
   end
 
   def each_recursive(parent_full_path, parent_relative_path, &block) # rubocop:disable Metrics/MethodLength, Metrics/AbcSize
     children = ::Dir.children(parent_full_path)
-
-    load_gitignore(parent_relative_path, soft: false) if @gitignore_enabled && children.include?('.gitignore')
+    load_gitignore(parent_relative_path, check_exists: false) if @gitignore_enabled && children.include?('.gitignore')
 
     children.each do |filename|
       begin
@@ -87,7 +92,7 @@ class FastIgnore
         relative_path = parent_relative_path + filename
         dir = @follow_symlinks_method.call(full_path).directory?
 
-        next unless @rule_sets.all? { |r| r.allowed_unrecursive?(relative_path, dir, full_path, filename, nil) }
+        next unless @rule_sets.allowed_unrecursive?(relative_path, dir, full_path, filename)
 
         if dir
           each_recursive(full_path + '/', relative_path + '/', &block)

data/lib/fast_ignore/backports.rb

@@ -2,12 +2,12 @@
 
 class FastIgnore
   module Backports
-    ruby_major, ruby_minor = RUBY_VERSION.split('.', 2)
+    ruby_major, ruby_minor = ::RUBY_VERSION.split('.', 2)
     unless ruby_major.to_i > 2 || ruby_major.to_i == 2 && ruby_minor.to_i > 5
       module DirEachChild
         refine ::Dir.singleton_class do
           def children(path)
-            Dir.entries(path) - ['.', '..']
+            ::Dir.entries(path) - ['.', '..']
           end
         end
       end

data/lib/fast_ignore/gitignore_rule_builder.rb

@@ -0,0 +1,212 @@
+# frozen_string_literal: true
+
+class FastIgnore
+  class GitignoreRuleBuilder # rubocop:disable Metrics/ClassLength
+    def initialize(rule, negation, dir_only, file_path, allow) # rubocop:disable Metrics/MethodLength
+      @re = ::String.new
+      @segment_re = ::String.new
+      @allow = allow
+      if @allow
+        @segments = 0
+        @parent_re = ::String.new
+      end
+
+      @s = ::StringScanner.new(rule)
+
+      @dir_only = dir_only
+      @file_path = (file_path if file_path && !file_path.empty?)
+      @negation = negation
+      @anchored = false
+      @trailing_stars = false
+    end
+
+    def process_escaped_char
+      @segment_re << ::Regexp.escape(@s.matched[1]) if @s.scan(/\\./)
+    end
+
+    def process_character_class
+      return unless @s.skip(/\[/)
+
+      @segment_re << '['
+      process_character_class_body(false)
+    end
+
+    def process_negated_character_class
+      return unless @s.skip(/\[\^/)
+
+      @segment_re << '[^'
+      process_character_class_body(true)
+    end
+
+    def unmatchable_rule!
+      throw :unmatchable_rule, (
+        @allow ? ::FastIgnore::UnmatchableRule : []
+      )
+    end
+
+    def process_character_class_end
+      return unless @s.skip(/\]/)
+
+      unmatchable_rule! unless @has_characters_in_group
+
+      @segment_re << ']'
+    end
+
+    def process_character_class_body(negated_class) # rubocop:disable Metrics/MethodLength
+      @has_characters_in_group = false
+      until process_character_class_end
+        if @s.eos?
+          unmatchable_rule!
+        elsif process_escaped_char
+          @has_characters_in_group = true
+        elsif @s.skip(%r{/})
+          next unless negated_class
+
+          @has_characters_in_group = true
+          @segment_re << '/'
+        elsif @s.skip(/-/)
+          @has_characters_in_group = true
+          @segment_re << '-'
+        else @s.scan(%r{[^/\]\-]+})
+             @has_characters_in_group = true
+             @segment_re << ::Regexp.escape(@s.matched)
+        end
+      end
+    end
+
+    def process_star_star_slash
+      return unless @s.skip(%r{\*{2,}/})
+
+      if @allow
+        if @segment_re.empty?
+          @parent_re << '.*'
+        else
+          process_slash_allow('.*')
+        end
+      end
+      process_slash('(?:.*/)?')
+    end
+
+    def process_star_slash
+      return unless @s.skip(%r{\*/})
+
+      process_slash_allow('[^/]*/') if @allow
+      process_slash('[^/]*/')
+    end
+
+    def process_no_star_slash
+      return unless @s.skip(%r{/})
+
+      process_slash_allow('/') if @allow
+      process_slash('/')
+    end
+
+    def process_slash(append)
+      @re << @segment_re
+      @re << append
+      @segment_re.clear
+      @anchored = true
+    end
+
+    def process_slash_allow(append)
+      @segments += 1
+      @parent_re << '(?:'
+      @parent_re << @segment_re
+      @parent_re << append
+    end
+
+    def process_stars
+      (@segment_re << '[^/]*') if @s.scan(%r{\*+(?=[^*/])})
+    end
+
+    def process_question_mark
+      (@segment_re << '[^/]') if @s.skip(/\?/)
+    end
+
+    def process_text
+      (@segment_re << ::Regexp.escape(@s.matched)) if @s.scan(%r{[^*/?\[\\]+})
+    end
+
+    def process_end
+      return unless @s.scan(/\*+\z/)
+
+      if @s.matched.length == 1
+        @segment_re << if @segment_re.empty? # at least something. this is to allow subdir negations to work
+          '[^/]+\\z'
+        else
+          '[^/]*\\z'
+        end
+      end
+      @trailing_stars = true
+    end
+
+    def process_rule
+      until @s.eos?
+        process_escaped_char ||
+          process_star_star_slash || process_star_slash || process_no_star_slash ||
+          process_stars || process_question_mark ||
+          process_negated_character_class || process_character_class ||
+          process_text || process_end
+      end
+    end
+
+    def build # rubocop:disable Metrics/MethodLength, Metrics/AbcSize
+      @anchored = true if @s.skip(%r{/})
+
+      catch :unmatchable_rule do # rubocop:disable Metrics/BlockLength
+        process_rule
+
+        @re << @segment_re
+
+        prefix = if @file_path
+          escaped_file_path = ::Regexp.escape @file_path
+          if @anchored
+            "\\A#{escaped_file_path}"
+          else
+            "\\A#{escaped_file_path}(?:.*/)?"
+          end
+        else
+          if @anchored
+            '\\A'
+          else
+            '(?:\\A|/)'
+          end
+        end
+
+        @re.prepend(prefix)
+        anchored_or_file_path = @anchored || @file_path
+        if @allow
+          if @file_path
+            allow_escaped_file_path = escaped_file_path.gsub(%r{(?<!\\)(?:\\\\)*/}) do |e|
+              @segments += 1
+              "#{e[0..-2]}(?:/"
+            end
+
+            prefix = if @anchored
+              "\\A#{allow_escaped_file_path}"
+            else
+              "\\A#{allow_escaped_file_path}(?:.*/)?"
+            end
+          end
+          @parent_re.prepend(prefix)
+          @parent_re << (')?' * @segments)
+          (@re << '(/|\\z)') unless @dir_only || @trailing_stars
+          rules = [
+            # Regexp::IGNORECASE = 1
+            ::FastIgnore::Rule.new(::Regexp.new(@re, 1), @negation, anchored_or_file_path, @dir_only),
+            ::FastIgnore::Rule.new(::Regexp.new(@parent_re, 1), true, anchored_or_file_path, true)
+          ]
+          if @dir_only
+            (rules << ::FastIgnore::Rule.new(::Regexp.new((@re << '/.*'), 1), @negation, anchored_or_file_path, false))
+          end
+          rules
+        else
+          (@re << '\\z') unless @trailing_stars
+
+          # Regexp::IGNORECASE = 1
+          ::FastIgnore::Rule.new(::Regexp.new(@re, 1), @negation, anchored_or_file_path, @dir_only)
+        end
+      end
+    end
+  end
+end

data/lib/fast_ignore/global_gitignore.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+class FastIgnore
+  module GlobalGitignore
+    class << self
+      def path(root:)
+        gitconfig_gitignore_path(::File.expand_path('.git/config', root)) ||
+          gitconfig_gitignore_path(::File.expand_path('~/.gitconfig')) ||
+          gitconfig_gitignore_path(xdg_config_path) ||
+          gitconfig_gitignore_path('/etc/gitconfig') ||
+          default_global_gitignore_path
+      end
+
+      def gitconfig_gitignore_path(config_path)
+        return unless config_path
+        return unless ::File.exist?(config_path)
+
+        ignore_path = ::File.readlines(config_path).find { |l| l.sub!(/\A\s*excludesfile\s*=/, '') }
+        return unless ignore_path
+
+        ignore_path.strip!
+        return ignore_path if ignore_path.empty? # don't expand path in this case
+
+        ::File.expand_path(ignore_path)
+      end
+
+      def xdg_config_path
+        return unless ::ENV['XDG_CONFIG_HOME'] && !::ENV['XDG_CONFIG_HOME'].empty?
+
+        ::File.expand_path('git/config', ::ENV['XDG_CONFIG_HOME'])
+      end
+
+      def default_global_gitignore_path
+        if ::ENV['XDG_CONFIG_HOME'] && !::ENV['XDG_CONFIG_HOME'].empty?
+          ::File.expand_path('git/ignore', ::ENV['XDG_CONFIG_HOME'])
+        else
+          ::File.expand_path('~/.config/git/ignore')
+        end
+      end
+    end
+  end
+end

data/lib/fast_ignore/rule.rb

@@ -6,24 +6,35 @@ class FastIgnore
     alias_method :negation?, :negation
     undef :negation
 
+    attr_reader :component_rules
+
     attr_reader :dir_only
     alias_method :dir_only?, :dir_only
     undef :dir_only
 
-    attr_reader :unanchored
-    alias_method :unanchored?, :unanchored
-    undef :unanchored
-
-    attr_reader :type
+    attr_reader :squashable_type
     attr_reader :rule
 
-    def initialize(rule, negation, unanchored = nil, dir_only = nil)
-      @rule = rule.is_a?(Regexp) ? rule : ::FastIgnore::FNMatchToRegex.call(rule)
-      @unanchored = unanchored
+    def squash(rules)
+      # component rules is to improve the performance of repos with many .gitignore files. e.g. linux.
+      rules = rules.flat_map(&:component_rules)
+      ::FastIgnore::Rule.new(::Regexp.union(rules.map(&:rule)).freeze, @negation, @anchored, @dir_only, rules)
+    end
+
+    def initialize(rule, negation, anchored, dir_only, component_rules = self) # rubocop:disable Metrics/MethodLength
+      @rule = rule
+      @anchored = anchored
       @dir_only = dir_only
       @negation = negation
+      @component_rules = component_rules
 
-      @type = negation ? 1 : 0
+      @squashable_type = if anchored && negation
+        1
+      elsif anchored
+        0
+      else
+        ::Float::NAN # because it doesn't equal itself
+      end
 
       freeze
     end
@@ -32,13 +43,13 @@ class FastIgnore
       false
     end
 
-    def shebang
-      nil
+    def shebang?
+      false
     end
 
     # :nocov:
     def inspect
-      "#<Rule #{'!' if @negation}#{@rule}#{'/' if @dir_only}>"
+      "#<Rule #{'!' if @negation}#{'/' if @anchored}#{@rule}#{'/' if @dir_only}>"
     end
     # :nocov:
 

data/lib/fast_ignore/rule_builder.rb

@@ -7,13 +7,13 @@ class FastIgnore
       using ::FastIgnore::Backports::DeletePrefixSuffix if defined?(::FastIgnore::Backports::DeletePrefixSuffix)
       # :nocov:
 
-      def build(rule, allow, expand_path, file_root)
+      def build(rule, allow, expand_path_with, file_root)
         return shebang_rules(rule, allow) if remove_shebang(rule)
 
         strip(rule)
         return [] if skip?(rule)
 
-        gitignore_rules(rule, allow, expand_path, file_root)
+        gitignore_rules(rule, allow, expand_path_with, file_root)
       end
 
       private
@@ -32,31 +32,23 @@ class FastIgnore
       end
 
       def shebang_rules(rule, allow)
-        rules = [::FastIgnore::ShebangRule.new(/\A#!.*\b#{Regexp.escape(rule)}\b/i.freeze, allow)]
-        return rules unless allow
+        rule = ::FastIgnore::ShebangRule.new(/\A#!.*\b#{::Regexp.escape(rule)}\b/i, allow)
+        return rule unless allow
 
-        rules << ::FastIgnore::Rule.new('**/*', true, true, true)
-        rules
+        [::FastIgnore::Rule.new(//, true, true, true), rule]
       end
 
       def skip?(rule)
         rule.empty? || rule.start_with?('#')
       end
 
-      def gitignore_rules(rule, allow, expand_path, file_root)
+      def gitignore_rules(rule, allow, expand_path_with, file_root)
         dir_only = extract_dir_only(rule)
         negation = extract_negation(rule, allow)
 
-        unanchored = if expand_path
-          expand_rule_path(rule, expand_path)
-        else
-          unanchored?(rule)
-        end
-        rule.delete_prefix!('/')
+        expand_rule_path(rule, expand_path_with) if expand_path_with
 
-        rule.prepend("#{file_root}#{'**/' if unanchored}") if file_root || unanchored
-
-        build_gitignore_rules(rule, unanchored, allow, dir_only, negation)
+        ::FastIgnore::GitignoreRuleBuilder.new(rule, negation, dir_only, file_root, allow).build
       end
 
       def extract_dir_only(rule)
@@ -69,34 +61,11 @@ class FastIgnore
         not allow
       end
 
-      EXPAND_PATH_RE = %r{^(?:[~/]|\.{1,2}/)}.freeze
+      EXPAND_PATH_RE = %r{(^(?:[~/]|\.{1,2}/)|/\.\./)}.freeze
       def expand_rule_path(rule, root)
         rule.replace(::File.expand_path(rule)) if rule.match?(EXPAND_PATH_RE)
         rule.delete_prefix!(root)
-        rule.start_with?('*')
-      end
-
-      def unanchored?(rule)
-        not rule.include?('/') # we've already removed the trailing '/' with extract_dir_only
-      end
-
-      def build_gitignore_rules(rule, unanchored, allow, dir_only, negation)
-        rules = [::FastIgnore::Rule.new(rule, negation, unanchored, dir_only)]
-        return rules unless allow
-
-        rules << ::FastIgnore::Rule.new("#{rule}/**/*", negation, unanchored, false)
-        rules + ancestor_rules(rule, unanchored)
-      end
-
-      def ancestor_rules(parent, unanchored)
-        ancestor_rules = []
-
-        while (parent = ::File.dirname(parent)) != '.'
-          rule = ::File.basename(parent) == '**' ? "#{parent}/*" : parent.freeze
-          ancestor_rules << ::FastIgnore::Rule.new(rule, true, unanchored, true)
-        end
-
-        ancestor_rules
+        rule.prepend('/') unless rule.start_with?('*')
       end
     end
   end

data/lib/fast_ignore/rule_set.rb

@@ -9,8 +9,7 @@ class FastIgnore
     def initialize(rules, allow, gitignore)
       @dir_rules = squash_rules(rules.reject(&:file_only?)).freeze
       @file_rules = squash_rules(rules.reject(&:dir_only?)).freeze
-      @any_not_anchored = rules.any?(&:unanchored?)
-      @has_shebang_rules = rules.any?(&:shebang)
+      @has_shebang_rules = rules.any?(&:shebang?)
 
       @allowed_recursive = { '.' => true }
       @allow = allow
@@ -22,10 +21,9 @@ class FastIgnore
     def <<(other)
       return unless other
 
-      @any_not_anchored ||= other.any_not_anchored
       @has_shebang_rules ||= other.has_shebang_rules
-      @dir_rules += other.dir_rules
-      @file_rules += other.file_rules
+      @dir_rules = squash_rules(@dir_rules + other.dir_rules)
+      @file_rules = squash_rules(@file_rules + other.file_rules)
     end
 
     def allowed_recursive?(relative_path, dir, full_path, filename, content = nil)
@@ -41,18 +39,16 @@ class FastIgnore
         return rule.negation? if rule.match?(relative_path, full_path, filename, content)
       end
 
-      (not @allow) || (dir && @any_not_anchored)
+      not @allow
     end
 
     def squash_rules(rules)
-      out = rules.chunk_while { |a, b| a.type == b.type }.map do |chunk|
+      rules.chunk_while { |a, b| a.squashable_type == b.squashable_type }.map do |chunk|
         first = chunk.first
         next first if chunk.length == 1
 
-        first.class.new(Regexp.union(chunk.map(&:rule)), first.negation?)
+        first.squash(chunk)
       end
-
-      out
     end
 
     def weight
@@ -65,6 +61,6 @@ class FastIgnore
 
     protected
 
-    attr_reader :dir_rules, :file_rules, :any_not_anchored, :has_shebang_rules
+    attr_reader :dir_rules, :file_rules, :has_shebang_rules
   end
 end

data/lib/fast_ignore/rule_sets.rb

@@ -0,0 +1,112 @@
+# frozen_string_literal: true
+
+class FastIgnore
+  class RuleSets
+    # :nocov:
+    using ::FastIgnore::Backports::DeletePrefixSuffix if defined?(::FastIgnore::Backports::DeletePrefixSuffix)
+    # :nocov:
+
+    def initialize( # rubocop:disable Metrics/ParameterLists
+      root:,
+      ignore_rules: nil,
+      ignore_files: nil,
+      gitignore: true,
+      include_rules: nil,
+      include_files: nil,
+      argv_rules: nil
+    )
+      @array = []
+      @project_root = root
+      append_root_gitignore(gitignore)
+      append_set_from_array(ignore_rules)
+      append_set_from_array(include_rules, allow: true)
+      append_set_from_array(argv_rules, allow: true, expand_path_with: @project_root)
+      append_sets_from_files(ignore_files)
+      append_sets_from_files(include_files, allow: true)
+      @array.sort_by!(&:weight)
+      @array.freeze if @gitignore_rule_set
+    end
+
+    def allowed_recursive?(relative_path, full_path, filename, content)
+      @array.all? { |r| r.allowed_recursive?(relative_path, false, full_path, filename, content) }
+    end
+
+    def allowed_unrecursive?(relative_path, dir, full_path, filename)
+      @array.all? { |r| r.allowed_unrecursive?(relative_path, dir, full_path, filename, nil) }
+    end
+
+    def append_subdir_gitignore(relative_path:, check_exists: true)
+      new_gitignore = build_set_from_file(relative_path, gitignore: true, check_exists: check_exists)
+      return if !new_gitignore || new_gitignore.empty?
+
+      if @gitignore_rule_set
+        @gitignore_rule_set << new_gitignore
+      else
+        @array << new_gitignore
+        @gitignore_rule_set = new_gitignore
+        @array.sort_by!(&:weight) && @array.freeze
+      end
+      new_gitignore
+    end
+
+    private
+
+    def append_and_return_if_present(value)
+      return unless value && !value.empty?
+
+      @array << value
+      value
+    end
+
+    def append_root_gitignore(gitignore)
+      return @gitignore_rule_set = nil unless gitignore
+
+      append_set_from_array('.git')
+      gi = ::FastIgnore::RuleSet.new([], false, true)
+      gi << build_from_root_gitignore_file(::FastIgnore::GlobalGitignore.path(root: @project_root))
+      gi << build_from_root_gitignore_file("#{@project_root}.git/info/exclude")
+      gi << build_from_root_gitignore_file("#{@project_root}.gitignore")
+      @gitignore_rule_set = append_and_return_if_present(gi)
+    end
+
+    def build_from_root_gitignore_file(path)
+      return unless ::File.exist?(path)
+
+      build_rule_set(::File.readlines(path), false, gitignore: true)
+    end
+
+    def build_rule_set(rules, allow, expand_path_with: nil, file_root: nil, gitignore: false)
+      rules = rules.flat_map do |rule|
+        ::FastIgnore::RuleBuilder.build(rule, allow, expand_path_with, file_root)
+      end
+
+      ::FastIgnore::RuleSet.new(rules, allow, gitignore)
+    end
+
+    def build_set_from_file(filename, allow: false, file_root: nil, gitignore: false, check_exists: false)
+      filename = ::File.expand_path(filename, @project_root)
+      return if check_exists && !::File.exist?(filename)
+      unless file_root || filename.start_with?(@project_root)
+        raise ::FastIgnore::Error, "#{filename} is not within #{@project_root}"
+      end
+
+      file_root ||= "#{::File.dirname(filename)}/".delete_prefix(@project_root)
+      build_rule_set(::File.readlines(filename), allow, file_root: file_root, gitignore: gitignore)
+    end
+
+    def append_sets_from_files(files, allow: false)
+      Array(files).each do |file|
+        append_and_return_if_present(build_set_from_file(file, allow: allow))
+      end
+    end
+
+    def append_set_from_array(rules, allow: false, expand_path_with: nil)
+      return unless rules
+
+      rules = Array(rules).flat_map { |string| string.to_s.lines }
+      return if rules.empty?
+
+      append_and_return_if_present(build_rule_set(rules, allow, expand_path_with: expand_path_with))
+    end
+  end
+end

data/lib/fast_ignore/shebang_rule.rb

@@ -7,15 +7,18 @@ class FastIgnore
     undef :negation
 
     attr_reader :rule
-    alias_method :shebang, :rule
 
-    attr_reader :type
+    attr_reader :squashable_type
+
+    def squash(rules)
+      ::FastIgnore::ShebangRule.new(::Regexp.union(rules.map(&:rule)).freeze, negation?)
+    end
 
     def initialize(rule, negation)
       @rule = rule
       @negation = negation
 
-      @type = negation ? 3 : 2
+      @squashable_type = negation ? 3 : 2
 
       freeze
     end
@@ -28,10 +31,6 @@ class FastIgnore
       false
     end
 
-    def unanchored?
-      true
-    end
-
     # :nocov:
     def inspect
       "#<ShebangRule #{'allow ' if @negation}#!:#{@rule.to_s[15..-4]}>"
@@ -44,12 +43,24 @@ class FastIgnore
       (content || first_line(full_path))&.match?(@rule)
     end
 
+    def shebang?
+      true
+    end
+
     private
 
-    def first_line(path)
+    def first_line(path) # rubocop:disable Metrics/MethodLength
       file = ::File.new(path)
-      first_line = file.sysread(25)
-      first_line += file.sysread(50) until first_line.include?("\n")
+      first_line = new_fragment = file.sysread(64)
+      if first_line.start_with?('#!')
+        until new_fragment.include?("\n")
+          new_fragment = file.sysread(64)
+          first_line += new_fragment
+        end
+      else
+        file.close
+        return
+      end
       file.close
       first_line
     rescue ::EOFError, ::SystemCallError

data/lib/fast_ignore/unmatchable_rule.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+class FastIgnore
+  class UnmatchableRule
+    class << self
+      def squash(_)
+        self
+      end
+
+      def squashable_type
+        5
+      end
+
+      def dir_only?
+        false
+      end
+
+      def file_only?
+        false
+      end
+
+      def shebang?
+        false
+      end
+
+      # :nocov:
+      def inspect
+        '#<UnmatchableRule>'
+      end
+      # :nocov:
+
+      def match?(_, _, _, _)
+        false
+      end
+    end
+  end
+end

data/lib/fast_ignore/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class FastIgnore
-  VERSION = '0.13.0'
+  VERSION = '0.14.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fast_ignore
 version: !ruby/object:Gem::Version
-  version: 0.13.0
+  version: 0.14.0
 platform: ruby
 authors:
 - Dana Sherson
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-06-06 00:00:00.000000000 Z
+date: 2020-06-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -148,12 +148,14 @@ files:
 - README.md
 - lib/fast_ignore.rb
 - lib/fast_ignore/backports.rb
-- lib/fast_ignore/fn_match_to_re.rb
+- lib/fast_ignore/gitignore_rule_builder.rb
+- lib/fast_ignore/global_gitignore.rb
 - lib/fast_ignore/rule.rb
 - lib/fast_ignore/rule_builder.rb
 - lib/fast_ignore/rule_set.rb
-- lib/fast_ignore/rule_set_builder.rb
+- lib/fast_ignore/rule_sets.rb
 - lib/fast_ignore/shebang_rule.rb
+- lib/fast_ignore/unmatchable_rule.rb
 - lib/fast_ignore/version.rb
 homepage: https://github.com/robotdana/fast_ignore
 licenses:

data/lib/fast_ignore/fn_match_to_re.rb

@@ -1,96 +0,0 @@
-# frozen_string_literal: true
-
-class FastIgnore
-  module FNMatchToRegex
-    # This doesn't look rubyish because i ported it from rust (the only rust i ever wrote that worked)
-    class << self
-      def call(pattern) # rubocop:disable Metrics/MethodLength, Metrics/AbcSize
-        re = '\\A'.dup
-
-        in_character_group = false
-        has_characters_in_group = false
-        escape_next_character = false
-        last_char_opened_character_group = false
-        negated_character_group = false
-        stars = 0
-
-        pattern.each_char do |char| # rubocop:disable Metrics/BlockLength
-          if escape_next_character
-            re << Regexp.escape(char)
-            escape_next_character = false
-          elsif char == '\\' # single char, just needs to be escaped
-            escape_next_character = true
-          elsif in_character_group
-            if char == '/'
-              if negated_character_group
-                has_characters_in_group = true
-                re << char
-              end
-            elsif char == '^'
-              if last_char_opened_character_group
-                re << char
-                negated_character_group = true
-              else
-                re << '\\^'
-                has_characters_in_group = true
-              end
-              # not characters in group
-            elsif char == ']'
-              break unless has_characters_in_group
-
-              re << ']'
-              in_character_group = false
-              has_characters_in_group = false
-              negated_character_group = false
-              last_char_opened_character_group = false
-            elsif char == '-'
-              has_characters_in_group = true
-              re << char
-            else
-              has_characters_in_group = true
-              re << Regexp.escape(char)
-            end
-            last_char_opened_character_group = false
-          elsif char == '*'
-            stars += 1
-          elsif char == '/'
-            re << if stars >= 2
-              '(?:.*/)?'
-            elsif stars.positive?
-              '[^/]*/'
-            else
-              char
-            end
-            stars = 0
-          else
-            if stars.positive?
-              re << '[^/]*'
-              stars = 0
-            end
-            if char == '?'
-              re << '[^/]'
-            elsif char == '['
-              re << '['
-              in_character_group = true
-              last_char_opened_character_group = true
-            else
-              re << Regexp.escape(char)
-            end
-          end
-        end
-
-        if in_character_group
-          return /(?!)/ # impossible to match anything
-        end
-
-        if stars >= 2
-          re << '.*'
-        elsif stars.positive?
-          re << '[^/]*'
-        end
-        re << '\\z'
-        Regexp.new(re, Regexp::IGNORECASE)
-      end
-    end
-  end
-end

data/lib/fast_ignore/rule_set_builder.rb

@@ -1,141 +0,0 @@
-# frozen_string_literal: true
-
-class FastIgnore
-  module RuleSetBuilder # rubocop:disable Metrics/ModuleLength
-    class << self
-      # :nocov:
-      using ::FastIgnore::Backports::DeletePrefixSuffix if defined?(::FastIgnore::Backports::DeletePrefixSuffix)
-      # :nocov:
-
-      def build( # rubocop:disable Metrics/ParameterLists
-        root:,
-        ignore_rules: nil,
-        ignore_files: nil,
-        gitignore: true,
-        include_rules: nil,
-        include_files: nil,
-        argv_rules: nil
-      )
-        prepare [
-          from_array(ignore_rules),
-          *from_files(ignore_files, project_root: root),
-          (from_array('.git') if gitignore),
-          from_gitignore_arg(gitignore, project_root: root),
-          from_array(include_rules, allow: true),
-          *from_files(include_files, allow: true, project_root: root),
-          from_array(argv_rules, allow: true, expand_path: root)
-        ]
-      end
-
-      def append_gitignore(rule_sets, project_root:, relative_path:, soft: true)
-        new_gitignore = from_file(relative_path, project_root: project_root, gitignore: true, soft: soft)
-        return unless new_gitignore
-
-        base_gitignore = rule_sets.find(&:gitignore?)
-        if base_gitignore
-          base_gitignore << new_gitignore
-        else
-          rule_sets << new_gitignore
-          prepare(rule_sets)
-        end
-      end
-
-      private
-
-      def prepare(rule_sets)
-        rule_sets.compact!
-        rule_sets.reject!(&:empty?)
-        rule_sets.sort_by!(&:weight)
-        rule_sets
-      end
-
-      def from_file(filename, project_root:, allow: false, file_root: nil, gitignore: false, soft: false) # rubocop:disable Metrics/ParameterLists
-        filename = ::File.expand_path(filename, project_root)
-        return if soft && !::File.exist?(filename)
-        unless file_root || filename.start_with?(project_root)
-          raise FastIgnore::Error, "#{filename} is not within #{project_root}"
-        end
-
-        file_root ||= "#{::File.dirname(filename)}/".delete_prefix(project_root)
-        build_rule_set(::File.readlines(filename), allow, file_root: file_root, gitignore: gitignore)
-      end
-
-      def from_files(files, project_root:, allow: false)
-        Array(files).map do |file|
-          from_file(file, project_root: project_root, allow: allow)
-        end
-      end
-
-      def from_gitignore_arg(gitignore, project_root:)
-        return unless gitignore
-
-        gi = ::FastIgnore::RuleSet.new([], false, true)
-        gi << from_root_gitignore_file(global_gitignore_path(root: project_root))
-        gi << from_root_gitignore_file(::File.join(project_root, '.git/info/exclude'))
-        gi << from_root_gitignore_file(::File.join(project_root, '.gitignore'))
-        gi
-      end
-
-      def from_root_gitignore_file(path)
-        return unless ::File.exist?(path)
-
-        build_rule_set(::File.readlines(path), false, file_root: '', gitignore: true)
-      end
-
-      def global_gitignore_path(root:)
-        gitconfig_gitignore_path(::File.expand_path('.git/config', root)) ||
-          gitconfig_gitignore_path(::File.expand_path('~/.gitconfig')) ||
-          gitconfig_gitignore_path(xdg_config_path) ||
-          gitconfig_gitignore_path('/etc/gitconfig') ||
-          default_global_gitignore_path
-      end
-
-      def gitconfig_gitignore_path(config_path)
-        return unless config_path
-        return unless ::File.exist?(config_path)
-
-        ignore_path = ::File.readlines(config_path).find { |l| l.sub!(/\A\s*excludesfile\s*=/, '') }
-        return unless ignore_path
-
-        ignore_path.strip!
-        return ignore_path if ignore_path.empty? # don't expand path in this case
-
-        ::File.expand_path(ignore_path)
-      end
-
-      def xdg_config_path
-        return unless ENV['XDG_CONFIG_HOME'] && !ENV['XDG_CONFIG_HOME'].empty?
-
-        ::File.expand_path('git/config', ENV['XDG_CONFIG_HOME'])
-      end
-
-      def default_global_gitignore_path
-        if ENV['XDG_CONFIG_HOME'] && !ENV['XDG_CONFIG_HOME'].empty?
-          ::File.expand_path('git/ignore', ENV['XDG_CONFIG_HOME'])
-        else
-          ::File.expand_path('~/.config/git/ignore')
-        end
-      end
-
-      def from_array(rules, allow: false, expand_path: false)
-        return unless rules
-
-        rules = Array(rules)
-
-        return if rules.empty?
-
-        rules = rules.flat_map { |string| string.to_s.lines }
-
-        build_rule_set(rules, allow, expand_path: expand_path)
-      end
-
-      def build_rule_set(rules, allow, expand_path: false, file_root: nil, gitignore: false)
-        rules = rules.flat_map do |rule|
-          ::FastIgnore::RuleBuilder.build(rule, allow, expand_path, file_root)
-        end
-
-        ::FastIgnore::RuleSet.new(rules, allow, gitignore)
-      end
-    end
-  end
-end