regexp_parser 1.4.0 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0ff2b9541be8d00d5a0f8a355ebb9ab6bc5bc2ac50ffa14df13144bf2d239b42
-  data.tar.gz: b5d4c720eaa3606a7973b110251a5fb1fe87e11714fed5a195908678098a4cbe
+  metadata.gz: 707834a32bc2295b448953730eabddabb11bafc68fdf3148174f7be61d8b1f30
+  data.tar.gz: 72d199d28c342d6aae178a5876a7df6f59abcdf40c6ac4f05ea9dc40d16d9f3a
 SHA512:
-  metadata.gz: e8759d373fdea7bbd455a5e2ff96ce1a64cb81f35c325fad49a886a99388897486c1904a847a072b14b245e6da0dded81c3ef031e74944b2fe5d8c67a4cffaab
-  data.tar.gz: 6e39afe8a277eced992c0508a99d022dba939401925e6e3e793cab364d0b3b2143cfade3a433b9d0445c49004dad781dc04f7a99c69229c471dff7095823d065
+  metadata.gz: 4c1402afedc1efb79f633ee93065598b64732519ba587ca3f682eb8bbb4aaa264e31dd916b95f9751f7c6e85e867efa260501b40e55409327efa5b769346a183
+  data.tar.gz: 6335cbc411b08adb64bfca9646eebc3a5c39d4651a2495d34f87fca21927da3a363fc320159a3732cdd9e2d8732986190fcd6c9d523b7308531f91848951ccbd

data/CHANGELOG.md

@@ -1,5 +1,31 @@
 ## [Unreleased]
 
+### [1.5.0] - 2019-05-14 - [Janosch Müller](mailto:janosch84@gmail.com)
+
+### Added
+
+- Added `#referenced_expression` for backrefs, subexp calls and conditionals
+  * returns the `Group` expression that is being referenced via name or number
+- Added `Expression#repetitions`
+  * returns a `Range` of allowed repetitions (`1..1` if there is no quantifier)
+  * like `#quantity` but with a more uniform interface
+- Added `Expression#match_length`
+  * allows to inspect and iterate over String lengths matched by the Expression
+
+### Fixed
+
+- Fixed `Expression#clone` "direction"
+  * it used to dup ivars onto the callee, leaving only the clone referencing the original objects
+  * this will affect you if you call `#eql?`/`#equal?` on expressions or use them as Hash keys
+- Fixed `#clone` results for `Sequences`, e.g. alternations and conditionals
+  * the inner `#text` was cloned onto the `Sequence` and thus duplicated
+  * e.g. `Regexp::Parser.parse(/(a|bc)/).clone.to_s # => (aa|bcbc)`
+- Fixed inconsistent `#to_s` output for `Sequences`
+  * it used to return only the "specific" text, e.g. "|" for an alternation
+  * now it includes nested expressions as it does for all other `Subexpressions`
+- Fixed quantification of codepoint lists with more than one entry (`\u{62 63 64}+`)
+  * quantifiers apply only to the last entry, so this token is now split up if quantified
+
 ### [1.4.0] - 2019-04-02 - [Janosch Müller](mailto:janosch84@gmail.com)
 
 ### Added
@@ -75,7 +101,7 @@ This release includes several breaking changes, mostly to character sets, #map a
 - Changed `(?m)` and the likes to emit as `:options_switch` token (@4ade4d1)
   * allows differentiating from group-local `:options`, e.g. `(?m:.)`
 - Changed name of `Backreference::..NestLevel` to `..RecursionLevel` (@4184339)
-- Changed B`ackreference::Number#number` from `String` to `Integer` (@40a2231)
+- Changed `Backreference::Number#number` from `String` to `Integer` (@40a2231)
 
 ### Added
 

data/Gemfile

@@ -5,5 +5,5 @@ gemspec
 group :development, :test do
   gem 'rake'
   gem 'regexp_property_values'
-  gem 'test-unit'
+  gem 'rspec'
 end

data/README.md

@@ -391,28 +391,22 @@ To run the tests simply run rake from the root directory, as 'test' is the defau
 
 It generates the scanner's code from the Ragel source files and runs all the tests, thus it requires Ragel to be installed.
 
-The tests use Ruby's test/unit. They can also be run with:
+The tests use RSpec. They can also be run with the test runner that whitelists some warnings:
 
 ```
 bin/test
 ```
 
-The test runner accepts all arguments accepted by test/unit.  You can run a specific test like so:
+You can run a specific test like so:
 
 ```
-bin/test test/scanner/test_properties.rb
+bin/test spec/scanner/properties_spec.rb
 ```
 
-It is sometimes helpful during development to focus on a specific test case, for example:
+Note that changes to Ragel files will not be reflected when running `rspec` or `bin/test`, so you might want to run:
 
 ```
-bin/test test/expression/test_base.rb -n test_expression_to_re
-```
-
-Note that changes to Ragel files will not be reflected when using `bin/test`, so you might want to run:
-
-```
-rake ragel:rb && bin/test test/scanner/test_properties.rb
+rake ragel:rb && bin/test spec/scanner/properties_spec.rb
 ```
 
 ## Building
@@ -440,7 +434,9 @@ Projects using regexp_parser.
 
 - [meta_re](https://github.com/ammar/meta_re) is a regular expression preprocessor with alias support.
 
-- [mutant](https://github.com/mbj/mutant) manipulates your regular expressions (amongst others) to see if your tests cover their behavior.
+- [mutant](https://github.com/mbj/mutant) (before v0.9.0) manipulates your regular expressions (amongst others) to see if your tests cover their behavior.
+
+- [twitter-cldr-rb](https://github.com/twitter/twitter-cldr-rb) uses regexp_parser to generate examples of postal codes.
 
 - [js_regex](https://github.com/janosch-x/js_regex) converts Ruby regular expressions to JavaScript-compatible regular expressions.
 
@@ -471,4 +467,4 @@ Documentation and books used while working on this project.
 
 ---
 ##### Copyright
-_Copyright (c) 2010-2016 Ammar Ali. See LICENSE file for details._
+_Copyright (c) 2010-2019 Ammar Ali. See LICENSE file for details._

data/lib/regexp_parser/expression.rb

@@ -21,10 +21,10 @@ module Regexp::Expression
       self.options           = options
     end
 
-    def initialize_clone(other)
-      other.text       = (text        ? text.dup         : nil)
-      other.options    = (options     ? options.dup      : nil)
-      other.quantifier = (quantifier  ? quantifier.clone : nil)
+    def initialize_clone(orig)
+      self.text       = (orig.text       ? orig.text.dup         : nil)
+      self.options    = (orig.options    ? orig.options.dup      : nil)
+      self.quantifier = (orig.quantifier ? orig.quantifier.clone : nil)
       super
     end
 
@@ -62,15 +62,28 @@ module Regexp::Expression
       self.quantifier = Quantifier.new(token, text, min, max, mode)
     end
 
+    def unquantified_clone
+      clone.tap { |exp| exp.quantifier = nil }
+    end
+
     def quantified?
       !quantifier.nil?
     end
 
+    # Deprecated. Prefer `#repetitions` which has a more uniform interface.
     def quantity
       return [nil,nil] unless quantified?
       [quantifier.min, quantifier.max]
     end
 
+    def repetitions
+      return 1..1 unless quantified?
+      min = quantifier.min
+      max = quantifier.max < 0 ? Float::INFINITY : quantifier.max
+      # fix Range#minmax - https://bugs.ruby-lang.org/issues/15807
+      (min..max).tap { |r| r.define_singleton_method(:minmax) { [min, max] } }
+    end
+
     def greedy?
       quantified? and quantifier.greedy?
     end
@@ -101,22 +114,20 @@ module Regexp::Expression
     alias :x? :free_spacing?
     alias :extended? :free_spacing?
 
-    if RUBY_VERSION >= '2.0'
-      def default_classes?
-        options[:d] == true
-      end
-      alias :d? :default_classes?
+    def default_classes?
+      options[:d] == true
+    end
+    alias :d? :default_classes?
 
-      def ascii_classes?
-        options[:a] == true
-      end
-      alias :a? :ascii_classes?
+    def ascii_classes?
+      options[:a] == true
+    end
+    alias :a? :ascii_classes?
 
-      def unicode_classes?
-        options[:u] == true
-      end
-      alias :u? :unicode_classes?
+    def unicode_classes?
+      options[:u] == true
     end
+    alias :u? :unicode_classes?
 
     def matches?(string)
       Regexp.new(to_s) =~ string ? true : false
@@ -161,10 +172,6 @@ module Regexp::Expression
 
 end # module Regexp::Expression
 
-require 'regexp_parser/expression/methods/tests'
-require 'regexp_parser/expression/methods/traverse'
-require 'regexp_parser/expression/methods/strfregexp'
-
 require 'regexp_parser/expression/quantifier'
 require 'regexp_parser/expression/subexpression'
 require 'regexp_parser/expression/sequence'
@@ -186,3 +193,8 @@ require 'regexp_parser/expression/classes/set'
 require 'regexp_parser/expression/classes/set/intersection'
 require 'regexp_parser/expression/classes/set/range'
 require 'regexp_parser/expression/classes/type'
+
+require 'regexp_parser/expression/methods/match_length'
+require 'regexp_parser/expression/methods/strfregexp'
+require 'regexp_parser/expression/methods/tests'
+require 'regexp_parser/expression/methods/traverse'

data/lib/regexp_parser/expression/classes/backref.rb

@@ -1,9 +1,12 @@
 module Regexp::Expression
   module Backreference
-    class Base < Regexp::Expression::Base; end
+    class Base < Regexp::Expression::Base
+      attr_accessor :referenced_expression
+    end
 
     class Number < Backreference::Base
       attr_reader :number
+      alias reference number
 
       def initialize(token, options = {})
         @number = token.text[token.token.equal?(:number) ? 1..-1 : 3..-2].to_i
@@ -13,6 +16,7 @@ module Regexp::Expression
 
     class Name < Backreference::Base
       attr_reader :name
+      alias reference name
 
       def initialize(token, options = {})
         @name = token.text[3..-2]
@@ -20,27 +24,31 @@ module Regexp::Expression
       end
     end
 
+    class NumberRelative     < Backreference::Number
+      attr_accessor :effective_number
+      alias reference effective_number
+    end
+
     class NumberCall         < Backreference::Number; end
-    class NumberRelative     < Backreference::Number; end
-    class NumberCallRelative < Backreference::Number; end
-    class NameCall < Backreference::Name; end
+    class NameCall           < Backreference::Name; end
+    class NumberCallRelative < Backreference::NumberRelative; end
 
-    class NumberRecursionLevel < Backreference::Base
-      attr_reader :number, :recursion_level
+    class NumberRecursionLevel < Backreference::Number
+      attr_reader :recursion_level
 
       def initialize(token, options = {})
-        @number, @recursion_level = token.text[3..-2].split(/(?=[+-])/).map(&:to_i)
         super
+        @number, @recursion_level = token.text[3..-2].split(/(?=[+-])/).map(&:to_i)
       end
     end
 
-    class NameRecursionLevel < Backreference::Base
-      attr_reader :name, :recursion_level
+    class NameRecursionLevel < Backreference::Name
+      attr_reader :recursion_level
 
       def initialize(token, options = {})
+        super
         @name, recursion_level = token.text[3..-2].split(/(?=[+-])/)
         @recursion_level = recursion_level.to_i
-        super
       end
     end
   end

data/lib/regexp_parser/expression/classes/conditional.rb

@@ -7,6 +7,8 @@ module Regexp::Expression
     end
 
     class Condition < Regexp::Expression::Base
+      attr_accessor :referenced_expression
+
       # Name or number of the referenced capturing group that determines state.
       # Returns a String if reference is by name, Integer if by number.
       def reference
@@ -18,6 +20,8 @@ module Regexp::Expression
     class Branch < Regexp::Expression::Sequence; end
 
     class Expression < Regexp::Expression::Subexpression
+      attr_accessor :referenced_expression
+
       def <<(exp)
         expressions.last << exp
       end

data/lib/regexp_parser/expression/classes/group.rb

@@ -19,20 +19,22 @@ module Regexp::Expression
 
     class Capture < Group::Base
       attr_accessor :number, :number_at_level
+      alias identifier number
 
       def capturing?; true end
     end
 
     class Named < Group::Capture
       attr_reader :name
+      alias identifier name
 
       def initialize(token, options = {})
         @name = token.text[3..-2]
         super
       end
 
-      def initialize_clone(other)
-        other.instance_variable_set(:@name, name.dup)
+      def initialize_clone(orig)
+        @name = orig.name.dup
         super
       end
     end

data/lib/regexp_parser/expression/classes/keep.rb

@@ -1,7 +1,5 @@
 module Regexp::Expression
-
   module Keep
-    class Mark          < Regexp::Expression::Base; end
+    class Mark < Regexp::Expression::Base; end
   end
-
 end

data/lib/regexp_parser/expression/methods/match_length.rb

@@ -0,0 +1,172 @@
+class Regexp::MatchLength
+  include Enumerable
+
+  def self.of(obj)
+    exp = obj.is_a?(Regexp::Expression::Base) ? obj : Regexp::Parser.parse(obj)
+    exp.match_length
+  end
+
+  def initialize(exp, opts = {})
+    self.exp_class = exp.class
+    self.min_rep = exp.repetitions.min
+    self.max_rep = exp.repetitions.max
+    if base = opts[:base]
+      self.base_min = base
+      self.base_max = base
+      self.reify = ->{ '.' * base }
+    else
+      self.base_min = opts.fetch(:base_min)
+      self.base_max = opts.fetch(:base_max)
+      self.reify = opts.fetch(:reify)
+    end
+  end
+
+  def each(opts = {})
+    return enum_for(__method__) unless block_given?
+    limit = opts[:limit] || 1000
+    yielded = 0
+    (min..max).each do |num|
+      next unless include?(num)
+      yield(num)
+      break if (yielded += 1) >= limit
+    end
+  end
+
+  def endless_each(&block)
+    return enum_for(__method__) unless block_given?
+    (min..max).each { |num| yield(num) if include?(num) }
+  end
+
+  def include?(length)
+    test_regexp.match?('X' * length)
+  end
+
+  def fixed?
+    min == max
+  end
+
+  def min
+    min_rep * base_min
+  end
+
+  def max
+    max_rep * base_max
+  end
+
+  def minmax
+    [min, max]
+  end
+
+  def inspect
+    type = exp_class.name.sub('Regexp::Expression::', '')
+    "#<#{self.class}<#{type}> min=#{min} max=#{max}>"
+  end
+
+  def to_re
+    "(?:#{reify.call}){#{min_rep},#{max_rep unless max_rep == Float::INFINITY}}"
+  end
+
+  private
+
+  attr_accessor :base_min, :base_max, :min_rep, :max_rep, :exp_class, :reify
+
+  def test_regexp
+    @test_regexp ||= Regexp.new("^#{to_re}$").tap do |regexp|
+      regexp.respond_to?(:match?) || def regexp.match?(str); !!match(str) end
+    end
+  end
+end
+
+module Regexp::Expression
+  MatchLength = Regexp::MatchLength
+
+  [
+    CharacterSet,
+    CharacterSet::Intersection,
+    CharacterSet::IntersectedSequence,
+    CharacterSet::Range,
+    CharacterType::Base,
+    EscapeSequence::Base,
+    PosixClass,
+    UnicodeProperty::Base,
+  ].each do |klass|
+    klass.class_eval <<-RUBY, __FILE__, __LINE__ + 1
+      def match_length
+        MatchLength.new(self, base: 1)
+      end
+    RUBY
+  end
+
+  class Literal
+    def match_length
+      MatchLength.new(self, base: text.length)
+    end
+  end
+
+  class Subexpression
+    def match_length
+      MatchLength.new(self,
+                       base_min: map { |exp| exp.match_length.min }.inject(0, :+),
+                       base_max: map { |exp| exp.match_length.max }.inject(0, :+),
+                       reify: ->{ map { |exp| exp.match_length.to_re }.join })
+    end
+
+    def inner_match_length
+      dummy = Regexp::Expression::Root.build
+      dummy.expressions = expressions.map(&:clone)
+      dummy.quantifier = quantifier && quantifier.clone
+      dummy.match_length
+    end
+  end
+
+  [
+    Alternation,
+    Conditional::Expression,
+  ].each do |klass|
+    klass.class_eval <<-RUBY, __FILE__, __LINE__ + 1
+      def match_length
+        MatchLength.new(self,
+                         base_min: map { |exp| exp.match_length.min }.min,
+                         base_max: map { |exp| exp.match_length.max }.max,
+                         reify: ->{ map { |exp| exp.match_length.to_re }.join('|') })
+      end
+    RUBY
+  end
+
+  [
+    Anchor::Base,
+    Assertion::Base,
+    Conditional::Condition,
+    FreeSpace,
+    Keep::Mark,
+  ].each do |klass|
+    klass.class_eval <<-RUBY, __FILE__, __LINE__ + 1
+      def match_length
+        MatchLength.new(self, base: 0)
+      end
+    RUBY
+  end
+
+  class Backreference::Base
+    def match_length
+      if referenced_expression.nil?
+        raise ArgumentError, 'Missing referenced_expression - not parsed?'
+      end
+      referenced_expression.unquantified_clone.match_length
+    end
+  end
+
+  class EscapeSequence::CodepointList
+    def match_length
+      MatchLength.new(self, base: codepoints.count)
+    end
+  end
+
+  # Special case. Absence group can match 0.. chars, irrespective of content.
+  # TODO: in theory, they *can* exclude match lengths with `.`: `(?~.{3})`
+  class Group::Absence
+    def match_length
+      MatchLength.new(self, base_min: 0, base_max: Float::INFINITY, reify: ->{ '.*' })
+    end
+  end
+end