regexp_parser 1.7.0 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d7b93dde993f6fe427ff43755738bf7de50f8613cf6e8097c9d791646d803e4c
-  data.tar.gz: 993a88720a4ee1d8a34f4c95e167089adc6455289bfeb356de8c028a9bbee63d
+  metadata.gz: dcf56dd42e703e7f1f846762c418e83792d46f2c7d9efffc1fb1612b4325e076
+  data.tar.gz: a6197d98af2325a93ed60a102f09bc98e2163dec0d2b57fbede82dbf0479ea8b
 SHA512:
-  metadata.gz: 0bf5c142591b2d5a65023c53f76a64a13106074050042d24614963cc14dabda197ea9140fccd93f26ad06885293369b076bb5e9198967a6e3762654df8033455
-  data.tar.gz: 1311b3dfa90633ef456edc12abf6ace2d7311c7be8450f3768a436f9c8491c3a87987f3d6ac24c6966b6f4de5363e0f6f874bfe9e1b038a6cf5d9c043553b58e
+  metadata.gz: 8b9db6543c87b63c49e24666e06bc012ea9a6e330711c9fbd35961c70d1222988bf2403927fdbe2f8797176d674b83c7c3f2d1215c5f92b0d6f00c6ab7fe37af
+  data.tar.gz: 6cf07796d7c6ab1520a63b0f3b65d2e513caccf0b81306321376473fc438c7573bcf867e74ea8465122e12a47fe2e7e09652473f83e26f1038e112c2d66b3d2c

data/CHANGELOG.md

@@ -1,10 +1,89 @@
 ## [Unreleased]
 
+## [2.0.0] - 2020-11-25 - [Janosch Müller](mailto:janosch84@gmail.com)
+
+### Changed
+
+- some methods that used to return byte-based indices now return char-based indices
+  * the returned values have only changed for Regexps that contain multibyte chars
+  * this is only a breaking change if you used such methods directly AND relied on them pointing to bytes
+  * affected methods:
+  * `Regexp::Token` `#length`, `#offset`, `#te`, `#ts`
+  * `Regexp::Expression::Base` `#full_length`, `#offset`, `#starts_at`, `#te`, `#ts`
+  * thanks to [Akinori MUSHA](https://github.com/knu) for the report
+- removed some deprecated methods/signatures
+  * these are rarely used and have been showing deprecation warnings for a long time
+  * `Regexp::Expression::Subexpression.new` with 3 arguments
+  * `Regexp::Expression::Root.new` without a token argument
+  * `Regexp::Expression.parsed`
+
+### Added
+
+- `Regexp::Expression::Base#base_length`
+  * returns the character count of an expression body, ignoring any quantifier
+- pragmatic, experimental support for chained quantifiers
+  * e.g.: `/^a{10}{4,6}$/` matches exactly 40, 50 or 60 `a`s
+  * successive quantifiers used to be silently dropped by the parser
+  * they are now wrapped with passive groups as if they were written `(?:a{10}){4,6}`
+  * thanks to [calfeld](https://github.com/calfeld) for reporting this a while back
+
+### Fixed
+
+- incorrect encoding output for non-ascii comments
+  * this led to a crash when calling `#to_s` on parse results containing such comments
+  * thanks to [Michael Glass](https://github.com/michaelglass) for the report
+- some crashes when scanning contrived patterns such as `'\😋'`
+
+### [1.8.2] - 2020-10-11 - [Janosch Müller](mailto:janosch84@gmail.com)
+
+### Fixed
+
+- fix `FrozenError` in `Expression::Base#repetitions` on Ruby 3.0
+  * thanks to [Thomas Walpole](https://github.com/twalpole)
+- removed "unknown future version" warning on Ruby 3.0
+
+### [1.8.1] - 2020-09-28 - [Janosch Müller](mailto:janosch84@gmail.com)
+
+### Fixed
+
+- fixed scanning of comment-like text in normal mode
+  * this was an old bug, but had become more prevalent in v1.8.0
+  * thanks to [Tietew](https://github.com/Tietew) for the report
+- specified correct minimum Ruby version in gemspec
+  * it said 1.9 but really required 2.0 as of v1.8.0
+
+### [1.8.0] - 2020-09-20 - [Janosch Müller](mailto:janosch84@gmail.com)
+
+### Changed
+
+- dropped support for running on Ruby 1.9.x
+
+### Added
+
+- regexp flags can now be passed when parsing a `String` as regexp body
+  * see the [README](/README.md#usage) for details
+  * thanks to [Owen Stephens](https://github.com/owst)
+- bare occurrences of `\g` and `\k` are now allowed and scanned as literal escapes
+  * matches Onigmo behavior
+  * thanks for the report to [Marc-André Lafortune](https://github.com/marcandre)
+
+### Fixed
+
+- fixed parsing comments without preceding space or trailing newline in x-mode
+  * thanks to [Owen Stephens](https://github.com/owst)
+
+### [1.7.1] - 2020-06-07 - [Ammar Ali](mailto:ammarabuali@gmail.com)
+
+### Fixed
+
+- Support for literals that include the unescaped delimiters `{`, `}`, and `]`. These
+  delimiters are informally supported by various regexp engines.
+
 ### [1.7.0] - 2020-02-23 - [Janosch Müller](mailto:janosch84@gmail.com)
 
 ### Added
 
-- `Expression#each_expression` and `1.#traverse` can now be called without a block
+- `Expression#each_expression` and `#traverse` can now be called without a block
   * this returns an `Enumerator` and allows chaining, e.g. `each_expression.select`
   * thanks to [Masataka Kuwabara](https://github.com/pocke)
 

data/README.md

@@ -8,8 +8,8 @@ A Ruby gem for tokenizing, parsing, and transforming regular expressions.
   * A scanner/tokenizer based on [Ragel](http://www.colm.net/open-source/ragel/)
   * A lexer that produces a "stream" of token objects.
   * A parser that produces a "tree" of Expression objects (OO API)
-* Runs on Ruby 1.9, 2.x, and JRuby (1.9 mode) runtimes.
-* Recognizes Ruby 1.8, 1.9, and 2.x regular expressions [See Supported Syntax](#supported-syntax)
+* Runs on Ruby 2.x, 3.x and JRuby runtimes
+* Recognizes Ruby 1.8, 1.9, 2.x and 3.x regular expressions [See Supported Syntax](#supported-syntax)
 
 
 _For examples of regexp_parser in use, see [Example Projects](#example-projects)._
@@ -18,7 +18,7 @@ _For examples of regexp_parser in use, see [Example Projects](#example-projects)
 ---
 ## Requirements
 
-* Ruby >= 1.9
+* Ruby >= 2.0
 * Ragel >= 6.0, but only if you want to build the gem or work on the scanner.
 
 
@@ -72,6 +72,17 @@ called with the results as follows:
 * **Parser**: after completion, the block gets passed the root expression.
   _The result of the block is returned._
 
+All three methods accept either a `Regexp` or `String` (containing the pattern)
+- if a String is passed, `options` can be supplied:
+
+```ruby
+require 'regexp_parser'
+
+Regexp::Parser.parse(
+  "a+ # Recognises a and A...",
+  options: ::Regexp::EXTENDED | ::Regexp::IGNORECASE
+)
+```
 
 ---
 ## Components
@@ -136,11 +147,8 @@ Regexp::Scanner.scan( /(cat?([bhm]at)){3,5}/ ).map {|token| token[2]}
     to the lexer.
 
   * The MRI implementation may accept expressions that either conflict with
-    the documentation or are undocumented. The scanner does not support such
-    implementation quirks.
-    _(See issues [#3](https://github.com/ammar/regexp_parser/issues/3) and
-    [#15](https://github.com/ammar/regexp_parser/issues/15) for examples)_
-
+    the documentation or are undocumented, like `{}` and `]` _(unescaped)_.
+    The scanner will try to support as many of these cases as possible.
 
 ---
 ### Syntax
@@ -309,7 +317,7 @@ Expression class. See the next section for details._
 
 ## Supported Syntax
 The three modules support all the regular expression syntax features of Ruby 1.8,
-1.9, and 2.x:
+1.9, 2.x and 3.x:
 
 _Note that not all of these are available in all versions of Ruby_
 
@@ -432,13 +440,17 @@ rake install
 ## Example Projects
 Projects using regexp_parser.
 
+- [capybara](https://github.com/teamcapybara/capybara) is an integration testing tool that uses regexp_parser to convert Regexps to css/xpath selectors.
+
+- [js_regex](https://github.com/janosch-x/js_regex) converts Ruby regular expressions to JavaScript-compatible regular expressions.
+
 - [meta_re](https://github.com/ammar/meta_re) is a regular expression preprocessor with alias support.
 
 - [mutant](https://github.com/mbj/mutant) (before v0.9.0) manipulates your regular expressions (amongst others) to see if your tests cover their behavior.
 
-- [twitter-cldr-rb](https://github.com/twitter/twitter-cldr-rb) uses regexp_parser to generate examples of postal codes.
+- [rubocop](https://github.com/rubocop-hq/rubocop) is a linter for Ruby that uses regexp_parser to lint Regexps.
 
-- [js_regex](https://github.com/janosch-x/js_regex) converts Ruby regular expressions to JavaScript-compatible regular expressions.
+- [twitter-cldr-rb](https://github.com/twitter/twitter-cldr-rb) is a localization helper that uses regexp_parser to generate examples of postal codes.
 
 
 ## References
@@ -467,4 +479,4 @@ Documentation and books used while working on this project.
 
 ---
 ##### Copyright
-_Copyright (c) 2010-2019 Ammar Ali. See LICENSE file for details._
+_Copyright (c) 2010-2020 Ammar Ali. See LICENSE file for details._

data/lib/regexp_parser/expression.rb

@@ -34,6 +34,10 @@ module Regexp::Expression
 
     alias :starts_at :ts
 
+    def base_length
+      to_s(:base).length
+    end
+
     def full_length
       to_s.length
     end
@@ -80,8 +84,12 @@ module Regexp::Expression
       return 1..1 unless quantified?
       min = quantifier.min
       max = quantifier.max < 0 ? Float::INFINITY : quantifier.max
-      # fix Range#minmax - https://bugs.ruby-lang.org/issues/15807
-      (min..max).tap { |r| r.define_singleton_method(:minmax) { [min, max] } }
+      range = min..max
+      # fix Range#minmax on old Rubies - https://bugs.ruby-lang.org/issues/15807
+      if RUBY_VERSION.to_f < 2.7
+        range.define_singleton_method(:minmax) { [min, max] }
+      end
+      range
     end
 
     def greedy?
@@ -114,23 +122,6 @@ module Regexp::Expression
     alias :to_h :attributes
   end
 
-  def self.parsed(exp)
-    warn('WARNING: Regexp::Expression::Base.parsed is buggy and '\
-         'will be removed in 2.0.0. Use Regexp::Parser.parse instead.')
-    case exp
-    when String
-      Regexp::Parser.parse(exp)
-    when Regexp
-      Regexp::Parser.parse(exp.source) # <- causes loss of root options
-    when Regexp::Expression            # <- never triggers
-      exp
-    else
-      raise ArgumentError, 'Expression.parsed accepts a String, Regexp, or '\
-                           'a Regexp::Expression as a value for exp, but it '\
-                           "was given #{exp.class.name}."
-    end
-  end
-
 end # module Regexp::Expression
 
 require 'regexp_parser/expression/quantifier'

data/lib/regexp_parser/expression/classes/group.rb

@@ -10,9 +10,24 @@ module Regexp::Expression
       def comment?; false end
     end
 
-    class Atomic  < Group::Base; end
-    class Passive < Group::Base; end
+    class Passive < Group::Base
+      attr_writer :implicit
+
+      def to_s(format = :full)
+        if implicit?
+          "#{expressions.join}#{quantifier_affix(format)}"
+        else
+          super
+        end
+      end
+
+      def implicit?
+        @implicit ||= false
+      end
+    end
+
     class Absence < Group::Base; end
+    class Atomic  < Group::Base; end
     class Options < Group::Base
       attr_accessor :option_changes
     end

data/lib/regexp_parser/expression/classes/root.rb

@@ -1,24 +1,12 @@
 module Regexp::Expression
 
   class Root < Regexp::Expression::Subexpression
-    # TODO: this override is here for backwards compatibility, remove in 2.0.0
-    def initialize(*args)
-      unless args.first.is_a?(Regexp::Token)
-        warn('WARNING: Root.new without a Token argument is deprecated and '\
-             'will be removed in 2.0.0. Use Root.build for the old behavior.')
-        return super(self.class.build_token, *args)
-      end
-      super
+    def self.build(options = {})
+      new(build_token, options)
     end
 
-    class << self
-      def build(options = {})
-        new(build_token, options)
-      end
-
-      def build_token
-        Regexp::Token.new(:expression, :root, '', 0)
-      end
+    def self.build_token
+      Regexp::Token.new(:expression, :root, '', 0)
     end
   end
 end

data/lib/regexp_parser/expression/quantifier.rb

@@ -40,5 +40,14 @@ module Regexp::Expression
       RUBY
     end
     alias :lazy? :reluctant?
+
+    def ==(other)
+      other.class == self.class &&
+        other.token == token &&
+        other.mode == mode &&
+        other.min == min &&
+        other.max == max
+    end
+    alias :eq :==
   end
 end

data/lib/regexp_parser/expression/sequence.rb

@@ -7,16 +7,6 @@ module Regexp::Expression
   # Used as the base class for the Alternation alternatives, Conditional
   # branches, and CharacterSet::Intersection intersected sequences.
   class Sequence < Regexp::Expression::Subexpression
-    # TODO: this override is here for backwards compatibility, remove in 2.0.0
-    def initialize(*args)
-      if args.count == 3
-        warn('WARNING: Sequence.new without a Regexp::Token argument is '\
-             'deprecated and will be removed in 2.0.0.')
-        return self.class.at_levels(*args)
-      end
-      super
-    end
-
     class << self
       def add_to(subexpression, params = {}, active_opts = {})
         sequence = at_levels(

data/lib/regexp_parser/lexer.rb

@@ -11,11 +11,11 @@ class Regexp::Lexer
 
   CLOSING_TOKENS = [:close].freeze
 
-  def self.lex(input, syntax = "ruby/#{RUBY_VERSION}", &block)
-    new.lex(input, syntax, &block)
+  def self.lex(input, syntax = "ruby/#{RUBY_VERSION}", options: nil, &block)
+    new.lex(input, syntax, options: options, &block)
   end
 
-  def lex(input, syntax = "ruby/#{RUBY_VERSION}", &block)
+  def lex(input, syntax = "ruby/#{RUBY_VERSION}", options: nil, &block)
     syntax = Regexp::Syntax.new(syntax)
 
     self.tokens = []
@@ -25,7 +25,7 @@ class Regexp::Lexer
     self.shift = 0
 
     last = nil
-    Regexp::Scanner.scan(input) do |type, token, text, ts, te|
+    Regexp::Scanner.scan(input, options: options) do |type, token, text, ts, te|
       type, token = *syntax.normalize(type, token)
       syntax.check! type, token
 
@@ -96,10 +96,10 @@ class Regexp::Lexer
 
     tokens.pop
     tokens << Regexp::Token.new(:literal, :literal, lead,
-              token.ts, (token.te - last.bytesize),
+              token.ts, (token.te - last.length),
               nesting, set_nesting, conditional_nesting)
     tokens << Regexp::Token.new(:literal, :literal, last,
-              (token.ts + lead.bytesize), token.te,
+              (token.ts + lead.length), token.te,
               nesting, set_nesting, conditional_nesting)
   end
 

data/lib/regexp_parser/parser.rb

@@ -18,12 +18,12 @@ class Regexp::Parser
     end
   end
 
-  def self.parse(input, syntax = "ruby/#{RUBY_VERSION}", &block)
-    new.parse(input, syntax, &block)
+  def self.parse(input, syntax = "ruby/#{RUBY_VERSION}", options: nil, &block)
+    new.parse(input, syntax, options: options, &block)
   end
 
-  def parse(input, syntax = "ruby/#{RUBY_VERSION}", &block)
-    root = Root.build(options_from_input(input))
+  def parse(input, syntax = "ruby/#{RUBY_VERSION}", options: nil, &block)
+    root = Root.build(extract_options(input, options))
 
     self.root = root
     self.node = root
@@ -35,7 +35,7 @@ class Regexp::Parser
 
     self.captured_group_counts = Hash.new(0)
 
-    Regexp::Lexer.scan(input, syntax) do |token|
+    Regexp::Lexer.scan(input, syntax, options: options) do |token|
       parse_token(token)
     end
 
@@ -54,14 +54,20 @@ class Regexp::Parser
                 :options_stack, :switching_options, :conditional_nesting,
                 :captured_group_counts
 
-  def options_from_input(input)
-    return {} unless input.is_a?(::Regexp)
+  def extract_options(input, options)
+    if options && !input.is_a?(String)
+      raise ArgumentError, 'options cannot be supplied unless parsing a String'
+    end
+
+    options = input.options if input.is_a?(::Regexp)
 
-    options = {}
-    options[:i] = true if input.options & ::Regexp::IGNORECASE != 0
-    options[:m] = true if input.options & ::Regexp::MULTILINE  != 0
-    options[:x] = true if input.options & ::Regexp::EXTENDED   != 0
-    options
+    return {} unless options
+
+    enabled_options = {}
+    enabled_options[:i] = true if options & ::Regexp::IGNORECASE != 0
+    enabled_options[:m] = true if options & ::Regexp::MULTILINE  != 0
+    enabled_options[:x] = true if options & ::Regexp::EXTENDED   != 0
+    enabled_options
   end
 
   def nest(exp)
@@ -432,6 +438,28 @@ class Regexp::Parser
     target_node || raise(ArgumentError, 'No valid target found for '\
                                         "'#{token.text}' ")
 
+    # in case of chained quantifiers, wrap target in an implicit passive group
+    # description of the problem: https://github.com/ammar/regexp_parser/issues/3
+    # rationale for this solution: https://github.com/ammar/regexp_parser/pull/69
+    if target_node.quantified?
+      new_token = Regexp::Token.new(
+        :group,
+        :passive,
+        '', # text
+        target_node.ts,
+        nil, # te (unused)
+        target_node.level,
+        target_node.set_level,
+        target_node.conditional_level
+      )
+      new_group = Group::Passive.new(new_token, active_opts)
+      new_group.implicit = true
+      new_group << target_node
+      increase_level(target_node)
+      node.expressions[offset] = new_group
+      target_node = new_group
+    end
+
     case token.token
     when :zero_or_one
       target_node.quantify(:zero_or_one, token.text, 0, 1, :greedy)
@@ -462,6 +490,11 @@ class Regexp::Parser
     end
   end
 
+  def increase_level(exp)
+    exp.level += 1
+    exp.respond_to?(:each) && exp.each { |subexp| increase_level(subexp) }
+  end
+
   def interval(target_node, token)
     text = token.text
     mchr = text[text.length-1].chr =~ /[?+]/ ? text[text.length-1].chr : nil