regexp-examples 1.1.4 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 43bddf0ebdc30ee2ae0e7ed35722165921d25c93
-  data.tar.gz: 3e52ef064a6dd7c484f06814c15135997f7fb19d
+  metadata.gz: 47da78b8a954d0a3cc88910fffd716e34675d953
+  data.tar.gz: 69705d5fa7a10b0e4980fd6036b8aa8b29486947
 SHA512:
-  metadata.gz: 9e53baeb48bce1a0e5d00e610b0ea933bc5a014d1fe0ea5b9678875c63b5507aaa32baab9f339e60610ce1b9c97e4173729a89340b8b75530912a6560352ca19
-  data.tar.gz: e63cf93bfb1ec76e4aa710bb15da91bd61e27f259c30d6011e0805c64f3b219420b20cbe2343cf6736ba2e7e40c7cc413b6b8a04779eb4cf7438cad3e4aa04cf
+  metadata.gz: 224d21492ee5097eac3700ea168866f04f32e62e0bb8a2e2fd955a2398df27e5ff97daf9907ed0e27e3523238b020ef676243ecc9bc888dc24aad49565d4fae0
+  data.tar.gz: 91c0d6413d6f7ef35e8cb3a20dda797eb33b26ec4e6da88e23afad5c02e0f61fea3fceeb8afd2727980efbbe7667051e6c0df92e8f2e576cbc99354113d7f18b

data/.travis.yml

@@ -2,6 +2,6 @@ language: ruby
 rvm:
   - 2.0.0-p598
   - 2.1.6
-  - 2.2.0
-  - 2.2.2
+  - 2.2.4
+  - 2.3.0
   - ruby-head

data/README.md

@@ -15,7 +15,7 @@ or a huge number of possible matches, such as `/.\w/`, then only a subset of the
 
 For more detail on this, see [configuration options](#configuration-options).
 
-If you'd like to understand how/why this gem works, please check out my [blog post](http://tom-lord.weebly.com/blog/reverse-engineering-regular-expressions) about it!
+If you'd like to understand how/why this gem works, please check out my [blog post](http://tom-lord.weebly.com/blog/reverse-engineering-regular-expressions) about it.
 
 ## Usage
 
@@ -46,6 +46,24 @@ If you'd like to understand how/why this gem works, please check out my [blog po
 /written by tom lord/i.random_example #=> "WrITtEN bY tOM LORD"
 ```
 
+## Supported ruby versions
+* MRI 2.0.x
+* MRI 2.1.x
+* MRI 2.2.x
+* MRI 2.3.x
+* MRI 2.4.0-dev
+
+MRI ≤ 1.9.3 are not supported. This is primarily because MRI 2.0.0 introduced a new
+regexp engine (`Oniguruma` was replaced by `Onigmo`). Whilst *most* of this gem could
+be made to work with MRI 1.9.x (or even 1.8.x), I feel the changes are too significant
+to implement backwards compatability (especially since [long-term support for MRI
+1.9.3 has now ended](https://www.ruby-lang.org/en/news/2014/01/10/ruby-1-9-3-will-end-on-2015/)).
+
+For example, named properties (e.g. `/\p{Alpha}/`) are illegal syntax on MRI 1.9.3.
+
+Other implementations, such as JRuby, could probably work fine -
+but I haven't fully tried/tested it. Pull requests are welcome.
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -86,12 +104,12 @@ Long answer:
   * Non-capture groups, e.g. `/(?:foo)/`
   * Comment groups, e.g. `/foo(?#comment)bar/`
 * Control characters, e.g. `/\ca/`, `/\cZ/`, `/\C-9/`
-* Escape sequences, e.g. `/\x42/`, `/\x5word/`, `/#{"\x80".force_encoding("ASCII-8BIT")}/`
+* Escape sequences, e.g. `/\x42/`, `/\x5word/`, `/#{"\x80".force\_encoding("ASCII-8BIT")}/`
 * Unicode characters, e.g. `/\u0123/`, `/\uabcd/`, `/\u{789}/`
 * Octal characters, e.g. `/\10/`, `/\177/`
 * Named properties, e.g. `/\p{L}/` ("Letter"), `/\p{Arabic}/` ("Arabic character")
-, `/\p{^Ll}/` ("Not a lowercase letter"), `/\P{^Canadian_Aboriginal}/` ("Not not a Canadian aboriginal character")
-  * ...Even between different ruby versions!! (e.g. `/\p{Arabic}/.examples(max_group_results: 999)` will give you a different answer in ruby v2.1.x and v2.2.x)
+, `/\p{^Ll}/` ("Not a lowercase letter"), `/\P{^Canadian\_Aboriginal}/` ("Not not a Canadian aboriginal character")
+  * ...Even between different ruby versions!! (e.g. `/\p{Arabic}/.examples(max\_group\_results: 999)` will give you a different answer in ruby v2.1.x and v2.2.x)
 * **Arbitrarily complex combinations of all the above!**
 
 * Regexp options can also be used:
@@ -102,12 +120,12 @@ Long answer:
 
 ## Bugs and Not-Yet-Supported syntax
 
-* There are some (rare) edge cases where backreferences do not work properly, e.g. `/(a*)a* \1/.examples` - which includes "aaaa aa". This is because each repeater is not context-aware, so the "greediness" logic is flawed. (E.g. in this case, the second `a*` should always evaluate to an empty string, because the previous `a*` was greedy!) However, patterns like this are highly unusual...
+* There are some (rare) edge cases where backreferences do not work properly, e.g. `/(a\*)a\* \1/.examples` - which includes "aaaa aa". This is because each repeater is not context-aware, so the "greediness" logic is flawed. (E.g. in this case, the second `a\*` should always evaluate to an empty string, because the previous `a\*` was greedy!) However, patterns like this are highly unusual...
 
 Since the Regexp language is so vast, it's quite likely I've missed something (please raise an issue if you find something)! The only missing feature that I'm currently aware of is:
 * Conditional capture groups, e.g. `/(group1)? (?(1)yes|no)/.examples` (which *should* return: `["group1 yes", " no"]`)
 
-Some of the most obscure regexp features are not even mentioned in the ruby docs! However, full documentation on all the intricate obscurities in the ruby (version 2.x) regexp parser can be found [here](https://raw.githubusercontent.com/k-takata/Onigmo/master/doc/RE).
+Some of the most obscure regexp features are not even mentioned in the ruby docs. However, full documentation on all the intricate obscurities in the ruby (version 2.x) regexp parser can be found [here](https://raw.githubusercontent.com/k-takata/Onigmo/master/doc/RE).
 
 ## Impossible features ("illegal syntax")
 
@@ -116,61 +134,65 @@ If you'd like to understand this in more detail, check out what I had to say in
 
 Using any of the following will raise a RegexpExamples::IllegalSyntax exception:
 
-* Lookarounds, e.g. `/foo(?=bar)/`, `/foo(?!bar)/`, `/(?<=foo)bar/`, `/(?<!foo)bar/`
+* Lookarounds, e.g. `/foo(?=bar)/`, `/foo(?!bar)/`, `/(?<=foo)bar/`, `/(?<\!foo)bar/`
 * [Anchors](http://ruby-doc.org/core-2.2.0/Regexp.html#class-Regexp-label-Anchors) (`\b`, `\B`, `\G`, `^`, `\A`, `$`, `\z`, `\Z`), e.g. `/\bword\b/`, `/line1\n^line2/`
   * However, a special case has been made to allow `^`, `\A` and `\G` at the start of a pattern; and to allow `$`, `\z` and `\Z` at the end of pattern. In such cases, the characters are effectively just ignored.
-* Subexpression calls (`\g`), e.g. `/(?<name> ... \g<name>* )/`
+* Subexpression calls (`\g`), e.g. `/(?<name> ... \g<name>\* )/`
 
-(Note: Backreferences are not really "regular" either, but I got these to work with a bit of hackery!)
+(Note: Backreferences are not really "regular" either, but I got these to work with a bit of hackery.)
 
 ##Configuration Options
 
-When generating examples, the gem uses 2 configurable values to limit how many examples are listed:
+When generating examples, the gem uses 3 configurable values to limit how many examples are listed:
 
-* `max_repeater_variance` (default = `2`) restricts how many examples to return for each repeater. For example:
-  * `.*` is equivalent to `.{0,2}`
+* `max\_repeater\_variance` (default = `2`) restricts how many examples to return for each repeater. For example:
+  * `.\*` is equivalent to `.{0,2}`
   * `.+` is equivalent to `.{1,3}`
   * `.{2,}` is equivalent to `.{2,4}`
   * `.{,3}` is equivalent to `.{0,2}`
   * `.{3,8}` is equivalent to `.{3,5}`
 
-* `max_group_results` (default = `5`) restricts how many characters to return for each "set". For example:
+* `max\_group\_results` (default = `5`) restricts how many characters to return for each "set". For example:
   * `\d` is equivalent to `[01234]`
   * `\w` is equivalent to `[abcde]`
   * `[h-s]` is equivalent to `[hijkl]`
   * `(1|2|3|4|5|6|7|8)` is equivalent to `[12345]`
 
-`Rexexp#examples` makes use of *both* these options; `Rexexp#random_example` only uses `max_repeater_variance`, since the other option is redundant!
+* `max\_results\_limit` (default = `10000`) restricts the maximum number of results that can possibly be generated. For example:
+  * `/(crazy){1,999} B\*I\*G\* regex/.examples.length <= 10000` -- Attempting this will NOT freeze your system
+
+`Rexexp#examples` makes use of *all* these options; `Rexexp#random\_example` only uses `max\_repeater\_variance`, since the other options are redundant.
 
 To use an alternative value, simply pass the configuration option as follows:
 
 ```ruby
-/a*/.examples(max_repeater_variance: 5)
+/a*/.examples(max\_repeater\_variance: 5)
   #=> [''. 'a', 'aa', 'aaa', 'aaaa' 'aaaaa']
-/[F-X]/.examples(max_group_results: 10)
+/[F-X]/.examples(max\_group\_results: 10)
   #=> ['F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O']
-/.*/.random_example(max_repeater_variance: 50)
+/[ab]{10}/.examples(max\_results\_limit: 64).length == 64 # NOT 1024
+/[slow]{9}/.examples(max\_results\_limit: 9999999).length == 4 \*\* 9 == 262144 # Warning - this will take a while!
+/.\*/.random\_example(max\_repeater\_variance: 50)
   #=> "A very unlikely result!"
 ```
 
-_**WARNING**: Choosing huge numbers for `Regexp#examples` and/or a sufficiently "complex" regex, could easily cause your system to freeze!_
+A sensible use case might be, for example, to generate all 1-5 digit strings:
 
-For example, if you try to generate a list of _all_ 5-letter words: `/\w{5}/.examples(max_group_results: 999)`, then since there are actually `63` "word" characters (upper/lower case letters, numbers and "\_"), this will try to generate `63**5 #=> 992436543` (almost 1 _billion_) examples!
-
-In other words, think twice before playing around with this config!
-
-A more sensible use case might be, for example, to generate all 1-4 digit strings:
-
-`/\d{1,4}/.examples(max_repeater_variance: 3, max_group_results: 10)`
+```ruby
+/\d{1,5}/.examples(max\_repeater\_variance: 4, max\_group\_results: 10, max\_results\_limit: 100000)
+  #=> ['0', '1', '2', ..., '99998', '99999']
+```
 
-Due to code optimisation, this is not something you need to worry about (much) for `Regexp#random_example`. For instance, the following takes no more than ~ 1 second on my machine:
+Due to code optimisation, `Regexp#random\_example` runs pretty fast even on very complex patterns.
+(I.e. It's a _lot_ faster than using `/pattern/.exammples.sample(1)`.)
+For instance, the following takes no more than ~ 1 second on my machine:
 
-`/.*\w+\d{100}/.random_example(max_repeater_variance: 1000)`
+`/.\*\w+\d{100}/.random\_example(max\_repeater\_variance: 1000)`
 
 ## TODO
 
-* Performance improvements:
-  * (Maybe?) add a `max_examples` configuration option and use lazy evaluation, to ensure the method never "freezes".
+* Make regexp#examples always return up to `max\_results\_limit` - currenty, it usually "aborts" before this limit is reached.
+* `\z` should be interpreted like `\n?\z`, not just `\z` like it is currently.
 
 ## Contributing
 

data/lib/core_extensions/regexp/examples.rb

@@ -1,30 +1,31 @@
 module CoreExtensions
   module Regexp
-    # A wrapper module to namespace/isolate the Regexp#examples and Regexp#random_exanple
+    # A wrapper module to namespace/isolate the Regexp#examples and Regexp#random_example
     # monkey patches.
     # No core classes are extended in any way, other than the above two methods.
     module Examples
       def examples(**config_options)
         RegexpExamples::ResultCountLimiters.configure!(
-          config_options[:max_repeater_variance],
-          config_options[:max_group_results]
+          max_repeater_variance: config_options[:max_repeater_variance],
+          max_group_results: config_options[:max_group_results],
+          max_results_limit: config_options[:max_results_limit]
         )
-        examples_by_method(:map_results)
+        examples_by_method(:result)
       end
 
       def random_example(**config_options)
         RegexpExamples::ResultCountLimiters.configure!(
-          config_options[:max_repeater_variance]
+          max_repeater_variance: config_options[:max_repeater_variance]
         )
-        examples_by_method(:map_random_result).first
+        examples_by_method(:random_result).first
       end
 
       private
 
       def examples_by_method(method)
-        full_examples = RegexpExamples.public_send(
-          method,
-          RegexpExamples::Parser.new(source, options).parse
+        full_examples = RegexpExamples.generic_map_result(
+          RegexpExamples::Parser.new(source, options).parse,
+          method
         )
         RegexpExamples::BackReferenceReplacer.new.substitute_backreferences(full_examples)
       end

data/lib/regexp-examples.rb

@@ -3,6 +3,7 @@ require_relative 'regexp-examples/chargroup_parser'
 require_relative 'regexp-examples/constants'
 require_relative 'regexp-examples/groups'
 require_relative 'regexp-examples/backreferences'
+require_relative 'regexp-examples/max_results_limiter'
 require_relative 'regexp-examples/helpers'
 require_relative 'regexp-examples/parser'
 require_relative 'regexp-examples/repeaters'

data/lib/regexp-examples/constants.rb

@@ -17,11 +17,17 @@ module RegexpExamples
     # \w is equivalent to [abcde]
     MAX_GROUP_RESULTS_DEFAULT = 5
 
+    # Maximum number of results to be generated, for Regexp#examples
+    # This is to prevent the system "freezing" when given instructions like:
+    # /[ab]{30}/.examples
+    # (Which would attempt to generate 2**30 == 1073741824 examples!!!)
+    MAX_RESULTS_LIMIT_DEFAULT = 10000
     class << self
-      attr_reader :max_repeater_variance, :max_group_results
-      def configure!(max_repeater_variance, max_group_results = nil)
+      attr_reader :max_repeater_variance, :max_group_results, :max_results_limit
+      def configure!(max_repeater_variance: nil, max_group_results: nil, max_results_limit: nil)
         @max_repeater_variance = (max_repeater_variance || MAX_REPEATER_VARIANCE_DEFAULT)
         @max_group_results = (max_group_results || MAX_GROUP_RESULTS_DEFAULT)
+        @max_results_limit = (max_results_limit || MAX_RESULTS_LIMIT_DEFAULT)
       end
     end
   end
@@ -32,6 +38,9 @@ module RegexpExamples
   def self.max_group_results
     ResultCountLimiters.max_group_results
   end
+  def self.max_results_limit
+    ResultCountLimiters.max_results_limit
+  end
 
   # Definitions of various special characters, used in regular expressions.
   # For example, `/\h/.examples` will return the value of `Hex` in this module

data/lib/regexp-examples/groups.rb

@@ -158,18 +158,20 @@ module RegexpExamples
     end
 
     def result
-      result_by_method(:map_results)
+      result_by_method(:result)
     end
 
     def random_result
-      result_by_method(:map_random_result).sample(1)
+      result_by_method(:random_result).sample(1)
     end
 
     private
 
     def result_by_method(method)
+      max_results_limiter = MaxResultsLimiterBySum.new
       repeaters_list
-        .map { |repeaters| RegexpExamples.public_send(method, repeaters) }
+        .map { |repeaters| RegexpExamples.generic_map_result(repeaters, method) }
+        .map { |result| max_results_limiter.limit_results(result)}
         .inject(:concat)
         .map { |result| GroupResult.new(result) }
         .uniq

data/lib/regexp-examples/helpers.rb

@@ -6,10 +6,14 @@ module RegexpExamples
   # For example:
   # permutations_of_strings [ ['a'], ['b'], ['c', 'd', 'e'] ] #=> ['abc', 'abd', 'abe']
   # permutations_of_strings [ ['a', 'b'], ['c', 'd'] ] #=> [ 'ac', 'ad', 'bc', 'bd' ]
-  def self.permutations_of_strings(arrays_of_strings)
-    first = arrays_of_strings.shift
-    return first if arrays_of_strings.empty?
-    first.product(permutations_of_strings(arrays_of_strings)).map do |result|
+  #
+  # Edge case:
+  # permutations_of_strings [ [] ] #=> nil
+  # (For example, ths occurs during /[^\d\D]/.examples #=> [])
+  def self.permutations_of_strings(arrays_of_strings, max_results_limiter = MaxResultsLimiterByProduct.new)
+    partial_result = max_results_limiter.limit_results(arrays_of_strings.shift)
+    return partial_result if arrays_of_strings.empty?
+    partial_result.product(permutations_of_strings(arrays_of_strings, max_results_limiter)).map do |result|
       join_preserving_capture_groups(result)
     end
   end
@@ -25,21 +29,9 @@ module RegexpExamples
     GroupResult.new(result.join, nil, subgroups)
   end
 
-  def self.map_results(repeaters)
-    generic_map_result(repeaters, :result)
-  end
-
-  def self.map_random_result(repeaters)
-    generic_map_result(repeaters, :random_result)
-  end
-
-  private
-
   def self.generic_map_result(repeaters, method)
-    repeaters
-      .map { |repeater| repeater.public_send(method) }
-      .instance_eval do |partial_results|
-        RegexpExamples.permutations_of_strings(partial_results)
-      end
+    permutations_of_strings(
+      repeaters.map { |repeater| repeater.public_send(method) }
+    )
   end
 end

data/lib/regexp-examples/max_results_limiter.rb

@@ -0,0 +1,58 @@
+module RegexpExamples
+  class MaxResultsLimiter # Base class
+    def initialize(initial_results_count)
+      @results_count = initial_results_count
+    end
+
+    private
+
+    # limiter_method and cumulator_method must be inverses
+    # i.e. :- and :+, or :/ and :*
+    def limit_results(partial_results, limiter_method, cumulator_method)
+      return [] if partial_results.empty? # guard clause
+      results_allowed = results_allowed_from(partial_results, limiter_method)
+      cumulate_total(results_allowed.length, cumulator_method)
+      results_allowed
+    end
+
+    def cumulate_total(new_results_count, cumulator_method)
+      if @results_count.zero?
+        @results_count = new_results_count
+      else
+        @results_count = @results_count.public_send(cumulator_method, new_results_count)
+      end
+    end
+
+    def results_allowed_from(partial_results, limiter_method)
+      partial_results.first(
+        RegexpExamples.max_results_limit.public_send(limiter_method, @results_count)
+      )
+    end
+  end
+
+  # For example:
+  # Needed when generating examples for /[ab]{10}/
+  # (here, results_count will reach 2**10 == 1024)
+  class MaxResultsLimiterByProduct < MaxResultsLimiter
+    def initialize
+      super(1)
+    end
+
+    def limit_results(partial_results)
+      super(partial_results, :/, :*)
+    end
+  end
+
+  # For example:
+  # Needed when generating examples for /[ab]{10}|{cd}{11}/             
+  # (here, results_count will reach 1024 + 2048 == 3072)
+  class MaxResultsLimiterBySum < MaxResultsLimiter
+    def initialize
+      super(0)
+    end
+
+    def limit_results(partial_results)
+      super(partial_results, :-, :+)
+    end
+  end
+end

data/lib/regexp-examples/repeaters.rb

@@ -12,14 +12,16 @@ module RegexpExamples
     def result
       group_results = group.result.first(RegexpExamples.max_group_results)
       results = []
+      max_results_limiter = MaxResultsLimiterBySum.new
       min_repeats.upto(max_repeats) do |repeats|
-        if repeats.zero?
-          results << [GroupResult.new('')]
-        else
-          results << RegexpExamples.permutations_of_strings(
-            [group_results] * repeats
-          )
-        end
+        result = if repeats.zero?
+                   [GroupResult.new('')]
+                 else
+                   RegexpExamples.permutations_of_strings(
+                     [group_results] * repeats
+                   )
+                 end
+        results << max_results_limiter.limit_results(result)
       end
       results.flatten.uniq
     end

data/lib/regexp-examples/version.rb

@@ -1,4 +1,4 @@
 # Gem version
 module RegexpExamples
-  VERSION = '1.1.4'
+  VERSION = '1.2.0'
 end

data/spec/regexp-examples_spec.rb

@@ -369,6 +369,41 @@ RSpec.describe Regexp, '#examples' do
           it { expect(/a(?-i:b)c/i.examples).to match_array %w(abc abC Abc AbC) }
         end
       end
+    end # context 'exact examples match'
+  end # context 'returns matching strings'
+
+  context 'max_results_limit config option' do
+    it 'with low limit' do
+      expect(/[A-Z]/.examples(max_results_limit: 5))
+        .to match_array %w(A B C D E)
+    end
+    it 'with (default) high limit' do
+      expect(/[ab]{14}/.examples.length)
+        .to be <= 10000 # NOT 2**14 == 16384, because it's been limited
+    end
+    it 'with (custom) high limit' do
+      expect(/[ab]{14}/.examples(max_results_limit: 20000).length)
+        .to eq 16384 # NOT 10000, because it's below the limit
+    end
+    it 'for boolean or groups' do
+      expect(/[ab]{3}|[cd]{3}/.examples(max_results_limit: 10).length)
+        .to eq 10
+    end
+    it 'for case insensitive examples' do
+      expect(/[ab]{3}/i.examples(max_results_limit: 10).length)
+        .to be <= 10
+    end
+    it 'for range repeaters' do
+      expect(/[ab]{2,3}/.examples(max_results_limit: 10).length)
+        .to be <= 10 # NOT 4 + 8 = 12
+    end
+    it 'for backreferences' do
+      expect(/([ab]{3})\1?/.examples(max_results_limit: 10).length)
+        .to be <= 10 # NOT 8 * 2 = 16
+    end
+    it 'for a complex pattern' do
+      expect(/(a|[bc]{2})\1{1,3}/.examples(max_results_limit: 14).length)
+        .to be <= 14 # NOT (1 + 4) * 3 = 15
     end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: regexp-examples
 version: !ruby/object:Gem::Version
-  version: 1.1.4
+  version: 1.2.0
 platform: ruby
 authors:
 - Tom Lord
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-12-31 00:00:00.000000000 Z
+date: 2016-02-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -65,6 +65,7 @@ files:
 - lib/regexp-examples/constants.rb
 - lib/regexp-examples/groups.rb
 - lib/regexp-examples/helpers.rb
+- lib/regexp-examples/max_results_limiter.rb
 - lib/regexp-examples/parser.rb
 - lib/regexp-examples/parser_helpers/charset_negation_helper.rb
 - lib/regexp-examples/parser_helpers/parse_after_backslash_group_helper.rb
@@ -100,7 +101,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.4.6
 signing_key: 
 specification_version: 4
 summary: Extends the Regexp class with '#examples' and '#random_example'

data/db/unicode_ranges_2.1.pstore

@@ -1 +0,0 @@
-unicode_ranges_2.0.pstore