re2 0.7.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 32a2347d40362a3014dfd03f948405964468a3c2
-  data.tar.gz: ea15968dae15568249200c6ba91bad66d0e28f6f
+SHA256:
+  metadata.gz: b7a33f3c27224496ec131c261c7c66172f21474bce66cdf1cc1631c83040b3ca
+  data.tar.gz: 4d4a5448aa271bbce4cf37ffadfaadfa0797691fad5cc13f2f722e281116d66f
 SHA512:
-  metadata.gz: 00588950e6c82fb5720043c04d0f833d819f3107a3c953f2284d9eaede74210c0c803f34b3e9fd89413c886fae1e4b853b5a85b18b3c2d788ff0a7e8b46036cf
-  data.tar.gz: af2d23503d089a15ea6dd32e00bc80622cd7b0b1773ed471edcc982ac6791a6d1bf0bf5332d2440446796b5a4da15a169fd7396e3cf729af8a300e52a5985bad
+  metadata.gz: f080eccb20bb599b374485981f4c23fd8a49e84903ebe32e99e00b458c8b6334430cef41cbb1227fcba057f81d2fed4ead67648e506348fa6c2f975b7e6fea7f
+  data.tar.gz: bbd0ca07b287ca7e212175b4f5eaafd2698b7be94aefa432e81c4ff108b4b4c660e47d03d48d6affb82c0cee5f98fcb37bdcd97651bf309cebb148ac5e654518

data/README.md

@@ -1,11 +1,12 @@
-re2 [![Build Status](https://travis-ci.org/mudge/re2.svg?branch=master)](http://travis-ci.org/mudge/re2)
+re2 [![Build Status](https://github.com/mudge/re2/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/mudge/re2/actions)
 ===
 
 A Ruby binding to [re2][], an "efficient, principled regular expression
 library".
 
-**Current version:** 0.7.0  
-**Supported Ruby versions:** 1.8.7, 1.9.2, 1.9.3, 2.0.0, 2.1.0, Rubinius 2.2
+**Current version:** 1.3.0  
+**Supported Ruby versions:** 1.8.7, 1.9.3, 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.0  
+**Supported re2 versions:** libre2.0 (< 2020-03-02), libre2.1 (2020-03-02), libre2.6 (2020-03-03), libre2.7 (2020-05-01), libre2.8 (2020-07-06), libre2.9 (2020-11-01)
 
 Installation
 ------------
@@ -21,13 +22,19 @@ If you are using Debian, you can install the [libre2-dev][] package like so:
 
     $ sudo apt-get install libre2-dev
 
+Recent versions of re2 require a compiler with C++11 support such as [clang](http://clang.llvm.org/) 3.4 or [gcc](https://gcc.gnu.org/) 4.8.
+
 If you are using a packaged Ruby distribution, make sure you also have the
 Ruby header files installed such as those provided by the [ruby-dev][] package
 on Debian and Ubuntu.
 
 You can then install the library via RubyGems with `gem install re2` or `gem
-install re2 -- --with-re2-dir=/opt/local/re2` if re2 is not installed in the
-default location of `/usr/local/`.
+install re2 -- --with-re2-dir=/path/to/re2/prefix` if re2 is not installed in
+any of the following default locations:
+
+* `/usr/local`
+* `/opt/homebrew`
+* `/usr`
 
 Documentation
 -------------
@@ -35,14 +42,17 @@ Documentation
 Full documentation automatically generated from the latest version is
 available at <http://mudge.name/re2/>.
 
-Bear in mind that re2's regular expression syntax differs from PCRE, see the
-[official syntax page][] for more details.
+Note that re2's regular expression syntax differs from PCRE and Ruby's
+built-in [`Regexp`][Regexp] library, see the [official syntax page][] for more
+details.
 
 Usage
 -----
 
-You can use re2 as a mostly drop-in replacement for Ruby's own [Regexp][] and
-[MatchData][] classes:
+While re2 uses the same naming scheme as Ruby's built-in regular expression
+library (with [`Regexp`](http://mudge.name/re2/RE2/Regexp.html) and
+[`MatchData`](http://mudge.name/re2/RE2/MatchData.html)), its API is slightly
+different:
 
 ```console
 $ irb -rubygems
@@ -67,9 +77,11 @@ $ irb -rubygems
 => nil
 ```
 
-As `RE2::Regexp.new` (or `RE2::Regexp.compile`) can be quite verbose, a helper
-method has been defined against `Kernel` so you can use a shorter version to
-create regular expressions:
+As
+[`RE2::Regexp.new`](http://mudge.name/re2/RE2/Regexp.html#initialize-instance_method)
+(or `RE2::Regexp.compile`) can be quite verbose, a helper method has been
+defined against `Kernel` so you can use a shorter version to create regular
+expressions:
 
 ```console
 > RE2('(\d+)')
@@ -123,7 +135,7 @@ Features
 --------
 
 * Pre-compiling regular expressions with
-  [`RE2::Regexp.new(re)`](http://code.google.com/p/re2/source/browse/re2/re2.h#96),
+  [`RE2::Regexp.new(re)`](https://github.com/google/re2/blob/2016-02-01/re2/re2.h#L100),
   `RE2::Regexp.compile(re)` or `RE2(re)` (including specifying options, e.g.
   `RE2::Regexp.new("pattern", :case_sensitive => false)`
 
@@ -152,21 +164,25 @@ Features
   `pattern.replace_all(replacement, original)`
 
 * Escaping regular expressions with
-  [`RE2.escape(unquoted)`](http://code.google.com/p/re2/source/browse/re2/re2.h#377) and
+  [`RE2.escape(unquoted)`](https://github.com/google/re2/blob/2016-02-01/re2/re2.h#L418) and
   `RE2.quote(unquoted)`
 
 Contributions
 -------------
 
-Thanks to [Jason Woods](https://github.com/driskell) who contributed the
-original implementations of `RE2::MatchData#begin` and `RE2::MatchData#end`.
+* Thanks to [Jason Woods](https://github.com/driskell) who contributed the
+  original implementations of `RE2::MatchData#begin` and `RE2::MatchData#end`;
+* Thanks to [Stefano Rivera](https://github.com/stefanor) who first contributed C++11 support;
+* Thanks to [Stan Hu](https://github.com/stanhu) for reporting a bug with empty patterns and `RE2::Regexp#scan`;
+* Thanks to [Sebastian Reitenbach](https://github.com/buzzdeee) for reporting
+  the deprecation and removal of the `utf8` encoding option in re2.
 
 Contact
 -------
 
-All feedback should go to the mailing list: <mailto:ruby.re2@librelist.com>
+All issues and suggestions should go to [GitHub Issues](https://github.com/mudge/re2/issues).
 
-  [re2]: http://code.google.com/p/re2/
+  [re2]: https://github.com/google/re2
   [gcc]: http://gcc.gnu.org/
   [ruby-dev]: http://packages.debian.org/ruby-dev
   [build-essential]: http://packages.debian.org/build-essential
@@ -174,5 +190,5 @@ All feedback should go to the mailing list: <mailto:ruby.re2@librelist.com>
   [MatchData]: http://ruby-doc.org/core/classes/MatchData.html
   [Homebrew]: http://mxcl.github.com/homebrew
   [libre2-dev]: http://packages.debian.org/search?keywords=libre2-dev
-  [official syntax page]: http://code.google.com/p/re2/wiki/Syntax
+  [official syntax page]: https://github.com/google/re2/wiki/Syntax
 

data/Rakefile

@@ -1,15 +1,10 @@
 require 'rake/extensiontask'
-require 'rake/testtask'
+require 'rspec/core/rake_task'
 
 Rake::ExtensionTask.new('re2')
 
-Rake::TestTask.new do |t|
-  t.libs << "spec"
-  t.test_files = FileList["spec/**/*_spec.rb"]
-  t.verbose = true
-end
+RSpec::Core::RakeTask.new(:spec)
 
-task :test    => :compile
-task :spec    => :test
-task :default => :test
+task :spec    => :compile
+task :default => :spec
 

data/ext/re2/extconf.rb

@@ -6,24 +6,72 @@
 
 require 'mkmf'
 
-incl, lib = dir_config("re2", "/usr/local/include", "/usr/local/lib")
+if ENV["CC"]
+  RbConfig::MAKEFILE_CONFIG["CC"] = ENV["CC"]
+  RbConfig::CONFIG["CC"] = ENV["CC"]
+end
+
+if ENV["CXX"]
+  RbConfig::MAKEFILE_CONFIG["CXX"] = ENV["CXX"]
+  RbConfig::CONFIG["CXX"] = ENV["CXX"]
+end
+
+header_dirs = [
+  "/usr/local/include",
+  "/opt/homebrew/include",
+  "/usr/include"
+]
+
+lib_dirs = [
+  "/usr/local/lib",
+  "/opt/homebrew/lib",
+  "/usr/lib"
+]
+
+dir_config("re2", header_dirs, lib_dirs)
 
 $CFLAGS << " -Wall -Wextra -funroll-loops"
 
+# Pass -x c++ to force gcc to compile the test program
+# as C++ (as it will end in .c by default).
+compile_options = "-x c++"
+
 have_library("stdc++")
 have_header("stdint.h")
 have_func("rb_str_sublen")
 
-if have_library("re2")
+unless have_library("re2")
+  abort "You must have re2 installed and specified with --with-re2-dir, please see https://github.com/google/re2/wiki/Install"
+end
 
-  # Determine which version of re2 the user has installed.
-  # Revision d9f8806c004d added an `endpos` argument to the
-  # generic Match() function.
-  #
-  # To test for this, try to compile a simple program that uses
-  # the newer form of Match() and set a flag if it is successful.
-  checking_for("RE2::Match() with endpos argument") do
-    test_re2_match_signature = <<SRC
+# Recent versions of re2 now require a compiler with C++11 support
+checking_for("re2 requires C++11 compiler") do
+  minimal_program = <<SRC
+#include <re2/re2.h>
+int main() { return 0; }
+SRC
+
+  unless try_compile(minimal_program, compile_options)
+    if try_compile(minimal_program, compile_options + " -std=c++11")
+      compile_options << " -std=c++11"
+      $CPPFLAGS << " -std=c++11"
+    elsif try_compile(minimal_program, compile_options + " -std=c++0x")
+      compile_options << " -std=c++0x"
+      $CPPFLAGS << " -std=c++0x"
+    else
+      abort "Cannot compile re2 with your compiler: recent versions require C++11 support."
+    end
+  end
+end
+
+# Determine which version of re2 the user has installed.
+# Revision d9f8806c004d added an `endpos` argument to the
+# generic Match() function.
+#
+# To test for this, try to compile a simple program that uses
+# the newer form of Match() and set a flag if it is successful.
+checking_for("RE2::Match() with endpos argument") do
+  test_re2_match_signature = <<SRC
 #include <re2/re2.h>
 
 int main() {
@@ -35,14 +83,9 @@ int main() {
 }
 SRC
 
-    # Pass -x c++ to force gcc to compile the test program
-    # as C++ (as it will end in .c by default).
-    if try_compile(test_re2_match_signature, "-x c++")
-      $defs.push("-DHAVE_ENDPOS_ARGUMENT")
-    end
+  if try_compile(test_re2_match_signature, compile_options)
+    $defs.push("-DHAVE_ENDPOS_ARGUMENT")
   end
-
-  create_makefile("re2")
-else
-  abort "You must have re2 installed and specified with --with-re2-dir, please see http://code.google.com/p/re2/wiki/Install"
 end
+
+create_makefile("re2")

data/ext/re2/re2.cc

@@ -6,8 +6,8 @@
  * Released under the BSD Licence, please see LICENSE.txt
  */
 
-#include <re2/re2.h>
 #include <ruby.h>
+#include <re2/re2.h>
 #include <stdint.h>
 #include <string>
 #include <sstream>
@@ -89,6 +89,7 @@ typedef struct {
 typedef struct {
   re2::StringPiece *input;
   int number_of_capturing_groups;
+  bool eof;
   VALUE regexp, text;
 } re2_scanner;
 
@@ -172,6 +173,21 @@ static VALUE re2_scanner_string(VALUE self) {
   return c->text;
 }
 
+/*
+ * Returns whether the scanner has consumed all input or not.
+ *
+ * @return [Boolean] whether the scanner has consumed all input or not
+ * @example
+ *   c = RE2::Regexp.new('(\d+)').scan("foo")
+ *   c.eof? #=> true
+ */
+static VALUE re2_scanner_eof(VALUE self) {
+  re2_scanner *c;
+  Data_Get_Struct(self, re2_scanner, c);
+
+  return BOOL2RUBY(c->eof);
+}
+
 /*
  * Rewind the scanner to the start of the string.
  *
@@ -188,6 +204,7 @@ static VALUE re2_scanner_rewind(VALUE self) {
   Data_Get_Struct(self, re2_scanner, c);
 
   c->input = new(nothrow) re2::StringPiece(StringValuePtr(c->text));
+  c->eof = false;
 
   return self;
 }
@@ -204,6 +221,8 @@ static VALUE re2_scanner_rewind(VALUE self) {
  */
 static VALUE re2_scanner_scan(VALUE self) {
   int i;
+  size_t original_input_size, new_input_size;
+  bool input_advanced;
   re2_pattern *p;
   re2_scanner *c;
   VALUE result;
@@ -215,6 +234,12 @@ static VALUE re2_scanner_scan(VALUE self) {
   vector<RE2::Arg*> args(c->number_of_capturing_groups);
   vector<string> matches(c->number_of_capturing_groups);
 
+  if (c->eof) {
+    return Qnil;
+  }
+
+  original_input_size = c->input->size();
+
   for (i = 0; i < c->number_of_capturing_groups; i++) {
     matches[i] = "";
     argv[i] = &matches[i];
@@ -224,15 +249,26 @@ static VALUE re2_scanner_scan(VALUE self) {
   if (RE2::FindAndConsumeN(c->input, *p->pattern, &args[0],
         c->number_of_capturing_groups)) {
     result = rb_ary_new2(c->number_of_capturing_groups);
+    new_input_size = c->input->size();
+    input_advanced = new_input_size < original_input_size;
+
     for (i = 0; i < c->number_of_capturing_groups; i++) {
       if (matches[i].empty()) {
         rb_ary_push(result, Qnil);
       } else {
         rb_ary_push(result, ENCODED_STR_NEW(matches[i].data(),
               matches[i].size(),
-              p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1"));
+              p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1"));
       }
     }
+
+    /* Check whether we've exhausted the input yet. */
+    c->eof = new_input_size == 0;
+
+    /* If the match didn't advance the input, we need to do this ourselves. */
+    if (!input_advanced && new_input_size > 0) {
+      c->input->remove_prefix(1);
+    }
   } else {
     result = Qnil;
   }
@@ -325,7 +361,7 @@ static VALUE re2_matchdata_begin(VALUE self, VALUE n) {
     offset = reinterpret_cast<uintptr_t>(match->data()) - reinterpret_cast<uintptr_t>(StringValuePtr(m->text));
 
     return ENCODED_STR_SUBLEN(StringValue(m->text), offset,
-           p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+           p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
   }
 }
 
@@ -356,7 +392,7 @@ static VALUE re2_matchdata_end(VALUE self, VALUE n) {
     offset = reinterpret_cast<uintptr_t>(match->data()) - reinterpret_cast<uintptr_t>(StringValuePtr(m->text)) + match->size();
 
     return ENCODED_STR_SUBLEN(StringValue(m->text), offset,
-           p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+           p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
   }
 }
 
@@ -420,7 +456,7 @@ static VALUE re2_matchdata_to_a(VALUE self) {
       rb_ary_push(array, Qnil);
     } else {
       rb_ary_push(array, ENCODED_STR_NEW(match->data(), match->size(),
-            p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1"));
+            p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1"));
     }
   }
 
@@ -444,7 +480,7 @@ static VALUE re2_matchdata_nth_match(int nth, VALUE self) {
       return Qnil;
     } else {
       return ENCODED_STR_NEW(match->data(), match->size(),
-          p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+          p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
     }
   }
 }
@@ -575,7 +611,7 @@ static VALUE re2_matchdata_inspect(VALUE self) {
   output << ">";
 
   result = ENCODED_STR_NEW(output.str().data(), output.str().length(),
-      p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+      p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
 
   return result;
 }
@@ -648,7 +684,7 @@ static VALUE re2_regexp_initialize(int argc, VALUE *argv, VALUE self) {
 
     utf8 = rb_hash_aref(options, ID2SYM(id_utf8));
     if (!NIL_P(utf8)) {
-      re2_options.set_utf8(RTEST(utf8));
+      re2_options.set_encoding(RTEST(utf8) ? RE2::Options::EncodingUTF8 : RE2::Options::EncodingLatin1);
     }
 
     posix_syntax = rb_hash_aref(options, ID2SYM(id_posix_syntax));
@@ -731,7 +767,7 @@ static VALUE re2_regexp_inspect(VALUE self) {
   output << "#<RE2::Regexp /" << p->pattern->pattern() << "/>";
 
   result = ENCODED_STR_NEW(output.str().data(), output.str().length(),
-      p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+      p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
 
   return result;
 }
@@ -749,7 +785,7 @@ static VALUE re2_regexp_to_s(VALUE self) {
   Data_Get_Struct(self, re2_pattern, p);
   return ENCODED_STR_NEW(p->pattern->pattern().data(),
       p->pattern->pattern().size(),
-      p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+      p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
 }
 
 /*
@@ -779,7 +815,7 @@ static VALUE re2_regexp_ok(VALUE self) {
 static VALUE re2_regexp_utf8(VALUE self) {
   re2_pattern *p;
   Data_Get_Struct(self, re2_pattern, p);
-  return BOOL2RUBY(p->pattern->options().utf8());
+  return BOOL2RUBY(p->pattern->options().encoding() == RE2::Options::EncodingUTF8);
 }
 
 /*
@@ -976,7 +1012,7 @@ static VALUE re2_regexp_error_arg(VALUE self) {
   } else {
     return ENCODED_STR_NEW(p->pattern->error_arg().data(),
         p->pattern->error_arg().size(),
-        p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+        p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
   }
 }
 
@@ -1007,7 +1043,7 @@ static VALUE re2_regexp_options(VALUE self) {
   options = rb_hash_new();
 
   rb_hash_aset(options, ID2SYM(id_utf8),
-      BOOL2RUBY(p->pattern->options().utf8()));
+      BOOL2RUBY(p->pattern->options().encoding() == RE2::Options::EncodingUTF8));
 
   rb_hash_aset(options, ID2SYM(id_posix_syntax),
       BOOL2RUBY(p->pattern->options().posix_syntax()));
@@ -1077,7 +1113,7 @@ static VALUE re2_regexp_named_capturing_groups(VALUE self) {
   for (iterator = groups.begin(); iterator != groups.end(); iterator++) {
     rb_hash_aset(capturing_groups,
         ENCODED_STR_NEW(iterator->first.data(), iterator->first.size(),
-          p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1"),
+          p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1"),
         INT2FIX(iterator->second));
   }
 
@@ -1216,6 +1252,7 @@ static VALUE re2_regexp_scan(VALUE self, VALUE text) {
   c->regexp = self;
   c->text = text;
   c->number_of_capturing_groups = p->pattern->NumberOfCapturingGroups();
+  c->eof = false;
 
   return scanner;
 }
@@ -1247,7 +1284,7 @@ static VALUE re2_Replace(VALUE self, VALUE str, VALUE pattern,
     RE2::Replace(&str_as_string, *p->pattern, StringValuePtr(rewrite));
 
     return ENCODED_STR_NEW(str_as_string.data(), str_as_string.size(),
-        p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+        p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
   } else {
     RE2::Replace(&str_as_string, StringValuePtr(pattern),
         StringValuePtr(rewrite));
@@ -1284,7 +1321,7 @@ static VALUE re2_GlobalReplace(VALUE self, VALUE str, VALUE pattern,
     RE2::GlobalReplace(&str_as_string, *p->pattern, StringValuePtr(rewrite));
 
     return ENCODED_STR_NEW(str_as_string.data(), str_as_string.size(),
-        p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
+        p->pattern->options().encoding() == RE2::Options::EncodingUTF8 ? "UTF-8" : "ISO-8859-1");
   } else {
     RE2::GlobalReplace(&str_as_string, StringValuePtr(pattern),
                        StringValuePtr(rewrite));
@@ -1349,6 +1386,8 @@ void Init_re2(void) {
 
   rb_define_method(re2_cScanner, "string",
       RUBY_METHOD_FUNC(re2_scanner_string), 0);
+  rb_define_method(re2_cScanner, "eof?",
+      RUBY_METHOD_FUNC(re2_scanner_eof), 0);
   rb_define_method(re2_cScanner, "regexp",
       RUBY_METHOD_FUNC(re2_scanner_regexp), 0);
   rb_define_method(re2_cScanner, "scan",