jasherai-oniguruma 1.1.1

data/History.txt

@@ -0,0 +1,56 @@
+== 1.1.1 /
+* Fixed oregexp.c issue with UTF16/32 etc. patch found at http://howflow.com/tricks/ruby_how_to_install_the_gem_oniguruma
+
+== 1.1.0 /
+* Fixed string escaping in ORegexp#to_str and ORegexp#inspect.
+* Added begin parameter to ORegexp#match.
+
+== 1.0.1 / 2007-03-28
+* Minimal recommended version of oniglib changed to be compatible with Ruby 1.9, now is 4.6 or higher.
+* Restore check for onig version to build with 4.6
+* In getting replacement do not create temp string object, but directly add to resulting buffer (performance impr.)
+* Included binary gems for windows.
+* Modified Rakefile to support win32 gems.
+
+== 1.0.0 / 2007-03-27
+* Added documentation for MatchData.
+* Added ogsub, ogsub!, sub and sub! to ::String.
+* Removed ::String definitions from tests.
+* Now the minimal recommended version of oniglib is 5.5 or higher.
+* Removed ugly #if statements from c code.
+* Do not create @named_captures hash if there are no named groups for regexp -- somewhat improve speed for repetive calls
+* Fixed usage of named backreferences in gsub with non-ascii names
+* Move ORegexp#=~ to C code, make it work just like Regexp#=~, i.e. set $~. Throw ArgumentError instead of Exception if pattern does not compile
+* Fix implementation of ORegexp#===, so it now does not raise errors in case statement anymore
+  (resembles plain Ruby Regexp#=== behaviour)
+* Modified begin, end and offset methods in MatchData to handle named groups and default to group 0.
+* Exception is not longer thrown when in oregexp_make_match_data.
+* Removed references to MultiMatchData from documentation
+* Removed class MultiMatchData
+* Fix off by one error in region->num_regs usage
+* Fix dumb bug with zero-width matches that made infinite loops. now consume at least one char in gsub and scan
+* ORegexp API changes:
+  * Pass only MatchData to sub/gsub with blocks
+      oregexp.sub( str ) {|match_data| ... }
+      oregexp.gsub( str ) {|match_data| ... }
+  * Add ORegexp#scan instead of match_all
+      oregexp.scan(str) {|match_data| ... } # => MultiMatchData
+  * Friendly way to set options 
+      ORegexp.new( pattern, options_str, encoding, syntax)
+      ORegexp.new('\w+', 'imsx', 'koi8r', 'perl')
+  *  Named backreferences in substitions 
+      ORegexp.new('(?<pre>\w+)\d+(?<after>\w+)').sub('abc123def', '\<after>123\<pre>')  #=> 'def123abc'
+* couple of bugfixes with region's num_regs
+* some  docs for substitution methods added
+
+== 0.9.1 / 2007-03-25
+* FIX: Buggy resolution of numeric codes for encoding and syntax options (Nikolai Lugovoi)
+* FIX: Buggy implementation of ORegexp#gsub and  ORegexp#gsub methods. Now code is all C (Nikolai Lugovoi)
+* Added documentation for class ORegexp
+* Added regexp syntax documentation.
+
+== 0.9.0 / 2007-03-19
+
+* 1 major enhancement
+  * Birthday!
+

data/Manifest.txt

@@ -0,0 +1,10 @@
+History.txt
+Manifest.txt
+README.txt
+Syntax.txt
+Rakefile
+lib/oniguruma.rb
+ext/oregexp.c
+ext/extconf.rb
+test/test_oniguruma.rb
+win/oregexp.so

data/README.txt

@@ -0,0 +1,71 @@
+== ONIGURUMA FOR RUBY:
+
+Ruby bindings to the Oniguruma[http://www.geocities.jp/kosako3/oniguruma/] regular expression library (no need to recompile Ruby).
+
+== FEATURES:
+  
+* Increased performance. 
+* Same interface than standard Regexp class (easy transition!). 
+* Support for named groups, look-ahead, look-behind, and other 
+  cool features!
+* Support for other regexp syntaxes (Perl, Python, Java, etc.)
+
+== SYNOPSIS:
+
+ reg = Oniguruma::ORegex.new( '(?<before>.*)(a)(?<after>.*)' )
+ match = reg.match( 'terraforming' )
+ puts match[0]         <= 'terraforming'
+ puts match[:before]   <= 'terr'
+ puts match[:after]    <= 'forming'
+ 
+== SYNTAX
+
+Consult the Syntax.txt[link:files/Syntax_txt.html] page.
+
+== REQUIREMENTS:
+
+* Oniguruma[http://www.geocities.jp/kosako3/oniguruma/] library v. 4.6 or higher
+
+== INSTALL:
+
+sudo gem install -r oniguruma
+
+== BUGS/PROBLEMS/INCOMPATIBILITIES:
+
+* <code>ORegexp#~</code> is not implemented.
+* <code>ORegexp#kcode</code> results are not compatible with <code>Regexp</code>.
+* <code>ORegexp</code> options set in the string are not visible, this affects
+  <code>ORegexp#options</code>, <code>ORegexp#to_s</code>, <code>ORegexp#inspect</code>
+  and <code>ORegexp#==</code>.
+
+== TODO:
+
+* Complete documentation (methods, oniguruma syntax).
+
+== CREDITS:
+
+* N. Lugovoi. ORegexp.sub and ORegexp.gsub code and lots of other stuff.
+* K. Kosako. For his great library.
+* A lot of the documentation has been copied from the original Ruby Regex documentation.
+
+== LICENSE:
+
+New BSD License
+
+Copyright (c) 2007, Dizan Vasquez
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the 
+  documentation and/or other materials provided with the distribution.
+* Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this 
+  software without specific prior written permission.
+            
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/Rakefile

@@ -0,0 +1,67 @@
+require 'rubygems'
+require 'hoe'
+
+rubyforge_name = "oniguruma"
+
+begin
+
+   class Hoe 
+      # Dirty hack to eliminate Hoe from gem dependencies
+      def extra_deps 
+         @extra_deps.delete_if{ |x| x.first == 'hoe' }
+      end
+      
+      # Dirty hack to package only the required files per platform
+      def spec= s 
+         if ENV['PLATFORM'] =~ /win32/
+            s.files = s.files.reject! {|f| f =~ /extconf\.rb/}
+         else
+            s.files = s.files.reject! {|f| f =~ /win\//}
+         end
+         @spec = s
+      end
+   end
+   
+   version = /^== *(\d+\.\d+\.\d+)/.match( File.read( 'History.txt' ) )[1]
+   
+   h = Hoe.new('oniguruma', version) do |p|
+      p.rubyforge_name = 'oniguruma'
+      p.author = ['Dizan Vasquez', 'Nikolai Lugovoi']
+      p.email = 'dichodaemon@gmail.com'
+      p.summary = 'Bindings for the oniguruma regular expression library'
+      p.description = p.paragraphs_of('README.txt', 1 ).join('\n\n')
+      p.url = 'http://oniguruma.rubyforge.org'
+      if ENV['PLATFORM'] =~ /win32/
+         p.lib_files = ["win/oregexp.so"]
+         p.spec_extras[:require_paths] = ["win", "lib", "ext" ]
+         p.spec_extras[:platform] = Gem::Platform::WIN32
+      else
+         p.spec_extras[:extensions] = ["ext/extconf.rb"]
+      end
+      p.rdoc_pattern = /^(lib|bin|ext)|txt$/
+      p.changes = p.paragraphs_of('History.txt', 0).join("\n\n")
+      p.clean_globs = ["manual/*"]
+   end
+
+   desc 'Create MaMa documentation'
+   task :mama => :clean do
+      system "mm -c -t refresh -o manual mm/manual.mm"
+   end
+   
+   desc 'Publish MaMa documentation to RubyForge'
+      task :mama_publish => [:clean, :mama] do
+      config = YAML.load(File.read(File.expand_path("~/.rubyforge/user-config.yml")))
+      host = "#{config["username"]}@rubyforge.org"
+      remote_dir = "/var/www/gforge-projects/#{h.rubyforge_name}"
+      local_dir = 'manual'
+      system "rsync -av #{local_dir}/ #{host}:#{remote_dir}"
+   end
+
+rescue LoadError => e
+   desc 'Run the test suite.'
+   task :test do
+      system "ruby -Ibin:lib:test test_#{rubyforge_name}.rb"
+   end
+end
+
+

data/Syntax.txt

@@ -0,0 +1,396 @@
+= RUBY REGULAR EXPRESSION SYNTAX
+
+
+== Syntax Elements
+
+[\]     escape (enable or disable meta character meaning)
+[|]     alternation
+[(...)] group
+[[...]] character class  
+
+
+== Characters
+
+[\t]           horizontal tab (0x09)
+[\v]           vertical tab   (0x0B)
+[\n]           newline        (0x0A)
+[\r]           return         (0x0D)
+[\b]           back space     (0x08)
+
+               \b is effective in character class [...] only
+[\f]           form feed      (0x0C)
+[\a]           bell           (0x07)
+[\e]           escape         (0x1B)
+[\nnn]         octal char            (encoded byte value)
+[\xHH]         hexadecimal char      (encoded byte value)
+[\x{7HHHHHHH}] wide hexadecimal char (character code point value)
+[\cx]          control char          (character code point value)
+[\C-x]         control char          (character code point value)
+[\M-x]         meta  (x|0x80)        (character code point value)
+[\M-\C-x]      meta control char     (character code point value)
+
+
+
+== Character types
+
+[.]        any character (except newline)
+[\w]       word character
+
+           Not Unicode:
+           * alphanumeric, "_" and multibyte char. 
+           Unicode:
+           * General_Category -- (Letter|Mark|Number|Connector_Punctuation)
+[\W]       non word char
+[\s]       whitespace char
+
+           Not Unicode:
+           * \t, \n, \v, \f, \r, \x20
+           Unicode:
+           * 0009, 000A, 000B, 000C, 000D, 0085(NEL), 
+           * General_Category:
+             * -- Line_Separator
+             * -- Paragraph_Separator
+             * -- Space_Separator
+[\S]       non whitespace char
+[\d]       decimal digit char
+
+           Unicode: General_Category -- Decimal_Number
+[\D]       non decimal digit char
+[\h]       hexadecimal digit char   [0-9a-fA-F]
+[\H]       non hexadecimal digit char
+
+
+== Character Properties
+
+ \p{property-name}
+ \p{^property-name}    (negative)
+ \P{property-name}     (negative)
+
+=== property-name:
+
+Works on all encodings:
+* Alnum, Alpha, Blank, Cntrl, Digit, Graph, Lower,
+  Print, Punct, Space, Upper, XDigit, Word, ASCII,
+Works on EUC_JP, Shift_JIS:
+* Hiragana, Katakana
+Works on UTF8, UTF16, UTF32:
+* Any, Assigned, C, Cc, Cf, Cn, Co, Cs, L, Ll, Lm, Lo, Lt, Lu,
+  M, Mc, Me, Mn, N, Nd, Nl, No, P, Pc, Pd, Pe, Pf, Pi, Po, Ps,
+  S, Sc, Sk, Sm, So, Z, Zl, Zp, Zs, 
+  Arabic, Armenian, Bengali, Bopomofo, Braille, Buginese,
+  Buhid, Canadian_Aboriginal, Cherokee, Common, Coptic,
+  Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian,
+  Glagolitic, Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul,
+  Hanunoo, Hebrew, Hiragana, Inherited, Kannada, Katakana,
+  Kharoshthi, Khmer, Lao, Latin, Limbu, Linear_B, Malayalam,
+  Mongolian, Myanmar, New_Tai_Lue, Ogham, Old_Italic, Old_Persian,
+  Oriya, Osmanya, Runic, Shavian, Sinhala, Syloti_Nagri, Syriac,
+  Tagalog, Tagbanwa, Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan,
+  Tifinagh, Ugaritic, Yi
+
+== Quantifiers
+
+=== Greedy
+
+[?]       1 or 0 times
+[*]       0 or more times
+[+]       1 or more times
+[{n,m}]   at least n but not more than m times
+[{n,}]    at least n times
+[{,n}]    at least 0 but not more than n times ({0,n})
+[{n}]     n times
+
+=== Reluctant
+
+[??]      1 or 0 times
+[*?]      0 or more times
+[+?]      1 or more times
+[{n,m}?]  at least n but not more than m times  
+[{n,}?]   at least n times
+[{,n}?]   at least 0 but not more than n times (== {0,n}?)
+
+=== Possessive (greedy and does not backtrack after repeated)
+
+[?+]      1 or 0 times
+[*+]      0 or more times
+[++]      1 or more times
+
+({n,m}+, {n,}+, {n}+ are possessive op. in ONIG_SYNTAX_JAVA only)
+
+
+== Anchors
+
+[^]       beginning of the line
+[$]       end of the line
+[\b]      word boundary
+[\B]      not word boundary
+[\A]      beginning of string
+[\Z]      end of string, or before newline at the end
+[\z]      end of string
+[\G]      matching start position 
+
+
+== Character class
+
+[^...]    negative class (lowest precedence operator)
+[x-y]     range from x to y
+[[...]]   set (character class in character class)
+[..&&..]  intersection (low precedence at the next of ^)
+          
+If you want to use '[', '-', ']' as a normal character
+in a character class, you should escape these characters by '\'.
+
+
+POSIX bracket ([:xxxxx:], negate [:^xxxxx:])
+
+=== Not Unicode Case:
+
+[alnum]    alphabet or digit char
+[alpha]    alphabet
+[ascii]    code value: [0 - 127]
+[blank]    \t, \x20
+[cntrl]    control
+[digit]    0-9
+[graph]    include all of multibyte encoded characters
+[lower]    lower case
+[print]    include all of multibyte encoded characters
+[punct]    punctuation
+[space]    \t, \n, \v, \f, \r, \x20
+[upper]    upper case
+[xdigit]   0-9, a-f, A-F
+[word]     alphanumeric, "_" and multibyte characters
+
+
+=== Unicode Case:
+
+[alnum]    Letter | Mark | Decimal_Number
+[alpha]    Letter | Mark
+[ascii]    0000 - 007F
+[blank]    Space_Separator | 0009
+[cntrl]    Control | Format | Unassigned | Private_Use | Surrogate
+[digit]    Decimal_Number
+[graph]    [[:^space:]] && ^Control && ^Unassigned && ^Surrogate
+[lower]    Lowercase_Letter
+[print]    [[:graph:]] | [[:space:]]
+[punct]    Connector_Punctuation | Dash_Punctuation | Close_Punctuation |
+           Final_Punctuation | Initial_Punctuation | Other_Punctuation |
+           Open_Punctuation
+[space]    Space_Separator | Line_Separator | Paragraph_Separator |
+           0009 | 000A | 000B | 000C | 000D | 0085
+[upper]    Uppercase_Letter
+[xdigit]   0030 - 0039 | 0041 - 0046 | 0061 - 0066
+           (0-9, a-f, A-F)
+[word]     Letter | Mark | Decimal_Number | Connector_Punctuation
+
+
+
+== Extended groups
+
+[(?#...)]            comment
+[(?imx-imx)]         option on/off:
+                     * i: ignore case
+                     * m: multi-line (dot(.) match newline)
+                     * x: extended form
+[(?imx-imx:subexp)]  option on/off for subexp
+[(?:subexp)]         not captured group
+[(subexp)]           captured group
+[(?=subexp)]         look-ahead
+[(?!subexp)]         negative look-ahead
+[(?<=subexp)]        look-behind
+[(?<!subexp)]        negative look-behind
+
+                     Subexp of look-behind must be fixed character length.
+                     But different character length is allowed in top level
+                     alternatives only.
+                     ex. (?<=a|bc) is OK. (?<=aaa(?:b|cd)) is not allowed.
+                     
+                     In negative-look-behind, captured group isn't allowed, 
+                     but shy group(?:) is allowed.
+[(?>subexp)]         atomic group
+                     don't backtrack in subexp.
+[(?<name>subexp)]    define named group
+                     (All characters of the name must be a word character.)
+                     
+                     Not only a name but a number is assigned like a captured
+                     group.
+
+                     Assigning the same name as two or more subexps is allowed.
+                     In this case, a subexp call can not be performed although
+                     the back reference is possible.
+
+
+== Back reference
+
+[\n]          back reference by group number (n >= 1)
+[\k<name>]    back reference by group name
+In the back reference by the multiplex definition name,
+a subexp with a large number is referred to preferentially.
+(When not matched, a group of the small number is referred to.)
+
+* Back reference by group number is forbidden if named group is defined 
+  in the pattern and ONIG_OPTION_CAPTURE_GROUP is not setted.
+
+
+=== Back reference with nest level
+
+[\k<name+n>]     n: 0, 1, 2, ...
+[\k<name-n>]     n: 0, 1, 2, ...
+
+Destinate relative nest level from back reference position.    
+
+Examples:
+   /\A(?<a>|.|(?:(?<b>.)\g<a>\k<b+0>))\z/.match("reer")
+
+   r = ORegexp.compile(<<'__REGEXP__'.strip, :options => Oniguruma::EXTENDED)
+   (?<element> \g<stag> \g<content>* \g<etag> ){0}
+   (?<stag> < \g<name> \s* > ){0}
+   (?<name> [a-zA-Z_:]+ ){0}
+   (?<content> [^<&]+ (\g<element> | [^<&]+)* ){0}
+   (?<etag> </ \k<name+1> >){0}
+   \g<element>
+   __REGEXP__
+   
+   p r.match('<foo>f<bar>bbb</bar>f</foo>').captures
+
+
+
+=== Subexp call ("Tanaka Akira special")
+
+[\g<name>]    call by group name
+[\g<n>]       call by group number (n >= 1)
+
+* left-most recursive call is not allowed.
+
+  Example:
+    (?<name>a|\g<name>b)   => error
+    (?<name>a|b\g<name>c)  => OK
+* Call by group number is forbidden if named group is defined in the pattern
+  and Oniguruma::OPTION_CAPTURE_GROUP is not set.
+* If the option status of called group is different from calling position
+  then the group's option is effective.
+  
+  Example:
+    (?-i:\g<name>)(?i:(?<name>a)){0}  <i>matches "A"</i>
+
+
+== Captured group
+
+Behavior of the no-named group (...) changes with the following conditions.
+(But named group is not changed.)
+
+[case 1]                         <code>ORegexp.new( '...' )</code>  (named group is not used, no option)
+
+                                 ... is treated as a captured group.
+[case 2]                         <code>ORegexp.new( '...', :options => OPTION_DONT_CAPTURE_GROUP )</code> (named group is not used, 'g' option)
+
+                                 ... is treated as a no-captured group (?:...).
+
+[case 3]                         <code>ORegexp.new( '...(?<name>...)...' )</code>  (named group is used, no option)
+
+                                 (?<name>...) is treated as a no-captured group (?:...)
+                                 
+                                 numbered-backref/call is not allowed.
+
+[case 2]                         <code>ORegexp.new( '...', :options => OPTION_CAPTURE_GROUP )</code> (named group is used, 'G' option)
+
+                                 (?<name>...) is treated as a captured group (?:...)
+
+                                 numbered-backref/call is allowed.
+
+where
+* g: OPTION_DONT_CAPTURE_GROUP
+* G: OPTION_CAPTURE_GROUP
+
+('g' and 'G' options are argued in ruby-dev ML)
+
+
+== Syntax dependent options
+
+=== ONIG_SYNTAX_RUBY
+
+[(?m)] dot(.) match newline
+
+=== ONIG_SYNTAX_PERL and ONIG_SYNTAX_JAVA
+
+[(?s)] dot(.) match newline
+[(?m)] ^ match after newline, $ match before newline
+
+== Original extensions
+
+* hexadecimal digit char type  \h, \H
+* named group                  (?<name>...)
+* named backref                \k<name>
+* subexp call                  \g<name>, \g<group-num>
+
+
+== Lacking features compare with perl 5.8.0
+
+* \N{name}
+* \l,\u,\L,\U, \X, \C
+* (?{code})
+* (??{code})
+* (?(condition)yes-pat|no-pat)
+* \Q...\E
+
+  This is effective on ONIG_SYNTAX_PERL and ONIG_SYNTAX_JAVA.
+
+
+== Differences with Japanized GNU regex(version 0.12) of Ruby 1.8
+
+* add character property (\p{property}, \P{property})
+* add hexadecimal digit char type (\h, \H)
+* add look-behind
+
+  (?<=fixed-char-length-pattern), (?<!fixed-char-length-pattern)
+* add possessive quantifier. ?+, *+, ++
+* add operations in character class. [], &&
+
+  ('[' must be escaped as an usual char in character class.)
+* add named group and subexp call.
+* octal or hexadecimal number sequence can be treated as 
+  a multibyte code char in character class if multibyte encoding
+  is specified.
+  
+  (ex. <code>[\xa1\xa2], [\xa1\xa7-\xa4\xa1]</code>)
+* allow the range of single byte char and multibyte char in character
+  class.
+  
+  ex. <code>[a-<<any EUC-JP character>>]</code> in EUC-JP encoding.
+* effect range of isolated option is to next ')'.
+  ex. (?:(?i)a|b) is interpreted as (?:(?i:a|b)), not (?:(?i:a)|b).
+* isolated option is not transparent to previous pattern.
+  ex. <code>a(?i)*</code> is a syntax error pattern.
+* allowed incompleted left brace as an usual string.
+  ex. /{/, /({)/, /a{2,3/ etc...
+* negative POSIX bracket [:^xxxx:] is supported.
+* POSIX bracket [:ascii:] is added.
+* repeat of look-ahead is not allowed.
+  ex. <code>(?=a)*</code>, <code>(?!b){5}</code>
+* Ignore case option is effective to numbered character.
+  ex. <code>/\x61/i =~ "A"<code>
+* In the range quantifier, the number of the minimum is omissible.
+
+  <code>/a{,n}/ == /a{0,n}/<code>
+  
+  The simultanious abbreviation of the number of times of the minimum
+  and the maximum is not allowed. (/a{,}/)
+* <code>a{n}?<code> is not a non-greedy operator.
+  <code>/a{n}?/ == /(?:a{n})?/<code>
+* invalid back reference is checked and cause error.
+  /\1/, /(a)\2/
+* Zero-length match in infinite repeat stops the repeat,
+  then changes of the capture group status are checked as stop condition.
+   /(?:()|())*\1\2/ =~ ""
+   /(?:\1a|())*/ =~ "a"
+
+
+== Problems
+
+* Invalid encoding byte sequence is not checked in UTF-8.
+
+* Invalid first byte is treated as a character.
+   /./u =~ "\xa3"
+
+* Incomplete byte sequence is not checked.
+   /\w+/ =~ "a\xf3\x8ec"
+