regexp_parser 0.1.0

data/ChangeLog

@@ -0,0 +1,4 @@
+
+Sat Nov 20 16:40:10 EET 2010 Ammar Ali <ammarabuali@gmail.com>
+
+	* Initial version bump to 0.1.0

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2010 Ammar Ali
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,307 @@
+= Regexp::Parser
+
+== What?
+A ruby library to help with lexing, parsing, and transforming regular expressions.
+
+* Multilayered
+
+  * A scanner based on ragel[http://www.complang.org/ragel/]
+  * A lexer that produces a "stream" of tokens
+  * A parser that produces a "tree" of Regexp::Expression objects (OO API)
+
+* Lexes and parses both 1.8 and 1.9 regular expression flavors
+* Supports ruby 1.8 and 1.9 runtime
+
+For an example of regexp_parser in use, see the meta_re project[https://github.com/ammar/meta_re]
+
+---
+== Requirements
+
+* ruby '1.8.6'..'1.9.2'
+* ragel, but only if you want to hack on the scanner
+
+
+---
+== Install
+
+  gem install regexp_parser
+
+---
+== Components
+=== Scanner
+A ragel generated scanner that recognizes the cumulative syntax of both
+supported flavors. Breaks the expression's text into tokens, including
+their type, token, text, and start/end offsets within the original
+pattern.
+
+==== Example
+The following scans the given pattern and prints out the type, token, text and
+start/end offsets for each token found.
+
+  require 'regexp_parser'
+
+  Regexp::Scanner.scan /(ab?(cd)*[e-h]+)/  do |type, token, text, ts, te|
+    puts "type: #{type}, token: #{token}, text: '#{text}' [#{ts}..#{te}]"
+  end
+
+A one-liner that returns an array of the textual parts of the given pattern:
+
+  Regexp::Scanner.scan( /(cat?([b]at)){3,5}/ ).map {|token| token[2]}
+  #=> ["(", "cat", "?", "(", "[", "b", "]", "at", ")", ")", "{3,5}"]
+
+
+==== Notes
+  * The scanner performs basic syntax error checking, like detecting missing
+    balancing punctuation and premature end of pattern. Flavor validity checks
+    are performed in the lexer.
+
+  * To keep the scanner simple(r) and fairly reusable for other uses, it
+    does not perform lexical analysis on the tokens, sticking to the task
+    of tokenizing and leaving lexical analysis upto to the lexer.
+
+  * If the input is a ruby Regexp object, the scanner calls #source on it to
+    get its string representation. #source does not include the options of
+    expression (m, i, and x) To include the options the scan, #to_s should
+    be called on the Regexp before passing it to the scanner, or any of the
+    higher layers.
+
+
+---
+=== Syntax
+Defines the supported tokens for a specific engine implementation (aka a
+flavor). Syntax classes act as lookup tables, and are layered to create
+flavor variations. Syntax only comes into play in the lexer.
+
+==== Example
+The following instantiates the syntax for Ruby 1.9 and checks a couple of its
+implementations features, and then does the same for Ruby 1.8:
+
+  require 'regexp_parser'
+
+  ruby_19 = Regexp::Syntax.new 'ruby/1.9'
+  ruby_19.implements? :quantifier, :zero_or_one             # => true
+  ruby_19.implements? :quantifier, :zero_or_one_reluctant   # => true
+  ruby_19.implements? :quantifier, :zero_or_one_possessive  # => true
+
+  ruby_18 = Regexp::Syntax.new 'ruby/1.8'
+  ruby_18.implements? :quantifier, :zero_or_one             # => true
+  ruby_18.implements? :quantifier, :zero_or_one_reluctant   # => true
+  ruby_18.implements? :quantifier, :zero_or_one_possessive  # => false
+
+
+==== Notes
+  * Variatiions on a token, for example a named group with < and > vs one with a
+    pair of single quotes, are specified with an underscore followed by two
+    characters appended to the base token. In the previous named group example,
+    the tokens would be :named_ab (angle brackets) and :named_sq (single quotes).
+    These variations are normalized by the syntax to :named.
+
+==== TODO
+  * Add flavor limits: like Ruby 1.8's maximum allowed number of grouped 
+    expressions (253).
+
+
+---
+=== Lexer
+Sits on top of the scanner and performs lexical analysis on the tokens that
+it emits. Among its tasks are breaking quantified literal runs, collecting the
+emitted token structures into an array of Token objects, calculating their
+nesting depth, normalizing tokens for the parser, and checkng if the tokens
+are implemented by the given syntax flavor.
+
+Tokens objects are Structs, basically data objects, with a few helper methods,
+like #next, #previous, #offsets and #length.
+
+==== Example
+The following example scans the given pattern, checks it against the ruby 1.8
+syntax, and prints the token objects' text.
+
+  require 'regexp_parser'
+
+  Regexp::Lexer.scan(/a?(b)*[c]+/, 'ruby/1.8') do |token|
+    puts "#{'  ' * token.depth}#{token.text}"
+  end
+
+A one-liner that returns an array of the textual parts of the given pattern.
+Compare the output with that of the one-liner example of the Scanner.
+
+  Regexp::Lexer.scan( /(cat?([b]at)){3,5}/ ).map {|token| token.text}
+  #=> ["(", "ca", "t", "?", "(", "[", "b", "]", "at", ")", ")", "{3,5}"]
+
+==== Notes
+  * The default syntax is that of the latest released version of ruby.
+
+  * The lexer performs some basic parsing to determine the depth of a the
+    emitted tokens. This responsibility might be relegated to the scanner.
+
+
+---
+=== Parser
+Sits on top of the lexer and transforms the "stream" of Token objects emitted
+by it into a tree of Expression objects represented by an instance of the
+Expression::Root class. See Expression below for more information.
+
+==== Example
+
+  require 'regexp_parser'
+
+  regex = /a?(b)*[c]+/m
+
+  # using #to_s on the Regexp object to include options
+  root = Regexp::Parser.parse( regex.to_s, 'ruby/1.8')
+
+  root.multiline?         # => true (aliased as m?)
+  root.case_insensitive?  # => false (aliased as i?)
+
+  # simple tree walking method
+  def walk(e, depth = 0)
+    puts "#{'  ' * depth}> #{e.class}"
+    unless e.expressions.empty?
+      e.each {|s| walk(s, depth+1) }
+    end
+  end
+
+  walk(root)
+
+  # output
+  > Regexp::Expression::Root
+    > Regexp::Expression::Literal
+    > Regexp::Expression::Group::Capture
+      > Regexp::Expression::Literal
+    > Regexp::Expression::CharacterSet
+
+Note: quantifiers do not appear in the output because they are members of the
+Expression class. See the next section for more details.
+
+---
+=== Expression
+The base class of all objects returned by the parser, implements most of the
+functions that are common to all expression classes.
+
+Each Expression object contains the following members:
+
+  * quantifier: an instance of Expression::Quantifier that holds the details
+    of repetition for the Expression. Has a nil value if the expressions is not
+    quantified.
+
+  * expressions: an array, holds the sub-expressions for the expression if it
+    is a group or alternation expression. Empty if the expression doesn't have
+    sub-expressions.
+
+  * options: a hash, holds the keys :i, :m, and :x with a boolean value that
+    indicates if the expression has a given option.
+
+Expressions also contain the following "lower level" members
+(from the scanner/lexer)
+
+  * type: a symbol, denoting the expression type, such as :group, :quantifier
+  * token: a symbol, for the object's token, or opening token (in the case of
+    groups and sets)
+  * text: a string, the text of the expression (same as token for nesting expressions)
+
+Every expressions also has the following methods:
+
+  * to_s: returns the string representation of the expression.
+  * <<: adds sub-expresions to the expression.
+  * each: iterates over the expressions sub-expressions, if any.
+  * []: access sub-expressions by index.
+  * quantified?: return true if the expressions was followed by a quantifier.
+  * quantity: returns an array of the expression's min and max repetitions.
+  * greedy?: returns true if the expression's quantifier is greedy.
+  * reluctant? or lazy?: returns true if the expression's quantifier is
+    reluctant.
+  * possessive?: returns true if the expression's quantifier is possessive.
+  * multiline? or m?: returns true if the expression has the m option
+  * case_insensitive? or ignore_case? or i?: returns true if the expression
+    has the i option
+  * free_spacing? or extended? or x?: returns true if the expression has the x
+    option
+
+A special expression class Expression::Sequence is used to hold the array of
+possible alternatives within an Expression::Alternation expression.
+
+
+== Scanner Syntax
+The following syntax elements are supported by the scanner. 
+
+  - Alternation: a|b|c, etc.
+  - Anchors: ^, $, \b, etc.
+  - Character Classes (aka Sets): [abc], [^\]]
+  - Character Types: \d, \H, \s, etc.
+  - Escape Sequences: \t, \+, \?, etc.
+  - Grouped Expressions
+    - Assertions
+      - Lookahead: (?=abc)
+      - Negative Lookahead: (?!abc)
+      - Lookabehind: (?<=abc)
+      - Negative Lookbehind: (?<\!abc)
+    - Atomic: (?>abc)
+    - Back-references:
+      - Named: \k<name>
+      - Nest Level: \k<n-1>
+      - Numbered: \k<1>
+      - Relative: \k<-2>
+    - Capturing: (abc)
+    - Comment: (?# comment)
+    - Named: (?<name>abc)
+    - Options: (?mi-x:abc)
+    - Passive: (?:abc)
+    - Sub-expression Calls: \g<name>, \g<1>
+  - Literals: abc, def?, etc.
+  - POSIX classes: [:alpha:], [:print:], etc.
+  - Quantifiers
+    - Greedy: ?, *, +, {m,M}
+    - Reluctant: ??, *?, +?, {m,M}?
+    - Possessive: ?+, *+, ++, {m,M}+
+  - String Escapes
+    - Control: \C-C, \cD, etc.
+    - Hex: \x20, \x{701230}, etc.
+    - Meta: \M-c, \M-\C-C etc.
+    - Octal: \0, \01, \012
+    - Unicode: \uHHHH, \u{H+ H+}
+  - Traditional Back-references: \1 thru \9
+  - Unicode Properties: 
+    - Age: \p{Age=2.1}, \P{age=5.2}, etc.
+    - Classes: \p{Alpha}, \P{Space}, etc.
+    - Derived Properties: \p{Math}, \P{Lowercase}, etc.
+    - General Categories: \p{Lu}, \P{Cs}, etc.
+    - Scripts: \p{Arabic}, \P{Hiragana}, etc.
+    - Simple Properties: \p{Dash}, \p{Extender}, etc.
+
+See something missing? Please submit an issue[https://github.com/ammar/regexp_parser/issues]
+
+== References
+Documentation and information being read while working on this project.
+
+==== Ruby Flavors
+* Oniguruma Regular Expressions link[http://www.geocities.jp/kosako3/oniguruma/doc/RE.txt]
+* Read Ruby > Regexps link[http://ruby.runpaint.org/regexps]
+
+
+==== General
+* Enumerating the strings of regular languages link[http://www.cs.dartmouth.edu/~doug/nfa.ps.gz]
+* Mastering Regular Expressions, By Jeffrey E.F. Friedl (2nd Edition) book[http://oreilly.com/catalog/9781565922570/]
+* Regular Expression Flavor Comparison link[http://www.regular-expressions.info/refflavors.html]
+
+
+==== Unicode
+* Unicode Derived Properties link[http://www.unicode.org/Public/UNIDATA/DerivedCoreProperties.txt]
+* Unicode Explained, By Jukka K. Korpela. book[http://oreilly.com/catalog/9780596101213]
+* Unicode Property Aliases link[http://www.unicode.org/Public/UNIDATA/PropertyAliases.txt]
+* Unicode Regular Expressions link[http://www.unicode.org/reports/tr18/]
+* Unicode Standard Annex #44 link[http://www.unicode.org/reports/tr44/]
+
+== Thanks
+This work is based on and inspired by the hard work and ideas of many people,
+directly or indirectly. The following are only a few of those that should be 
+thanked.
+
+* Adrian Thurston, for developing ragel[http://www.complang.org/ragel/].
+* Caleb Clausen, for feedback, which inspired this,  valuable insights on structuring the parser,
+  and lots of cool code[http://github.com/coatl].
+* Jan Goyvaerts, for his excellent resource[http://www.regular-expressions.info] on regular expressions. I owe him a "steak dinner", at least.
+* Run Paint Run Run, for his work on Read[http://ruby.runpaint.org/] Ruby
+* Yukihiro Matsumoto, of course! For "The Ruby", of course!
+
+== Copyright
+Copyright (c) 2010 Ammar Ali. See LICENSE file for details.

data/Rakefile

@@ -0,0 +1,91 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/gempackagetask'
+
+task :default => [:test]
+
+RAGEL_SOURCE_DIR = File.expand_path '../lib/regexp_parser/scanner', __FILE__
+RAGEL_OUTPUT_DIR = File.expand_path '../lib/regexp_parser', __FILE__
+
+RAGEL_SOURCE_FILES = %w{scanner}
+
+desc "Find and run all unit tests under test/ directory"
+Rake::TestTask.new("test") do |t|
+  t.libs << "test"
+  t.test_files = FileList['test/**/test_*.rb']
+end
+
+task :test
+
+namespace :test do
+  desc "Run all scanner tests"
+  Rake::TestTask.new("scanner") do |t|
+    t.libs << "test"
+    t.test_files = ['test/scanner/test_all.rb']
+  end
+
+  desc "Run all lexer tests"
+  Rake::TestTask.new("lexer") do |t|
+    t.libs << "test"
+    t.test_files = ['test/lexer/test_all.rb']
+  end
+
+  desc "Run all parser tests"
+  Rake::TestTask.new("parser") do |t|
+    t.libs << "test"
+    t.test_files = ['test/parser/test_all.rb']
+  end
+
+  desc "Run all syntax tests"
+  Rake::TestTask.new("syntax") do |t|
+    t.libs << "test"
+    t.test_files = ['test/syntax/test_all.rb']
+  end
+end
+
+namespace :ragel do
+  desc "Process the ragel source files and output ruby code"
+  task :rb do |t|
+    RAGEL_SOURCE_FILES.each do |file|
+      # using faster flat table driven FSM, about 25% larger code, but about 30% faster
+      sh "ragel -F1 -R #{RAGEL_SOURCE_DIR}/#{file}.rl -o #{RAGEL_OUTPUT_DIR}/#{file}.rb"
+    end
+  end
+
+  desc "Delete the ragel generated source file(s)"
+  task :clean do |t|
+    RAGEL_SOURCE_FILES.each do |file|
+      sh "rm -f #{RAGEL_OUTPUT_DIR}/#{file}.rb"
+    end
+  end
+end
+
+spec = Gem::Specification.new do |s|
+  s.name = 'regexp_parser'
+  s.version = '0.1.0'
+  s.summary = %q{Scanner, lexer, parser for ruby's regular expressions}
+  s.description = %q{A library for tokenizing, lexing, and parsing Ruby regular expressions.}
+  s.date = '2010-10-01'
+  s.authors = ["Ammar Ali"]
+  s.email = 'ammarabuali@gmail.com'
+  s.homepage = %q{http://github.com/ammar/regexp_parser}
+  s.rdoc_options = ["--inline-source", "--charset=UTF-8"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.7}
+
+  s.files = Dir.glob("{lib,test}/**/*.rb") + Dir.glob("lib/**/*.rl") +
+            %w(Rakefile LICENSE README.rdoc ChangeLog)
+
+  s.test_files = Dir.glob("test/**/*.rb")
+  s.extra_rdoc_files = ["ChangeLog", "LICENSE", "README.rdoc"]
+  s.required_rubygems_version = ">= 1.3.7"
+  s.rubyforge_project = "regexp_parser"
+  s.require_path = 'lib'
+end
+
+Rake::GemPackageTask.new(spec) do |pkg|
+  Rake::Task['ragel:rb'].execute
+
+  pkg.need_zip = true
+  pkg.need_tar = true
+end

data/lib/regexp_parser/ctype.rb

@@ -0,0 +1,48 @@
+# defines character type constants (as arrays) and methods that test
+# whether a given character belongs in one of them.
+module CType
+  Digit   = ('0'..'9').to_a.freeze
+  Lower   = ('a'..'z').to_a.freeze
+  Upper   = ('A'..'Z').to_a.freeze
+
+  Alpha   = [Lower, Upper].flatten.freeze
+  Alnum   = [Alpha, Digit].flatten.freeze
+  Word    = [Alnum, '_'].flatten.freeze
+
+  Blank   = [' ', "\t"].freeze
+  Space   = [" ", "\t", "\r", "\n", "\v", "\f"].freeze
+
+  Cntrl   = ( 0..31 ).map {|c| c.chr}.freeze
+  Graph   = (33..126).map {|c| c.chr}.freeze
+  Print   = (32..126).map {|c| c.chr}.freeze
+  ASCII   = ( 0..127).map {|c| c.chr}.freeze
+
+  Punct   = [
+    ('!'..'/').to_a,
+    (':'..'@').to_a,
+    ('['..'`').to_a,
+    ('{'..'~').to_a
+  ].flatten.freeze
+
+  XDigit  = [
+    Digit,
+    ('a'..'f').to_a,
+    ('A'..'F').to_a
+  ].flatten.freeze
+
+  def self.alnum?(c);  Alnum.include?(c)  end 
+  def self.alpha?(c);  Alpha.include?(c)  end 
+  def self.blank?(c);  Blank.include?(c)  end 
+  def self.cntrl?(c);  Cntrl.include?(c)  end 
+  def self.digit?(c);  Digit.include?(c)  end 
+  def self.graph?(c);  Graph.include?(c)  end 
+  def self.lower?(c);  Lower.include?(c)  end 
+  def self.print?(c);  Print.include?(c)  end 
+  def self.punct?(c);  Punct.include?(c)  end 
+  def self.space?(c);  Space.include?(c)  end 
+  def self.upper?(c);  Upper.include?(c)  end 
+  def self.xdigit?(c); XDigit.include?(c) end 
+
+  def self.word?(c);   Word.include?(c)   end 
+  def self.ascii?(c);  ASCII.include?(c)  end 
+end