regex-treetop 1.4.8

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2007 Nathan Sobo.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,164 @@
+Tutorial
+========
+Languages can be split into two components, their *syntax* and their *semantics*. It's your understanding of English syntax that tells you the stream of words "Sleep furiously green ideas colorless" is not a valid sentence. Semantics is deeper. Even if we rearrange the above sentence to be "Colorless green ideas sleep furiously", which is syntactically correct, it remains nonsensical on a semantic level. With Treetop, you'll be dealing with languages that are much simpler than English, but these basic concepts apply. Your programs will need to address both the syntax and the semantics of the languages they interpret.
+
+Treetop equips you with powerful tools for each of these two aspects of interpreter writing. You'll describe the syntax of your language with a *parsing expression grammar*. From this description, Treetop will generate a Ruby parser that transforms streams of characters written into your language into *abstract syntax trees* representing their structure. You'll then describe the semantics of your language in Ruby by defining methods on the syntax trees the parser generates.
+
+Parsing Expression Grammars, The Basics
+=======================================
+The first step in using Treetop is defining a grammar in a file with the `.treetop` extension. Here's a grammar that's useless because it's empty:
+    
+    # my_grammar.treetop
+    grammar MyGrammar
+    end
+
+Next, you start filling your grammar with rules. Each rule associates a name with a parsing expression, like the following:
+
+    # my_grammar.treetop
+    # You can use a .tt extension instead if you wish
+    grammar MyGrammar
+      rule hello
+        'hello chomsky'
+      end
+    end
+
+The first rule becomes the *root* of the grammar, causing its expression to be matched when a parser for the grammar is fed a string. The above grammar can now be used in a Ruby program. Notice how a string matching the first rule parses successfully, but a second nonmatching string does not.
+
+    # use_grammar.rb
+    require 'rubygems'
+    require 'treetop'
+    Treetop.load 'my_grammar'
+    # or just:
+    # require 'my_grammar'                     # This works because Polyglot hooks "require" to find and load Treetop files
+    
+    parser = MyGrammarParser.new
+    puts parser.parse('hello chomsky')         # => Treetop::Runtime::SyntaxNode
+    puts parser.parse('silly generativists!')  # => nil
+
+Users of *regular expressions* will find parsing expressions familiar. They share the same basic purpose, matching strings against patterns. However, parsing expressions can recognize a broader category of languages than their less expressive brethren. Before we get into demonstrating that, lets cover some basics. At first parsing expressions won't seem much different. Trust that they are.
+
+Terminal Symbols
+----------------
+The expression in the grammar above is a terminal symbol. It will only match a string that matches it exactly. There are two other kinds of terminal symbols, which we'll revisit later. Terminals are called *atomic expressions* because they aren't composed of smaller expressions.
+
+Ordered Choices
+---------------
+Ordered choices are *composite expressions*, which allow for any of several subexpressions to be matched. These should be familiar from regular expressions, but in parsing expressions, they are delimited by the `/` character. Its important to note that the choices are prioritized in the order they appear. If an earlier expression is matched, no subsequent expressions are tried. Here's an example:
+
+    # my_grammar.treetop
+    grammar MyGrammar
+      rule hello
+        'hello chomsky' / 'hello lambek'
+      end
+    end
+    
+    # fragment of use_grammar.rb
+    puts parser.parse('hello chomsky')         # => Treetop::Runtime::SyntaxNode
+    puts parser.parse('hello lambek')          # => Treetop::Runtime::SyntaxNode
+    puts parser.parse('silly generativists!')  # => nil
+
+Note that once a choice rule has matched the text using a particular alternative at a particular location in the input and hence has succeeded, that choice will never be reconsidered, even if the chosen alternative causes another rule to fail where a later alternative wouldn't have. It's always a later alternative, since the first to succeed is final - why keep looking when you've found what you wanted? This is a feature of PEG parsers that you need to understand if you're going to succeed in using Treetop. In order to memoize success and failures, such decisions cannot be reversed. Luckily Treetop provides a variety of clever ways you can tell it to avoid making the wrong decisions. But more on that later.
+
+Sequences
+---------
+Sequences are composed of other parsing expressions separated by spaces. Using sequences, we can tighten up the above grammar.
+
+    # my_grammar.treetop
+    grammar MyGrammar
+      rule hello
+        'hello ' ('chomsky' / 'lambek')
+      end
+    end
+
+Note the use of parentheses to override the default precedence rules, which bind sequences more tightly than choices.
+
+Once the whole sequence has been matched, the result is memoized and the details of the match will not be reconsidered for that location in the input.
+
+Nonterminal Symbols
+-------------------
+Here we leave regular expressions behind. Nonterminals allow expressions to refer to other expressions by name. A trivial use of this facility would allow us to make the above grammar more readable should the list of names grow longer.
+
+    # my_grammar.treetop
+    grammar MyGrammar
+      rule hello
+        'hello ' linguist
+      end
+      
+      rule linguist
+        'chomsky' / 'lambek' / 'jacobsen' / 'frege'
+      end
+    end
+
+The true power of this facility, however, is unleashed when writing *recursive expressions*. Here is a self-referential expression that can match any number of open parentheses followed by any number of closed parentheses. This is theoretically impossible with regular expressions due to the *pumping lemma*.
+
+    # parentheses.treetop
+    grammar Parentheses
+      rule parens
+        '(' parens ')' / ''
+      end
+    end
+
+
+The `parens` expression simply states that a `parens` is a set of parentheses surrounding another `parens` expression or, if that doesn't match, the empty string. If you are uncomfortable with recursion, its time to get comfortable, because it is the basis of language. Here's a tip: Don't try and imagine the parser circling round and round through the same rule. Instead, imagine the rule is *already* defined while you are defining it. If you imagine that `parens` already matches a string of matching parentheses, then its easy to think of `parens` as an open and closing parentheses around another set of matching parentheses, which conveniently, you happen to be defining. You know that `parens` is supposed to represent a string of matched parentheses, so trust in that meaning, even if you haven't fully implemented it yet.
+
+Repetition
+----------
+Any item in a rule may be followed by a '+' or a '*' character, signifying one-or-more and zero-or-more occurrences of that item. Beware though; the match is greedy, and if it matches too many items and causes subsequent items in the sequence to fail, the number matched will never be reconsidered. Here's a simple example of a rule that will never succeed:
+
+    # toogreedy.treetop
+    grammar TooGreedy
+      rule a_s
+      	'a'* 'a'
+      end
+    end
+
+The 'a'* will always eat up any 'a's that follow, and the subsequent 'a' will find none there, so the whole rule will fail. You might need to use lookahead to avoid matching too much.
+
+Negative Lookahead
+------------------
+
+When you need to ensure that the following item *doesn't* match in some case where it might otherwise, you can use negat!ve lookahead, which is an item preceeded by a ! - here's an example:
+
+    # postcondition.treetop
+    grammar PostCondition
+      rule conditional_sentence
+        ( !conditional_keyword word )+ conditional_keyword [ \t]+ word*
+      end
+
+      rule word
+        ([a-zA-Z]+ [ \t]+) 
+      end
+
+      rule conditional_keyword
+        'if' / 'while' / 'until'
+      end
+    end
+
+Even though the rule `word` would match any of the conditional keywords, the first words of a conditional_sentence must not be conditional_keywords. The negative lookahead prevents that matching, and prevents the repetition from matching too much input. Note that the lookahead may be a grammar rule of any complexity, including one that isn't used elsewhere in your grammar.
+
+Positive lookahead
+------------------
+
+Sometimes you want an item to match, but only if the *following* text would match some pattern. You don't want to consume that following text, but if it's not there, you want this rule to fail. You can append a positive lookahead like this to a rule by appending the lookahead rule preceeded by an & character.
+
+
+
+Features to cover in the talk
+=============================
+
+* Treetop files
+* Grammar definition
+* Rules
+* Loading a grammar
+* Compiling a grammar with the `tt` command
+* Accessing a parser for the grammar from Ruby
+* Parsing Expressions of all kinds
+? Left recursion and factorization
+  - Here I can talk about function application, discussing how the operator
+    could be an arbitrary expression
+* Inline node class eval blocks
+* Node class declarations
+* Labels
+* Use of super within within labels
+* Grammar composition with include
+* Use of super with grammar composition

data/Rakefile

@@ -0,0 +1,19 @@
+dir = File.dirname(__FILE__)
+require 'rubygems'
+require 'rake'
+$LOAD_PATH.unshift(File.join(dir, 'vendor', 'rspec', 'lib'))
+require 'spec/rake/spectask'
+
+require 'rake/gempackagetask'
+
+task :default => :spec
+
+Spec::Rake::SpecTask.new do |t|
+  t.pattern = 'spec/**/*spec.rb'
+end
+
+load "./treetop.gemspec"
+
+Rake::GemPackageTask.new($gemspec) do |pkg|
+  pkg.need_tar = true
+end

data/bin/tt

@@ -0,0 +1,112 @@
+#!/usr/bin/env ruby
+require 'optparse'
+require 'rubygems'
+require 'treetop'
+
+$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + "/../lib"))
+require 'treetop'
+require 'treetop/version'
+
+options = {}
+parser = OptionParser.new do |opts|
+  exts = Treetop::VALID_GRAMMAR_EXT.collect { |i| '.' + i }
+
+  opts.banner = "Treetop Parsing Expression Grammar (PEG) Comand Line Compiler"
+  opts.define_head "Usage: tt [options] grammar_file[#{exts.join('|')}] ..."
+  opts.separator ''
+  opts.separator 'Examples:'
+  opts.separator '  tt foo.tt               # 1 grammar -> 1 parser source'
+  opts.separator '  tt foo bar.treetop      # 2 grammars -> 2 separate parsers'
+  opts.separator '  tt -o alt_name.rb foo   # alternately named output file'
+  opts.separator ''
+  opts.separator ''
+  opts.separator 'NOTE: while treetop grammar files *must* have one of the following'
+  opts.separator 'filename extensions, the extension name is not required when calling'
+  opts.separator 'the compiler with grammar file names.'
+  opts.separator ''
+  opts.separator "  Valid extensions: #{exts.join(', ')}"
+  opts.separator ''
+  opts.separator ''
+  opts.separator 'Options:'
+
+  opts.on('-o', '--output FILENAME', 'Write parser source to FILENAME') do |fn|
+    options[:out_file] = fn
+  end
+
+  opts.on('-f', '--force', 'Overwrite existing output file(s)') do
+    options[:force] = true
+  end
+
+  opts.on_tail('-v', '--version', 'Show Treetop version') do
+    puts "Treetop v#{Treetop::VERSION::STRING}"
+    exit
+  end
+
+  opts.on_tail('-h', '--help', 'Show this help message') do
+    puts opts
+    exit
+  end
+
+end
+file_list = parser.parse!
+
+# check options and arg constraints
+if file_list.empty? || (options[:out_file] && file_list.size > 1)
+  puts parser
+  exit 1
+end
+
+def grammar_exist?(filename)
+  if File.extname(filename).empty?
+    Treetop::VALID_GRAMMAR_EXT.each do |ext|
+      fn_ext = "#{filename}.#{ext}"
+      return true if File.exist?(fn_ext) && !File.zero?(fn_ext)
+    end
+  end
+  File.exist?(filename) && !File.zero?(filename)
+end
+
+def full_grammar_filename(filename)
+  return filename if !File.extname(filename).empty?
+  Treetop::VALID_GRAMMAR_EXT.each do |ext|
+    fn_ext = "#{filename}.#{ext}"
+    return fn_ext if File.exist?(fn_ext) && !File.zero?(fn_ext)
+  end
+end
+
+def protect_output?(filename, forced=false)
+  if !forced and
+      File.exist?(filename) and
+      (l=File.open(filename) { |f| f.gets rescue "" }) != Treetop::Compiler::AUTOGENERATED
+    puts "ERROR: '#{filename}' output already exists; skipping compilation...\n"
+    return true
+  end
+  false
+end
+
+compiler = Treetop::Compiler::GrammarCompiler.new
+
+while !file_list.empty?
+  treetop_file = file_list.shift
+
+  # handle nonexistent and existent grammar files mixed together
+  if !grammar_exist?(treetop_file)
+    puts "ERROR: input grammar file '#{treetop_file}' does not exist; continuing...\n"
+    next
+  end
+
+  # try to compile
+  treetop_file = full_grammar_filename(treetop_file)
+  std_output_file = treetop_file.gsub(Treetop::VALID_GRAMMAR_EXT_REGEXP, '.rb')
+
+  if options[:out_file]
+    # explicit output file name option; never overwrite unless forced
+    next if protect_output?(options[:out_file], options[:force])
+    compiler.compile(treetop_file, options[:out_file])
+  else
+    # compile one input file from input file list option; never overwrite unless forced
+    next if protect_output?(std_output_file, options[:force])
+    compiler.compile(treetop_file)
+  end
+
+end

data/doc/contributing_and_planned_features.markdown

@@ -0,0 +1,103 @@
+#Google Group
+I've created a <a href="http://groups.google.com/group/treetop-dev">Google Group</a> as a better place to organize discussion and development.
+treetop-dev@google-groups.com
+
+#Contributing
+Visit <a href="http://github.com/nathansobo/treetop/tree/master">the Treetop repository page on GitHub</a> in your browser for more information about checking out the source code.
+
+I like to try Rubinius's policy regarding commit rights. If you submit one patch worth integrating, I'll give you commit rights. We'll see how this goes, but I think it's a good policy.
+
+
+##Getting Started with the Code
+Treetop compiler is interesting in that it is implemented in itself. Its functionality revolves around `metagrammar.treetop`, which specifies the grammar for Treetop grammars. I took a hybrid approach with regard to definition of methods on syntax nodes in the metagrammar. Methods that are more syntactic in nature, like those that provide access to elements of the syntax tree, are often defined inline, directly in the grammar. More semantic methods are defined in custom node classes.
+
+Iterating on the metagrammar is tricky. The current testing strategy uses the last stable version of Treetop to parse the version under test. Then the version under test is used to parse and functionally test the various pieces of syntax it should recognize and translate to Ruby. As you change `metagrammar.treetop` and its associated node classes, note that the node classes you are changing are also used to support the previous stable version of the metagrammar, so must be kept backward compatible until such time as a new stable version can be produced to replace it.
+
+##Tests
+Most of the compiler's tests are functional in nature. The grammar under test is used to parse and compile piece of sample code. Then I attempt to parse input with the compiled output and test its results.
+
+#What Needs to be Done
+##Small Stuff
+* Improve the `tt` command line tool to allow `.treetop` extensions to be elided in its arguments.
+* Generate and load temp files with `Treetop.load` rather than evaluating strings to improve stack trace readability.
+* Allow `do/end` style blocks as well as curly brace blocks. This was originally omitted because I thought it would be confusing. It probably isn't.
+
+##Big Stuff
+####Transient Expressions
+Currently, every parsing expression instantiates a syntax node. This includes even very simple parsing expressions, like single characters. It is probably unnecessary for every single expression in the parse to correspond to its own syntax node, so much savings could be garnered from a transient declaration that instructs the parser only to attempt a match without instantiating nodes.
+
+###Generate Rule Implementations in C
+Parsing expressions are currently compiled into simple Ruby source code that comprises the body of parsing rules, which are translated into Ruby methods. The generator could produce C instead of Ruby in the body of these method implementations.
+
+###Global Parsing State and Semantic Backtrack Triggering
+Some programming language grammars are not entirely context-free, requiring that global state dictate the behavior of the parser in certain circumstances. Treetop does not currently expose explicit parser control to the grammar writer, and instead automatically constructs the syntax tree for them. A means of semantic parser control compatible with this approach would involve callback methods defined on parsing nodes. Each time a node is successfully parsed it will be given an opportunity to set global state and optionally trigger a parse failure on _extrasyntactic_ grounds. Nodes will probably need to define an additional method that undoes their changes to global state when there is a parse failure and they are backtracked.
+
+Here is a sketch of the potential utility of such mechanisms. Consider the structure of YAML, which uses indentation to indicate block structure.
+
+    level_1:
+      level_2a:
+      level_2b:
+        level_3a:
+      level_2c:    
+
+Imagine a grammar like the following:
+
+    rule yaml_element
+      name ':' block
+      /
+      name ':' value
+    end
+    
+    rule block
+      indent yaml_elements outdent
+    end
+
+    rule yaml_elements
+      yaml_element (samedent yaml_element)*
+    end
+    
+    rule samedent
+      newline spaces {
+        def after_success(parser_state)
+          spaces.length == parser_state.indent_level
+        end
+      }
+    end
+    
+    rule indent
+      newline spaces {
+        def after_success(parser_state)
+          if spaces.length == parser_state.indent_level + 2
+            parser_state.indent_level += 2
+            true
+          else
+            false # fail the parse on extrasyntactic grounds
+          end
+        end
+      
+        def undo_success(parser_state)
+          parser_state.indent_level -= 2
+        end
+      }
+    end
+    
+    rule outdent
+      newline spaces {
+        def after_success(parser_state)
+          if spaces.length == parser_state.indent_level - 2
+            parser_state.indent_level -= 2
+            true
+          else
+            false # fail the parse on extrasyntactic grounds
+          end
+        end
+      
+        def undo_success(parser_state)
+          parser_state.indent_level += 2
+        end
+      }
+    end
+
+In this case a block will be detected only if a change in indentation warrants it. Note that this change in the state of indentation must be undone if a subsequent failure causes this node not to ultimately be incorporated into a successful result.
+
+I am by no means sure that the above sketch is free of problems, or even that this overall strategy is sound, but it seems like a promising path.

data/doc/grammar_composition.markdown

@@ -0,0 +1,65 @@
+#Grammar Composition
+A unique property of parsing expression grammars is that they are _closed under composition_. This means that when you compose two grammars they yield another grammar that can be composed yet again. This is a radical departure from parsing frameworks require on lexical scanning, which makes compositionally impossible. Treetop's facilities for composition are built upon those of Ruby.
+
+##The Mapping of Treetop Constructs to Ruby Constructs
+When Treetop compiles a grammar definition, it produces a module and a class. The module contains methods implementing all of the rules defined in the grammar. The generated class is a subclass of Treetop::Runtime::CompiledParser and includes the module. For example:
+
+    grammar Foo
+      ...
+    end
+    
+results in a Ruby module named `Foo` and a Ruby class named `FooParser` that `include`s the `Foo` module.
+
+##Using Mixin Semantics to Compose Grammars
+Because grammars are just modules, they can be mixed into one another. This enables grammars to share rules.
+
+    grammar A
+      rule a
+        'a'
+      end
+    end
+    
+    grammar B
+      include A
+    
+      rule ab
+        a 'b'
+      end
+    end
+    
+Grammar `B` above references rule `a` defined in a separate grammar that it includes. Because module inclusion places modules in the ancestor chain, rules may also be overridden with the use of the `super` keyword accessing the overridden rule.
+
+    grammar A
+      rule a
+        'a'
+      end
+    end
+
+    grammar B
+      include A
+
+      rule a
+        super / 'b'
+      end
+    end
+
+Now rule `a` in grammar `B` matches either `'a'` or `'b'`.
+
+##Motivation
+Imagine a grammar for Ruby that took account of SQL queries embedded in strings within the language. That could be achieved by combining two existing grammars.
+
+    grammar RubyPlusSQL
+      include Ruby
+      include SQL
+      
+      rule expression
+        ruby_expression
+      end
+      
+      rule ruby_string
+        ruby_quote sql_expression ruby_quote / ruby_string
+      end
+    end
+    
+##Work to be Done
+It has become clear that the include facility in grammars would be more useful if it had the ability to name prefix all rules from the included grammar to avoid collision. This is a planned but currently unimplemented feature.