regex-treetop 1.4.8

data/doc/index.markdown

@@ -0,0 +1,90 @@
+<p class="intro_text">
+	
+Treetop is a language for describing languages. Combining the elegance of Ruby with cutting-edge <em>parsing expression grammars</em>, it helps you analyze syntax with revolutionarily ease.	
+
+</p>
+
+    sudo gem install treetop
+
+#Intuitive Grammar Specifications
+Parsing expression grammars (PEGs) are simple to write and easy to maintain. They are a simple but powerful generalization of regular expressions that are easier to work with than the LALR or LR-1 grammars of traditional parser generators. There's no need for a tokenization phase, and _lookahead assertions_ can be used for a limited degree of context-sensitivity. Here's an extremely simple Treetop grammar that matches a subset of arithmetic, respecting operator precedence:
+
+    grammar Arithmetic
+      rule additive
+        multitive '+' additive / multitive
+      end
+      
+      rule multitive
+        primary '*' multitive / primary
+      end
+      
+      rule primary
+        '(' additive ')' / number
+      end
+      
+      rule number
+        [1-9] [0-9]*
+      end
+    end
+  
+
+#Syntax-Oriented Programming
+Rather than implementing semantic actions that construct parse trees, Treetop lets you define methods on trees that it constructs for you automatically. You can define these methods directly within the grammar...
+
+    grammar Arithmetic
+      rule additive
+        multitive '+' additive {
+          def value
+            multitive.value + additive.value
+          end
+        }
+        /
+        multitive
+      end
+      
+      # other rules below ...
+    end
+    
+...or associate rules with classes of nodes you wish your parsers to instantiate upon matching a rule.
+
+    grammar Arithmetic
+      rule additive
+        multitive '+' additive <AdditiveNode>
+        /
+        multitive
+      end
+      
+      # other rules below ...
+    end
+
+
+#Reusable, Composable Language Descriptions
+Because PEGs are closed under composition, Treetop grammars can be treated like Ruby modules. You can mix them into one another and override rules with access to the `super` keyword. You can break large grammars down into coherent units or make your language's syntax modular. This is especially useful if you want other programmers to be able to reuse your work.
+
+    grammar RubyWithEmbeddedSQL
+      include SQL
+      
+      rule string
+        quote sql_expression quote / super
+      end
+    end
+
+
+#Acknowledgements
+
+
+<a href="http://pivotallabs.com"><img id="pivotal_logo" src="./images/pivotal.gif"></a>
+
+First, thank you to my employer Rob Mee of <a href="http://pivotallabs.com"/>Pivotal Labs</a> for funding a substantial portion of Treetop's development. He gets it.
+
+
+I'd also like to thank:
+
+* Damon McCormick for several hours of pair programming.
+* Nick Kallen for lots of well-considered feedback and a few afternoons of programming.
+* Brian Takita for a night of pair programming.
+* Eliot Miranda for urging me rewrite as a compiler right away rather than putting it off.
+* Ryan Davis and Eric Hodel for hurting my code.
+* Dav Yaginuma for kicking me into action on my idea.
+* Bryan Ford for his seminal work on Packrat Parsers.
+* The editors of Lambda the Ultimate, where I discovered parsing expression grammars.

data/doc/pitfalls_and_advanced_techniques.markdown

@@ -0,0 +1,51 @@
+#Pitfalls
+##Left Recursion
+An weakness shared by all recursive descent parsers is the inability to parse left-recursive rules. Consider the following rule:
+
+    rule left_recursive
+      left_recursive 'a' / 'a'
+    end
+    
+Logically it should match a list of 'a' characters. But it never consumes anything, because attempting to recognize `left_recursive` begins by attempting to recognize `left_recursive`, and so goes an infinite recursion. There's always a way to eliminate these types of structures from your grammar. There's a mechanistic transformation called _left factorization_ that can eliminate it, but it isn't always pretty, especially in combination with automatically constructed syntax trees. So far, I have found more thoughtful ways around the problem. For instance, in the interpreter example I interpret inherently left-recursive function application right recursively in syntax, then correct the directionality in my semantic interpretation. You may have to be clever.
+
+#Advanced Techniques
+Here are a few interesting problems I've encountered. I figure sharing them may give you insight into how these types of issues are addressed with the tools of parsing expressions.
+
+##Matching a String
+
+    rule string
+      '"' (!'"' . / '\"')* '"'
+    end
+
+This expression says: Match a quote, then zero or more of any character but a quote or an escaped quote followed by a quote. Lookahead assertions are essential for these types of problems.
+
+##Matching Nested Structures With Non-Unique Delimeters
+Say I want to parse a diabolical wiki syntax in which the following interpretations apply.
+
+    ** *hello* ** --> <strong><em>hello</em></strong>
+    * **hello** * --> <em><strong>hello</strong></em>
+
+    rule strong
+      '**' (em / !'*' . / '\*')+ '**'
+    end
+    
+    rule em
+      '**' (strong / !'*' . / '\*')+ '**'    
+    end
+    
+Emphasized text is allowed within strong text by virtue of `em` being the first alternative. Since `em` will only successfully parse if a matching `*` is found, it is permitted, but other than that, no `*` characters are allowed unless they are escaped.
+
+##Matching a Keyword But Not Words Prefixed Therewith
+Say I want to consider a given string a characters only when it occurs in isolation. Lets use the `end` keyword as an example. We don't want the prefix of `'enders_game'` to be considered a keyword. A naiive implementation might be the following.
+
+    rule end_keyword
+      'end' &space
+    end
+    
+This says that `'end'` must be followed by a space, but this space is not consumed as part of the matching of `keyword`. This works in most cases, but is actually incorrect. What if `end` occurs at the end of the buffer? In that case, it occurs in isolation but will not match the above expression. What we really mean is that `'end'` cannot be followed by a _non-space_ character.
+
+    rule end_keyword
+      'end' !(!' ' .)
+    end
+    
+In general, when the syntax gets tough, it helps to focus on what you really mean. A keyword is a character not followed by another character that isn't a space.

data/doc/semantic_interpretation.markdown

@@ -0,0 +1,189 @@
+#Semantic Interpretation
+Lets use the below grammar as an example. It describes parentheses wrapping a single character to an arbitrary depth.
+
+    grammar ParenLanguage
+      rule parenthesized_letter
+        '(' parenthesized_letter ')'
+        /
+        [a-z]
+      end
+    end
+
+Matches:
+
+* `'a'`
+* `'(a)'`
+* `'((a))'`
+* etc.
+
+
+Output from a parser for this grammar looks like this:
+
+![Tree Returned By ParenLanguageParser](./images/paren_language_output.png)
+
+This is a parse tree whose nodes are instances of `Treetop::Runtime::SyntaxNode`. What if we could define methods on these node objects? We would then have an object-oriented program whose structure corresponded to the structure of our language. Treetop provides two techniques for doing just this.
+
+##Associating Methods with Node-Instantiating Expressions
+Sequences and all types of terminals are node-instantiating expressions. When they match, they create instances of `Treetop::Runtime::SyntaxNode`. Methods can be added to these nodes in the following ways:
+
+###Inline Method Definition
+Methods can be added to the nodes instantiated by the successful match of an expression
+
+    grammar ParenLanguage
+      rule parenthesized_letter
+        '(' parenthesized_letter ')' {
+          def depth
+            parenthesized_letter.depth + 1
+          end
+        }
+        /
+        [a-z] {
+          def depth
+            0
+          end
+        }
+      end
+    end
+
+Note that each alternative expression is followed by a block containing a method definition. A `depth` method is defined on both expressions. The recursive `depth` method defined in the block following the first expression determines the depth of the nested parentheses and adds one to it. The base case is implemented in the block following the second expression; a single character has a depth of 0.
+
+
+###Custom `SyntaxNode` Subclass Declarations
+You can instruct the parser to instantiate a custom subclass of Treetop::Runtime::SyntaxNode for an expression by following it by the name of that class enclosed in angle brackets (`<>`). The above inline method definitions could have been moved out into a single class like so.
+
+    # in .treetop file
+    grammar ParenLanguage
+      rule parenthesized_letter
+        '(' parenthesized_letter ')' <ParenNode>
+        /
+        [a-z] <ParenNode>
+      end
+    end
+
+    # in separate .rb file
+    class ParenNode < Treetop::Runtime::SyntaxNode
+      def depth
+        if nonterminal?
+          parenthesized_letter.depth + 1
+        else
+          0
+        end
+      end
+    end
+
+##Automatic Extension of Results
+Nonterminal and ordered choice expressions do not instantiate new nodes, but rather pass through nodes that are instantiated by other expressions. They can extend nodes they propagate with anonymous or declared modules, using similar constructs used with expressions that instantiate their own syntax nodes.
+
+###Extending a Propagated Node with an Anonymous Module
+    rule parenthesized_letter
+      ('(' parenthesized_letter ')' / [a-z]) {
+        def depth
+          if nonterminal?
+            parenthesized_letter.depth + 1
+          else
+            0
+          end
+        end
+      }
+    end
+
+The parenthesized choice above can result in a node matching either of the two choices. Than node will be extended with methods defined in the subsequent block. Note that a choice must always be parenthesized to be associated with a following block.
+
+###Extending A Propagated Node with a Declared Module
+    # in .treetop file
+    rule parenthesized_letter
+      ('(' parenthesized_letter ')' / [a-z]) <ParenNode>
+    end
+    
+    # in separate .rb file
+    module ParenNode
+      def depth
+        if nonterminal?
+          parenthesized_letter.depth + 1
+        else
+          0
+        end
+      end
+    end
+
+Here the result is extended with the `ParenNode` module. Note the previous example for node-instantiating expressions, the constant in the declaration must be a module because the result is extended with it.
+
+##Automatically-Defined Element Accessor Methods
+###Default Accessors
+Nodes instantiated upon the matching of sequences have methods automatically defined for any nonterminals in the sequence.
+
+    rule abc
+      a b c {
+        def to_s
+          a.to_s + b.to_s + c.to_s
+        end
+      }
+    end
+
+In the above code, the `to_s` method calls automatically-defined element accessors for the nodes returned by parsing nonterminals `a`, `b`, and `c`. 
+
+###Labels
+Subexpressions can be given an explicit label to have an element accessor method defined for them. This is useful in cases of ambiguity between two references to the same nonterminal or when you need to access an unnamed subexpression.
+
+    rule labels
+      first_letter:[a-z] rest_letters:(', ' letter:[a-z])* {
+        def letters
+          [first_letter] + rest_letters.map do |comma_and_letter|
+            comma_and_letter.letter
+          end
+        end
+      }
+    end
+
+The above grammar uses label-derived accessors to determine the letters in a comma-delimited list of letters. The labeled expressions _could_ have been extracted to their own rules, but if they aren't used elsewhere, labels still enable them to be referenced by a name within the expression's methods.
+
+###Overriding Element Accessors
+The module containing automatically defined element accessor methods is an ancestor of the module in which you define your own methods, meaning you can override them with access to the `super` keyword. Here's an example of how this fact can improve the readability of the example above.
+
+    rule labels
+      first_letter:[a-z] rest_letters:(', ' letter:[a-z])* {
+        def letters
+          [first_letter] + rest_letters
+        end
+        
+        def rest_letters
+          super.map { |comma_and_letter| comma_and_letter.letter }
+        end
+      }
+    end
+
+
+##Methods Available on `Treetop::Runtime::SyntaxNode`
+
+<table>
+  <tr>
+    <td>
+      <code>terminal?</code>
+    </td>
+    <td>
+      Was this node produced by the matching of a terminal symbol?
+    </td>
+  </tr>
+  <tr>
+    <td>
+      <code>nonterminal?</code>
+    </td>
+    <td>
+      Was this node produced by the matching of a nonterminal symbol?
+    </td>
+  <tr>
+    <td>
+      <code>text_value</code>
+    </td>
+    <td>
+      The substring of the input represented by this node.
+    </td>
+  <tr>
+    <td>
+      <code>elements</code>
+    </td>
+    <td>
+      Available only on nonterminal nodes, returns the nodes parsed by the elements of the matched sequence.
+    </td>
+  </tr>
+</table>

data/doc/site.rb

@@ -0,0 +1,112 @@
+require 'rubygems'
+require 'erector'
+require "#{File.dirname(__FILE__)}/sitegen"
+require 'fileutils'
+require 'bluecloth'
+
+class Layout < Erector::Widget
+  def render
+    html do
+      head do
+        link :rel => "stylesheet",
+        :type => "text/css",
+        :href => "./screen.css"
+        
+        rawtext %(
+          <script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
+          </script>
+          <script type="text/javascript">
+          _uacct = "UA-3418876-1";
+          urchinTracker();
+          </script>
+        )
+      end
+
+      body do
+        div :id => 'top' do
+          div :id => 'main_navigation' do
+            main_navigation
+          end
+        end
+        div :id => 'middle' do
+          div :id => 'content' do
+            content
+          end
+        end
+        div :id => 'bottom' do
+
+        end
+      end
+    end
+  end
+
+  def main_navigation
+    ul do
+      li { link_to "Documentation", SyntacticRecognition, Documentation }
+      li { link_to "Contribute", Contribute }
+      li { link_to "Home", Index }
+    end
+  end
+
+  def content
+  end
+end
+
+class Index < Layout
+  def content
+    bluecloth "index.markdown"
+  end
+end
+
+class Documentation < Layout
+  abstract
+
+  def content
+    div :id => 'secondary_navigation' do
+      ul do
+        li { link_to 'Syntax', SyntacticRecognition }
+        li { link_to 'Semantics', SemanticInterpretation }
+        li { link_to 'Using In Ruby', UsingInRuby }
+        li { link_to 'Advanced Techniques', PitfallsAndAdvancedTechniques }
+      end
+    end
+    
+    div :id => 'documentation_content' do
+      documentation_content
+    end
+  end
+end
+
+class SyntacticRecognition < Documentation
+  def documentation_content
+    bluecloth "syntactic_recognition.markdown"
+  end
+end
+
+class SemanticInterpretation < Documentation
+  def documentation_content
+    bluecloth "semantic_interpretation.markdown"
+  end
+end
+
+class UsingInRuby < Documentation
+  def documentation_content
+    bluecloth "using_in_ruby.markdown"
+  end
+end
+
+class PitfallsAndAdvancedTechniques < Documentation
+  def documentation_content
+    bluecloth "pitfalls_and_advanced_techniques.markdown"
+  end
+end
+
+
+class Contribute < Layout
+  def content
+    bluecloth "contributing_and_planned_features.markdown"
+  end
+end
+
+
+Layout.generate_site

data/doc/sitegen.rb

@@ -0,0 +1,65 @@
+class Layout < Erector::Widget
+
+  class << self
+    def inherited(page_class)
+      puts page_class
+      (@@page_classes ||= []) << page_class
+    end
+
+    def generate_site
+      FileUtils.mkdir_p(site_dir)
+      @@page_classes.each do |page_class|
+        page_class.generate_html unless page_class.abstract?
+        puts page_class
+      end
+    end
+
+    def generate_html
+      File.open(absolute_path, 'w') do |file|
+        file.write(new.render)
+      end
+    end
+
+    def absolute_path
+      absolutize(relative_path)
+    end
+
+    def relative_path
+      "#{name.gsub('::', '_').underscore}.html"
+    end
+
+    def absolutize(relative_path)
+      File.join(site_dir, relative_path)
+    end
+
+    def abstract
+      @abstract = true
+    end
+
+    def abstract?
+      @abstract
+    end
+
+    def site_dir
+      File.join(File.dirname(__FILE__), "site")
+    end
+  end
+
+  def bluecloth(relative_path)
+    File.open(File.join(File.dirname(__FILE__), relative_path)) do |file|
+      rawtext BlueCloth.new(file.read).to_html
+    end
+  end
+
+  def absolutize(relative_path)
+    self.class.absolutize(relative_path)
+  end
+
+  def link_to(link_text, page_class, section_class=nil)
+    if instance_of?(page_class) || section_class && is_a?(section_class)
+      text link_text
+    else
+      a link_text, :href => page_class.relative_path
+    end
+  end
+end