treetop 1.4.8 → 1.4.9

data/Rakefile

@@ -26,3 +26,18 @@ end
 task :version do
   puts RUBY_VERSION
 end
+
+desc 'Generate website files'
+task :website_generate do
+  `cd doc; ruby ./site.rb`
+end
+
+desc 'Upload website files'
+task :website_upload do
+  rubyforge_config_file = "#{ENV['HOME']}/.rubyforge/user-config.yml"
+  rubyforge_config = YAML.load_file(rubyforge_config_file)
+  `rsync -aCv doc/site/ #{rubyforge_config['username']}@rubyforge.org:/var/www/gforge-projects/treetop/`
+end
+
+desc 'Generate and upload website files'
+task :website => [:website_generate, :website_upload]

data/doc/contributing_and_planned_features.markdown

@@ -7,7 +7,6 @@ Visit <a href="http://github.com/nathansobo/treetop/tree/master">the Treetop rep
 
 I like to try Rubinius's policy regarding commit rights. If you submit one patch worth integrating, I'll give you commit rights. We'll see how this goes, but I think it's a good policy.
 
-
 ##Getting Started with the Code
 Treetop compiler is interesting in that it is implemented in itself. Its functionality revolves around `metagrammar.treetop`, which specifies the grammar for Treetop grammars. I took a hybrid approach with regard to definition of methods on syntax nodes in the metagrammar. Methods that are more syntactic in nature, like those that provide access to elements of the syntax tree, are often defined inline, directly in the grammar. More semantic methods are defined in custom node classes.
 

data/doc/index.markdown

@@ -1,6 +1,6 @@
 <p class="intro_text">
-	
-Treetop is a language for describing languages. Combining the elegance of Ruby with cutting-edge <em>parsing expression grammars</em>, it helps you analyze syntax with revolutionarily ease.	
+
+Treetop is a language for describing languages. Combining the elegance of Ruby with cutting-edge <em>parsing expression grammars</em>, it helps you analyze syntax with revolutionary ease.
 
 </p>
 
@@ -11,49 +11,47 @@ Parsing expression grammars (PEGs) are simple to write and easy to maintain. The
 
     grammar Arithmetic
       rule additive
-        multitive '+' additive / multitive
+        multitive ( '+' multitive )*
       end
-      
+
       rule multitive
-        primary '*' multitive / primary
+        primary ( [*/%] primary )*
       end
-      
+
       rule primary
         '(' additive ')' / number
       end
-      
+
       rule number
-        [1-9] [0-9]*
+        '-'? [1-9] [0-9]*
       end
     end
-  
+
 
 #Syntax-Oriented Programming
 Rather than implementing semantic actions that construct parse trees, Treetop lets you define methods on trees that it constructs for you automatically. You can define these methods directly within the grammar...
 
     grammar Arithmetic
       rule additive
-        multitive '+' additive {
+        multitive a:( '+' multitive )* {
           def value
-            multitive.value + additive.value
+            a.elements.inject(multitive.value) { |sum, e|
+              sum+e.multitive.value
+            }
           end
         }
-        /
-        multitive
       end
-      
+
       # other rules below ...
     end
-    
+
 ...or associate rules with classes of nodes you wish your parsers to instantiate upon matching a rule.
 
     grammar Arithmetic
       rule additive
-        multitive '+' additive <AdditiveNode>
-        /
-        multitive
+        multitive ('+' multitive)* <AdditiveNode>
       end
-      
+
       # other rules below ...
     end
 
@@ -63,7 +61,7 @@ Because PEGs are closed under composition, Treetop grammars can be treated like
 
     grammar RubyWithEmbeddedSQL
       include SQL
-      
+
       rule string
         quote sql_expression quote / super
       end
@@ -87,4 +85,4 @@ I'd also like to thank:
 * Ryan Davis and Eric Hodel for hurting my code.
 * Dav Yaginuma for kicking me into action on my idea.
 * Bryan Ford for his seminal work on Packrat Parsers.
-* The editors of Lambda the Ultimate, where I discovered parsing expression grammars.
+* The editors of Lambda the Ultimate, where I discovered parsing expression grammars.

data/doc/semantic_interpretation.markdown

@@ -87,7 +87,7 @@ Nonterminal and ordered choice expressions do not instantiate new nodes, but rat
       }
     end
 
-The parenthesized choice above can result in a node matching either of the two choices. Than node will be extended with methods defined in the subsequent block. Note that a choice must always be parenthesized to be associated with a following block.
+The parenthesized choice above can result in a node matching either of the two choices. The node will be extended with methods defined in the subsequent block. Note that a choice must always be parenthesized to be associated with a following block, otherwise the block will apply to just the last alternative.
 
 ###Extending A Propagated Node with a Declared Module
     # in .treetop file
@@ -185,5 +185,34 @@ The module containing automatically defined element accessor methods is an ances
     <td>
       Available only on nonterminal nodes, returns the nodes parsed by the elements of the matched sequence.
     </td>
+  <tr>
+    <td>
+      <code>input</code>
+    </td>
+    <td>
+       The entire input string, which is useful mainly in conjunction with <code>interval</code>
+    </td>
+  <tr>
+    <td>
+      <code>interval</code>
+    </td>
+    <td>
+       The Range of characters in <code>input</code> matched by this rule
+    </td>
+  <tr>
+    <td>
+      <code>empty?</code>
+    </td>
+    <td>
+       returns true if this rule matched no characters of input
+    </td>
+  <tr>
+    <td>
+      <code>inspect</code>
+    </td>
+    <td>
+       Handy-dandy method that returns an indented subtree dump of the syntax tree starting here.
+       This dump includes, for each node, the offset and a snippet of the text this rule matched, and the names of mixin modules and the accessor and extension methods.
+    </td>
   </tr>
 </table>

data/doc/site.rb

@@ -5,7 +5,7 @@ require 'fileutils'
 require 'bluecloth'
 
 class Layout < Erector::Widget
-  def render
+  def content
     html do
       head do
         link :rel => "stylesheet",
@@ -29,8 +29,8 @@ class Layout < Erector::Widget
           end
         end
         div :id => 'middle' do
-          div :id => 'content' do
-            content
+          div :id => 'main_content' do
+            main_content
           end
         end
         div :id => 'bottom' do
@@ -48,12 +48,12 @@ class Layout < Erector::Widget
     end
   end
 
-  def content
+  def main_content
   end
 end
 
 class Index < Layout
-  def content
+  def main_content
     bluecloth "index.markdown"
   end
 end
@@ -61,7 +61,7 @@ end
 class Documentation < Layout
   abstract
 
-  def content
+  def main_content
     div :id => 'secondary_navigation' do
       ul do
         li { link_to 'Syntax', SyntacticRecognition }
@@ -103,7 +103,7 @@ end
 
 
 class Contribute < Layout
-  def content
+  def main_content
     bluecloth "contributing_and_planned_features.markdown"
   end
 end

data/doc/site/contribute.html

@@ -0,0 +1,124 @@
+<html><head><link href="./screen.css" rel="stylesheet" type="text/css" />
+          <script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
+          </script>
+          <script type="text/javascript">
+          _uacct = "UA-3418876-1";
+          urchinTracker();
+          </script>
+        </head><body><div id="top"><div id="main_navigation"><ul><li><a href="syntactic_recognition.html">Documentation</a></li><li>Contribute</li><li><a href="index.html">Home</a></li></ul></div></div><div id="middle"><div id="main_content"><h1>Google Group</h1>
+
+<p>I've created a <a href="http://groups.google.com/group/treetop-dev">Google Group</a> as a better place to organize discussion and development.
+treetop-dev@google-groups.com</p>
+
+<h1>Contributing</h1>
+
+<p>Visit <a href="http://github.com/nathansobo/treetop/tree/master">the Treetop repository page on GitHub</a> in your browser for more information about checking out the source code.</p>
+
+<p>I like to try Rubinius's policy regarding commit rights. If you submit one patch worth integrating, I'll give you commit rights. We'll see how this goes, but I think it's a good policy.</p>
+
+<h2>Getting Started with the Code</h2>
+
+<p>Treetop compiler is interesting in that it is implemented in itself. Its functionality revolves around <code>metagrammar.treetop</code>, which specifies the grammar for Treetop grammars. I took a hybrid approach with regard to definition of methods on syntax nodes in the metagrammar. Methods that are more syntactic in nature, like those that provide access to elements of the syntax tree, are often defined inline, directly in the grammar. More semantic methods are defined in custom node classes.</p>
+
+<p>Iterating on the metagrammar is tricky. The current testing strategy uses the last stable version of Treetop to parse the version under test. Then the version under test is used to parse and functionally test the various pieces of syntax it should recognize and translate to Ruby. As you change <code>metagrammar.treetop</code> and its associated node classes, note that the node classes you are changing are also used to support the previous stable version of the metagrammar, so must be kept backward compatible until such time as a new stable version can be produced to replace it.</p>
+
+<h2>Tests</h2>
+
+<p>Most of the compiler's tests are functional in nature. The grammar under test is used to parse and compile piece of sample code. Then I attempt to parse input with the compiled output and test its results.</p>
+
+<h1>What Needs to be Done</h1>
+
+<h2>Small Stuff</h2>
+
+<ul>
+<li>Improve the <code>tt</code> command line tool to allow <code>.treetop</code> extensions to be elided in its arguments.</li>
+<li>Generate and load temp files with <code>Treetop.load</code> rather than evaluating strings to improve stack trace readability.</li>
+<li>Allow <code>do/end</code> style blocks as well as curly brace blocks. This was originally omitted because I thought it would be confusing. It probably isn't.</li>
+</ul>
+
+
+<h2>Big Stuff</h2>
+
+<h4>Transient Expressions</h4>
+
+<p>Currently, every parsing expression instantiates a syntax node. This includes even very simple parsing expressions, like single characters. It is probably unnecessary for every single expression in the parse to correspond to its own syntax node, so much savings could be garnered from a transient declaration that instructs the parser only to attempt a match without instantiating nodes.</p>
+
+<h3>Generate Rule Implementations in C</h3>
+
+<p>Parsing expressions are currently compiled into simple Ruby source code that comprises the body of parsing rules, which are translated into Ruby methods. The generator could produce C instead of Ruby in the body of these method implementations.</p>
+
+<h3>Global Parsing State and Semantic Backtrack Triggering</h3>
+
+<p>Some programming language grammars are not entirely context-free, requiring that global state dictate the behavior of the parser in certain circumstances. Treetop does not currently expose explicit parser control to the grammar writer, and instead automatically constructs the syntax tree for them. A means of semantic parser control compatible with this approach would involve callback methods defined on parsing nodes. Each time a node is successfully parsed it will be given an opportunity to set global state and optionally trigger a parse failure on <em>extrasyntactic</em> grounds. Nodes will probably need to define an additional method that undoes their changes to global state when there is a parse failure and they are backtracked.</p>
+
+<p>Here is a sketch of the potential utility of such mechanisms. Consider the structure of YAML, which uses indentation to indicate block structure.</p>
+
+<pre><code>level_1:
+  level_2a:
+  level_2b:
+    level_3a:
+  level_2c:    
+</code></pre>
+
+<p>Imagine a grammar like the following:</p>
+
+<pre><code>rule yaml_element
+  name ':' block
+  /
+  name ':' value
+end
+
+rule block
+  indent yaml_elements outdent
+end
+
+rule yaml_elements
+  yaml_element (samedent yaml_element)*
+end
+
+rule samedent
+  newline spaces {
+    def after_success(parser_state)
+      spaces.length == parser_state.indent_level
+    end
+  }
+end
+
+rule indent
+  newline spaces {
+    def after_success(parser_state)
+      if spaces.length == parser_state.indent_level + 2
+        parser_state.indent_level += 2
+        true
+      else
+        false # fail the parse on extrasyntactic grounds
+      end
+    end
+
+    def undo_success(parser_state)
+      parser_state.indent_level -= 2
+    end
+  }
+end
+
+rule outdent
+  newline spaces {
+    def after_success(parser_state)
+      if spaces.length == parser_state.indent_level - 2
+        parser_state.indent_level -= 2
+        true
+      else
+        false # fail the parse on extrasyntactic grounds
+      end
+    end
+
+    def undo_success(parser_state)
+      parser_state.indent_level += 2
+    end
+  }
+end
+</code></pre>
+
+<p>In this case a block will be detected only if a change in indentation warrants it. Note that this change in the state of indentation must be undone if a subsequent failure causes this node not to ultimately be incorporated into a successful result.</p>
+
+<p>I am by no means sure that the above sketch is free of problems, or even that this overall strategy is sound, but it seems like a promising path.</p></div></div><div id="bottom"></div></body></html>

data/doc/site/index.html

@@ -0,0 +1,102 @@
+<html><head><link href="./screen.css" rel="stylesheet" type="text/css" />
+          <script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
+          </script>
+          <script type="text/javascript">
+          _uacct = "UA-3418876-1";
+          urchinTracker();
+          </script>
+        </head><body><div id="top"><div id="main_navigation"><ul><li><a href="syntactic_recognition.html">Documentation</a></li><li><a href="contribute.html">Contribute</a></li><li>Home</li></ul></div></div><div id="middle"><div id="main_content"><p class="intro_text">
+
+Treetop is a language for describing languages. Combining the elegance of Ruby with cutting-edge <em>parsing expression grammars</em>, it helps you analyze syntax with revolutionary ease.
+
+</p>
+
+
+<pre><code>sudo gem install treetop
+</code></pre>
+
+<h1>Intuitive Grammar Specifications</h1>
+
+<p>Parsing expression grammars (PEGs) are simple to write and easy to maintain. They are a simple but powerful generalization of regular expressions that are easier to work with than the LALR or LR-1 grammars of traditional parser generators. There's no need for a tokenization phase, and <em>lookahead assertions</em> can be used for a limited degree of context-sensitivity. Here's an extremely simple Treetop grammar that matches a subset of arithmetic, respecting operator precedence:</p>
+
+<pre><code>grammar Arithmetic
+  rule additive
+    multitive ( '+' multitive )*
+  end
+
+  rule multitive
+    primary ( [*/%] primary )*
+  end
+
+  rule primary
+    '(' additive ')' / number
+  end
+
+  rule number
+    '-'? [1-9] [0-9]*
+  end
+end
+</code></pre>
+
+<h1>Syntax-Oriented Programming</h1>
+
+<p>Rather than implementing semantic actions that construct parse trees, Treetop lets you define methods on trees that it constructs for you automatically. You can define these methods directly within the grammar...</p>
+
+<pre><code>grammar Arithmetic
+  rule additive
+    multitive a:( '+' multitive )* {
+      def value
+        a.elements.inject(multitive.value) { |sum, e|
+          sum+e.multitive.value
+        }
+      end
+    }
+  end
+
+  # other rules below ...
+end
+</code></pre>
+
+<p>...or associate rules with classes of nodes you wish your parsers to instantiate upon matching a rule.</p>
+
+<pre><code>grammar Arithmetic
+  rule additive
+    multitive ('+' multitive)* &lt;AdditiveNode&gt;
+  end
+
+  # other rules below ...
+end
+</code></pre>
+
+<h1>Reusable, Composable Language Descriptions</h1>
+
+<p>Because PEGs are closed under composition, Treetop grammars can be treated like Ruby modules. You can mix them into one another and override rules with access to the <code>super</code> keyword. You can break large grammars down into coherent units or make your language's syntax modular. This is especially useful if you want other programmers to be able to reuse your work.</p>
+
+<pre><code>grammar RubyWithEmbeddedSQL
+  include SQL
+
+  rule string
+    quote sql_expression quote / super
+  end
+end
+</code></pre>
+
+<h1>Acknowledgements</h1>
+
+<p><a href="http://pivotallabs.com"><img id="pivotal_logo" src="./images/pivotal.gif"></a></p>
+
+<p>First, thank you to my employer Rob Mee of <a href="http://pivotallabs.com"/>Pivotal Labs</a> for funding a substantial portion of Treetop's development. He gets it.</p>
+
+<p>I'd also like to thank:</p>
+
+<ul>
+<li>Damon McCormick for several hours of pair programming.</li>
+<li>Nick Kallen for lots of well-considered feedback and a few afternoons of programming.</li>
+<li>Brian Takita for a night of pair programming.</li>
+<li>Eliot Miranda for urging me rewrite as a compiler right away rather than putting it off.</li>
+<li>Ryan Davis and Eric Hodel for hurting my code.</li>
+<li>Dav Yaginuma for kicking me into action on my idea.</li>
+<li>Bryan Ford for his seminal work on Packrat Parsers.</li>
+<li>The editors of Lambda the Ultimate, where I discovered parsing expression grammars.</li>
+</ul>
+</div></div><div id="bottom"></div></body></html>