treetop 1.4.14 → 1.4.15

data/README.md

@@ -30,17 +30,18 @@ Next, you start filling your grammar with rules. Each rule associates a name wit
 
 The first rule becomes the *root* of the grammar, causing its expression to be matched when a parser for the grammar is fed a string. The above grammar can now be used in a Ruby program. Notice how a string matching the first rule parses successfully, but a second nonmatching string does not.
 
-    # use_grammar.rb
-    require 'rubygems'
-    require 'treetop'
-    Treetop.load 'my_grammar'
-    # or just:
-    # require 'my_grammar'                     # This works because Polyglot hooks "require" to find and load Treetop files
+```ruby
+# use_grammar.rb
+require 'rubygems'
+require 'treetop'
+Treetop.load 'my_grammar'
+# or just:
+# require 'my_grammar'                     # This works because Polyglot hooks "require" to find and load Treetop files
     
-    parser = MyGrammarParser.new
-    puts parser.parse('hello chomsky')         # => Treetop::Runtime::SyntaxNode
-    puts parser.parse('silly generativists!')  # => nil
-
+parser = MyGrammarParser.new
+puts parser.parse('hello chomsky')         # => Treetop::Runtime::SyntaxNode
+puts parser.parse('silly generativists!')  # => nil
+```
 Users of *regular expressions* will find parsing expressions familiar. They share the same basic purpose, matching strings against patterns. However, parsing expressions can recognize a broader category of languages than their less expressive brethren. Before we get into demonstrating that, lets cover some basics. At first parsing expressions won't seem much different. Trust that they are.
 
 Terminal Symbols
@@ -57,12 +58,13 @@ Ordered choices are *composite expressions*, which allow for any of several sube
         'hello chomsky' / 'hello lambek'
       end
     end
-    
-    # fragment of use_grammar.rb
-    puts parser.parse('hello chomsky')         # => Treetop::Runtime::SyntaxNode
-    puts parser.parse('hello lambek')          # => Treetop::Runtime::SyntaxNode
-    puts parser.parse('silly generativists!')  # => nil
 
+```ruby    
+# fragment of use_grammar.rb
+puts parser.parse('hello chomsky')         # => Treetop::Runtime::SyntaxNode
+puts parser.parse('hello lambek')          # => Treetop::Runtime::SyntaxNode
+puts parser.parse('silly generativists!')  # => nil
+```
 Note that once a choice rule has matched the text using a particular alternative at a particular location in the input and hence has succeeded, that choice will never be reconsidered, even if the chosen alternative causes another rule to fail where a later alternative wouldn't have. It's always a later alternative, since the first to succeed is final - why keep looking when you've found what you wanted? This is a feature of PEG parsers that you need to understand if you're going to succeed in using Treetop. In order to memoize success and failures, such decisions cannot be reversed. Luckily Treetop provides a variety of clever ways you can tell it to avoid making the wrong decisions. But more on that later.
 
 Sequences

data/Rakefile

@@ -15,7 +15,11 @@ Jeweler::Tasks.new do |gem|
   gem.homepage = "https://github.com/cjheath/treetop"
   gem.platform = Gem::Platform::RUBY
   gem.summary = "A Ruby-based text parsing and interpretation DSL"
-  gem.files = ["LICENSE", "README.md", "Rakefile", "treetop.gemspec", "{spec,lib,bin,doc,examples}/**/*"].map{|p| Dir[p]}.flatten
+  gem.files = [
+      "LICENSE", "README.md", "Rakefile", "treetop.gemspec",
+      "{spec,lib,bin,examples}/**/*",
+      "doc/*"
+    ].map{|p| Dir[p] }.flatten
   gem.bindir = "bin"
   gem.executables = ["tt"]
   gem.require_path = "lib"

data/doc/tt.1

@@ -0,0 +1,83 @@
+.\" treetop - Bringing the simplicity of Ruby to syntactic analysis
+.\"
+.\" Copyright (c) 2007 Nathan Sobo.
+.\"
+.\" Permission is hereby granted, free of charge, to any person obtaining a copy
+.\" of this software and associated documentation files (the "Software"), to deal
+.\" in the Software without restriction, including without limitation the rights
+.\" to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+.\" copies of the Software, and to permit persons to whom the Software is
+.\" furnished to do so, subject to the following conditions:
+.\"
+.\" The above copyright notice and this permission notice shall be included in
+.\" all copies or substantial portions of the Software.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+.\" IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+.\" FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+.\" AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+.\" LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+.\" OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+.\" THE SOFTWARE.
+.TH tt 1 2013-06-19 Treetop "Treetop v1.4.14"
+.SH NAME
+tt \- Compile a treetop grammar file to ruby source code
+.SH SYNOPSIS
+.B tt
+.RI [ options "] " grammar_file "[.treetop|.tt] ..."
+
+.SH DESCRIPTION
+The
+.B tt
+program is a command-line script to compile .treetop files into Ruby
+source code.
+
+The
+.B tt
+program takes a list of files with a .treetop extension and compiles
+them into .rb files of the same name. You can then require these files
+like any other Ruby script.
+
+Alternately, you can supply just one .treetop file and a \-o flag to
+specify the name of the output file.
+
+Note: while treetop grammar files
+.B must
+have a supported filename extensions, (.treetop or .tt), the extension
+name is not required when calling the compiler with grammar file
+names.
+.SH OPTIONS
+.TP 4
+.BI "\-o, \-\-output" " FILENAME"
+
+Write parser source to 
+.I FILENAME.
+.TP 4
+.B \-f, \-\-force
+
+Overwrite existing output file(s)
+.TP 4
+.B \-v, \-\-version
+
+Show Treetop version
+.TP 4
+.B \-h, \-\-help
+
+.SH EXAMPLES
+.TP 4
+1 grammar -> 1 parser source
+
+tt foo.tt
+.TP 4
+2 grammars -> 2 separate parsers
+
+tt foo bar.treetop
+.TP 4
+Alternately named output file
+
+tt \-o alterate_name.rb foo
+.SH SEE ALSO
+
+The treetop website:
+
+.B http://treetop.rubyforge.org

data/lib/treetop/ruby_extensions/string.rb

@@ -22,23 +22,19 @@ class String
   # The following methods are lifted from Facets 2.0.2
   def tabto(n)
     if self =~ /^( *)\S/
-      indent(n - $1.length)
-    else
-      self
-    end
-  end
-
-  unless method_defined?(:indent)
-    def indent(n)
-      if n >= 0
-        gsub(/^/, ' ' * n)
+      # Inlined due to collision with ActiveSupport 4.0: indent(n - $1.length)
+      m = n - $1.length
+      if m >= 0
+        gsub(/^/, ' ' * m)
       else
-        gsub(/^ {0,#{-n}}/, "")
+        gsub(/^ {0,#{-m}}/, "")
       end
+    else
+      self
     end
   end
 
   def treetop_camelize
     to_s.gsub(/\/(.?)/){ "::" + $1.upcase }.gsub(/(^|_)(.)/){ $2.upcase }
   end
-end
+end

data/lib/treetop/version.rb

@@ -2,7 +2,7 @@ module Treetop #:nodoc:
   module VERSION #:nodoc:
     MAJOR = 1
     MINOR = 4
-    TINY  = 14
+    TINY  = 15
 
     STRING = [MAJOR, MINOR, TINY].join('.')
   end

data/spec/compiler/character_class_spec.rb

@@ -86,7 +86,7 @@ module CharacterClassSpec
   end
 
   describe "a character class with a negated POSIX bracket expression" do
-    testing_expression "[[:^space:]]"
+    testing_expression "[^[:space:]]"
     it "matches a character outside the negated class" do
       parse('a').should_not be_nil
     end

data/spec/compiler/grammar_compiler_spec.rb

@@ -12,8 +12,12 @@ describe Compiler::GrammarCompiler do
     @source_path_with_treetop_extension = "#{dir}/test_grammar.treetop"
     @source_path_with_do = "#{dir}/test_grammar_do.treetop"
     @source_path_with_tt_extension = "#{dir}/test_grammar.tt"
+    @source_path_with_magic_coding = "#{dir}/test_grammar_magic_coding.treetop"
+    @source_path_with_magic_encoding = "#{dir}/test_grammar_magic_encoding.treetop"
     @target_path = "#{@tmpdir}/test_grammar.rb"
     @target_path_with_do = "#{@tmpdir}/test_grammar_do.rb"
+    @target_path_with_magic_coding = "#{@tmpdir}/test_grammar_magic_coding.rb"
+    @target_path_with_magic_encoding = "#{@tmpdir}/test_grammar_magic_encoding.rb"
     @alternate_target_path = "#{@tmpdir}/test_grammar_alt.rb"
     delete_target_files
   end
@@ -82,6 +86,24 @@ describe Compiler::GrammarCompiler do
     Test::GrammarParser.new.parse('foo').should_not be_nil
   end
 
+  specify "grammars with magic 'encoding' comments keep those comments at the top" do
+    src_copy = "#{@tmpdir}/test_grammar_magic_encoding.treetop"
+    File.open(@source_path_with_magic_encoding) do |f|
+      File.open(src_copy,'w'){|o|o.write(f.read)}
+    end
+    compiler.compile(src_copy)
+    File.open(@target_path_with_magic_encoding).readline.should == "# encoding: UTF-8\n"
+  end
+
+  specify "grammars with magic 'coding' comments keep those comments at the top" do
+    src_copy = "#{@tmpdir}/test_grammar_magic_coding.treetop"
+    File.open(@source_path_with_magic_coding) do |f|
+      File.open(src_copy,'w'){|o|o.write(f.read)}
+    end
+    compiler.compile(src_copy)
+    File.open(@target_path_with_magic_coding).readline.should == "# coding: UTF-8\n"
+  end
+
   def delete_target_files
     File.delete(target_path) if File.exists?(target_path)
     File.delete(@target_path_with_do) if File.exists?(@target_path_with_do)

data/spec/compiler/test_grammar_magic_coding.treetop

@@ -0,0 +1,8 @@
+# coding: UTF-8
+module Test
+  grammar Grammar do
+    rule foo do
+      'foo'
+    end
+  end
+end

data/spec/compiler/test_grammar_magic_encoding.treetop

@@ -0,0 +1,8 @@
+# encoding: UTF-8
+module Test
+  grammar Grammar do
+    rule foo do
+      'foo'
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -63,7 +63,7 @@ module Treetop
     def parse_multibyte(input, options = {})
       require 'active_support/all'
 
-      if RUBY_VERSION !~ /^1.9/ && 'NONE' == $KCODE then $KCODE = 'UTF8' end
+      if RUBY_VERSION !~ /^(1\.9|2\.0)/ && 'NONE' == $KCODE then $KCODE = 'UTF8' end
       # rspec 1.3 used to do something similar (set it to 'u') that we need
       # for activerecord multibyte wrapper to kick in (1.8 only? @todo)
 

data/treetop.gemspec

@@ -5,12 +5,12 @@
 
 Gem::Specification.new do |s|
   s.name = "treetop"
-  s.version = "1.4.14"
+  s.version = "1.4.15"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Nathan Sobo", "Clifford Heath"]
   s.autorequire = "treetop"
-  s.date = "2013-06-04"
+  s.date = "2013-08-17"
   s.email = "cliffordheath@gmail.com"
   s.executables = ["tt"]
   s.extra_rdoc_files = [
@@ -28,21 +28,9 @@ Gem::Specification.new do |s|
     "doc/pitfalls_and_advanced_techniques.markdown",
     "doc/semantic_interpretation.markdown",
     "doc/site.rb",
-    "doc/site/contribute.html",
-    "doc/site/images/bottom_background.png",
-    "doc/site/images/middle_background.png",
-    "doc/site/images/paren_language_output.png",
-    "doc/site/images/pivotal.gif",
-    "doc/site/images/top_background.png",
-    "doc/site/index.html",
-    "doc/site/pitfalls_and_advanced_techniques.html",
-    "doc/site/robots.txt",
-    "doc/site/screen.css",
-    "doc/site/semantic_interpretation.html",
-    "doc/site/syntactic_recognition.html",
-    "doc/site/using_in_ruby.html",
     "doc/sitegen.rb",
     "doc/syntactic_recognition.markdown",
+    "doc/tt.1",
     "doc/using_in_ruby.markdown",
     "examples/lambda_calculus/arithmetic.rb",
     "examples/lambda_calculus/arithmetic.treetop",
@@ -120,6 +108,8 @@ Gem::Specification.new do |s|
     "spec/compiler/test_grammar.treetop",
     "spec/compiler/test_grammar.tt",
     "spec/compiler/test_grammar_do.treetop",
+    "spec/compiler/test_grammar_magic_coding.treetop",
+    "spec/compiler/test_grammar_magic_encoding.treetop",
     "spec/compiler/tt_compiler_spec.rb",
     "spec/compiler/zero_or_more_spec.rb",
     "spec/composition/a.treetop",

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: treetop
 version: !ruby/object:Gem::Version
-  version: 1.4.14
+  version: 1.4.15
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: treetop
 bindir: bin
 cert_chain: []
-date: 2013-06-04 00:00:00.000000000 Z
+date: 2013-08-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: polyglot
@@ -159,21 +159,9 @@ files:
 - doc/pitfalls_and_advanced_techniques.markdown
 - doc/semantic_interpretation.markdown
 - doc/site.rb
-- doc/site/contribute.html
-- doc/site/images/bottom_background.png
-- doc/site/images/middle_background.png
-- doc/site/images/paren_language_output.png
-- doc/site/images/pivotal.gif
-- doc/site/images/top_background.png
-- doc/site/index.html
-- doc/site/pitfalls_and_advanced_techniques.html
-- doc/site/robots.txt
-- doc/site/screen.css
-- doc/site/semantic_interpretation.html
-- doc/site/syntactic_recognition.html
-- doc/site/using_in_ruby.html
 - doc/sitegen.rb
 - doc/syntactic_recognition.markdown
+- doc/tt.1
 - doc/using_in_ruby.markdown
 - examples/lambda_calculus/arithmetic.rb
 - examples/lambda_calculus/arithmetic.treetop
@@ -251,6 +239,8 @@ files:
 - spec/compiler/test_grammar.treetop
 - spec/compiler/test_grammar.tt
 - spec/compiler/test_grammar_do.treetop
+- spec/compiler/test_grammar_magic_coding.treetop
+- spec/compiler/test_grammar_magic_encoding.treetop
 - spec/compiler/tt_compiler_spec.rb
 - spec/compiler/zero_or_more_spec.rb
 - spec/composition/a.treetop
@@ -289,7 +279,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 622706517614693275
+      hash: 2062680504230675145
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:

data/doc/site/contribute.html

@@ -1,124 +0,0 @@
-<html><head><link href="./screen.css" rel="stylesheet" type="text/css" />
-          <script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
-          </script>
-          <script type="text/javascript">
-          _uacct = "UA-3418876-1";
-          urchinTracker();
-          </script>
-        </head><body><div id="top"><div id="main_navigation"><ul><li><a href="syntactic_recognition.html">Documentation</a></li><li>Contribute</li><li><a href="index.html">Home</a></li></ul></div></div><div id="middle"><div id="main_content"><h1>Google Group</h1>
-
-<p>I've created a <a href="http://groups.google.com/group/treetop-dev">Google Group</a> as a better place to organize discussion and development.
-treetop-dev@google-groups.com</p>
-
-<h1>Contributing</h1>
-
-<p>Visit <a href="http://github.com/nathansobo/treetop/tree/master">the Treetop repository page on GitHub</a> in your browser for more information about checking out the source code.</p>
-
-<p>I like to try Rubinius's policy regarding commit rights. If you submit one patch worth integrating, I'll give you commit rights. We'll see how this goes, but I think it's a good policy.</p>
-
-<h2>Getting Started with the Code</h2>
-
-<p>Treetop compiler is interesting in that it is implemented in itself. Its functionality revolves around <code>metagrammar.treetop</code>, which specifies the grammar for Treetop grammars. I took a hybrid approach with regard to definition of methods on syntax nodes in the metagrammar. Methods that are more syntactic in nature, like those that provide access to elements of the syntax tree, are often defined inline, directly in the grammar. More semantic methods are defined in custom node classes.</p>
-
-<p>Iterating on the metagrammar is tricky. The current testing strategy uses the last stable version of Treetop to parse the version under test. Then the version under test is used to parse and functionally test the various pieces of syntax it should recognize and translate to Ruby. As you change <code>metagrammar.treetop</code> and its associated node classes, note that the node classes you are changing are also used to support the previous stable version of the metagrammar, so must be kept backward compatible until such time as a new stable version can be produced to replace it.</p>
-
-<h2>Tests</h2>
-
-<p>Most of the compiler's tests are functional in nature. The grammar under test is used to parse and compile piece of sample code. Then I attempt to parse input with the compiled output and test its results.</p>
-
-<h1>What Needs to be Done</h1>
-
-<h2>Small Stuff</h2>
-
-<ul>
-<li>Improve the <code>tt</code> command line tool to allow <code>.treetop</code> extensions to be elided in its arguments.</li>
-<li>Generate and load temp files with <code>Treetop.load</code> rather than evaluating strings to improve stack trace readability.</li>
-<li>Allow <code>do/end</code> style blocks as well as curly brace blocks. This was originally omitted because I thought it would be confusing. It probably isn't.</li>
-</ul>
-
-
-<h2>Big Stuff</h2>
-
-<h4>Transient Expressions</h4>
-
-<p>Currently, every parsing expression instantiates a syntax node. This includes even very simple parsing expressions, like single characters. It is probably unnecessary for every single expression in the parse to correspond to its own syntax node, so much savings could be garnered from a transient declaration that instructs the parser only to attempt a match without instantiating nodes.</p>
-
-<h3>Generate Rule Implementations in C</h3>
-
-<p>Parsing expressions are currently compiled into simple Ruby source code that comprises the body of parsing rules, which are translated into Ruby methods. The generator could produce C instead of Ruby in the body of these method implementations.</p>
-
-<h3>Global Parsing State and Semantic Backtrack Triggering</h3>
-
-<p>Some programming language grammars are not entirely context-free, requiring that global state dictate the behavior of the parser in certain circumstances. Treetop does not currently expose explicit parser control to the grammar writer, and instead automatically constructs the syntax tree for them. A means of semantic parser control compatible with this approach would involve callback methods defined on parsing nodes. Each time a node is successfully parsed it will be given an opportunity to set global state and optionally trigger a parse failure on <em>extrasyntactic</em> grounds. Nodes will probably need to define an additional method that undoes their changes to global state when there is a parse failure and they are backtracked.</p>
-
-<p>Here is a sketch of the potential utility of such mechanisms. Consider the structure of YAML, which uses indentation to indicate block structure.</p>
-
-<pre><code>level_1:
-  level_2a:
-  level_2b:
-    level_3a:
-  level_2c:    
-</code></pre>
-
-<p>Imagine a grammar like the following:</p>
-
-<pre><code>rule yaml_element
-  name ':' block
-  /
-  name ':' value
-end
-
-rule block
-  indent yaml_elements outdent
-end
-
-rule yaml_elements
-  yaml_element (samedent yaml_element)*
-end
-
-rule samedent
-  newline spaces {
-    def after_success(parser_state)
-      spaces.length == parser_state.indent_level
-    end
-  }
-end
-
-rule indent
-  newline spaces {
-    def after_success(parser_state)
-      if spaces.length == parser_state.indent_level + 2
-        parser_state.indent_level += 2
-        true
-      else
-        false # fail the parse on extrasyntactic grounds
-      end
-    end
-
-    def undo_success(parser_state)
-      parser_state.indent_level -= 2
-    end
-  }
-end
-
-rule outdent
-  newline spaces {
-    def after_success(parser_state)
-      if spaces.length == parser_state.indent_level - 2
-        parser_state.indent_level -= 2
-        true
-      else
-        false # fail the parse on extrasyntactic grounds
-      end
-    end
-
-    def undo_success(parser_state)
-      parser_state.indent_level += 2
-    end
-  }
-end
-</code></pre>
-
-<p>In this case a block will be detected only if a change in indentation warrants it. Note that this change in the state of indentation must be undone if a subsequent failure causes this node not to ultimately be incorporated into a successful result.</p>
-
-<p>I am by no means sure that the above sketch is free of problems, or even that this overall strategy is sound, but it seems like a promising path.</p></div></div><div id="bottom"></div></body></html>