rattler 0.5.0 → 0.6.0

data/README.rdoc

@@ -12,183 +12,11 @@ as possible. To that end, the normal parsing expression grammar syntax is
 extended with simple features to overcome some of the limitations of pure
 PEG grammars.
 
-A language syntax is specified in a grammar using the Rattler syntax. Parser
-classes and modules can be generated statically using the "rtlr" command or
-dynamically from strings.
+=== Documentation
 
-The cucumber features are currently the best documentation.
+For the best documentation see the {cucumber features}[http://www.relishapp.com/jarhart/rattler]
 
-{RDoc}[http://rubydoc.info/gems/rattler/0.5.0/frames]
-
-== FEATURES:
-
-* Uses readable PEG-like grammars
-* Supports directly and indirectly left-recursive grammars
-* Whitespace can be specified in one place and skipped automatically
-* Automatic and custom parse error reporting
-* Optimizing parser generator generates efficient pure-ruby parsers
-* Compatible with Ruby 1.8.7 and 1.9.2, tested with MRI and JRuby on Linux and
-  Windows and with Rubinius on Linux
-
-== PROBLEMS/LIMITATIONS:
-
-* Only strings can be parsed, so files have to be read completely before parsing
-* There are many holes in the tests so there are undoubtedly many bugs
-* Support for composable grammars is not yet implemented
-* Semantics of automatic whitespace skipping could be refined
-* Incomplete documentation
-
-== EXAMPLES:
-
-=== Example 1: Statically generated parser class
-
-==== json.rtlr:
-  
-  # JSON parser based on the grammar at http://www.json.org
-
-  require File.expand_path('json_helper', File.dirname(__FILE__))
-
-  parser JsonParser < Rattler::Runtime::PackratParser
-
-  include JsonHelper
-
-  %whitespace (SPACE+ / comment)* {
-
-    json_text <-  (object / array) EOF
-
-    object    <-  ~'{' members ~'}'                                   { object _ }
-
-    members   <-  pair *, ','
-
-    pair      <-  string ~':' value
-
-    array     <-  ~'[' elements ~']'
-
-    elements  <-  value *, ','
-
-    value     <-  string
-                / number
-                / object
-                / array
-                / `true`                                              { :true }
-                / `false`                                             { :false }
-                / `null`                                              { :null }
-                / fail "value expected"
-
-    string    <-  @('"' char* '"')                                    { string _ }
-
-    number    <-  @(int frac? exp?)                                   { number _ }
-  }
-
-  %inline {
-
-    char      <-  !('"' / '\\' / CNTRL) .
-                / '\\' (["\\/bfnrt] / 'u' XDIGIT XDIGIT XDIGIT XDIGIT)
-
-    int       <-  '-'? ('0' !DIGIT / [1-9] DIGIT*)
-
-    frac      <-  '.' DIGIT+
-
-    exp       <-  [eE] [+-]? DIGIT+
-
-    comment   <-  '/*' (! '*/' .)* '*/'
-                / '//' [^\n]*
-  }
-
-==== json_helper.rb:
-
-  module JsonHelper
-
-    def object(members)
-      Hash[*members.flatten(1)]
-    end
-
-    def string(expr)
-      eval "%q#{expr}", TOPLEVEL_BINDING
-    end
-
-    def number(expr)
-      eval expr, TOPLEVEL_BINDING
-    end
-
-  end
-
-==== $ rtlr json.rtlr
-
-  json.rtlr -> json_parser.rb
-
-==== parse_json.rb:
-
-  require 'rubygems'
-  require 'rattler'
-  require 'json_parser'
-  
-  begin
-    p JsonParser.parse!(open(ARGV[0]) {|io| io.read })
-  rescue Rattler::Runtime::SyntaxError => e
-    puts e
-  end
-
-=== Example 2: Statically generated grammar module
-
-==== json.rtlr:
-  
-  # JSON parser based on the grammar at http://www.json.org
-
-  grammar JsonGrammar
-
-  %whitespace (SPACE+ / comment)* {
-
-  ...
-
-==== $ rtlr json.rtlr
-
-  json.rtlr -> json_grammar.rb
-
-==== json_parser.rb:
-
-  require 'rubygems'
-  require 'rattler'
-  require 'json_grammar'
- 
-  class JsonParser < Rattler::Runtime::PackratParser
-
-    include JsonGrammar
-
-    def object(members)
-      Hash[*members.flatten(1)]
-    end
-
-    ...
-
-=== Example 3: Dynamically generated parser class
-
-  require 'rubygems'
-  require 'rattler'
-  
-  Calculator = Rattler::compile_parser %{
-    %whitespace SPACE*
-    
-    start   <-  expr EOF
-    
-    expr    <-  expr ~'+' term                  {|a, b| a + b }
-              / expr ~'-' term                  {|a, b| a - b }
-              / term
-    
-    term    <-  term ~'*' primary               {|a, b| a * b }
-              / term ~'/' primary               {|a, b| a / b }
-              / primary
-    
-    primary <-  ~'(' expr ~')'
-              / @('-'? DIGIT+ ('.' DIGIT+)?)    { _.to_f }
-  },
-  :type => :extended_packrat
- 
-  begin
-    puts Calculator.parse!(expr)
-  rescue Rattler::Runtime::SyntaxError => e
-    puts e
-  end
+You may also wish to browse the {API Docs}[http://rubydoc.info/gems/rattler/0.5.0/frames]
 
 == REQUIREMENTS:
 

data/features/README.markdown

@@ -0,0 +1,27 @@
+# Rattler - Ruby Tool for Language Recognition
+
+Parsing for Ruby that's so easy it feels like cheating.
+
+Rattler is a parser generator for Ruby based on parsing expression grammars.
+Rattler's purpose is to make language design and recognition as simple and fun
+as possible. To that end, the normal parsing expression grammar syntax is
+extended with simple features to overcome many of the limitations of pure
+PEG grammars.
+
+A language syntax is specified in a grammar using the Rattler syntax. Parser
+classes and modules can be generated statically using the "rtlr" command or
+dynamically from strings.
+
+## Features
+
+* Languages are described using readable PEG-based grammars
+* Directly and indirectly left-recursive grammars can be used
+* Whitespace can be specified in one place and skipped automatically
+* Useful parse error messages are created automatically
+* Custom parse error messages can be specified easily
+* Generates efficient optimized pure-ruby parsers
+* Works with MRI 1.8.7 and 1.9.2, JRuby and Rubinius
+
+[Source](https://github.com/jarhart/rattler)
+
+[API Docs](http://rubydoc.info/gems/rattler/0.5.0/frames)

data/features/Tutorial.md

@@ -0,0 +1,224 @@
+To demonstrate how to get started using Rattler, we'll walk through writing the
+ubiquitous arithmetic expression parser. This tutorial assumes that your are
+familiar with parsing expression grammars. If you are not, you may wish to read
+the [Wikipedia article](http://en.wikipedia.org/wiki/Parsing_expression_grammar)
+on them to familiarize yourself.
+
+### Install Rattler
+
+    $ gem install rattler
+
+### Write a grammar
+
+We'll start by writing just enough to describe the syntax of simple arithmetic
+expressions in a way that automatically honors the standard order of operations.
+
+We'll write the following and save it to a file called "`arithmetic.rtlr`":
+
+    require 'rattler'
+
+    parser ArithmeticParser < Rattler::Runtime::ExtendedPackratParser
+
+    %whitespace SPACE*
+
+    expr    <-  expr "+" term
+              / expr "-" term
+              / term
+
+    term    <-  term "*" primary
+              / term "/" primary
+              / primary
+
+    primary <-  "(" expr ")"
+              / @("-"? DIGIT+ ("." DIGIT+)?)
+
+
+If you have experience with recursive descent parsing, you might think the
+above grammar won't work because recursive descent parsers can't handle left
+recursion. Actually, there is an
+[algorithm](http://www.cs.ucla.edu/~todd/research/pepm08.html) for packrat
+parsing that handles left recursion. Rattler includes an implementation of that
+algorithm, so left-recursive grammars are fine.
+
+Our grammar starts with a [require](/jarhart/rattler/docs/grammar-heading/require)
+statement to require the Rattler library. Any `require` statements in the
+grammar heading are placed at the top of the generated parser.
+
+Next we declare our parser class with a
+[grammar declaration](/jarhart/rattler/docs/grammar-heading/grammar-declaration).
+We extend the ExtendedPackratParser base class, which implements the packrat
+parsing algorithm that supports left recursion.
+
+After the declaring our parser class, we use the
+[`%whitespace`](/jarhart/rattler/docs/extended-matching-syntax/whitespace-skipping)
+directive to tell Rattler to generate a parser that automatically ignores any
+whitespace characters. `SPACE` is a
+[POSIX character class](/jarhart/rattler/docs/extended-matching-syntax/posix-character-classes)
+that matches any whitespace character.
+
+The rest of the grammar is a standard parsing expression grammar until we get
+to the last line. The expression "`@("-"? DIGIT+ ("." DIGIT+)?)`" will match a
+decimal number. The `@` is the 
+[Token Operator](/jarhart/rattler/docs/extended-matching-syntax/token-operator)
+which causes the expression to be matched as a single string instead of
+producing a parse tree. The `DIGIT` character class matches decimal digits.
+
+### Generate the parser
+
+    $ rtlr arithmetic.rtlr
+    arithmetic.rtlr -> arithmetic_parser.rb
+
+This generates a file called `arithmetic_parser.rb`. We can test out the parser
+by running the file as a script.
+
+    $ echo "1 + 2 / 3" | ruby arithmetic_parser.rb
+
+This will print out the parse tree:
+
+    ["1", "+", ["2", "/", "3"]]
+
+If you have GraphViz and the ruby-graphviz gem installed you can view the parse
+tree with dotty:
+
+    $ echo "1 + 2 / 3" | ruby arithmetic_parser.rb --graphviz
+
+You can use the `--help` option to see all of the available command-line
+options:
+
+    $ arithmetic_parser.rb --help
+    Usage: arithmetic_parser.rb [filenames] [options]
+
+        -g, --graphviz                   Display the parse tree using GraphViz dotty
+            --jpg FILENAME               Output the parse tree as a JPG file using GraphViz
+            --png FILENAME               Output the parse tree as a PNG file using GraphViz
+            --svg FILENAME               Output the parse tree as an SVG file using GraphViz
+            --pdf FILENAME               Output the parse tree as a PDF file using GraphViz
+            --ps FILENAME                Output the parse tree as a PS file using GraphViz
+            --ps2 FILENAME               Output the parse tree as a PS2 file using GraphViz
+            --eps FILENAME               Output the parse tree as an EPS file using GraphViz
+        -f, --file FILENAME              Output the parse tree as a GraphViz DOT language file
+        -d, --dot                        Output the parse tree in the GraphViz DOT language
+
+        -h, --help                       Show this message
+
+### Iterate
+
+Let's try parentheses to see if the parser groups the operations correctly:
+
+    $ echo "(1 + 2) / 3" | ruby arithmetic_parser.rb
+    [["(", ["1", "+", "2"], ")"], "/", "3"]
+
+The grouping works, but we get the parentheses in the parse tree, which we
+really don't need. Let's fix that by using the `~` operator to
+[skip](/jarhart/rattler/docs/extended-matching-syntax/skip-operator) those:
+
+Change the line
+
+    primary <-  "(" expr ")"
+
+To read:
+
+    primary <-  ~"(" expr ~")"
+
+
+And regenerate the parser, using `-f` to force overwriting the file.
+
+    $ rtlr arithmetic.rtlr -f
+    arithmetic.rtlr -> arithmetic_parser.rb
+
+Now when we try parsing the expression with parentheses:
+
+    $ echo "(1 + 2) / 3" | ruby arithmetic_parser.rb
+    [["1", "+", "2"], "/", "3"]
+
+We get the correct grouping without the superflous parentheses in the result.
+
+We're still just getting arrays of tokens, and while that makes it easy to see
+the parse tree in this case, it doesn't help much when we want to add semantics.
+Each type of operation is matched by a separate sequence expression, but that
+information is partially lost by making everything an array. We can add
+[semantic attributes](/jarhart/rattler/docs/semantics/node-actions) to capture
+the different operations.
+
+We'll start by just using generic parse nodes and giving them mnemonic names.
+We can also discard the matched operator tokens themselves at this point.
+
+Change the `expr` and `term` rules to look like this:
+
+    expr    <-  expr ~"+" term      <"ADD">
+              / expr ~"-" term      <"SUB">
+              / term
+
+    term    <-  term ~"*" primary   <"MUL">
+              / term ~"/" primary   <"DIV">
+              / primary
+
+Don't forget the `~` in front of the operators to omit them from the parse
+results.
+
+Using parse nodes like this makes it very easy to see the parse tree with
+graphviz, especially with more complex expressions.
+
+    $ rtlr arithmetic.rtlr -f
+    arithmetic.rtlr -> arithmetic_parser.rb
+    $ echo "1+2*3/4-5+6-7*8/9" | ruby arithmetic_parser.rb -g
+
+This feature is useful for debugging more complex grammars, but we can see that
+our parser is parsing correctly, so let's finish by replacing the parse node
+attributes with
+[semantic actions](/jarhart/rattler/docs/semantics/semantic-actions) to perform
+the arithmetic operations as it parses.
+
+Change the `expr` and `term` rules to look like this:
+
+    expr    <-  expr ~"+" term                  {|a, b| a + b }
+              / expr ~"-" term                  {|a, b| a - b }
+              / term
+
+    term    <-  term ~"*" primary               {|a, b| a * b }
+              / term ~"/" primary               {|a, b| a / b }
+              / primary
+
+Semantic actions look like ruby blocks and, for the most part, act like them
+too. The parse results are bound to the parameters and the actions are
+performed as soon as the parsing expression matches. The special `_` variable
+can be used to refer to the entire parse results.
+
+    $ echo "1+2*3/4-5+6-7*8/9" | bundle exec ruby arithmetic_parser.rb
+    -2.7222222222222223
+
+Success! Of course we could have accomplished the same thing with
+"`eval ARGV.join`" but it's just a tutorial.
+
+### Finish
+
+There's one last finishing touch to add. As it is now, the parser will accept
+any input the starts with a valid expression no matter what comes after it.
+We'll add a new rule at the top so that it only matches if the entire input is
+a valid expression:
+
+    start   <-  expr EOF
+
+The [`EOF`](/jarhart/rattler/docs/extended-matching-syntax/eof-symbol) symbol
+matches only if there is no more input to parse.
+
+Our final grammar looks like this:
+
+    require 'rattler'
+
+    parser ArithmeticParser < Rattler::Runtime::ExtendedPackratParser
+
+    %whitespace SPACE*
+
+    start   <-  expr EOF
+
+    expr    <-  expr ~"+" term                  {|a, b| a + b }
+              / expr ~"-" term                  {|a, b| a - b }
+              / term
+
+    term    <-  term ~"*" primary               {|a, b| a * b }
+              / term ~"/" primary               {|a, b| a / b }
+              / primary
+
+    primary <-  ~"(" expr ~")"
+              / @("-"? DIGIT+ ("." DIGIT+)?)    { _.to_f }

data/features/command_line/output_option.feature

@@ -1,7 +1,8 @@
 Feature: --output option
 
   Use the --output (or -o) option to specify a different output file name, or
-  "-" for STDOUT.
+  "-" for STDOUT. By default the output filename is the grammar or parser name
+  converted from camel-case to underscore.
 
   @command-line
   Scenario: Specify a different file name

data/features/command_line/{parser_generator.feature → rtlr.feature}

@@ -1,7 +1,8 @@
-Feature: Parser Generator
+Feature: rtlr
 
   The "rtlr" command is used to generate a grammar module or parser class from
-  a Rattler grammar file.
+  a Rattler grammar file.  The generated file is itself a parser script that
+  makes it easy to try out the parser.
   
   @command-line
   Scenario: Getting help
@@ -14,7 +15,6 @@ Feature: Parser Generator
           -d, --dest DIRECTORY             Specify an explicit destination directory
           -o, --output FILENAME            Specify a different output filename ("-" = STDOUT)
           -f, --force                      Force overwrite if the output file exists
-          -s, --standalone                 Optimize for use as a standalone parser
           -n, --no-optimize                Disable optimization
 
           -h, --help                       Show this message
@@ -27,7 +27,7 @@ Feature: Parser Generator
       grammar BinaryGrammar
       expr <- [01]*
       """
-    When I run `rtlr binary.rtlr --standalone`
+    When I run `rtlr binary.rtlr`
     Then the output should contain "binary.rtlr -> binary_grammar.rb"
       And the file "binary_grammar.rb" should contain:
         """
@@ -41,13 +41,27 @@ Feature: Parser Generator
           
           # @private
           def match_expr #:nodoc:
-            a0 = []
-            while r = @scanner.scan(/[01]/)
-              a0 << r
-            end
-            a0
+            apply :match_expr!
           end
           
+          # @private
+          def match_expr! #:nodoc:
+            begin
+              a0 = []
+              while r = @scanner.scan(/[01]/)
+                a0 << r
+              end
+              a0
+            end ||
+            fail { :expr }
+          end
+          
+        end
+
+        if __FILE__ == $0
+          require 'rubygems'
+          require 'rattler'
+          Rattler::Util::GrammarCLI.run(BinaryGrammar)
         end
         """
   
@@ -59,7 +73,7 @@ Feature: Parser Generator
       parser BinaryParser < Rattler::Runtime::PackratParser
       expr <- [01]*
       """
-    When I run `rtlr binary.rtlr --standalone`
+    When I run `rtlr binary.rtlr`
     Then the output should contain "binary.rtlr -> binary_parser.rb"
       And the file "binary_parser.rb" should contain:
         """
@@ -75,12 +89,26 @@ Feature: Parser Generator
           
           # @private
           def match_expr #:nodoc:
-            a0 = []
-            while r = @scanner.scan(/[01]/)
-              a0 << r
-            end
-            a0
+            apply :match_expr!
           end
           
+          # @private
+          def match_expr! #:nodoc:
+            begin
+              a0 = []
+              while r = @scanner.scan(/[01]/)
+                a0 << r
+              end
+              a0
+            end ||
+            fail { :expr }
+          end
+          
+        end
+        
+        if __FILE__ == $0
+          require 'rubygems'
+          require 'rattler'
+          Rattler::Util::ParserCLI.run(BinaryParser)
         end
         """