rattler 0.4.2 → 0.5.0

data/README.rdoc

@@ -4,6 +4,8 @@
 
 === Rattler - Ruby Tool for Language Recognition
 
+Parsing for Ruby that's so easy it feels like cheating.
+
 Rattler is a parser generator for Ruby based on parsing expression grammars.
 Rattler's purpose is to make language design and recognition as simple and fun
 as possible. To that end, the normal parsing expression grammar syntax is
@@ -14,7 +16,9 @@ A language syntax is specified in a grammar using the Rattler syntax. Parser
 classes and modules can be generated statically using the "rtlr" command or
 dynamically from strings.
 
-{RDoc}[http://rubydoc.info/gems/rattler/0.4.2/frames]
+The cucumber features are currently the best documentation.
+
+{RDoc}[http://rubydoc.info/gems/rattler/0.5.0/frames]
 
 == FEATURES:
 
@@ -30,9 +34,9 @@ dynamically from strings.
 
 * Only strings can be parsed, so files have to be read completely before parsing
 * There are many holes in the tests so there are undoubtedly many bugs
-* Optimzations are not complete
 * Support for composable grammars is not yet implemented
-* Symantics of automatic whitespace skipping could be refined
+* Semantics of automatic whitespace skipping could be refined
+* Incomplete documentation
 
 == EXAMPLES:
 
@@ -50,6 +54,8 @@ dynamically from strings.
 
   %whitespace (SPACE+ / comment)* {
 
+    json_text <-  (object / array) EOF
+
     object    <-  ~'{' members ~'}'                                   { object _ }
 
     members   <-  pair *, ','

data/features/command_line/dest_option.feature

@@ -12,7 +12,7 @@ Feature: --dest option
       expr <- [01]*
       """
       And a directory named "lib/my_examples"
-    When I run "rtlr --dest lib/my_examples binary.rtlr"
+    When I run `rtlr --dest lib/my_examples binary.rtlr`
     Then the output should contain "binary.rtlr -> lib/my_examples/binary_grammar.rb"
       And the file "lib/my_examples/binary_grammar.rb" should contain:
         """

data/features/command_line/lib_option.feature

@@ -14,7 +14,7 @@ Feature: --lib option
       expr <- [01]*
       """
       And a directory named "lib"
-    When I run "rtlr --lib lib binary.rtlr"
+    When I run `rtlr --lib lib binary.rtlr`
     Then the output should contain "binary.rtlr -> lib/binary_grammar.rb"
       And the file "lib/binary_grammar.rb" should contain:
         """
@@ -29,7 +29,7 @@ Feature: --lib option
       expr <- [01]*
       """
       And a directory named "lib/examples"
-    When I run "rtlr --lib lib binary.rtlr"
+    When I run `rtlr --lib lib binary.rtlr`
     Then the output should contain "binary.rtlr -> lib/examples/binary_grammar.rb"
       And the file "lib/examples/binary_grammar.rb" should contain:
         """

data/features/command_line/output_option.feature

@@ -11,7 +11,7 @@ Feature: --output option
       expr <- [01]*
       """
       And a directory named "lib"
-    When I run "rtlr --output my_grammar.rb binary.rtlr"
+    When I run `rtlr --output my_grammar.rb binary.rtlr`
     Then the output should contain "binary.rtlr -> my_grammar.rb"
       And the file "my_grammar.rb" should contain:
         """
@@ -26,7 +26,7 @@ Feature: --output option
       expr <- [01]*
       """
       And a directory named "lib"
-    When I run "rtlr --output - binary.rtlr"
+    When I run `rtlr --output - binary.rtlr`
     Then the output should contain:
       """
       module BinaryGrammar

data/features/command_line/parser_generator.feature

@@ -5,7 +5,7 @@ Feature: Parser Generator
   
   @command-line
   Scenario: Getting help
-    When I run "rtlr --help"
+    When I run `rtlr --help`
     Then the output should contain:
       """
       Usage: rtlr FILENAME [options]
@@ -27,7 +27,7 @@ Feature: Parser Generator
       grammar BinaryGrammar
       expr <- [01]*
       """
-    When I run "rtlr binary.rtlr --standalone"
+    When I run `rtlr binary.rtlr --standalone`
     Then the output should contain "binary.rtlr -> binary_grammar.rb"
       And the file "binary_grammar.rb" should contain:
         """
@@ -59,7 +59,7 @@ Feature: Parser Generator
       parser BinaryParser < Rattler::Runtime::PackratParser
       expr <- [01]*
       """
-    When I run "rtlr binary.rtlr --standalone"
+    When I run `rtlr binary.rtlr --standalone`
     Then the output should contain "binary.rtlr -> binary_parser.rb"
       And the file "binary_parser.rb" should contain:
         """

data/features/grammar/any_character.feature

@@ -2,10 +2,6 @@ Feature: "Any Character" Symbol
 
   The "." (period) means match any single character.
 
-  In order to match any single character
-  As a language designer
-  I want to use the "." symbol in my grammar
-  
   Scenario Outline: Parsing
     Given a grammar with:
       """

data/features/grammar/back_reference.feature

@@ -4,10 +4,6 @@ Feature: Back References
   The reference refers to the previous parse result and it means to match the
   same exact input again.
 
-  In order to reuse a previous parse result to match the same text again
-  As a language designer
-  I want to use back references in my grammar
-
   Scenario Outline: Parsing
     Given a grammar with:
       """

data/features/grammar/character_class.feature

@@ -4,10 +4,6 @@ Feature: Character Classes
   any single character in the set. They are identical to character classes in
   Ruby's regular expressions.
   
-  In order to match specific sets of character
-  As a language designer
-  I want to use character classes in my grammar
-  
   Scenario Outline: Parsing
     Given a grammar with:
       """

data/features/grammar/comments.feature

@@ -2,10 +2,6 @@ Feature: Whitespace
   
   Comments start with "#" and continue to the end of the line.
   
-  In order to clarify my grammars
-  As a language designer
-  I want to use comments
-  
   Scenario: Entire line as a comment
     Given a grammar with:
       """

data/features/grammar/e_symbol.feature

@@ -0,0 +1,18 @@
+Feature: EOF Symbol
+  
+  The symbol "E" always matches without consuming any input.
+  
+  Background: A grammar with the EOF symbol
+    Given a grammar with:
+      """
+      foo <- E
+      """
+
+  Scenario: Before end-of-input
+    When I parse "foo"
+    Then the parse result should be true
+  
+  Scenario: At end-of-input
+    When I parse "foo"
+      And the parse position is 3
+    Then the parse result should be true

data/features/grammar/eof.feature

@@ -3,10 +3,6 @@ Feature: EOF Symbol
   The symbol "EOF" means to match the end of the source, i.e. match only if
   there is no more input.
   
-  In order to match the end of the input
-  As a language designer
-  I want to use the EOF symbol in my grammar
-  
   Background: A grammar with the EOF symbol
     Given a grammar with:
       """

data/features/grammar/fail.feature

@@ -6,10 +6,6 @@ Feature: Fail Expressions
   as a string literal. Using "fail_rule" causes the entire rule to fail and
   "fail_parse" causes the entire parse to fail.
   
-  In order to define my own failure messages
-  As a language designer
-  I want to use fail expressions in my grammar
-  
   Scenario: Fail-expression
     Given a grammar with:
       """

data/features/grammar/list_matching.feature

@@ -4,9 +4,10 @@ Feature: List Matching
   to match a list of terms with separators between them. "*," matches a list of
   zero or more terms, "+," matches a list of one or more term.
 
-  In order to clearly and easily match list expressions
-  As a language designer
-  I want to use a list-matching expression
+  Repetition counts or ranges can also be used for generalized list matching
+  expressions. A count by itself means to match the preceding expression exactly
+  that many times. A range can be written with ".." between the lower and upper
+  bounds. The upper bound is optional.
 
   Scenario Outline: Zero or more terms
     Given a grammar with:
@@ -40,6 +41,54 @@ Feature: List Matching
     | "foo,bar,"    | ["foo", "bar"]        | 7   |
     | "   "         | FAIL                  | 0   |
 
+  Scenario Outline: Range
+    Given a grammar with:
+      """
+      words <-  @WORD+ 2..4, ','
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+      And the parse position should be <pos>
+  
+  Examples:
+    | input         | result                | pos |
+    | "foo,bar"     | ["foo", "bar"]        | 7   |
+    | "foo,bar,"    | ["foo", "bar"]        | 7   |
+    | "a,b,c,d,e"   | ["a", "b", "c", "d"]  | 7   |
+    | "foo"         | FAIL                  | 0   |
+
+  Scenario Outline: Lower bound only
+    Given a grammar with:
+      """
+      words <-  @WORD+ 2.., ','
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+      And the parse position should be <pos>
+  
+  Examples:
+    | input         | result                | pos |
+    | "foo,bar"     | ["foo", "bar"]        | 7   |
+    | "foo,bar,"    | ["foo", "bar"]        | 7   |
+    | "a,b,c,d,e"   | ["a","b","c","d","e"] | 9   |
+    | "foo"         | FAIL                  | 0   |
+
+  Scenario Outline: Specific count
+    Given a grammar with:
+      """
+      words <-  @WORD+ 2, ','
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+      And the parse position should be <pos>
+  
+  Examples:
+    | input         | result          | pos |
+    | "foo,bar"     | ["foo", "bar"]  | 7   |
+    | "foo,bar,"    | ["foo", "bar"]  | 7   |
+    | "a,b,c,d,e"   | ["a","b"]       | 3   |
+    | "foo"         | FAIL            | 0   |
+
   Scenario: Using whitespace
     Given a grammar with:
       """

data/features/grammar/literal.feature

@@ -3,10 +3,6 @@ Feature: Literal Expressions
   A string literal has the same syntax as a Ruby string literal and it means to
   match if that exact text is next in the input.
   
-  In order to define symbols and keywords
-  As a language designer
-  I want to use string literals in my grammar
-  
   Scenario Outline: Normal String
     Given a grammar with:
       """

data/features/grammar/negative_lookahead.feature

@@ -4,10 +4,6 @@ Feature: Negative Lookahead Operator
   would not match here and never consume any input. This is known as zero-width
   negative lookahead.
   
-  In order to define zero-width negative lookahead expressions
-  As a language designer
-  I want to use the negative lookahead operator in my grammar
-  
   Scenario Outline: Parsing
     Given a grammar with:
       """

data/features/grammar/negative_semantic_predicate.feature

@@ -0,0 +1,99 @@
+Feature: Negative Symantic Predicates
+
+  A negative symantic predicate can be defined by placing a "!" in front of a
+  symantic action. If the action results in a false value the parse succeeds
+  with no parse result, otherwise the parse fails.
+
+  Scenario Outline: Single token
+    Given a grammar with:
+      """
+      expr <- @DIGIT+ !{|s| s.to_i > 20 }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+  
+  Examples:
+    | input | result |
+    | "17"  | "17"   |
+    | "42"  | FAIL   |
+
+  Scenario Outline: Sequence
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ !{|a,b| a.to_i < b.to_i }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input  | result      |
+    | "16 3" | ["16", "3"] |
+    | "3 16" | FAIL        |
+
+  Scenario Outline: Sequence with non-capturing expressions
+    Given a grammar with:
+      """
+      expr <- ~"(" @DIGIT+ ~">" @DIGIT+ ~")" !{|a,b| a.to_i <= b.to_i }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input     | result       |
+    | "(23>17)" | ["23", "17"] |
+    | "(17>23)" | FAIL         |
+
+  Scenario Outline: Sequence with labeled expressions
+    Given a grammar with:
+      """
+      expr <- "(" left:@DIGIT+ "<" right:@DIGIT+ ")" !{ left.to_i >= right.to_i }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input     | result       |
+    | "(17<23)" | ["(", "17", "<", "23", ")"] |
+    | "(23<17)" | FAIL         |
+
+  Scenario Outline: Single token using "_"
+    Given a grammar with:
+      """
+      expr <- @DIGIT+ !{ _.to_i > 20 }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input | result |
+    | "17"  | "17"   |
+    | "42"  | FAIL   |
+  
+  Scenario Outline: Sequence using "_"
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ !{ _[0].to_i < _[1].to_i }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input  | result      |
+    | "16 3" | ["16", "3"] |
+    | "3 16" | FAIL        |
+
+  Scenario Outline: Sequence using "_" as a parameter name
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ !{|_| _.to_i < 20 }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input  | result      |
+    | "42 7" | ["42", "7"] |
+    | "3 16" | FAIL        |

data/features/grammar/node_action.feature

@@ -7,10 +7,6 @@ Feature: Node Actions
   Array and any attributes are included in a Hash. The action can optionally
   include a name.
   
-  In order to create node objects as parse results
-  As a language designer
-  I want to use node actions in my grammar
-  
   Scenario: Explicit node class
     Given a grammar with:
       """
@@ -42,3 +38,20 @@ Feature: Node Actions
         """
     When I parse "42+23"
     Then the parse result should be Expr[["42", "23"], {:name => "sum"}] 
+
+  Scenario: Just a node name
+    Given a grammar with:
+      """
+      integer <- @DIGIT+ <"num">
+      """
+    When I parse "42"
+    Then the parse result should be Rattler::Runtime::ParseNode["42", {:name => "num"}]
+
+  Scenario: Lone action
+    Given a grammar with:
+      """
+      default <- <>
+      """
+    When I parse "anything"
+    Then the parse result should be Rattler::Runtime::ParseNode[]
+      And the parse position should be 0

data/features/grammar/nonterminal.feature

@@ -1,12 +1,7 @@
 Feature: Nonterminals
   
   A nonterminal is an identifier that refers to a parse rule and means to parse
-  according to the rule. A nonterminal can recursively refer to the rule it is
-  used in.
-  
-  In order to define context-free languages with recursive definitions
-  As a language designer
-  I want to use nonterminal expressions in my grammar
+  according to the rule. Nonterminals can be used to define recursive rules.
   
   Scenario: Simple example
     Given a grammar with:
@@ -25,4 +20,5 @@ Feature: Nonterminals
               / "a"
       """
     When I parse "aaab"
-    Then the parse result should be [["a", ["a", "a"]], "b"]
+    Then the parse result should be [["a", ["a", "a"]], "b"]
+