rattler 0.5.0 → 0.6.0

data/features/error_reporting/automatic_error_messages.feature

@@ -0,0 +1,40 @@
+Feature: Automatic Error Messages
+
+  Rattler parsers automatically generate parse error messages for failed rules
+  using the name of the failed rule.
+
+  Rattler usually uses the name of the latest and deepest failed rule to ensure
+  the message is as specific as possible. The exception is when a rule is an
+  ordered choice. In that case using the name of the ordered choice rule makes
+  more sense.
+
+  Scenario: Automatic Error Message
+    Given a grammar with:
+      """
+      expr <- DIGIT
+      """
+    When I parse "a"
+    Then the parse should fail
+      And the failure message should be "expr expected"
+
+  Scenario: Deepest Rule Name
+    Given a grammar with:
+      """
+      expr    <-  digits
+      digits  <-  DIGIT+
+      """
+    When I parse "a"
+    Then the parse should fail
+      And the failure message should be "digits expected"
+
+  Scenario: Ordered Choice
+    Given a grammar with:
+      """
+      alnums  <-  alphas
+                / digits
+      alphas  <-  ALPHA+
+      digits  <-  DIGIT+
+      """
+    When I parse "@"
+    Then the parse should fail
+      And the failure message should be "alnums expected"

data/features/error_reporting/custom_error_messages.feature

@@ -0,0 +1,28 @@
+Feature: Custom Error Messages
+
+  A fail expression always fails with a specified message.
+
+  The keyword "expected" can be used as a shortcut for error messages like
+  "number expected."
+
+  Scenario: Custom Error Message
+    Given a grammar with:
+      """
+      expr  <-  "a"
+              / "b"
+              / fail "'a' or 'b' expected"
+      """
+    When I parse ""
+    Then the parse should fail
+      And the failure message should be "'a' or 'b' expected"
+
+  Scenario: "expected" keyword
+    Given a grammar with:
+      """
+      expr  <-  "a"
+              / "b"
+              / expected "'a' or 'b'"
+      """
+    When I parse ""
+    Then the parse should fail
+      And the failure message should be "'a' or 'b' expected"

data/features/examples/json_parser.markdown

@@ -0,0 +1,88 @@
+The following is an example of a complete JSON parser. It parses JSON syntax
+into Ruby hashes, arrays, and values.
+
+`json.rtlr`:
+
+    # JSON parser based on the grammar at http://www.json.org
+
+    require 'rattler'
+    require_relative 'json_helper'
+
+    parser JsonParser < Rattler::Runtime::PackratParser
+
+    include JsonHelper
+
+    %whitespace (SPACE+ / comment)*
+
+    json_text <-  (object / array) EOF
+
+    object    <-  ~'{' members ~'}'                         { object _ }
+
+    members   <-  pair *, ','
+
+    pair      <-  string ~':' value
+
+    array     <-  ~'[' elements ~']'                        { array _ }
+
+    elements  <-  value *, ','
+
+    value     <-  object
+                / array
+                / number
+                / string
+                / `true`                                    { :true }
+                / `false`                                   { :false }
+                / `null`                                    { :null }
+
+    string    <-  @('"' char* '"')                          { string _ }
+
+    number    <-  @(int frac? exp?)                         { number _ }
+
+    %fragments
+
+    char      <-  !('"' / '\\' / CNTRL) .
+                / '\\' (["\\/bfnrt] / 'u' XDIGIT XDIGIT XDIGIT XDIGIT)
+
+    int       <-  '-'? ('0' !DIGIT / [1-9] DIGIT*)
+
+    frac      <-  '.' DIGIT+
+
+    exp       <-  [eE] [+-]? DIGIT+
+
+    comment   <-  '/*' (! '*/' .)* '*/'
+                / '//' [^\n]*
+
+Because `true`, `false`, and `nil` all have special meaning to Rattler's
+parsers, we encode these as symbols, then decode the symbols when we match
+objects and arrays.
+
+`json_helper`:
+
+    module JsonHelper
+
+      def object(members)
+        Hash[members.map {|k, v| [k, decode(v)] }]
+      end
+
+      def array(a)
+        a.map {|_| decode _ }
+      end
+
+      def string(expr)
+        eval "%q#{expr}", TOPLEVEL_BINDING
+      end
+
+      def number(expr)
+        eval expr, TOPLEVEL_BINDING
+      end
+
+      def decode(v)
+        case v
+        when :true then true
+        when :false then false
+        when :null then nil
+        else v
+        end
+      end
+
+    end

data/features/{grammar → extended_matching_syntax}/back_reference.feature

@@ -1,8 +1,10 @@
 Feature: Back References
 
-  A back reference is a "$" followed by a label defined earlier in a sequence.
-  The reference refers to the previous parse result and it means to match the
-  same exact input again.
+  A back reference refers to a previous labelled parsing expression and matches
+  the exact string matched by the referenced expression.
+
+  A back reference is written as a "$" followed by a label defined earlier in a
+  sequence.
 
   Scenario Outline: Parsing
     Given a grammar with:

data/features/{grammar → extended_matching_syntax}/e_symbol.feature

@@ -1,8 +1,8 @@
-Feature: EOF Symbol
+Feature: E Symbol
   
   The symbol "E" always matches without consuming any input.
   
-  Background: A grammar with the EOF symbol
+  Background: A grammar with the E symbol
     Given a grammar with:
       """
       foo <- E

data/features/{grammar → extended_matching_syntax}/eof.feature

@@ -1,7 +1,8 @@
 Feature: EOF Symbol
   
-  The symbol "EOF" means to match the end of the source, i.e. match only if
-  there is no more input.
+  The symbol "EOF" matches the end of the source, i.e. it matches only if there
+  is no more input.  This can be used to ensure that a parse fails unless the
+  entire input is matched.
   
   Background: A grammar with the EOF symbol
     Given a grammar with:
@@ -17,4 +18,4 @@ Feature: EOF Symbol
   Scenario: Before end-of-input
     When I parse "foo"
       And the parse position is 2
-    Then the parse should fail
+    Then the parse should fail

data/features/{grammar → extended_matching_syntax}/fail.feature

@@ -1,10 +1,12 @@
 Feature: Fail Expressions
-  
-  A "fail" expression means alway fail here with a specified message. It can
-  used to provide more meaningful messages when a parse fails. The syntax is
-  either "fail", "fail_rule" or "fail_parse" followed by the message specified
-  as a string literal. Using "fail_rule" causes the entire rule to fail and
-  "fail_parse" causes the entire parse to fail.
+
+  A fail expression always fails with a specified message.  It can used to
+  provide more meaningful messages when a parse fails.
+
+  A fail expression is written as either "fail", "fail_rule" or "fail_parse"
+  followed by the message specified as a string literal.  Using "fail_rule"
+  causes the entire rule to fail and "fail_parse" causes the entire parse to
+  fail.
   
   Scenario: Fail-expression
     Given a grammar with:

data/features/extended_matching_syntax/fragments.feature

@@ -0,0 +1,29 @@
+Feature: Fragments
+  
+  The "%fragments" directive disables whitepsace skipping and marks parse rules
+  to be inlined. This is useful for factoring complex token rules.
+
+  Scenario: Matching a number
+    Given a grammar with:
+      """
+      %whitespace SPACE+
+
+      number  <-  @(int frac?)
+
+      %fragments
+
+      int     <-  '-'? DIGIT+
+
+      frac    <-  '.' DIGIT+
+      """
+    When I generate parser code
+    Then the code should contain:
+      """
+      def match_number! #:nodoc:
+        begin
+          @scanner.skip(/(?>(?>[[:space:]])+)((?>(?>(?>\-)?)(?>(?>[[:digit:]])+))(?>(?>(?>\.)(?>(?>[[:digit:]])+))?))/) &&
+          @scanner[1]
+        end ||
+        fail { :number }
+      end
+      """

data/features/extended_matching_syntax/include.feature

@@ -0,0 +1,42 @@
+Feature: include
+
+  A grammar can include another grammar and use or override its rules.
+
+  Scenario Outline: Included grammar
+    Given a grammar called "ExprGrammar" with:
+      """
+      a <- "A"
+      """
+    And a grammar with:
+      """
+      include ExprGrammar
+      
+      expr <- a
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input | result |
+    | "A"   | "A"    |
+    | "B"   | FAIL   |
+  
+  Scenario Outline: Overriding a rule
+    Given a grammar called "ExprGrammar" with:
+      """
+      expr  <- a
+      a     <- "A"
+      """
+    And a grammar with:
+      """
+      include ExprGrammar
+      
+      a <- "B"
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input | result |
+    | "B"   | "B"    |
+    | "A"   | FAIL   |

data/features/{grammar → extended_matching_syntax}/list_matching.feature

@@ -1,13 +1,14 @@
 Feature: List Matching
 
-  A term expression followed by "*," or "+," and a separator expression means
-  to match a list of terms with separators between them. "*," matches a list of
-  zero or more terms, "+," matches a list of one or more term.
+  A list matching expression defines a term expression and a separator and
+  matches a list of terms.
 
+  A list matching expression is written by following a parsing expression for
+  the term with "*," or "+," followed by a parsing expression for the separator.
   Repetition counts or ranges can also be used for generalized list matching
-  expressions. A count by itself means to match the preceding expression exactly
-  that many times. A range can be written with ".." between the lower and upper
-  bounds. The upper bound is optional.
+  expressions.  With a count by itself the expression matches a exactly that
+  many terms.  A range is written as a lower bound followed by "..", optionally
+  followed by an upper bound.
 
   Scenario Outline: Zero or more terms
     Given a grammar with:

data/features/extended_matching_syntax/posix_class.feature

@@ -0,0 +1,127 @@
+Feature: POSIX Character Classes
+  
+  The uppercase names of POSIX character classes can be used as a shortcut to
+  using them in character class expressions, e.g. "DIGIT" and "[[:digit:]]" are
+  equivalent.  In addition, "WORD" is a shortcut for "[[:alnum:]_]", equivalent
+  to "[[:word:]]" in Ruby 1.9.
+  
+  The POSIX character classes are:
+
+    ALNUM - Alphanumeric characters  
+    ALPHA - Alphabetic characters  
+    ASCII - ASCII characters  
+    BLANK - Space and tab  
+    CNTRL - Control characters  
+    DIGIT - Digits  
+    GRAPH - Visible characters  
+    LOWER - Lowercase characters  
+    PRINT - Visible characters and spaces  
+    PUNCT - Punctuation characters  
+    SPACE - Whitespace characters  
+    UPPER - Uppercase characters  
+    XDIGIT - Hexadecimal digits  
+    WORD - Alphanumeric characters plus "_"  
+  
+  Scenario: ALNUM
+    Given a grammar with:
+      """
+      expr <- ALNUM+
+      """
+    When I parse "abc123"
+    Then the parse result should be ["a", "b", "c", "1", "2", "3"]
+  
+  Scenario: ALPHA
+    Given a grammar with:
+      """
+      expr <- ALPHA+
+      """
+    When I parse "abc123"
+    Then the parse result should be ["a", "b", "c"]
+  
+  Scenario: BLANK
+    Given a grammar with:
+      """
+      expr <- BLANK+
+      """
+    When I parse " \t\nabc"
+    Then the parse result should be [" ", "\t"]
+  
+  Scenario: CNTRL
+    Given a grammar with:
+      """
+      expr <- CNTRL+
+      """
+    When I parse "\t\r\n abc"
+    Then the parse result should be ["\t", "\r", "\n"]
+  
+  Scenario: DIGIT
+    Given a grammar with:
+      """
+      expr <- DIGIT+
+      """
+    When I parse "123abc"
+    Then the parse result should be ["1", "2", "3"]
+  
+  Scenario: GRAPH
+    Given a grammar with:
+      """
+      expr <- GRAPH+
+      """
+    When I parse "a1;  "
+    Then the parse result should be ["a", "1", ";"]
+  
+  Scenario: LOWER
+    Given a grammar with:
+      """
+      expr <- LOWER+
+      """
+    When I parse "abcDEF"
+    Then the parse result should be ["a", "b", "c"]
+  
+  Scenario: PRINT
+    Given a grammar with:
+      """
+      expr <- PRINT+
+      """
+    When I parse " abc\tdef"
+    Then the parse result should be [" ", "a", "b", "c"]
+  
+  Scenario: PUNCT
+    Given a grammar with:
+      """
+      expr <- PUNCT+
+      """
+    When I parse ",.!;abc"
+    Then the parse result should be [",", ".", "!", ";"]
+  
+  Scenario: SPACE
+    Given a grammar with:
+      """
+      expr <- SPACE+
+      """
+    When I parse " \t\r\nabc"
+    Then the parse result should be [" ", "\t", "\r", "\n"]
+  
+  Scenario: UPPER
+    Given a grammar with:
+      """
+      expr <- UPPER+
+      """
+    When I parse "ABCdef"
+    Then the parse result should be ["A", "B", "C"]
+  
+  Scenario: XDIGIT
+    Given a grammar with:
+      """
+      expr <- XDIGIT+
+      """
+    When I parse "1aFg"
+    Then the parse result should be ["1", "a", "F"]
+  
+  Scenario: WORD
+    Given a grammar with:
+      """
+      expr <- WORD+
+      """
+    When I parse "Ab_1-d"
+    Then the parse result should be ["A", "b", "_", "1"]

data/features/{grammar → extended_matching_syntax}/repeat.feature

@@ -1,9 +1,9 @@
 Feature: Generalized Repeat
 
   Repetition counts or ranges can be used for generalized repeat expressions.
-  A count by itself means to match the preceding expression exactly that many
-  times. A range can be written with ".." between the lower and upper bounds.
-  The upper bound is optional.
+  With a count by itself the expression matches a exactly that many times.
+  A range is written as a lower bound followed by "..", optionally followed by
+  an upper bound.
 
   Scenario Outline: Range
     Given a grammar with:
@@ -48,3 +48,29 @@ Feature: Generalized Repeat
     | input   | result     | pos |
     | "abc"   | ["a", "b"] | 2   |
     | "a"     | FAIL       | 0   |
+
+  Scenario Outline: Choice of capturing or non-capturing
+    Given a grammar with:
+      """
+      expr  <- ("a" / ~"b") 2..
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+      And the parse position should be <pos>
+  
+  Examples:
+    | input | result     | pos |
+    | "aa"  | ["a", "a"] | 2   |
+    | "aba" | ["a", "a"] | 3   |
+    | "abc" | ["a"]      | 2   |
+    | "bbc" | []         | 2   |
+    | "acb" | FAIL       | 0   |
+
+  Scenario: Semantic attribute returning true
+    Given a grammar with:
+      """
+      expr  <- (ALPHA { true }) 2..
+      """
+    When I parse "foo "
+    Then the parse result should be []
+      And the parse position should be 3