rattler 0.4.2 → 0.5.0

data/features/grammar/one_or_more.feature

@@ -3,10 +3,6 @@ Feature: One-Or-More
   The "+" operator following an expression means to match one or more times,
   i.e. match repeatedly and succeed if input was matched at least once.
   
-  In order to define an expression that matches one or more times
-  As a language designer
-  I want to use a "one-or-more" postfix operator in my grammar
-  
   Scenario Outline: Capturing Expression
     Given a grammar with:
       """

data/features/grammar/optional.feature

@@ -3,10 +3,6 @@ Feature: Optional Operator
   The "?" operator following an expression makes the it optional, meaning it
   succeeds even if no input is matched.
   
-  In order to define an expression that matches optionally
-  As a language designer
-  I want to use an "optional" postfix operator in my grammar
-  
   Scenario Outline: Capturing Expression
     Given a grammar with:
       """

data/features/grammar/ordered_choice.feature

@@ -4,10 +4,6 @@ Feature: Ordered Choice Expressions
   and it means to try each sub-expression in order until one matches, and fail
   if none of the sub-expressions match.
   
-  In order to define multiple alternatives for an expression
-  As a language designer
-  I want to use ordered choice expressions in my grammar
-
   Scenario Outline: Parsing
     Given a grammar with:
       """

data/features/grammar/positive_lookahead.feature

@@ -4,10 +4,6 @@ Feature: Positive Lookahead Operator
   would match here but never consume any input. This is known as zero-width
   positive lookahead.
   
-  In order to define zero-width positive lookahead expressions
-  As a language designer
-  I want to use the positive lookahead operator in my grammar
-  
   Scenario Outline: Parsing
     Given a grammar with:
       """

data/features/grammar/positive_semantic_predicate.feature

@@ -0,0 +1,99 @@
+Feature: Positive Semantic Predicates
+
+  A positive semantic predicate can be defined by placing a "&" in front of a
+  semantic action. If the action results in a true value the parse succeeds,
+  otherwise the parse fails.
+
+  Scenario Outline: Single token
+    Given a grammar with:
+      """
+      expr <- @DIGIT+ &{|s| s.to_i > 20 }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+  
+  Examples:
+    | input | result |
+    | "42"  | "42"   |
+    | "17"  | FAIL   |
+
+  Scenario Outline: Sequence
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ &{|a,b| a.to_i < b.to_i }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input  | result      |
+    | "3 16" | ["3", "16"] |
+    | "16 3" | FAIL        |
+
+  Scenario Outline: Sequence with non-capturing expressions
+    Given a grammar with:
+      """
+      expr <- ~"(" @DIGIT+ ~">" @DIGIT+ ~")" &{|a,b| a.to_i > b.to_i }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input     | result       |
+    | "(23>17)" | ["23", "17"] |
+    | "(17>23)" | FAIL         |
+
+  Scenario Outline: Sequence with labeled expressions
+    Given a grammar with:
+      """
+      expr <- "(" left:@DIGIT+ "<" right:@DIGIT+ ")" &{ left.to_i < right.to_i }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input     | result       |
+    | "(17<23)" | ["(", "17", "<", "23", ")"] |
+    | "(23<17)" | FAIL         |
+
+  Scenario Outline: Single token using "_"
+    Given a grammar with:
+      """
+      expr <- @DIGIT+ &{ _.to_i > 20 }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input | result |
+    | "42"  | "42"   |
+    | "17"  | FAIL   |
+  
+  Scenario Outline: Sequence using "_"
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ &{ _[0].to_i < _[1].to_i }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input  | result      |
+    | "3 16" | ["3", "16"] |
+    | "16 3" | FAIL        |
+
+  Scenario Outline: Sequence using "_" as a parameter name
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ &{|_| _.to_i < 20 }
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+
+  Examples:
+    | input  | result      |
+    | "3 16" | ["3", "16"] |
+    | "42 7" | FAIL        |

data/features/grammar/posix_class.feature

@@ -18,11 +18,7 @@ Feature: POSIX Character Classes
     SPACE - Whitespace characters
     UPPER - Uppercase characters
     XDIGIT - Hexadecimal digits
-    WORD - Alphanumeric characters plus "_" (Ruby 1.9 only)
-  
-  In order to write more consistent and readable grammars
-  As a language designer
-  I want to use POSIX character classes in my grammar
+    WORD - Alphanumeric characters plus "_"
   
   Scenario: ALNUM
     Given a grammar with:
@@ -70,4 +66,5 @@ Feature: POSIX Character Classes
       expr <- UPPER+
       """
     When I parse "ABCdef"
-    Then the parse result should be ["A", "B", "C"]
+    Then the parse result should be ["A", "B", "C"]
+    

data/features/grammar/repeat.feature

@@ -0,0 +1,50 @@
+Feature: Generalized Repeat
+
+  Repetition counts or ranges can be used for generalized repeat expressions.
+  A count by itself means to match the preceding expression exactly that many
+  times. A range can be written with ".." between the lower and upper bounds.
+  The upper bound is optional.
+
+  Scenario Outline: Range
+    Given a grammar with:
+      """
+      expr <- ALPHA 2..4
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+      And the parse position should be <pos>
+  
+  Examples:
+    | input   | result               | pos |
+    | "abc"   | ["a", "b", "c"]      | 3   |
+    | "abcde" | ["a", "b", "c", "d"] | 4   |
+    | "a"     | FAIL                 | 0   |
+  
+  Scenario Outline: Lower bound only
+    Given a grammar with:
+      """
+      expr <- ALPHA 2..
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+      And the parse position should be <pos>
+  
+  Examples:
+    | input   | result                    | pos |
+    | "abc"   | ["a", "b", "c"]           | 3   |
+    | "abcde" | ["a", "b", "c", "d", "e"] | 5   |
+    | "a"     | FAIL                      | 0   |
+
+  Scenario Outline: Specific count
+    Given a grammar with:
+      """
+      expr <- ALPHA 2
+      """
+    When I parse <input>
+    Then the parse result should be <result>
+      And the parse position should be <pos>
+  
+  Examples:
+    | input   | result     | pos |
+    | "abc"   | ["a", "b"] | 2   |
+    | "a"     | FAIL       | 0   |

data/features/grammar/sequence.feature

@@ -3,10 +3,6 @@ Feature: Sequence Expressions
   A sequence expression is a series of sub-expressions and it means to match
   all of the sub-expressions in sequence, otherwise fail and consume no input.
   
-  In order to define an expression as a sequence of sub-expressions
-  As a language designer
-  I want to use sequence expressions in my grammar
-  
   Scenario Outline: Parsing
     Given a grammar with:
       """

data/features/grammar/side_effect.feature

@@ -0,0 +1,81 @@
+Feature: Semantic side-effects
+
+  A semantic side-effect can be defined by placing a "~" in front of a symantic
+  action. The expression is evaulated for effect and always succeeds with no
+  parse result.
+
+  Scenario: Single token
+    Given a grammar with:
+      """
+      expr <- @DIGIT+ ~{|s| $x = s.to_i }
+      """
+    When I parse "42 "
+    Then the parse result should be "42"
+      And $x should be 42
+
+  Scenario: Sequence
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ ~{|a,b| $x = a.to_i * b.to_i }
+      """
+    When I parse "3 16"
+    Then the parse result should be ["3", "16"]
+      And $x should be 48
+
+  Scenario: Sequence with non-capturing expressions
+    Given a grammar with:
+      """
+      expr <- ~"(" @DIGIT+ ~"+" @DIGIT+ ~")" ~{|a,b| $x = a.to_i + b.to_i }
+      """
+    When I parse "(23+17)"
+    Then the parse result should be ["23", "17"]
+      And $x should be 40
+
+  Scenario: Sequence with labeled expressions
+    Given a grammar with:
+      """
+      expr <- "(" l:@DIGIT+ "+" r:@DIGIT+ ")" ~{ $x = l.to_i + r.to_i }
+      """
+    When I parse "(17+29)"
+    Then the parse result should be ["(", "17", "+", "29", ")"]
+      And $x should be 46
+  
+  Scenario: Single token using "_"
+    Given a grammar with:
+      """
+      expr <- @DIGIT+ ~{ $x = _.to_i }
+      """
+    When I parse "23"
+    Then the parse result should be "23"
+      And $x should be 23
+  
+  Scenario: Sequence using "_"
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ ~{ $x = _.reverse }
+      """
+    When I parse "3 16"
+    Then the parse result should be ["3", "16"]
+      And $x should be ["16", "3"]
+  
+  Scenario: Sequence using "_" as a parameter name
+    Given a grammar with:
+      """
+      %whitespace SPACE*
+      expr <- @DIGIT+ @DIGIT+ ~{|_| $x = _.to_i }
+      """
+    When I parse "3 16"
+    Then the parse result should be ["3", "16"]
+      And $x should be 3
+
+  Scenario: Lone action
+    Given a grammar with:
+      """
+      expr <- ~{ $x = 42 }
+      """
+    When I parse "anything"
+    Then the parse result should be true
+      And $x should be 42
+      And the parse position should be 0

data/features/grammar/skip_operator.feature

@@ -2,10 +2,6 @@ Feature: Skip Operator
   
   The "~" operator before an expression means to match but ignore the result.
   
-  In order to match syntax without it being included in the parse results
-  As a language designer
-  I want to use a "skip" prefix operator in my grammar
-  
   Scenario: Sequence with skipped sub-expressions
     Given a grammar with:
       """
@@ -21,4 +17,5 @@ Feature: Skip Operator
       """
     When I parse "if "
     Then the parse result should be true
-      And the parse position should be 2
+      And the parse position should be 2
+      

data/features/grammar/start_rule.feature

@@ -2,10 +2,6 @@ Feature: Start Rule
   
   The first rule defined in a grammar is the start rule.
   
-  In order to make my grammars readable
-  As a language designer
-  I want to give my start rule a meaningful name
-  
   Scenario Outline: Default Start Rule
     Given a grammar with:
       """
@@ -18,4 +14,4 @@ Feature: Start Rule
   Examples:
     | input | result |
     | "42a" | "42"   |
-    | "foo" | FAIL   |
+    | "foo" | FAIL   |

data/features/grammar/symantic_action.feature

@@ -8,10 +8,6 @@ Feature: Symantic Actions
   are simply ignored. Labeled parse results can be refered to as identifiers in
   the action. The special identifier "_" refers to the entire parse results.
   
-  In order to add simple symantics to parse results
-  As a language designer
-  I want to use symantic actions in my grammar
-  
   Scenario: Single token
     Given a grammar with:
       """
@@ -70,3 +66,12 @@ Feature: Symantic Actions
       """
     When I parse "3 16"
     Then the parse result should be 3
+
+  Scenario: Lone action
+    Given a grammar with:
+      """
+      default <- { 42 }
+      """
+    When I parse "anything"
+    Then the parse result should be 42
+      And the parse position should be 0

data/features/grammar/token.feature

@@ -5,10 +5,6 @@ Feature: Token Operator
   automatically skipped. The expression becomes atomic, so any defined
   whitespace will be automatically skipped before the token.
   
-  In order to match arbitrarily complex expressions as tokens
-  As a language designer
-  I want to use a "token" prefix operator in my grammar
-  
   Scenario: Nested sequences and repeats
     Given a grammar with:
       """

data/features/grammar/whitespace.feature

@@ -6,10 +6,6 @@ Feature: Whitespace
   form defines whitespace for expressions in the block, otherwise it is defined
   for the rest of the grammar.
   
-  In order to focus on the structure of my language
-  As a language designer
-  I want to specify whitespace in my grammar once and only once
-  
   Scenario: Block form
     Given a grammar with:
       """

data/features/grammar/word_literal.feature

@@ -6,10 +6,6 @@ Feature: Word Literal Expressions
   word character can be defined using the %word_character directive. By default
   a word charactacter is an alphanumeric character or the underscore character.
   
-  In order to define keywords
-  As a language designer
-  I want to use word literals in my grammar
-  
   Scenario Outline: Default word character definition
     Given a grammar with:
       """

data/features/grammar/zero_or_more.feature

@@ -3,10 +3,6 @@ Feature: Zero-Or-More
   The "*" operator following an expression means to match zero or more times,
   i.e. match repeatedly and succeed even if no input was matched.
   
-  In order to define an expression that matches zero or more times
-  As a language designer
-  I want to use a "zero-or-more" postfix operator in my grammar
-  
   Scenario Outline: Capturing Expression
     Given a grammar with:
       """

data/features/step_definitions/grammar_steps.rb

@@ -36,4 +36,8 @@ end
 
 Then /failure position should be (.+)$/ do |expr|
   @parser.failure.pos.should == eval(expr, TOPLEVEL_BINDING)
-end
+end
+
+Then /^\$x should be (.+)$/ do |expr|
+  $x.should == eval(expr, TOPLEVEL_BINDING)
+end

data/lib/rattler/back_end/optimizer.rb

@@ -76,6 +76,7 @@ module Rattler::BackEnd
     autoload :OptimizationSequence, 'rattler/back_end/optimizer/optimization_sequence'
     autoload :OptimizeChildren, 'rattler/back_end/optimizer/optimize_children'
     autoload :InlineRegularRules, 'rattler/back_end/optimizer/inline_regular_rules'
+    autoload :SpecializeRepeat, 'rattler/back_end/optimizer/specialize_repeat'
     autoload :SimplifyRedundantRepeat, 'rattler/back_end/optimizer/simplify_redundant_repeat'
     autoload :RemoveMeaninglessWrapper, 'rattler/back_end/optimizer/remove_meaningless_wrapper'
     autoload :SimplifyTokenMatch, 'rattler/back_end/optimizer/simplify_token_match'

data/lib/rattler/back_end/optimizer/reduce_repeat_match.rb

@@ -20,10 +20,8 @@ module Rattler::BackEnd::Optimizer
 
     def _applies_to?(parser, context)
       context.matching? and
-      case parser
-      when ZeroOrMore, OneOrMore, Optional
-        parser.child.is_a?(Match)
-      end
+      parser.is_a? Repeat and
+      parser.child.is_a? Match
     end
 
     def _apply(parser, context)
@@ -33,10 +31,24 @@ module Rattler::BackEnd::Optimizer
     private
 
     def suffix(parser)
-      case parser
-      when ZeroOrMore then '*'
-      when OneOrMore then '+'
-      when Optional then '?'
+      if parser.zero_or_more?
+        '*'
+      elsif parser.one_or_more?
+        '+'
+      elsif parser.optional?
+        '?'
+      else
+        general_suffix parser.lower_bound, parser.upper_bound
+      end
+    end
+
+    def general_suffix(lower_bound, upper_bound)
+      if lower_bound == upper_bound
+        "{#{lower_bound}}"
+      elsif !upper_bound
+        "{#{lower_bound},}"
+      else
+        "{#{lower_bound},#{upper_bound}}"
       end
     end