packrat_parser 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3329c290118a8d488a68b9d83f645eee1095730ee11d51446bd1e76ff232b95e
-  data.tar.gz: d39e34dd833b9bb48079b6e342a5b36162396a257b239697bef76d5f796b0679
+  metadata.gz: 5600ad17276add802741ba1c0bda893d7757dcdab4a22dc8ae10e2b18a928fd9
+  data.tar.gz: ac53b87a45954f9d319f0b3298222217c2636fb478281c7596c4d440d81742b2
 SHA512:
-  metadata.gz: 9d51853503af9f290bbb062f4a0b3e8543ddb032aeb69d3f3d1ec555c02101975780e0cad5e4ab364a267cfd6fd361259b0cd1ca4136dbae00ce81dea954a753
-  data.tar.gz: afd0c1223b0bc3d392b947a319cedc9696280c6cf6d5e10cfbc638aaa1fdf7669a5c88c532542bc9744b645312cab8b88fb559d3fa90be4f1c77d50145f23abd
+  metadata.gz: 8ca4a20c8f1f3e4c3063db70c39441b88401bd144c1ac1cc50b29a03ec58b845029d93c3a5879176b454e7b79b0fd47b35193943a327b928a50552c055d46d29
+  data.tar.gz: 882cb9d1845f4c06f360f2fd933ff6f84d80c03e749dff3ba5aa6e687fed0b98cdd75a247043913ffbb5f8da89349a4fa6b4dfe0c6d3f5fae4b84a2b669f357c

data/README.md

@@ -49,9 +49,14 @@ The default Prism parser rejects `for ... then`. The library itself
 
 ## API
 
-Subclass `PackratParser`. **Every method you define in the subclass is a grammar
-rule** and must return a parser; rule methods are automatically made lazy and
-memoized so they can reference one another (and themselves) recursively.
+Subclass `PackratParser`. **Every public method you define in the subclass is a
+grammar rule** and must return a parser; rule methods are automatically made lazy
+and memoized so they can reference one another (and themselves) recursively.
+
+**Private methods are left as ordinary helpers** (not rules), so you can factor
+out plain Ruby logic. Use the `private` section form (`private` on its own line,
+then the defs); `private def foo` is not detected, because the method is still
+public when it is defined.
 
 ### Building blocks (available inside rule methods)
 
@@ -66,18 +71,26 @@ memoized so they can reference one another (and themselves) recursively.
 - `map { |v| new_value }` — transform the result.
 - `filter { |v| bool }` — succeed only when the predicate holds.
 - `a | b` — ordered choice: try `a`, and if it fails try `b`.
-- `a + b` — sequence, keep **both** results (Scala's `~`): run `a` then `b`,
-  yield the pair `[a, b]`. Left-associative and nesting, so `a + b + c` yields
-  `[[a, b], c]`; Ruby's block-parameter destructuring takes them apart the way
-  Scala's `case a ~ b ~ c` does: `(a + b + c).map { |(x, y), z| ... }`.
+- `a * b` — sequence, keep **both** results (Scala's `~`): run `a` then `b`,
+  yield the pair `[a, b]`. `*` reads as a product: the result is the product of
+  the two values, just as `|` (choice) is the algebraic sum. Left-associative and
+  nesting, so `a * b * c` yields `[[a, b], c]`; Ruby's block-parameter
+  destructuring takes them apart the way Scala's `case a ~ b ~ c` does:
+  `(a * b * c).map { |(x, y), z| ... }`.
 - `a << b` — sequence, keep the **left** result (Scala's `<~`): run `a` then
   `b`, yield `a`'s value and discard `b`'s.
 - `a >> b` — sequence, keep the **right** result (Scala's `~>`): run `a` then
   `b`, yield `b`'s value and discard `a`'s.
+- `p.rep` — zero or more (Scala's `rep`): yields an array of results (empty when
+  there are no matches). Always succeeds.
+- `p.rep1` — one or more (Scala's `rep1`): like `rep` but fails unless `p`
+  matches at least once; yields a non-empty array.
+- `p.opt` — optional (Scala's `opt`): yields `p`'s value, or `nil` (consuming
+  nothing) when `p` does not match.
 
 The arrow direction is a useful mnemonic: `<<`/`>>` keeps whichever side it
 points to. They are handy for discarding punctuation, e.g. `( expr )` is
-`term("(") >> expr << term(")")`. Ruby's precedence (`+` over `<<`/`>>` over
+`term("(") >> expr << term(")")`. Ruby's precedence (`*` over `<<`/`>>` over
 `|`) means sequencing binds tighter than ordered choice, as you'd want.
 
 `flat_map`, `map`, and `filter` are exactly what the `for ... then`
@@ -114,6 +127,10 @@ The setting is inherited by subclasses, so a base parser can enable it once.
   If omitted, the first defined method is used as the start symbol.
 - `Klass.parse(input)` / `Klass.new.parse(input)` — parse, returning the value.
   Raises `PackratParser::ParseError` on failure or leftover input.
+- Parse from any rule, not just the start symbol:
+  `Klass.new.number.parse("123")` (call `parse` on the rule) or
+  `Klass.new.parse("123", :number)` (pass the start rule). Both apply the same
+  full-consumption check and whitespace handling.
 
 ## Notes / limitations
 
@@ -123,11 +140,29 @@ The setting is inherited by subclasses, so a base parser can enable it once.
 - **No implicit whitespace by default.** `term` matches exactly. Enable implicit
   whitespace skipping with `skip_whitespace` (see above) when your grammar needs
   it.
+- **Byte-oriented positions.** The parser tracks byte offsets and matches with
+  `byteslice` / `byteindex`, so it stays efficient on UTF-8 input (indexing a
+  multibyte string by character is O(n)). Positions are byte offsets throughout,
+  including the `pos` reported on a raised `ParseError`.
 
 ## Running the tests
 
 ```sh
+rake            # or: rake test
+# or directly:
 bin/test
 # or:
-ruby --parser=parse.y -Ilib -Itest test/test_packrat_parser.rb
+ruby --parser=parse.y -Ilib test/test_packrat_parser.rb
 ```
+
+The `rake test` task runs the suite under `--parser=parse.y` for you (the test
+grammars need the fork's legacy parser).
+
+## Benchmark
+
+`benchmark/` compares this library against [racc](https://github.com/ruby/racc)
+(LALR) and [parslet](https://kschiess.github.io/parslet/) (PEG combinators) on
+the same calculator grammar. Run it with `rake bench` (needs `parslet`). Roughly,
+all three are linear on this grammar, with **racc ≫ packrat_parser > parslet** on
+constant factors (packrat_parser ~3× racc, parslet ~8× packrat_parser). See
+[benchmark/README.md](benchmark/README.md) for numbers and analysis.

data/lib/packrat_parser/base.rb

@@ -53,10 +53,17 @@ class PackratParser
   # Rule. Guards against rewriting the base class's own infrastructure and
   # against re-entering while we install the replacement (define_method itself
   # fires method_added).
+  #
+  # Private methods are left alone, so they can be used as ordinary helpers
+  # rather than grammar rules. Note this only sees the visibility in effect when
+  # the method is defined, so it recognises the `private` section form but not
+  # `private def foo ... end` (which is still public at this point and only made
+  # private afterwards).
   def self.method_added(name)
     return if self == PackratParser
     return if name == :initialize
     return if @__defining_rule
+    return if private_method_defined?(name)
 
     @__defining_rule = true
     @start_symbol ||= name
@@ -70,15 +77,27 @@ class PackratParser
     end
   end
 
-  # Per-input packrat memo table, keyed by [rule_name, pos].
+  # Per-input packrat memo table: a two-level hash, rule_name => (pos => result).
   def __memo
     @__memo ||= {}
   end
 
+  # Cache of built combinators, keyed by rule name. The combinator graph for a
+  # rule is stable, so it is built once and reused (loop variables are
+  # block-local, so reusing a closure across recursive activations is safe).
+  def __built
+    @__built ||= {}
+  end
+
   # A terminal parser. A String matches that exact literal at the current
   # position; a Regexp is matched anchored at the current position. The matched
   # substring is the parser's value.
   #
+  # Positions are *byte* offsets, not character offsets: indexing a UTF-8 string
+  # by character is O(n), so matching by byte (+byteslice+ for literals,
+  # +byteindex+ with a `\G` anchor for regexps) keeps each step O(match length)
+  # regardless of how far into the input we are.
+  #
   # When the class enables +skip_whitespace+, leading whitespace is consumed
   # before the match is attempted, mirroring Scala's RegexParsers.
   def term(pattern)
@@ -86,22 +105,30 @@ class PackratParser
     ws = /\G(?:#{ws})/ if ws
     case pattern
     when String
+      bytes = pattern.bytesize
+      # The failure message is constant for this terminal, and failures are
+      # common (ordered choice discards them) while the message is only read if
+      # the whole parse fails. Build it once and share the frozen string rather
+      # than interpolating on every failed match.
+      msg = "expected #{pattern.inspect}".freeze
       Parser.new do |input, pos|
         pos = __skip_ws(ws, input, pos)
-        if input[pos, pattern.length] == pattern
-          Success.new(pattern, pos + pattern.length)
+        if input.byteslice(pos, bytes) == pattern
+          Success.new(pattern, pos + bytes)
         else
-          Failure.new(pos, "expected #{pattern.inspect}")
+          Failure.new(pos, msg)
         end
       end
     when Regexp
       anchored = /\G(?:#{pattern})/
+      msg = "expected #{pattern.inspect}".freeze
       Parser.new do |input, pos|
         pos = __skip_ws(ws, input, pos)
-        if (m = anchored.match(input, pos))
-          Success.new(m[0], pos + m[0].length)
+        if input.byteindex(anchored, pos)
+          s = Regexp.last_match[0]
+          Success.new(s, pos + s.bytesize)
         else
-          Failure.new(pos, "expected #{pattern.inspect}")
+          Failure.new(pos, msg)
         end
       end
     else
@@ -109,11 +136,11 @@ class PackratParser
     end
   end
 
-  # Advance +pos+ past whitespace matched by the anchored regexp +ws+ (nil when
-  # skipping is disabled). Returns the new position.
+  # Advance the byte offset +pos+ past whitespace matched by the anchored regexp
+  # +ws+ (nil when skipping is disabled). Returns the new byte offset.
   def __skip_ws(ws, input, pos)
     return pos unless ws
-    (m = ws.match(input, pos)) ? pos + m[0].length : pos
+    input.byteindex(ws, pos) ? pos + Regexp.last_match[0].bytesize : pos
   end
 
   # A parser that succeeds with +value+ without consuming any input (monadic
@@ -122,11 +149,16 @@ class PackratParser
     Parser.new { |_input, pos| Success.new(value, pos) }
   end
 
-  # Parse +input+ starting from the configured start symbol. Returns the parsed
-  # value on success; raises ParseError on failure or on leftover input.
-  def parse(input)
+  # Parse +input+, starting from rule +start+ (defaults to the configured start
+  # symbol). Returns the parsed value on success; raises ParseError on failure or
+  # on leftover input. Pass +start+ to parse from any rule, e.g.
+  # +parser.parse("123", :number)+ or, equivalently, +parser.number.parse("123")+.
+  #
+  # Positions are byte offsets throughout, including the +pos+ reported on a
+  # ParseError (see +term+ for why matching is byte-oriented).
+  def parse(input, start = nil)
     @__memo = {}
-    name = self.class.start_symbol
+    name = start || self.class.start_symbol
     raise ParseError.new("no start symbol defined", 0) unless name
 
     result = send(name).call(input, 0)
@@ -138,7 +170,7 @@ class PackratParser
     # all input was used.
     ws = self.class.whitespace
     end_pos = __skip_ws(ws && /\G(?:#{ws})/, input, result.pos)
-    if end_pos < input.length
+    if end_pos < input.bytesize
       raise ParseError.new("unexpected trailing input", end_pos)
     end
     result.value

data/lib/packrat_parser/parser.rb

@@ -76,7 +76,18 @@ class PackratParser
     #
     #   number << term(";")   # parse a number followed by ";", yield the number
     def <<(other)
-      flat_map { |x| other.map { |_| x } }
+      # Equivalent to flat_map { |x| other.map { |_| x } }, but written directly
+      # so the combinator graph is built once (at bind time) instead of
+      # allocating a fresh `map` parser on every successful match.
+      Parser.new do |input, pos|
+        a = call(input, pos)
+        if a.success?
+          b = other.call(input, a.pos)
+          b.success? ? Success.new(a.value, b.pos) : b
+        else
+          a
+        end
+      end
     end
 
     # Sequence, keeping the *right* result (Scala's `~>`). Run this parser, then
@@ -84,18 +95,71 @@ class PackratParser
     #
     #   term("(") >> additive   # skip "(", yield whatever additive produces
     def >>(other)
-      flat_map { |_| other }
+      # Equivalent to flat_map { |_| other }, written directly to avoid the
+      # per-call block dispatch.
+      Parser.new do |input, pos|
+        a = call(input, pos)
+        a.success? ? other.call(input, a.pos) : a
+      end
     end
 
     # Sequence, keeping *both* results (Scala's `~`). Run this parser, then
-    # +other+, and on success return the pair +[left, right]+. Like Scala's `~`
-    # this is left-associative and nests, so `p + q + r` yields `[[a, b], c]`;
-    # Ruby's block-parameter destructuring takes them apart the way Scala's
-    # `case a ~ b ~ c` does:
+    # +other+, and on success return the pair +[left, right]+. The result type is
+    # the product of the operands' types, so `*` (product) is the natural
+    # spelling -- and it dovetails with `|` for choice, mirroring how a regular
+    # language is a semiring with choice as the sum and sequence as the product.
+    #
+    # Like Scala's `~` this is left-associative and nests, so `p * q * r` yields
+    # `[[a, b], c]`; Ruby's block-parameter destructuring takes them apart the
+    # way Scala's `case a ~ b ~ c` does:
     #
-    #   (p + q + r).map { |(a, b), c| ... }
-    def +(other)
-      flat_map { |x| other.map { |y| [x, y] } }
+    #   (p * q * r).map { |(a, b), c| ... }
+    def *(other)
+      # Equivalent to flat_map { |x| other.map { |y| [x, y] } }, written directly
+      # so only the result pair (and its Success) is allocated per match, not a
+      # fresh intermediate `map` parser.
+      Parser.new do |input, pos|
+        a = call(input, pos)
+        if a.success?
+          b = other.call(input, a.pos)
+          b.success? ? Success.new([a.value, b.value], b.pos) : b
+        else
+          a
+        end
+      end
+    end
+
+    # Zero or more repetitions (Scala's `rep` / `p.*`). Always succeeds, yielding
+    # an array of the collected values (empty when there are no matches). A match
+    # that consumes no input stops the loop, so a nullable parser can't spin
+    # forever.
+    def rep
+      Parser.new do |input, pos|
+        values = []
+        cur = pos
+        loop do
+          result = call(input, cur)
+          break if !result.success? || result.pos == cur
+          values << result.value
+          cur = result.pos
+        end
+        Success.new(values, cur)
+      end
+    end
+
+    # One or more repetitions (Scala's `rep1` / `p.+`). Fails if the first match
+    # fails; otherwise yields a non-empty array of values.
+    def rep1
+      flat_map { |first| rep.map { |rest| [first, *rest] } }
+    end
+
+    # Optional (Scala's `opt` / `p.?`). Yields the parsed value, or nil (consuming
+    # nothing) when this parser does not match.
+    def opt
+      Parser.new do |input, pos|
+        result = call(input, pos)
+        result.success? ? result : Success.new(nil, pos)
+      end
     end
   end
 
@@ -110,20 +174,14 @@ class PackratParser
   # Memoizing the *result* per (rule, pos) gives the packrat property: each rule
   # is evaluated at most once per input position, so parsing stays linear.
   #
-  # The combinator graph is rebuilt on every (memo-missed) entry rather than
-  # cached, and that is deliberate. The `for ... then` comprehension's loop
-  # variables currently leak into the enclosing rule-method scope instead of
-  # being block-local, so a single built closure shares those slots. If the same
-  # closure were reused for a recursive activation (e.g. additive calling
-  # additive), the inner activation would clobber the outer's leaked variables.
-  # Building fresh gives each activation its own scope. The rebuild is bounded:
-  # result memoization means a build happens at most once per (rule, pos).
-  #
-  # NOTE: this rebuild is a workaround for the loop-variable leak in the fork's
-  # comprehension (parse.y: new_for_comp_gen leaves the loop var in the
-  # surrounding scope, like the legacy `for`). If the comprehension is changed to
-  # make loop variables block-local, this clobbering goes away and `built` can be
-  # cached once per (owner, name) again -- e.g. `@owner.__built[@name] ||= ...`.
+  # The combinator graph is built once per rule and cached on the owner. This
+  # relies on the comprehension's loop variables being block-local: each closure
+  # built from the rule body owns its loop-variable bindings, so reusing one
+  # cached closure across recursive activations (e.g. additive calling additive)
+  # is safe. (An earlier fork leaked the loop variables into the rule-method
+  # scope, which forced a rebuild per entry so activations would not clobber each
+  # other's bindings; that workaround is no longer needed now that the
+  # comprehension scopes them.)
   class Rule < Parser
     def initialize(owner, name, body)
       @owner = owner
@@ -132,11 +190,20 @@ class PackratParser
     end
 
     def call(input, pos)
-      memo = @owner.__memo
-      key = [@name, pos]
-      return memo[key] if memo.key?(key)
-      combinator = @body.bind(@owner).call
-      memo[key] = combinator.call(input, pos)
+      # Two-level memo table (rule -> pos -> result). Keying on a single [name,
+      # pos] Array would allocate one Array per rule invocation and hash it;
+      # nesting keeps the per-call key a bare Integer.
+      memo = (@owner.__memo[@name] ||= {})
+      return memo[pos] if memo.key?(pos)
+      combinator = (@owner.__built[@name] ||= @body.bind(@owner).call)
+      memo[pos] = combinator.call(input, pos)
+    end
+
+    # Parse +input+ starting from this rule, e.g. +parser.number.parse("123")+.
+    # Delegates to the owner so memo reset, whitespace handling, and the
+    # full-consumption check are applied exactly as for the start symbol.
+    def parse(input)
+      @owner.parse(input, @name)
     end
   end
 end

data/lib/packrat_parser/version.rb

@@ -1,3 +1,3 @@
 class PackratParser
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,52 +1,118 @@
 --- !ruby/object:Gem::Specification
 name: packrat_parser
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
-- Shugo Maeda
+  - Shugo Maeda
 bindir: bin
 cert_chain: []
 date: 1980-01-02 00:00:00.000000000 Z
-dependencies: []
+dependencies:
+  - !ruby/object:Gem::Dependency
+    name: rake
+    requirement: !ruby/object:Gem::Requirement
+      requirements:
+        -
+          - ~>
+          - !ruby/object:Gem::Version
+            version: "13.0"
+    type: :development
+    prerelease: false
+    version_requirements: !ruby/object:Gem::Requirement
+      requirements:
+        -
+          - ~>
+          - !ruby/object:Gem::Version
+            version: "13.0"
+  - !ruby/object:Gem::Dependency
+    name: test-unit
+    requirement: !ruby/object:Gem::Requirement
+      requirements:
+        -
+          - ~>
+          - !ruby/object:Gem::Version
+            version: "3.0"
+    type: :development
+    prerelease: false
+    version_requirements: !ruby/object:Gem::Requirement
+      requirements:
+        -
+          - ~>
+          - !ruby/object:Gem::Version
+            version: "3.0"
+  - !ruby/object:Gem::Dependency
+    name: racc
+    requirement: !ruby/object:Gem::Requirement
+      requirements:
+        -
+          - ~>
+          - !ruby/object:Gem::Version
+            version: "1.8"
+    type: :development
+    prerelease: false
+    version_requirements: !ruby/object:Gem::Requirement
+      requirements:
+        -
+          - ~>
+          - !ruby/object:Gem::Version
+            version: "1.8"
+  - !ruby/object:Gem::Dependency
+    name: parslet
+    requirement: !ruby/object:Gem::Requirement
+      requirements:
+        -
+          - ~>
+          - !ruby/object:Gem::Version
+            version: "2.0"
+    type: :development
+    prerelease: false
+    version_requirements: !ruby/object:Gem::Requirement
+      requirements:
+        -
+          - ~>
+          - !ruby/object:Gem::Version
+            version: "2.0"
 description: |
-  packrat_parser is a small PEG/packrat parser-combinator library. Grammar
-  rules are plain methods that return parsers and can be written using the
-  `for ... then` comprehension (flat_map/map/filter), giving a Scala-style
-  for-comprehension feel.
+    packrat_parser is a small PEG/packrat parser-combinator library. Grammar
+    rules are plain methods that return parsers and can be written using the
+    `for ... then` comprehension (flat_map/map/filter), giving a Scala-style
+    for-comprehension feel.
 email:
-- shugo.maeda@gmail.com
+  - shugo.maeda@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- LICENSE
-- README.md
-- examples/simple_calc.rb
-- lib/packrat_parser.rb
-- lib/packrat_parser/base.rb
-- lib/packrat_parser/parser.rb
-- lib/packrat_parser/result.rb
-- lib/packrat_parser/version.rb
-homepage: https://github.com/shugo/packrat_parser
+  - LICENSE
+  - README.md
+  - examples/simple_calc.rb
+  - lib/packrat_parser.rb
+  - lib/packrat_parser/base.rb
+  - lib/packrat_parser/parser.rb
+  - lib/packrat_parser/result.rb
+  - lib/packrat_parser/version.rb
+homepage: "https://github.com/shugo/packrat_parser"
 licenses:
-- MIT
+  - MIT
 metadata: {}
 rdoc_options: []
 require_paths:
-- lib
+  - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
-    - !ruby/object:Gem::Version
-      version: 3.0.0
+    -
+      - ">="
+      - !ruby/object:Gem::Version
+        version: 3.0.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
-    - !ruby/object:Gem::Version
-      version: '0'
+    -
+      - ">="
+      - !ruby/object:Gem::Version
+        version: "0"
 requirements: []
-rubygems_version: 4.0.10
+rubygems_version: 4.1.0.dev
 specification_version: 4
 summary: A packrat / PEG parser-combinator library with a Scala-inspired API.
 test_files: []