packrat_parser 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3329c290118a8d488a68b9d83f645eee1095730ee11d51446bd1e76ff232b95e
+  data.tar.gz: d39e34dd833b9bb48079b6e342a5b36162396a257b239697bef76d5f796b0679
+SHA512:
+  metadata.gz: 9d51853503af9f290bbb062f4a0b3e8543ddb032aeb69d3f3d1ec555c02101975780e0cad5e4ab364a267cfd6fd361259b0cd1ca4136dbae00ce81dea954a753
+  data.tar.gz: afd0c1223b0bc3d392b947a319cedc9696280c6cf6d5e10cfbc638aaa1fdf7669a5c88c532542bc9744b645312cab8b88fb559d3fa90be4f1c77d50145f23abd

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shugo Maeda
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,133 @@
+# packrat_parser
+
+A small [packrat](https://bford.info/packrat/) / PEG parser-combinator library
+with a Scala-inspired, monadic API. Grammar rules are plain methods that return
+parsers, and they can be written with the `for ... then` comprehension so the
+grammar reads like a Scala for-comprehension.
+
+```ruby
+require "packrat_parser"
+
+class SimpleCalcParser < PackratParser
+  def additive
+    (for x in multitive, _ in term("+"), y in additive then x + y end) |
+      (for x in multitive, _ in term("-"), y in additive then x - y end) |
+      multitive
+  end
+
+  def multitive
+    (for x in primary, _ in term("*"), y in multitive then x * y end) |
+      (for x in primary, _ in term("/"), y in multitive then x / y end) |
+      primary
+  end
+
+  def primary
+    (for _l in term("("), x in additive, _r in term(")") then x end) |
+      number
+  end
+
+  def number
+    for s in term(/\d+/) then s.to_i end
+  end
+end
+
+SimpleCalcParser.parse("1+2*3")   # => 7
+SimpleCalcParser.parse("(1+2)*3") # => 9
+```
+
+## Requirements
+
+The `for ... then` comprehension is a feature of a Ruby fork and is only
+recognized by its **legacy parser**. Run grammars with `--parser=parse.y`:
+
+```sh
+ruby --parser=parse.y -Ilib examples/simple_calc.rb
+```
+
+The default Prism parser rejects `for ... then`. The library itself
+(`lib/`) is ordinary Ruby — only files that *write* grammars need the flag.
+
+## API
+
+Subclass `PackratParser`. **Every method you define in the subclass is a grammar
+rule** and must return a parser; rule methods are automatically made lazy and
+memoized so they can reference one another (and themselves) recursively.
+
+### Building blocks (available inside rule methods)
+
+- `term(string)` — match an exact literal at the current position.
+- `term(regexp)` — match a regexp anchored at the current position.
+  Both yield the matched substring (e.g. `term(/\d+/)`).
+- `pure(value)` — succeed with `value` without consuming input.
+
+### Combinators (methods on every parser)
+
+- `flat_map { |v| parser }` — sequence: run `parser` after this one succeeds.
+- `map { |v| new_value }` — transform the result.
+- `filter { |v| bool }` — succeed only when the predicate holds.
+- `a | b` — ordered choice: try `a`, and if it fails try `b`.
+- `a + b` — sequence, keep **both** results (Scala's `~`): run `a` then `b`,
+  yield the pair `[a, b]`. Left-associative and nesting, so `a + b + c` yields
+  `[[a, b], c]`; Ruby's block-parameter destructuring takes them apart the way
+  Scala's `case a ~ b ~ c` does: `(a + b + c).map { |(x, y), z| ... }`.
+- `a << b` — sequence, keep the **left** result (Scala's `<~`): run `a` then
+  `b`, yield `a`'s value and discard `b`'s.
+- `a >> b` — sequence, keep the **right** result (Scala's `~>`): run `a` then
+  `b`, yield `b`'s value and discard `a`'s.
+
+The arrow direction is a useful mnemonic: `<<`/`>>` keeps whichever side it
+points to. They are handy for discarding punctuation, e.g. `( expr )` is
+`term("(") >> expr << term(")")`. Ruby's precedence (`+` over `<<`/`>>` over
+`|`) means sequencing binds tighter than ordered choice, as you'd want.
+
+`flat_map`, `map`, and `filter` are exactly what the `for ... then`
+comprehension desugars to (a non-final generator → `flat_map`, the final
+generator → `map`, a `when` guard → `filter`), so:
+
+```ruby
+for x in p, y in q when y > 0 then x + y end
+# == p.flat_map { |x| q.filter { |y| y > 0 }.map { |y| x + y } }
+```
+
+### Whitespace skipping (optional)
+
+By default `term` matches exactly. To skip whitespace implicitly — like Scala's
+`RegexParsers` — declare `skip_whitespace` at the class level. Each `term` then
+consumes leading whitespace before matching, and `parse` consumes trailing
+whitespace before requiring full input consumption:
+
+```ruby
+class CalcParser < PackratParser
+  skip_whitespace            # default pattern: /\s+/
+  # skip_whitespace(/[ \t]+/)  # or a custom pattern (e.g. spaces/tabs only)
+  # ... rules using term(...) ...
+end
+
+CalcParser.parse("  1 + 2 * 3  ")  # => 7
+```
+
+The setting is inherited by subclasses, so a base parser can enable it once.
+
+### Entry point
+
+- `start_symbol :name` (class level) — choose the rule to start from.
+  If omitted, the first defined method is used as the start symbol.
+- `Klass.parse(input)` / `Klass.new.parse(input)` — parse, returning the value.
+  Raises `PackratParser::ParseError` on failure or leftover input.
+
+## Notes / limitations
+
+- **Classic packrat: no left recursion.** Write rules right-recursively. A
+  consequence is that `-` and `/` in the example calculator associate to the
+  right (`12/4/3` parses as `12/(4/3)` == `12`).
+- **No implicit whitespace by default.** `term` matches exactly. Enable implicit
+  whitespace skipping with `skip_whitespace` (see above) when your grammar needs
+  it.
+
+## Running the tests
+
+```sh
+bin/test
+# or:
+ruby --parser=parse.y -Ilib -Itest test/test_packrat_parser.rb
+```

data/examples/simple_calc.rb

@@ -0,0 +1,41 @@
+# Run with the Ruby fork's legacy parser, which understands `for ... then`:
+#
+#   /workspace/ruby/ruby --disable-gems --parser=parse.y -Ilib examples/simple_calc.rb
+#
+require "packrat_parser"
+
+# A four-function integer calculator.
+#
+# The grammar is right-recursive (additive -> multitive "+" additive), which is
+# what classic packrat parsing supports without left-recursion handling. A
+# consequence is that "-" and "/" associate to the right, e.g. 8-3-2 parses as
+# 8-(3-2) == 7. That trade-off comes from the monadic-core / no-`rep` API.
+class SimpleCalcParser < PackratParser
+  def additive
+    for x in multitive << term("+"), y in additive then x + y end |
+      for x in multitive << term("-"), y in additive then x - y end |
+      multitive
+  end
+
+  def multitive
+    for x in primary << term("*"), y in multitive then x * y end |
+      for x in primary << term("/"), y in multitive then x / y end |
+      primary
+  end
+
+  def primary
+    for x in term("(") >> additive << term(")") then x end |
+      number
+  end
+
+  def number
+    for s in term(/\d+/) then s.to_i end
+  end
+end
+
+if __FILE__ == $PROGRAM_NAME
+  examples = ["1+2*3", "(1+2)*3", "2*3+4", "12/4/3", "((7))"]
+  examples.each do |src|
+    puts "#{src} => #{SimpleCalcParser.parse(src)}"
+  end
+end

data/lib/packrat_parser/base.rb

@@ -0,0 +1,146 @@
+class PackratParser
+  # Subclass this and define grammar rules as plain methods that return parsers.
+  # Every method defined in a subclass is automatically treated as a rule: it is
+  # rewritten to return a lazy, memoizing Rule (see Rule), so rules may reference
+  # each other (and themselves) without infinitely recursing while the combinator
+  # graph is built.
+  #
+  #   class SimpleCalcParser < PackratParser
+  #     def additive
+  #       (for x in multitive, _ in term("+"), y in additive then x + y end) | multitive
+  #     end
+  #     ...
+  #   end
+  #
+  #   SimpleCalcParser.parse("1+2*3")  # => 7
+
+  # Set (or read) the rule the parser starts from.
+  # If omitted, the first defined method is used as the start symbol.
+  def self.start_symbol(name = nil)
+    if name
+      @start_symbol = name
+    else
+      @start_symbol
+    end
+  end
+
+  # Enable implicit whitespace skipping (Scala's RegexParsers mode). When set,
+  # every +term+ skips leading whitespace matching +pattern+ before attempting
+  # its match, and +parse+ also consumes trailing whitespace before requiring
+  # full input consumption. Off by default (terminals match exactly).
+  #
+  #   class CalcParser < PackratParser
+  #     skip_whitespace            # default /\s+/
+  #     # skip_whitespace(/[ \t]+/)  # or a custom pattern
+  #   end
+  def self.skip_whitespace(pattern = /\s+/)
+    @__whitespace = pattern
+  end
+
+  # The configured whitespace pattern, or nil when skipping is disabled.
+  # Inherited by subclasses so a base parser can turn the mode on once.
+  def self.whitespace
+    return @__whitespace if defined?(@__whitespace)
+    superclass.respond_to?(:whitespace) ? superclass.whitespace : nil
+  end
+
+  # Convenience: parse +input+ with a fresh instance.
+  def self.parse(input)
+    new.parse(input)
+  end
+
+  # Rewrite every method defined on a subclass into a rule that returns a lazy
+  # Rule. Guards against rewriting the base class's own infrastructure and
+  # against re-entering while we install the replacement (define_method itself
+  # fires method_added).
+  def self.method_added(name)
+    return if self == PackratParser
+    return if name == :initialize
+    return if @__defining_rule
+
+    @__defining_rule = true
+    @start_symbol ||= name
+    begin
+      body = instance_method(name)
+      define_method(name) do
+        Rule.new(self, name, body)
+      end
+    ensure
+      @__defining_rule = false
+    end
+  end
+
+  # Per-input packrat memo table, keyed by [rule_name, pos].
+  def __memo
+    @__memo ||= {}
+  end
+
+  # A terminal parser. A String matches that exact literal at the current
+  # position; a Regexp is matched anchored at the current position. The matched
+  # substring is the parser's value.
+  #
+  # When the class enables +skip_whitespace+, leading whitespace is consumed
+  # before the match is attempted, mirroring Scala's RegexParsers.
+  def term(pattern)
+    ws = self.class.whitespace
+    ws = /\G(?:#{ws})/ if ws
+    case pattern
+    when String
+      Parser.new do |input, pos|
+        pos = __skip_ws(ws, input, pos)
+        if input[pos, pattern.length] == pattern
+          Success.new(pattern, pos + pattern.length)
+        else
+          Failure.new(pos, "expected #{pattern.inspect}")
+        end
+      end
+    when Regexp
+      anchored = /\G(?:#{pattern})/
+      Parser.new do |input, pos|
+        pos = __skip_ws(ws, input, pos)
+        if (m = anchored.match(input, pos))
+          Success.new(m[0], pos + m[0].length)
+        else
+          Failure.new(pos, "expected #{pattern.inspect}")
+        end
+      end
+    else
+      raise ArgumentError, "term expects a String or Regexp, got #{pattern.class}"
+    end
+  end
+
+  # Advance +pos+ past whitespace matched by the anchored regexp +ws+ (nil when
+  # skipping is disabled). Returns the new position.
+  def __skip_ws(ws, input, pos)
+    return pos unless ws
+    (m = ws.match(input, pos)) ? pos + m[0].length : pos
+  end
+
+  # A parser that succeeds with +value+ without consuming any input (monadic
+  # unit / Scala's `success`).
+  def pure(value)
+    Parser.new { |_input, pos| Success.new(value, pos) }
+  end
+
+  # Parse +input+ starting from the configured start symbol. Returns the parsed
+  # value on success; raises ParseError on failure or on leftover input.
+  def parse(input)
+    @__memo = {}
+    name = self.class.start_symbol
+    raise ParseError.new("no start symbol defined", 0) unless name
+
+    result = send(name).call(input, 0)
+    unless result.success?
+      raise ParseError.new(result.message, result.pos)
+    end
+    # The last terminal skips only *leading* whitespace, so trailing whitespace
+    # after the final token is left for parse to consume before requiring that
+    # all input was used.
+    ws = self.class.whitespace
+    end_pos = __skip_ws(ws && /\G(?:#{ws})/, input, result.pos)
+    if end_pos < input.length
+      raise ParseError.new("unexpected trailing input", end_pos)
+    end
+    result.value
+  end
+end

data/lib/packrat_parser/parser.rb

@@ -0,0 +1,142 @@
+class PackratParser
+  # A parser combinator. Wraps a function (input, pos) -> Success | Failure.
+  #
+  # The four monadic operations (+flat_map+, +map+, +filter+, +pure+) are what
+  # the `for ... then` comprehension in the Ruby fork desugars to, so grammar
+  # rules can be written with comprehension syntax:
+  #
+  #   for x in multitive, _ in term("+"), y in additive then x + y end
+  #     # => multitive.flat_map { |x| term("+").flat_map { |_| additive.map { |y| x + y } } }
+  class Parser
+    def initialize(&fn)
+      @fn = fn
+    end
+
+    # Run this parser against +input+ starting at +pos+.
+    def call(input, pos)
+      @fn.call(input, pos)
+    end
+
+    # Sequencing / monadic bind. On success, +yield+ the value to obtain the
+    # next parser and run it where this one stopped. Failures short-circuit.
+    def flat_map
+      Parser.new do |input, pos|
+        result = call(input, pos)
+        if result.success?
+          next_parser = yield(result.value)
+          next_parser.call(input, result.pos)
+        else
+          result
+        end
+      end
+    end
+
+    # Transform the successful value without consuming further input.
+    def map
+      Parser.new do |input, pos|
+        result = call(input, pos)
+        result.success? ? Success.new(yield(result.value), result.pos) : result
+      end
+    end
+
+    # Succeed only when the block returns a truthy value for the parsed result;
+    # otherwise fail at the position where this parser started. This is what a
+    # `when` guard in a comprehension desugars to.
+    def filter
+      Parser.new do |input, pos|
+        result = call(input, pos)
+        if result.success? && yield(result.value)
+          result
+        else
+          Failure.new(pos, "guard failed")
+        end
+      end
+    end
+
+    # Ordered choice (PEG `/`). Try this parser; if it fails, try +other+ at the
+    # same position. Reports whichever failure reached furthest into the input.
+    def |(other)
+      Parser.new do |input, pos|
+        result = call(input, pos)
+        if result.success?
+          result
+        else
+          alt = other.call(input, pos)
+          if alt.success?
+            alt
+          else
+            alt.pos >= result.pos ? alt : result
+          end
+        end
+      end
+    end
+
+    # Sequence, keeping the *left* result (Scala's `<~`). Run this parser, then
+    # +other+, and on success return this parser's value, discarding +other+'s.
+    #
+    #   number << term(";")   # parse a number followed by ";", yield the number
+    def <<(other)
+      flat_map { |x| other.map { |_| x } }
+    end
+
+    # Sequence, keeping the *right* result (Scala's `~>`). Run this parser, then
+    # +other+, and on success return +other+'s value, discarding this one's.
+    #
+    #   term("(") >> additive   # skip "(", yield whatever additive produces
+    def >>(other)
+      flat_map { |_| other }
+    end
+
+    # Sequence, keeping *both* results (Scala's `~`). Run this parser, then
+    # +other+, and on success return the pair +[left, right]+. Like Scala's `~`
+    # this is left-associative and nests, so `p + q + r` yields `[[a, b], c]`;
+    # Ruby's block-parameter destructuring takes them apart the way Scala's
+    # `case a ~ b ~ c` does:
+    #
+    #   (p + q + r).map { |(a, b), c| ... }
+    def +(other)
+      flat_map { |x| other.map { |y| [x, y] } }
+    end
+  end
+
+  # A lazy, memoizing reference to a named grammar rule.
+  #
+  # Rule methods on a PackratParser subclass return a Rule instead of building
+  # their combinator immediately. This is essential: a comprehension evaluates
+  # its generator *receivers* eagerly (to call flat_map/map on them), so a
+  # self-referential rule like `additive` would recurse forever at build time if
+  # the body ran on every reference. Returning a lazy Rule breaks that cycle.
+  #
+  # Memoizing the *result* per (rule, pos) gives the packrat property: each rule
+  # is evaluated at most once per input position, so parsing stays linear.
+  #
+  # The combinator graph is rebuilt on every (memo-missed) entry rather than
+  # cached, and that is deliberate. The `for ... then` comprehension's loop
+  # variables currently leak into the enclosing rule-method scope instead of
+  # being block-local, so a single built closure shares those slots. If the same
+  # closure were reused for a recursive activation (e.g. additive calling
+  # additive), the inner activation would clobber the outer's leaked variables.
+  # Building fresh gives each activation its own scope. The rebuild is bounded:
+  # result memoization means a build happens at most once per (rule, pos).
+  #
+  # NOTE: this rebuild is a workaround for the loop-variable leak in the fork's
+  # comprehension (parse.y: new_for_comp_gen leaves the loop var in the
+  # surrounding scope, like the legacy `for`). If the comprehension is changed to
+  # make loop variables block-local, this clobbering goes away and `built` can be
+  # cached once per (owner, name) again -- e.g. `@owner.__built[@name] ||= ...`.
+  class Rule < Parser
+    def initialize(owner, name, body)
+      @owner = owner
+      @name = name
+      @body = body
+    end
+
+    def call(input, pos)
+      memo = @owner.__memo
+      key = [@name, pos]
+      return memo[key] if memo.key?(key)
+      combinator = @body.bind(@owner).call
+      memo[key] = combinator.call(input, pos)
+    end
+  end
+end

data/lib/packrat_parser/result.rb

@@ -0,0 +1,53 @@
+class PackratParser
+  # The result of running a parser at some position in the input.
+  #
+  # +Success+ carries the parsed +value+ and +pos+, the index of the next
+  # unconsumed character. +Failure+ carries the +pos+ where parsing failed and
+  # a human-readable +message+.
+  class Success
+    attr_reader :value, :pos
+
+    def initialize(value, pos)
+      @value = value
+      @pos = pos
+    end
+
+    def success?
+      true
+    end
+
+    def to_s
+      "Success(#{@value.inspect} @#{@pos})"
+    end
+    alias inspect to_s
+  end
+
+  class Failure
+    attr_reader :pos, :message
+
+    def initialize(pos, message)
+      @pos = pos
+      @message = message
+    end
+
+    def success?
+      false
+    end
+
+    def to_s
+      "Failure(#{@message} @#{@pos})"
+    end
+    alias inspect to_s
+  end
+
+  # Raised by PackratParser#parse when the input cannot be parsed, or when the
+  # start symbol succeeds but does not consume the whole input.
+  class ParseError < StandardError
+    attr_reader :pos
+
+    def initialize(message, pos)
+      @pos = pos
+      super("#{message} (at position #{pos})")
+    end
+  end
+end

data/lib/packrat_parser/version.rb

@@ -0,0 +1,3 @@
+class PackratParser
+  VERSION = "0.1.0"
+end

data/lib/packrat_parser.rb

@@ -0,0 +1,9 @@
+# A small packrat / PEG parser-combinator library whose grammar rules can be
+# written with the `for ... then` comprehension from the Ruby fork.
+class PackratParser
+end
+
+require_relative "packrat_parser/version"
+require_relative "packrat_parser/result"
+require_relative "packrat_parser/parser"
+require_relative "packrat_parser/base"

metadata

@@ -0,0 +1,52 @@
+--- !ruby/object:Gem::Specification
+name: packrat_parser
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Shugo Maeda
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies: []
+description: |
+  packrat_parser is a small PEG/packrat parser-combinator library. Grammar
+  rules are plain methods that return parsers and can be written using the
+  `for ... then` comprehension (flat_map/map/filter), giving a Scala-style
+  for-comprehension feel.
+email:
+- shugo.maeda@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- examples/simple_calc.rb
+- lib/packrat_parser.rb
+- lib/packrat_parser/base.rb
+- lib/packrat_parser/parser.rb
+- lib/packrat_parser/result.rb
+- lib/packrat_parser/version.rb
+homepage: https://github.com/shugo/packrat_parser
+licenses:
+- MIT
+metadata: {}
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.10
+specification_version: 4
+summary: A packrat / PEG parser-combinator library with a Scala-inspired API.
+test_files: []