birman 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b986fd1949d2c3755859b6acd8753027417e40a382e18a0bdd02e350133bb1fa
+  data.tar.gz: cd8f04b2b86f234cf915b7b1ad941574f03bfe4f2f1f35fd6488bc94ec27911a
+SHA512:
+  metadata.gz: 430c9e5d8b26483919f9eb0caaf24c39614752b117e4e61b45faa2919598453a75558a6e928d592055b765563a5fe1a531022ce4311687cc3689bdc6c3f961b2
+  data.tar.gz: 7856edc44f834226c32f014a467de9d89fd5f4e4e840475f3c14b021199ae6de4b62b59520fc5eb9d813e17eb4e521e1e64fc3d74c7db1b49ea983b4ed090d1c

data/CHANGELOG.md

@@ -0,0 +1,42 @@
+## [Unreleased]
+
+## [0.1.0] - 2026-10-17
+
+First release: a small PEG parser library for Ruby 3.2+.
+
+- **Grammars** are classes that `include Birman::Grammar`. Rules are
+  methods (`rule(:name) { ... }`), so they compose and override through
+  `include`, and a module that includes `Birman::Grammar` can share
+  rules between grammars. Settings: `root`, `ignore` (a rule matched and
+  discarded between tokens, usually `:spacing`) and `keywords`, all
+  inherited and clearable with `nil`.
+- **Combinators**: `seq`, `either`, `one_of`, `maybe`, `rep`,
+  `absent`/`present`, `&` guards, `sep_by`, `between`, `parens`,
+  left-associative `infix`, `label`, `quiet`, `map`, plus operator forms
+  (`>>`, `|`, `&`, `>`). Bare strings in `seq`/`either` are literals that
+  skip trailing spacing; word-like ones are keywords.
+- **Captures and nodes**: `.as(:key)` names parts of a sequence and
+  `.to(Node)` builds a node from them. `node(:a, :b)` defines immutable
+  `Data` AST nodes with a source `span` (line, column, text), which
+  equality ignores; nodes may define their own `initialize`.
+- **Commit points**: `seq("let", commit, ...)` (or `a > b`) makes a
+  failure after the commit end the parse with an error at that spot.
+  Lookaheads contain commits; `rep` and `maybe` don't.
+- **Packrat memoization**, with failure diagnostics replayed from the
+  memo so cached results report errors correctly. Linear-time matching
+  on non-ASCII input (byte offsets internally, character offsets in the
+  API).
+- **Results**: `parse` returns `Success`/`Failure` and doesn't raise on
+  bad input, including input nested too deeply for Ruby's stack;
+  `parse!` raises `ParseError`. Failures carry position, expectations
+  and a pluggable formatter (`Failure#message(formatter)`,
+  `parse!(input, formatter:)`).
+- **Grammar mistakes fail loudly**: left recursion raises naming the
+  rule, as do rule names that would replace a method the grammar relies
+  on (`send`, `match`, `token`, ...) and a `commit` at the start of a
+  `seq`.
+- **Built-in rules**: `Birman::Rules::Whitespace`, `Comments`,
+  `Numbers`, `Strings` and `Identifiers`.
+- **Introspection**: `Grammar.to_ast` and `Grammar.to_ebnf`.
+- `examples/calc.rb`: a small language with an interpreter, as a
+  complete worked example.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 TODO: Write your name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,193 @@
+<p align="center">
+  <img src="assets/birman.png" alt="Birman logo: a Birman cat's face" width="200">
+</p>
+
+# Birman
+
+A small PEG parser library for Ruby 3.2+: module-based grammars,
+immutable `Data` matchers and AST nodes with source spans, packrat
+memoization, commit points, labels, and `Success`/`Failure` results.
+
+## Installation
+
+```sh
+bundle add birman
+```
+
+```ruby
+require "birman"
+
+module Lang
+  # A grammar is a class that includes Birman::Grammar. Each rule is a
+  # method on it, so rules can call each other by name.
+  class Grammar
+    include Birman::Grammar
+
+    # Ready-made rules: `spacing` (whitespace), `number` and `identifier`.
+    include Birman::Rules::Whitespace
+    include Birman::Rules::Numbers
+    include Birman::Rules::Identifiers
+
+    # AST nodes: immutable Data classes that also record where in the
+    # input they came from (node.span).
+    Let = node(:name, :value)
+    Num = node(:value)
+
+    # Parsing starts at this rule.
+    root :let_statement
+
+    # Match `spacing` after every literal and token, and throw it away,
+    # so the rules below never have to mention whitespace.
+    ignore :spacing
+
+    # Reserved words: `identifier` won't accept "let" as a name.
+    keywords "let"
+
+    # let_statement: "let", a name, "=", a number. Reading left to right:
+    #   "let"                  - a literal keyword ("let", not "letter")
+    #   commit                 - after "let", this must be a let statement;
+    #                            a mistake later is reported right there
+    #   identifier.as(:name)   - match a name and save it as :name
+    #   "="                    - another literal
+    #   number.to(Num)         - match a number and wrap it in a Num node,
+    #     .as(:value)            saved as :value
+    #   .to(Let)               - build a Let from the saved :name and :value
+    rule(:let_statement) do
+      seq("let", commit, identifier.as(:name), "=", number.to(Num).as(:value)).to(Let)
+    end
+  end
+end
+
+# parse returns Success or Failure (bad input never raises), so
+# pattern-match on the result.
+# (parse! returns the value or raises Birman::ParseError instead.)
+case Lang::Grammar.parse("let x = 42")
+in Birman::Success(ast)   then p ast        # => (Let "x" (Num 42))
+in Birman::Failure => f   then warn f.message
+end
+```
+
+See `examples/calc.rb` for a complete small language with an
+interpreter, and `Grammar.to_ebnf` to print any grammar.
+
+## Building blocks
+
+| Form                          | Meaning                                              |
+|-------------------------------|------------------------------------------------------|
+| `seq(a, b, c)`                | sequence                                             |
+| `seq(a, commit, b, c)`        | after `a`, the rest must match or parsing stops here |
+| `either(a, b, c)`             | ordered choice                                       |
+| `x.as(:key)`                  | name a part; the sequence's value becomes a hash     |
+| `x.to(Node)`                  | build a node from that hash (or from a single value) |
+| `infix(operand, "+", "-", node: BinOp)` | left-associative binary operators; `BinOp` needs `op`, `left`, `right` fields |
+| `parens(x)`, `between(o, c, x)` | wrapped                                            |
+| `sep_by(x, ",")`, `sep_by1`   | separated lists                                      |
+| `.rep(min, max)`, `.maybe`    | repetition, optional                                 |
+| `.absent`, `.present`         | negative / positive lookahead                        |
+| `.map { }`                    | transform the value                                  |
+| `.label("name")`              | "expected name" in errors                            |
+| `.quiet`                      | never mention in errors                              |
+
+The operator forms still work for short rules: `a >> b` (sequence),
+`a | b` (choice), `a & b` (guard), `a > b` (commit). Precedence, tightest
+first: `>>`, `&`, `|`, `>`.
+
+## Literals, spacing, and lexemes
+
+Declare `ignore :spacing` and whitespace (and comments, with
+`Rules::Comments`) takes care of itself:
+
+- Strings passed to `seq`, `either`, `infix`, `parens` and friends are
+  **literals**. They skip trailing spacing, and word-like ones
+  (`"let"`) are keywords, so `"let"` won't match the start of `letter`.
+- `str("...")` and `match(/.../)` are **raw**: exactly those characters,
+  no skipping.
+- Rules for tokens (numbers, names, strings) should be declared with
+  **`lexeme`** instead of `rule`, so they skip trailing spacing too. The
+  built-in `Birman::Rules` already do this.
+
+## Commit
+
+`seq("let", commit, name, "=", expr)` says: once `"let"` has matched,
+this is definitely a let statement. If anything after `commit` fails,
+the whole parse fails right there, with that error. No enclosing
+`either`, `rep` or `maybe` gets to backtrack and try something else.
+That reach is what keeps errors pointing at the real problem instead
+of at whatever a looser alternative accepted.
+
+- `rep` and `maybe` don't stop a commit. `seq("else", commit, expr).maybe`
+  rejects `else` followed by garbage rather than quietly treating it as
+  "no else clause".
+- Lookaheads do. `x.absent`, `x.present` and the guard in `a & b` only
+  ask whether `x` would match, so a commit failure inside them is just
+  "no".
+- `commit` can't come first in a `seq`. Nothing has matched yet, so
+  there's nothing to be sure about.
+
+This is Parsec-style commitment, not the choice-local "cut" from the
+PEG literature.
+
+## Nodes
+
+`node(:a, :b)` (in any grammar class, from `extend Birman::Nodes`, or
+as `Birman.node`) is `Data.define` plus:
+
+- a `span` the parser fills in (`node.span.line`, `.column`, `.text`),
+  which equality and pattern matching ignore
+- s-expression `inspect`: `(BinOp "+" (Num 1) (Num 2))`
+
+A node can define its own `initialize`, for example to give a field a
+default: `node(:name, :args) { def initialize(name:, args: []) = super }`.
+The parser and `with` set the span without going through it. (Passing
+`span:` to `new` by hand only works for nodes without their own
+`initialize`.) `span` can't be a field name, since every node already
+has one.
+
+## Pieces
+
+- `Birman::Rules`: `Whitespace`, `Comments`, `Numbers`, `Strings`,
+  `Identifiers`
+- Grammar settings: `root`, `ignore`, `keywords`. Subclasses inherit
+  them; pass `nil` to clear one (`ignore nil`, `keywords nil`)
+- `Grammar.parse` returns `Success`/`Failure`; `parse!` raises `ParseError`
+  (`parse!(input, formatter: f)` words it your way).
+  Input nested too deeply for Ruby's stack (several hundred levels)
+  is a `Failure` too, with `reason` set.
+- `Failure#message(formatter)` takes anything with `#call(failure)`
+- `Grammar.to_ast` and `Grammar.to_ebnf` for introspection
+- Rules are methods, so they compose and override through `include`.
+  Write your own rule module with `extend Birman::RuleSet`, or
+  `include Birman::Grammar` in a module to share rules, nodes and
+  combinators; any class that includes it is a grammar. Settings
+  (`root`, `ignore`, `keywords`) still go on the class.
+- A rule can't be named after a method the grammar relies on (`send`,
+  `class`, `match`, `token`...). Defining one raises `ArgumentError`
+  instead of silently breaking the grammar. Redefining a rule is fine.
+
+## Not yet
+
+- Left recursion. A rule that can reach itself without consuming input
+  raises `ArgumentError` naming the rule; use `infix`, or
+  `term (op term)*` with a fold, instead.
+- Compiling grammars to Ruby methods for speed
+- Railroad diagrams from `to_ast`
+
+## Development
+
+```sh
+bin/setup          # install dependencies
+bundle exec rake   # tests + rubocop
+ruby examples/calc.rb
+```
+
+## License
+
+MIT. See [LICENSE.txt](LICENSE.txt).
+
+## The name
+
+Birman is named for Alexander Birman, whose 1970 Ph.D. thesis, *The
+TMG Recognition Schema*, was the origin of packrat parsing and parsing
+expression grammars.
+
+The cat in the logo is a Birman too.

data/lib/birman/ast.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+module Birman
+  # Helpers for the plain-array ASTs that matchers produce via #to_ast:
+  #
+  #   [:str, "if"]  [:pat, "/\\d+/"]  [:ref, :expr]
+  #   [:seq, commit_at, [parts...]]  [:choice, [alts...]]
+  #   [:rep, min, max, inner]  [:not, inner]  [:and, inner]
+  #   [:both, left, right]  [:action, inner]  [:label, name, inner]
+  #   [:quiet, inner]
+  module AST
+    module_function
+
+    # Rule names referenced in an AST, in first-use order.
+    def refs(node)
+      case node
+      in [:ref, name] then [name]
+      in Array then node.flat_map { refs(_1) }.uniq
+      else []
+      end
+    end
+  end
+
+  # Prints a grammar AST in PEG-flavored EBNF:
+  #
+  #   a b  sequence    a / b  choice     ~  commit (rest must match)
+  #   a*  a+  a?  a{2,3}    !a  not      &a  lookahead / guard
+  #
+  # Actions and labels don't change what's accepted, so they're omitted.
+  module EBNF
+    module_function
+
+    # Precedence levels, loosest to tightest.
+    CHOICE = 0
+    SEQ    = 1
+    PREFIX = 2
+    ATOM   = 3
+
+    def call(rules)
+      width = rules.keys.map { _1.to_s.size }.max
+      rules.map { |name, ast| "#{name.to_s.ljust(width)} = #{expr(ast, CHOICE)} ;" }.join("\n")
+    end
+
+    # One branch per AST node type, so its size grows with the matcher set.
+    def expr(node, prec) # rubocop:disable Metrics
+      case node
+      in [:str, s]           then s.inspect
+      in [:pat, source]      then source
+      in [:ref, name]        then name.to_s
+      in [:action, inner]    then expr(inner, prec)
+      in [:label, _, inner]  then expr(inner, prec)
+      in [:quiet, inner]     then expr(inner, prec)
+      in [:both, [:not, guard], right]
+        wrap("!#{expr(guard, ATOM)} #{expr(right, PREFIX)}", prec > SEQ)
+      in [:choice, alts]     then wrap(alts.map { expr(_1, SEQ) }.join(" / "), prec > CHOICE)
+      in [:seq, commit_at, parts]
+        items = parts.map { expr(_1, PREFIX) }
+        items.insert(commit_at, "~") if commit_at
+        wrap(items.join(" "), prec > SEQ)
+      in [:both, left, right] then wrap("&#{expr(left, ATOM)} #{expr(right, PREFIX)}", prec > SEQ)
+      in [:not, inner]        then wrap("!#{expr(inner, ATOM)}", prec > PREFIX)
+      in [:and, inner]        then wrap("&#{expr(inner, ATOM)}", prec > PREFIX)
+      in [:rep, min, max, inner] then "#{expr(inner, ATOM)}#{suffix(min, max)}"
+      end
+    end
+
+    def suffix(min, max)
+      case [min, max]
+      in [0, nil] then "*"
+      in [1, nil] then "+"
+      in [0, 1]   then "?"
+      in [n, nil] then "{#{n},}"
+      in [n, ^n]  then "{#{n}}"
+      in [m, n]   then "{#{m},#{n}}"
+      end
+    end
+
+    def wrap(text, parens) = parens ? "(#{text})" : text
+  end
+end

data/lib/birman/combinators.rb

@@ -0,0 +1,75 @@
+# frozen_string_literal: true
+
+module Birman
+  # Marker for seq: everything after it is committed (see Matcher#>).
+  COMMIT = Object.new.tap { |o| def o.inspect = "Birman::COMMIT" }.freeze
+
+  # Building blocks and parameterized rule factories, available in
+  # every grammar. They're plain methods, so a module can override
+  # them or add more.
+  #
+  # Strings passed to seq, either, infix, one_of, sep_by, between and
+  # parens are literals (see #lit): they skip trailing spacing when
+  # the grammar declares `ignore`, and word-like ones are keywords. Use
+  # str and match for exact, raw matching.
+  module Combinators
+    # --- Raw matchers ---
+
+    def str(s)    = Str.new(s)
+    def match(re) = Pat.new(re)
+    def any       = Pat.new(/./m)
+    def eof       = any.absent.label("end of input")
+    def never     = Pat.new(/(?!)/)
+
+    # A whole word: "if" matches in "if x" but not in "iffy".
+    def keyword(word) = (str(word) >> match(/\w/).absent).map(&:first).label(word.inspect)
+
+    # --- Literals and the readable forms ---
+
+    # A string literal as a token. "let" becomes a keyword, "=" an
+    # exact string; either way followed by the grammar's ignore rule.
+    # Non-strings pass through unchanged.
+    def lit(x)
+      return Birman.coerce(x) unless x.is_a?(String)
+
+      base = x.match?(/\A\w+\z/) ? keyword(x) : str(x)
+      Lexeme.new(base, ignore_rule, x)
+    end
+
+    # Sequence. Put `commit` where the construct becomes unambiguous:
+    #   seq("let", commit, name.as(:name), "=", expr.as(:value))
+    def seq(*items)
+      commit_at = items.index(COMMIT)
+      raise ArgumentError, "commit can't come first in a seq: nothing has matched yet" if commit_at&.zero?
+
+      parts = items.reject { _1.equal?(COMMIT) }.map { lit(_1) }
+      Seq.new(parts, commit_at)
+    end
+
+    def commit = COMMIT
+
+    # Ordered choice: the first alternative that matches wins.
+    def either(*alts) = alts.size == 1 ? lit(alts.first) : Choice.new(alts.map { lit(_1) })
+
+    def one_of(*strings) = either(*strings)
+
+    # Left-associative binary operators, folded into nodes:
+    #   infix(product, "+", "-", node: BinOp)   # BinOp = node(:op, :left, :right)
+    def infix(operand, *ops, node:) = Infix.new(Birman.coerce(operand), either(*ops).label("operator"), node)
+
+    # One or more items separated by sep; value is an array of items.
+    def sep_by1(item, sep) = SepBy.new(lit(item), lit(sep))
+
+    # Zero or more items separated by sep.
+    def sep_by(item, sep) = sep_by1(item, sep).maybe.map { _1 || [] }
+
+    # inner wrapped in open/close; value is inner's.
+    def between(open, close, inner) = Pick.new([lit(open), Birman.coerce(inner), lit(close)], 1)
+
+    def parens(inner) = between("(", ")", inner)
+
+    private
+
+    def ignore_rule = self.class.respond_to?(:ignore) ? self.class.ignore : nil
+  end
+end

data/lib/birman/context.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+module Birman
+  # Per-parse state: input, memo table, and failure tracking. It
+  # mutates constantly, so it's a plain class rather than Data.
+  #
+  # Positions here, and in every matcher, are byte offsets: slicing a
+  # non-ASCII string by character offset costs time proportional to
+  # the offset, which made parsing quadratic. Failure and Span convert
+  # back to characters for anything public.
+  class Context
+    # Marks a memo entry whose rule is still being matched.
+    IN_PROGRESS = Object.new.freeze
+
+    # position: where the most recently entered rule started. When the
+    # input nests too deeply for Ruby's stack, that's where it happened.
+    attr_reader :input, :grammar, :memo, :furthest, :expected, :position
+
+    def initialize(input, grammar)
+      @input = input
+      @grammar = grammar
+      @memo = {}
+      @furthest = 0
+      @expected = []
+      @token_ends = {}
+    end
+
+    # A lexeme ended at token_end, then the ignore rule consumed spacing
+    # (whitespace, comments) up to pos. Spans use this to stop at the
+    # token instead of including what was ignored after it.
+    def ignored_after(token_end, pos)
+      @token_ends[pos] = token_end if pos > token_end
+    end
+
+    # Where the last token ended, for a match that ended at pos.
+    def token_end(pos) = @token_ends.fetch(pos, pos)
+
+    # fixed_anchor: \A, ^ and lookbehind see the whole input, not just
+    # the text from the current position on.
+    def scanner = @scanner ||= StringScanner.new(input, fixed_anchor: true)
+
+    # Record that something was expected at pos. Only the furthest
+    # position matters: that's where the parser got stuck.
+    def fail_at(pos, what)
+      if pos > @furthest
+        @furthest = pos
+        @expected = [what]
+      elsif pos == @furthest
+        @expected << what
+      end
+      nil
+    end
+
+    def merge(pos, expected) = expected.each { fail_at(pos, _1) }
+
+    # Run the block with fresh failure tracking starting at pos, then
+    # restore the outer state. Returns [result, furthest, expected] so
+    # the caller decides what to keep: Ref memoizes it, Label rewrites
+    # it, Not throws it away.
+    def isolate(pos)
+      outer = [@furthest, @expected]
+      @furthest = pos
+      @expected = []
+      result = yield
+      [result, @furthest, @expected]
+    ensure
+      @furthest, @expected = outer
+    end
+
+    def commit_failure! = raise(Committed.new(@furthest, @expected.dup))
+
+    # A positive lookahead: does matcher match at pos? Nothing is
+    # consumed, and failures inside it describe input the parse never
+    # moves past, so they're dropped. If it fails, that's recorded at
+    # pos, as what it expected there or as the lookahead itself.
+    def peek?(matcher, pos)
+      result, furthest, expected = isolate(pos) { lookahead { matcher.match(self, pos) } }
+      return true if result
+
+      if furthest == pos && expected.any?
+        merge(pos, expected)
+      else
+        fail_at(pos, EBNF.expr(matcher.to_ast, EBNF::CHOICE))
+      end
+      false
+    end
+
+    # Rule name's memo entry at pos, computed by the block the first
+    # time. Meeting the rule again at the same position while it is
+    # still being matched means it reaches itself without consuming
+    # input: left recursion, which would never terminate. That's a bug
+    # in the grammar, not the input, so it raises.
+    def memoize(name, pos, &)
+      key = [name, pos]
+      entry = @memo[key]
+      return entry if entry && !entry.equal?(IN_PROGRESS)
+      raise ArgumentError, "left recursion in rule :#{name}: it can reach itself without consuming input" if entry
+
+      @position = pos
+      compute(key, &)
+    end
+
+    # Run a lookahead's inner match. A lookahead only asks whether
+    # something would match here, so a commit failure inside it is an
+    # ordinary failure (nil) rather than the end of the parse.
+    def lookahead
+      yield
+    rescue Committed
+      nil
+    end
+
+    private
+
+    def compute(key)
+      @memo[key] = IN_PROGRESS
+      @memo[key] = yield
+    ensure
+      # A commit failure caught by a lookahead unwinds through here; the
+      # rule may be tried again at this position afterwards.
+      @memo.delete(key) if @memo[key].equal?(IN_PROGRESS)
+    end
+  end
+end

data/lib/birman/error_formatter.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module Birman
+  # Turns a Failure into readable text. Anything that responds to
+  # #call(failure) works in its place, for custom wording or i18n:
+  #
+  #   failure.message(MyFormatter.new)
+  class ErrorFormatter
+    def call(failure)
+      source = source_line(failure)
+      <<~MSG.chomp
+        line #{failure.line}, column #{failure.column}: #{problem(failure)}
+          #{source}
+          #{caret_indent(source, failure.column)}^
+      MSG
+    end
+
+    private
+
+    def problem(failure)
+      failure.reason || "expected #{expected_list(failure.expected)}, found #{failure.found}"
+    end
+
+    def expected_list(items)
+      case items
+      in [] then "nothing"
+      in [only] then only
+      in [*rest, last] then "#{rest.join(", ")} or #{last}"
+      end
+    end
+
+    # Spaces up to the column, except tabs stay tabs, so the caret lands
+    # under the right character however wide the terminal draws a tab.
+    def caret_indent(source, column) = source[0, column - 1].to_s.gsub(/[^\t]/, " ").ljust(column - 1)
+
+    def source_line(failure) = failure.input.lines.fetch(failure.line - 1, "").chomp
+  end
+end