birman 0.2.2 → 0.3.0

data/README.md

@@ -20,7 +20,7 @@ bundle add birman
 or in your Gemfile:
 
 ```ruby
-gem "birman", "~> 0.2.2"
+gem "birman", "~> 0.3.0"
 ```
 
 ```ruby
@@ -77,8 +77,41 @@ end
 ```
 
 See `examples/calc.rb` for a complete small language with an
-interpreter, `examples/json.rb` for a full JSON parser (tested against
-`JSON.parse`), and `Grammar.to_ebnf` to print any grammar.
+interpreter, `examples/leftcalc.rb` for its arithmetic written as left
+recursion instead of `infix`, `examples/json.rb` for a full JSON parser (tested against
+`JSON.parse`), `examples/toml.rb` for TOML 1.0 (tested against the
+official toml-test suite), `examples/query.rb` for a GitHub-style
+search query (lookahead, implicit AND, a custom error formatter) with
+an evaluator that filters records and a compiler to SQL for
+ActiveRecord's `where`, and `Grammar.to_ebnf` to print any grammar.
+
+## Why not a regex?
+
+A Ruby regex can match nested input with `\g<name>`, but it can't
+build the structure (a capture keeps only its last match), and it
+can't say what's wrong when the match fails.
+
+```ruby
+re = /\A(?<l>\[\s*(?:(?:\d+|\g<l>)(?:\s*,\s*(?:\d+|\g<l>))*)?\s*\])\z/
+re.match?("[1, [2, 3], []]")      # => true
+re.match("[1, [2, 3], []]")[:l]   # => "[1, [2, 3], []]"  (just the string back)
+re.match?("[1, [2, 3]")           # => false              (no hint why)
+
+class Lists
+  include Birman::Grammar
+  include Birman::Rules::Whitespace
+  ignore :spacing
+  root :list
+  rule(:list) { seq("[", commit, only(sep_by(item, ",")), "]") }
+  rule(:item) { either(match(/\d+/).map(&:to_i), list) }
+end
+
+Lists.parse("[1, [2, 3], []]")   # => Success([1, [2, 3], []])
+puts Lists.parse("[1, [2, x], []]").message
+# line 1, column 9: expected /\d+/ or "[", found "x"
+#   [1, [2, x], []]
+#           ^
+```
 
 ## Building blocks
 
@@ -94,15 +127,43 @@ interpreter, `examples/json.rb` for a full JSON parser (tested against
 | `infix(operand, "+", "-", node: BinOp)` | left-associative binary operators; `BinOp` needs `op`, `left`, `right` fields |
 | `parens(x)`, `between(o, c, x)` | wrapped                                            |
 | `sep_by(x, ",")`, `sep_by1`   | separated lists                                      |
+| `sep_by(x, ",", trailing: true)` | and one more `","` may end the list (not an empty one) |
+| `match(re).captures`          | a hash of `re`'s named groups (`{date: "1979-05-27", time: nil}`; `nil` for a group that didn't match) |
 | `.rep(min, max)`, `.maybe`    | repetition, optional                                 |
 | `.absent`, `.present`         | negative / positive lookahead                        |
-| `.map { }`                    | transform the value                                  |
+| `.map(:to_i)`                 | call that method on the value                        |
+| `.action(:name)`              | pass the value to the grammar's method `name`        |
+| `.check(:name, "label")`      | fail as "expected label" unless the grammar's method `name` accepts the value |
+| `.means(v)`                   | the value is `v` (`lit("null").means(nil)`)          |
+| `.map { }`                    | transform the value with a block (prefer the above)  |
 | `.label("name")`              | "expected name" in errors                            |
 | `.quiet`                      | never mention in errors                              |
 
+Rules can be left-recursive, written the way a textbook grammar writes
+left-associative operators (see [Left recursion](#left-recursion)):
+
+```ruby
+rule(:expr) { either(seq(expr.as(:left), either("+", "-").as(:op), term.as(:right)).to(BinOp), term) }
+```
+
+`1 - 2 - 3` parses as `(1 - 2) - 3`, the same tree
+`infix(term, "+", "-", node: BinOp)` builds.
+
+A name used twice in one `seq` raises `ArgumentError`, and so does a
+`seq` of named parts (or `match(re).captures`) inside another whose
+value would drop it, even through `maybe`, `label`, `either` and the
+like: list the parts in the outer `seq`, or name the inner one,
+`seq(...).as(:pair)`. A name every alternative of an `either` shares
+passes outward; alternatives named differently pass on none. A rule
+keeps its names to itself, so using one inside a `seq` is always fine.
+
 The operator forms still work for short rules: `a >> b` (sequence),
 `a | b` (choice), `a & b` (guard), `a > b` (commit). Precedence, tightest
-first: `>>`, `&`, `|`, `>`.
+first: `>>`, `&`, `|`, `>`. A chain of `>>` is one sequence:
+`a.as(:x) >> b.as(:y) >> c.as(:z)` is `seq(a.as(:x), b.as(:y), c.as(:z))`,
+however it's grouped, and `a > b >> c` is `seq(a, commit, b, c)`. A seq
+built any other way (by `seq(...)`, or with a commit or `only`) stays
+one part, as it would in `seq`: `pair >> c` is `seq(pair, c)`.
 
 ## Literals, spacing, and lexemes
 
@@ -115,7 +176,11 @@ Declare `ignore :spacing` and whitespace (and comments, with
   word-like ones (`"let"`) are keywords, so `"let"` won't match the
   start of `letter`.
 - `str("...")` and `match(/.../)` are **raw**: exactly those characters,
-  no skipping.
+  no skipping. `match(/.../).captures` is raw too, its value a hash of
+  the named groups with Symbol keys, every name always present. Each
+  name must be used once (`(?<n>a)|(?<n>b)` raises `ArgumentError`), and
+  unnamed groups aren't included: Ruby stops capturing them once a group
+  is named. `.to(Node)` builds a node from it like any hash.
 - Rules for tokens (numbers, names, strings) should be declared with
   **`lexeme`** instead of `rule`, so they skip trailing spacing too. The
   built-in `Birman::Rules` already do this.
@@ -132,6 +197,9 @@ of at whatever a looser alternative accepted.
 - `rep` and `maybe` don't stop a commit. `seq("else", commit, expr).maybe`
   rejects `else` followed by garbage rather than quietly treating it as
   "no else clause".
+- Neither does a trailing separator. `sep_by(x, ",", trailing: true)`
+  ends the list at a last `","` only when the `x` after it fails
+  without committing.
 - Lookaheads do. `x.absent`, `x.present` and the guard in `a & b` only
   ask whether `x` would match, so a commit failure inside them is just
   "no".
@@ -141,6 +209,182 @@ of at whatever a looser alternative accepted.
 This is Parsec-style commitment, not the choice-local "cut" from the
 PEG literature.
 
+## Left recursion
+
+A rule can call itself before consuming anything, as grammars usually
+write left-associative operators:
+
+```ruby
+rule(:expr) { either(seq(expr, "-", term).map { |l, _, r| l - r }, term) }
+rule(:term) { either(seq(term, "*", integer).map { |l, _, r| l * r }, integer) }
+
+parse!("5*4*2-5*9*2") # => -50, as ((5*4)*2) - ((5*9)*2)
+```
+
+`expr op term` is the left-associative form, the same shape `infix`
+builds. Birman grows the match from a seed (Warth et al., 2008): it
+tries the rule with its own call failing, then again with that call
+returning the last match, for as long as the match gets further. So
+each level is a value or node of its own, `((3-4)-5)`, and a node's
+span covers its whole subexpression. A long chain uses no more stack
+than a short one, and errors say what the last attempt needed
+(`1-` expects an integer after the `-`). `to_ruby` generates parsers
+that do the same.
+
+The recursion can go through other rules too (indirect left
+recursion), as when each level of precedence is split in two:
+
+```ruby
+rule(:sum)   { terms }
+rule(:terms) { either(seq(sum, "-", term).map { |l, _, r| l - r }, term) }
+```
+
+Each cycle grows at one rule, its leader, as pegen does: the first rule
+(in the order the grammar uses them) that every way round the cycle
+passes through, here `sum`. While it grows, the cycle's other rules
+are matched afresh each round, since what they match depends on how
+far it has got.
+
+- The rule needs an alternative that doesn't start with itself, its
+  base case (`term` above), and the alternatives that do start with it
+  come first. Tried first, `term` would win every round, so
+  `either(term, seq(expr, "-", term))` could never grow past `3`. Either
+  mistake in the rule's own `either` (not one nested inside a `seq`)
+  raises `ArgumentError` on the first parse, or `to_ruby`, saying which: ``in left-recursive rule :expr, the alternative `term`
+  comes before `expr "-" term`, so the rule can never grow past it``.
+  In a cycle, any of its rules counts as the rule itself: one of them
+  needs a base case, and in each, the alternatives that start with a
+  rule of the cycle come first.
+- `expr op expr`, with the rule on both sides, comes out
+  right-associative: `3-4-5` is `3-(4-5)`. The call on the right grows
+  to the end of the input before the left one can. Write `expr op term`.
+- A cycle is grown only at its leader, so a parse that enters it at
+  another rule gets that rule's match with the leader's grown one,
+  which may already have taken what that rule needed. In
+  `l ← p ".x" / "x"; p ← p "(n)" / l` (Medeiros et al., 2014), every
+  way round passes through `p`, so `p` leads even when the parse
+  starts at `l`; then `x.x` matches only `x`, since `p` has grown over
+  the `.x`. Starting at `p` parses it. Usually this doesn't come up: of
+  the rules that could lead, the one the grammar uses first does.
+- Cycles with no rule in common (`a` and `b` each call themselves and
+  each other) have nowhere to grow both from, so they raise
+  `ArgumentError` on the first parse, or `to_ruby`, naming them:
+  `left recursion through cycles with no rule in common isn't
+  supported: :a -> :a; :b -> :b`.
+- Left recursion through a custom matcher, which Birman can't see into,
+  is caught only when the parse gets there.
+
+## Checks
+
+An action can only return a value, and raising escapes `parse`, so an
+action can't turn a match down. `.check` can:
+
+```ruby
+rule(:date) { match(/\d{4}-\d\d-\d\d/).check(:real_date?, "a real date") }
+
+private
+
+def real_date?(text) = Date.valid_date?(*text.split("-").map(&:to_i))
+```
+
+The method (it may be private) gets the value and returns truthy to
+accept it; the value passes through unchanged. A falsy result is an
+ordinary failure, "expected a real date", where the match began, and
+whatever the inner match recorded (a `rep`'s stopping point, say) is
+dropped. (This example uses `Date` and `to_i`, so it won't go into a
+generated parser; see below.)
+
+- It backtracks like any failure. The furthest failure still wins, so
+  if a later alternative gets further (`1979` matching a number before
+  `-02-30` trips it up), that alternative owns the error.
+- Inside `rep` it ends the loop, and inside `maybe` it gives `nil`.
+- It commits only when an enclosing `seq` already has:
+  `seq("on", commit, date)` stops at the rejected date. To commit to the
+  check itself, look first: `pattern.present > pattern.check(...)`.
+- An outer `.label` at the same position replaces its label.
+- Like actions, the method must depend only on the value: it also runs
+  inside lookaheads, and its result is memoized with the rule.
+
+Some checks need more than one value: a name defined twice, a `break`
+outside a loop. Make those in a second pass over the AST, and report
+one at the node it's about with `Birman::Failure.at(node.span, reason)`.
+It's a `Failure` like `parse` returns, committed, with `reason` as
+its message:
+
+```ruby
+def self.check(program)
+  seen = {}
+  program.each do |let|
+    return Birman::Failure.at(let.span, "#{let.name} is already defined") if seen[let.name]
+
+    seen[let.name] = true
+  end
+  Birman::Success.new(program)
+end
+
+case Grammar.parse(source)
+in Birman::Success(program) then check(program)
+in failure then failure
+end
+# line 2, column 1: x is already defined
+#   let x = 2
+#   ^
+```
+
+## Without a grammar
+
+For a one-off pattern, skip the class: the combinators are on `Birman`
+too, and `Birman.parse` runs a matcher as a whole grammar.
+
+```ruby
+B = Birman
+date = B.seq(B.match(/\d{4}/), "-", B.match(/\d\d/)).map { _1.join }
+
+B.parse(date, "2026-10")    # => Success("2026-10")
+B.parse!(date, "26-10")     # raises Birman::ParseError
+
+B.parse(B.seq("let", B.match(/\w+/)), "let  x", ignore: /\s*/)
+# => Success(["let", "x"])
+```
+
+`parse` returns `Success`/`Failure` and never raises on bad input, as
+`Grammar.parse` does; `parse!` takes `formatter:` too. `ignore:` is a
+matcher (a regexp, a string or any matcher) rather than a rule name,
+since there's no grammar for a name to refer to; literals skip it, as
+in a grammar with `ignore`, but `str` and `match` stay raw, so skip
+spacing after them yourself if you need to. Matchers are values, so the grammar built
+for one is cached and reused for an equal one.
+
+To drop the `Birman.` prefix, mix `Birman::Combinators` into a class
+of your own: `extend` it for matchers held in constants, `include` it
+for matchers built by instance methods.
+
+```ruby
+class Dates
+  extend Birman::Combinators
+  DATE = seq(match(/\d{4}/), "-", match(/\d\d/))
+  def self.parse(s) = Birman.parse(DATE, s)
+end
+Dates.parse("2026-10")  # => Success(["2026", "-", "10"])
+
+class Inst
+  include Birman::Combinators
+  def date = seq(match(/\d{4}/).as(:y), "-", match(/\d\d/).as(:m))
+end
+Birman.parse(Inst.new.date, "2026-10")  # => Success({y: "2026", m: "10"})
+```
+
+The class is just a namespace, not a grammar: matchers have no `parse`
+of their own, so run them with `Birman.parse`, and bare strings skip
+spacing only when you pass `ignore:`.
+
+`rule` and `lexeme` aren't in `Combinators`, so anything named or
+recursive, and `Birman::Rules` (`number`, `identifier`, ...), still
+needs `include Birman::Grammar`; so do `keywords` and `.action`/`.check`
+with methods of your own (`large_integer` works). A rule reference,
+or an action or check naming a method of your own, raises
+`ArgumentError` before parsing starts.
+
 ## Nodes
 
 `node(:a, :b)` (in any grammar class, from `extend Birman::Nodes`, or
@@ -166,7 +410,8 @@ has one.
 - `Grammar.parse` returns `Success`/`Failure`; `parse!` raises `ParseError`
   (`parse!(input, formatter: f)` words it your way).
   Input nested too deeply for Ruby's stack is a `Failure` too, with
-  `reason` set (see Limits).
+  `reason` set (see Limits), and so is invalid UTF-8: `reason` is
+  "invalid UTF-8", at the first bad byte.
 - `Failure#message(formatter)` takes anything with `#call(failure)`
 - `Grammar.to_ast` and `Grammar.to_ebnf` for introspection
 - Rules are methods, so they compose and override through `include`.
@@ -175,8 +420,118 @@ has one.
   combinators; any class that includes it is a grammar. Settings
   (`root`, `ignore`, `keywords`) still go on the class.
 - A rule can't be named after a method the grammar relies on (`send`,
-  `class`, `match`, `token`...). Defining one raises `ArgumentError`
-  instead of silently breaking the grammar. Redefining a rule is fine.
+  `class`, `match`, `token`, or a private helper in one of Birman's
+  modules, like `Rules::Strings`'s `decode_escape`), nor after a public
+  method of your own. Defining one raises `ArgumentError` saying whose
+  method it would replace, instead of silently breaking the grammar.
+  Redefining a rule is fine, and so is replacing your grammar's own
+  private helpers.
+
+## Generating a parser
+
+`Grammar.to_ruby` writes a standalone parser: Ruby source with no
+dependency on Birman that gives exactly the results `parse` gives
+(values, spans, error positions, expected lists and messages).
+
+```ruby
+File.write("calc_parser.rb", Calc::Grammar.to_ruby(namespace: "CalcParser"))
+require_relative "calc_parser"
+CalcParser::Parser.new("print 1 + 2").parse   # => Success or Failure
+```
+
+The output is written in the Ruby subset that
+[Spinel](https://github.com/matz/spinel) compiles to a native binary.
+Add an entry point and compile it:
+
+```ruby
+source = Calc::Grammar.to_ruby(namespace: "CalcParser") + <<~RUBY
+  result = CalcParser::Parser.new($stdin.read).parse
+  puts result.success? ? result.value.inspect : result.message
+RUBY
+File.write("calc_parser.rb", source)  # then: spinel calc_parser.rb -o calc
+```
+
+Nodes come out as plain classes under `<namespace>::BirmanRT::Node`,
+with the same fields, `span`, `==` and `inspect`. Values transformed
+with `.map(:sym)`, `.action(:name)` and `.means(v)` compile, as does
+`.check(:name, "label")`: an action's (or check's) method is copied from
+its source, with the grammar methods it calls
+and the constants it reads (`Float::INFINITY` and `Float::NAN` too), so
+copying needs the `prism` gem (bundled from Ruby 3.3). So is a node's
+`initialize` that only gives fields defaults (see Nodes): each node the
+generated parser builds gets the default for a field its value leaves
+out, a new one each time, as in Ruby. The generator raises `Birman::Compiler::Error`, naming
+the rule, for what it can't translate faithfully:
+
+- `.map { }` blocks (their source is lost) and custom matcher classes
+- regexps using `\A`, `^`, lookbehind, or `\b`/`\B` at their start:
+  the generated parser scans from the current position, where those
+  would see less of the input than the interpreter's do
+- regexps with a wide `{m,n}` (more than 16 apart) alongside a
+  lookaround, atomic group, backreference or lazy interval: Spinel's
+  regexp engine tries every length on those, exponentially. Making the
+  repeat atomic on its own, `(?>[a-z]{1,63})`, is allowed
+- `.to(Node)` that may leave out a field with no default: named parts
+  that miss one, or one value (or a value that may or may not be a Hash
+  of fields) for a node with a field past its first that has none
+- node classes with methods of their own (a `node(...) { }` body),
+  except an `initialize` that takes exactly the fields as keywords,
+  gives some literal defaults (nil, true, false, numbers, strings,
+  symbols, and arrays and hashes of them) and only calls `super`, as
+  `def initialize(name:, args: []) = super` does; and
+  names that would clash in the generated code (a node called `Success`,
+  a copied constant called `FAIL`, two copied constants of one name,
+  rules `:a?` and `:a_3f`, which are both `a_3f` there)
+- actions that call `super`, Birman's own API (`keywords`, rules), or
+  classes the generated parser won't have, or that read a constant path
+  other than `Float::INFINITY` and `Float::NAN` (`Math::PI`; copy it
+  into a constant of the grammar)
+- actions that call `Integer(...)`, `.hex`, `.oct` or `.to_i` (on
+  anything but an Integer literal, since a receiver's type can't be
+  told), or name them (`map(&:to_i)`): past 63 bits, in a Spinel
+  binary they raise or, where Spinel can't tell the receiver is a
+  String, silently give 2**63 - 1. Use `large_integer(text)` or
+  `large_integer(text, base)` (`16` for `hex`, `8` for `oct`), which
+  Birman gives every grammar: it is `Integer(text, base)` in Ruby, and
+  in the generated parser it returns Bignums under Spinel too (from
+  Floats as well: exactly, as Ruby's `Integer` does). `.map(:to_i)` is
+  fine on a String, an optional String (`.maybe`; nil is 0) or an
+  Integer; `.map(:hex)`, `.map(:oct)`, and `.map(:to_i)` on anything
+  else are refused, as are `send` and `method` with a name that isn't
+  a literal symbol. This is a net for the common forms, not a proof:
+  `Float(text).round`, `text.to_f.to_int` and whatever else slips past
+  it may raise, or give a wrong number without a word. `large_integer`
+  takes Integer bases and Floats (truncated) as Ruby does; Rational and
+  NaN bases differ (`TypeError`, `FloatDomainError`), a base that isn't
+  a number raises `TypeError` as in Ruby 3.3+ (3.2's `Integer` ignores
+  it), and in a binary negative bases (Ruby's "prefix or this base")
+  raise
+
+```ruby
+def json_number(text) = text.match?(/[.eE]/) ? Float(text) : large_integer(text)
+```
+
+Actions and `.map(:name)` always run, as in the interpreter, even when
+nothing reads their value. The generated parser reads UTF-8 or binary input (other
+encodings raise `ArgumentError`). It also nests deeper before reporting
+"input is nested too deeply": its methods use less stack than the
+interpreter's matchers, and a Spinel binary has a 64 MB stack (JSON
+arrays nest about 55,000 deep), so input that fails with the interpreter
+can parse when compiled. One limit runs the other way: Spinel allows 64
+`begin` blocks active at once (and exits the process past that), and
+each lookahead (`absent`, `present`, `&`) holds one while its inner
+matcher runs. So lookaheads nested more than 56 deep are a Failure,
+"input is nested too deeply". To allow more, build with a bigger stack
+and say so when generating:
+
+```ruby
+File.write("p.rb", G.to_ruby(namespace: "P", lookahead_depth: 4000))
+# spinel p.rb -o p --cc="cc -DSP_EXC_STACK_MAX=4096"
+```
+
+Copied actions are exactly what you wrote. Under Spinel, a string that
+isn't valid UTF-8 (JSON's lone `\ud83d`) has the same bytes but
+`inspect`s differently.
 
 ## Limits
 
@@ -194,6 +549,12 @@ has one.
   That took the same grammar to about 12,000 levels on a thread. The
   main thread gets less (about 2,000), because its machine stack comes
   from the operating system (`ulimit -s`).
+- **Left recursion on both sides.** `expr op expr` groups to the right
+  (see [Left recursion](#left-recursion)); write `expr op term` for
+  left-associative operators.
+- **Entering a left-recursive cycle away from its leader** gets a
+  shorter match than growing from there would (see
+  [Left recursion](#left-recursion)).
 - **Changing rules after parsing.** `Grammar.parse` reuses one
   instance per class, so each rule is built once. Defining a rule or
   changing a setting on a class resets that class's instance, but a
@@ -203,10 +564,9 @@ has one.
 
 ## Not yet
 
-- Left recursion. A rule that can reach itself without consuming input
-  raises `ArgumentError` naming the rule; use `infix`, or
-  `term (op term)*` with a fold, instead.
-- Compiling grammars to Ruby methods for speed
+- Left recursion through cycles with no rule in common (see
+  [Left recursion](#left-recursion)); give them a rule in common, or
+  use `infix`.
 - Railroad diagrams from `to_ast`
 
 ## Development
@@ -214,9 +574,16 @@ has one.
 ```sh
 bin/setup           # install dependencies
 bundle exec rake    # tests + rubocop
+bundle exec rake spinel   # generated parsers compiled by Spinel vs the interpreter
+                          # (SPINEL=/path/to/spinel; skipped if not found)
 ruby examples/calc.rb
+ruby examples/leftcalc.rb
 ruby examples/json.rb
-ruby bench/json.rb  # vs JSON.parse, Parslet, Treetop, Citrus (installs them)
+ruby examples/email.rb
+ruby examples/toml.rb
+ruby examples/query.rb
+ruby bench/json.rb  # vs JSON.parse, Parslet, Treetop, Citrus (installs them);
+                    # also to_ruby, and its Spinel binary if Spinel is found
 ```
 
 ## License

data/lib/birman/ast.rb

@@ -7,7 +7,11 @@ module Birman
   #   [:seq, commit_at, [parts...]]  [:choice, [alts...]]
   #   [:rep, min, max, inner]  [:not, inner]  [:and, inner]
   #   [:both, left, right]  [:action, inner]  [:label, name, inner]
-  #   [:quiet, inner]
+  #   [:check, name, inner]  [:quiet, inner]  [:lexeme, inner]
+  #
+  # A lexeme is inner, then the grammar's ignore rule. Literals are
+  # lexemes too, but show as [:str, text]: the ignore rule runs only
+  # after the text, so only an empty literal shows as [:lexeme, ...].
   module AST
     module_function
 
@@ -27,6 +31,9 @@ module Birman
   #   a*  a+  a?  a{2,3}    !a  not      &a  lookahead / guard
   #
   # Actions and labels don't change what's accepted, so they're omitted.
+  # Checks do, but by running Ruby on the value, which EBNF can't show
+  # (`&{name}` would only repeat the error message), so they're omitted
+  # too: the EBNF accepts a superset of what the grammar does.
   module EBNF
     module_function
 
@@ -49,7 +56,9 @@ module Birman
       in [:ref, name]        then name.to_s
       in [:action, inner]    then expr(inner, prec)
       in [:label, _, inner]  then expr(inner, prec)
+      in [:check, _, inner]  then expr(inner, prec)
       in [:quiet, inner]     then expr(inner, prec)
+      in [:lexeme, inner]    then expr(inner, prec)
       in [:both, [:not, guard], right]
         wrap("!#{expr(guard, ATOM)} #{expr(right, PREFIX)}", prec > SEQ)
       in [:choice, alts]     then wrap(alts.map { expr(_1, SEQ) }.join(" / "), prec > CHOICE)

data/lib/birman/combinators.rb

@@ -57,7 +57,7 @@ module Birman
     # params = ["word: String"]
     # returns = "Matcher for word when not followed by a word character"
     # value = "word"
-    def keyword(word) = (str(word) >> match(/\w/).absent).map(&:first).label(word.inspect)
+    def keyword(word) = Pick.new([str(word), match(/\w/).absent], 0).label(word.inspect)
 
     # --- Literals and the readable forms ---
 
@@ -70,8 +70,8 @@ module Birman
     def lit(x) = Birman.coerce(x)
 
     # The matcher a literal stands for: "let" is a keyword, "=" an exact
-    # string, either way followed by the grammar's ignore rule. Built
-    # once per grammar instance.
+    # string, either way followed by the ignore rule of the grammar
+    # running it. Built once per grammar instance.
     # [method.literal_matcher]
     # params = ["text: String"]
     # returns = "The matcher a literal stands for: keyword or exact string, then the ignore rule"
@@ -80,7 +80,7 @@ module Birman
     def literal_matcher(text)
       (@literal_matchers ||= {})[text] ||= begin
         base = text.match?(/\A\w+\z/) ? keyword(text) : str(text)
-        Lexeme.new(base, ignore_rule, text)
+        Lexeme.new(base, text)
       end
     end
 
@@ -90,7 +90,8 @@ module Birman
     # params = ["*items: String | Regexp | Matcher | commit | only(...)"]
     # returns = "Matcher for the items in order"
     # value = "Array of item values; a Hash of the .as-named parts if any; the only(...) part's value if used"
-    # raises = ["ArgumentError (commit first; more than one only; only mixed with .as names)"]
+    # raises = ["ArgumentError (commit first; more than one only; only mixed with .as names)",
+    #           "ArgumentError (a name used twice; a nested seq's names the value would drop)"]
     def seq(*items)
       commit_at = items.index(COMMIT)
       raise ArgumentError, "commit can't come first in a seq: nothing has matched yet" if commit_at&.zero?
@@ -118,7 +119,7 @@ module Birman
     # [method.either]
     # params = ["*alts: String | Regexp | Matcher"]
     # returns = "Ordered choice: the first alternative that matches wins"
-    # value = "The winning alternative's value"
+    # value = "The winning alternative's value (a name all the alternatives share passes outward)"
     def either(*alts) = alts.size == 1 ? lit(alts.first) : Choice.new(alts.map { lit(_1) })
 
     # One of several literals. Given a hash, each literal produces its
@@ -131,7 +132,7 @@ module Birman
     def one_of(*choices)
       return either(*choices) unless choices in [Hash => table]
 
-      either(*table.map { |text, value| lit(text).map { value } })
+      either(*table.map { |text, value| lit(text).means(value) })
     end
 
     # Left-associative binary operators, folded into nodes:
@@ -144,18 +145,21 @@ module Birman
     def infix(operand, *ops, node:) = Infix.new(Birman.coerce(operand), either(*ops).label("operator"), node)
 
     # One or more items separated by sep; value is an array of items.
+    # With trailing: true, one separator may follow the last item:
+    #   seq("[", only(sep_by(value, ",", trailing: true)), "]")   # [1, 2,]
     # [method.sep_by1]
-    # params = ["item: String | Regexp | Matcher", "sep: String | Regexp | Matcher"]
-    # returns = "Matcher for one or more items separated by sep"
+    # params = ["item: String | Regexp | Matcher", "sep: String | Regexp | Matcher", "trailing: Boolean = false"]
+    # returns = "Matcher for one or more items separated by sep (and, if trailing, maybe one more sep)"
     # value = "Array of item values (item's .as name passes outward)"
-    def sep_by1(item, sep) = SepBy.new(lit(item), lit(sep))
+    def sep_by1(item, sep, trailing: false) = SepBy.new(lit(item), lit(sep), 1, trailing)
 
-    # Zero or more items separated by sep.
+    # Zero or more items separated by sep. A trailing separator needs an
+    # item before it: "," alone isn't an empty list.
     # [method.sep_by]
-    # params = ["item: String | Regexp | Matcher", "sep: String | Regexp | Matcher"]
-    # returns = "Matcher for zero or more items separated by sep"
+    # params = ["item: String | Regexp | Matcher", "sep: String | Regexp | Matcher", "trailing: Boolean = false"]
+    # returns = "Matcher for zero or more items separated by sep (and, if trailing, maybe one more sep)"
     # value = "Array of item values, [] when there are none"
-    def sep_by(item, sep) = sep_by1(item, sep).maybe.map { _1 || [] }
+    def sep_by(item, sep, trailing: false) = SepBy.new(lit(item), lit(sep), 0, trailing)
 
     # inner wrapped in open/close; value is inner's.
     # [method.between]
@@ -180,4 +184,11 @@ module Birman
 
     def ignore_rule = self.class.respond_to?(:ignore) ? self.class.ignore : nil
   end
+
+  # Birman.seq, Birman.str and the rest, for building matchers outside a
+  # grammar (see Birman.parse). literal_matcher stays private: it's what
+  # a grammar runs for a literal, and Birman isn't one.
+  extend Combinators
+
+  singleton_class.send(:private, :literal_matcher)
 end