ibex 0.1.0 → 0.2.0

data/docs/racc-migration.md

@@ -5,11 +5,14 @@ copy racc's internal table arrays, internal method names, native runtime, or gen
 
 ## Typical migration
 
-1. Run `ibex -o parser.rb grammar.y` in place of `racc -o parser.rb grammar.y`.
-2. Change the generated-file runtime dependency from deployment packaging only; application calls to `do_parse`, `yyparse`,
+1. Run `ibex migrate-check grammar.y` and resolve any errors. Use `--format=json` for CI.
+2. Generate a reviewable parity harness with `ibex migrate-harness -o migration_harness.rb grammar.y`, add explicit token
+   cases, and run it inside an isolation boundary appropriate for the grammar code.
+3. Run `ibex -o parser.rb grammar.y` in place of `racc -o parser.rb grammar.y`.
+4. Change the generated-file runtime dependency from deployment packaging only; application calls to `do_parse`, `yyparse`,
    `next_token`, `on_error`, `token_to_str`, `yyerror`, `yyerrok`, and `yyaccept` remain the same.
-3. Use `-E` if the generated parser must be a single file with no installed Ibex gem.
-4. Keep the default `--mode=racc` until intentionally adopting EBNF or names. Extended grammars can make that choice locally by
+5. Use `-E` if the generated parser must be a single file with no installed Ibex gem.
+6. Keep the default `--mode=default` until intentionally adopting EBNF or names. Extended grammars can make that choice locally by
    placing `pragma extended` immediately after their class header instead of requiring `--mode=extended` at each invocation.
 
 ## CLI mapping
@@ -23,6 +26,7 @@ copy racc's internal table arrays, internal method names, native runtime, or gen
 | `-E`, `--embedded` | Embed the Pure Ruby runtime |
 | `-F`, `--frozen` | Accepted; Ibex always emits frozen-string magic comments |
 | `--rbs[=FILE]` | Ibex extension; emit a generated parser signature |
+| `--action-source[=FILE]` | Ibex extension; emit a non-executable Steep shadow of semantic methods |
 | `--warnings=all,error` | Ibex extension; display or promote structured grammar diagnostics |
 | `--line-convert-all`, `-l` | Map header/inner/footer too, or disable all source mapping |
 | `-a` | Generate methods for implicit actions |
@@ -34,6 +38,32 @@ copy racc's internal table arrays, internal method names, native runtime, or gen
 Ibex defaults to `<input>.rb`; racc 1.8.1 was observed to default to `<input>.tab.rb`. Use `-o` for portable scripts.
 By default, semantic actions and `inner` methods report grammar-file lines while `header` and `footer` retain generated-file
 lines. `--line-convert-all` maps every user-code section; `-l` maps none. The mapping is retained through IR JSON resumption.
+For static action checking, combine `--rbs --action-source`; configure the generated `.rbs` and `.actions.rb` in the
+application's Steep target, and run Steep separately. The shadow is check-only input and must not replace or be required by the
+runtime parser.
+
+`migrate-check` never executes semantic actions or `header`/`inner`/`footer` code. `migrate-harness` also only writes source. The
+generated harness is the explicit execution step: after reviewed cases are added, it invokes both generators and executes both
+generated parsers in bounded child processes. This is not a sandbox; use a container or VM for untrusted grammar code. See
+[ADR 0012](decisions/0012-bounded-nonexecuting-analysis.md).
+
+## Compatibility baseline
+
+The compatibility claims above were checked on 2026-07-22 against racc 1.8.1 using only its public documentation, `racc --help`,
+and black-box execution. Ibex does not inspect racc implementation files or generated source. Self-authored probes compare
+observable results for arithmetic precedence, empty rules, string tokens, `convert`, `no_result_var`, inline actions,
+dangling-else `expect`, error recovery, source-line conversion, and a generated 500-production grammar.
+
+Precedence-resolved conflicts remain visible in Automaton IR but are excluded from the CLI conflict count and `expect`. Error
+recovery probes compare result values and the public `on_error` arguments. These tests skip when the `racc` executable is absent.
+
+Application-defined parser initializers do not have to call `super`; the runtime completes its isolated session state on first
+use while retaining application-owned instance variables. During parsing, the historical `@vstack` and `@racc_vstack` names
+both refer to the live semantic-value stack for read compatibility. They are internal aliases: applications may inspect them
+while tokenizing but must not mutate, replace, or retain them across parser sessions.
+
+An unqualified `ParseError` in a parser method or semantic action resolves through `Runtime::Parser`, so application actions can
+keep their existing explicit rejection path. Structured failures remain available as `Ibex::Runtime::ParseError`.
 
 ## Known differences
 
@@ -43,5 +73,3 @@ lines. `--line-convert-all` maps every user-code section; `-l` maps none. The ma
   unknown lookahead through `on_error` first, then recovers if the callback returns. Declared invalid tokens match in the
   black-box recovery probe.
 - `require "racc/parser"` replacement and previously generated racc parser table compatibility are out of scope.
-
-See [compatibility observations](compat-notes.md) for the tested version and probe set.

data/docs/release-readiness.md

@@ -0,0 +1,163 @@
+# v1.0 readiness report
+
+This is the outcome-based v1.0 decision required by the project design. It
+records measurements taken on 2026-07-27 with Ruby 4.0.0 on
+`arm64-darwin24`. Passing the implementation checklist is not sufficient for a
+stable release.
+
+## Decision
+
+**HOLD v1.0.** Compatibility behavior, scale construction, semantic-value
+signatures, and the repository compatibility suite have evidence. The external
+generator/runtime performance target is not met, and the published ten-case
+error comparison has not received independent third-party review.
+
+The feature freeze remains in force while those two release gates are open.
+The format-v6 batch Red/Green CST has been selected as part of the initial
+Stable v1 API. Its syntax-only incremental layer remains Experimental. Other Preview and
+Experimental features may continue to ship in prereleases without promotion.
+
+## KPI result
+
+| KPI | Result | Evidence |
+|---|---|---|
+| Three public-gem migrations | Pass with documented adapters | Namae, BCDice, and Nokogiri behavior suites below |
+| Hundreds-of-productions scale | Pass | 501 productions, 503 states, 125.008 ms average complete build |
+| Ten-case error UX comparison | Partial | 10/10 snapshots and 8/10 useful repairs are public; independent review is missing |
+| Generator and parser performance at least racc | Fail | Reuse medians are at parity, but only Nokogiri's interval passes; all cold-generation and new-instance rows fail |
+| Semantic-value RBS and typed ratchet | Pass | Generated reduction signatures include declared RHS/LHS types; whole-library Steep is 89.6% typed |
+| Compatibility suite unbeaten | Pass at the measured revision | Current black-box, self-host, IR, property, and runtime suites are green |
+
+## Public-gem migration evidence
+
+Only public grammar files, public commands, and observable test results were
+used. Neither racc implementation files nor generated source were inspected.
+Each repository was cloned at the recorded revision:
+
+| Project | Revision and grammar | Static check | Behavior result |
+|---|---|---|---|
+| Namae | [`d33875a`](https://github.com/berkmancenter/namae/commit/d33875aaf1fc420a8dfe946a3b29cc3e19710061), `lib/namae/parser.y` | compatible | 151 RSpec examples, 85 Cucumber scenarios, 198 steps |
+| BCDice | [`21b4a03`](https://github.com/bcdice/BCDice/commit/21b4a03789bf2080ad41aaf31299b609ee7bda86), `lib/bcdice/command/parser.y` | compatible; runtime-constant warning | 25 tests, 103 assertions |
+| Nokogiri | [`04a4c29`](https://github.com/sparklemotion/nokogiri/commit/04a4c29c6a605ad40a78f4ce343ced0832a1805c), `lib/nokogiri/css/parser.y` | compatible; runtime-constant warning | 6 tests, 14 assertions |
+
+Namae generated and ran as one embedded file. The other two grammars contain a
+user header whose superclass is named `Racc::Parser`. Their PoCs preload this
+explicit migration adapter before loading the generated parser:
+
+```ruby
+require "ibex/runtime"
+
+module Racc
+  Parser = Ibex::Runtime::Parser
+end
+```
+
+This is the adapter recommended by `ibex migrate-check`; it does not load or
+execute the racc runtime. The PoCs uncovered and fixed three compatibility
+gaps: an omitted `end` before a user-code section, an application initializer
+that omits `super` plus historical value-stack reads, and an unqualified
+`ParseError`.
+
+## Formal Pure Ruby performance comparison
+
+The formal artifact was collected from clean revision `3fda78c` with ten
+alternating isolated runs per implementation, 50 warm-up workloads, 250
+measured workloads, and 10,000 bootstrap samples. Each workload parses the
+five fixed public inputs. Ruby 4.0.0 ran with YJIT disabled; Racc 1.8.1 was
+forced to its Ruby backend and every worker verified the selected runtime.
+Ratios below are Ibex divided by Racc. Bold values meet the gate, whose timing
+and allocation metrics require the bootstrap interval's upper bound to be no
+greater than 1.0.
+
+| Project | Cold generation time | Reuse time | Reuse allocations | New-instance time | New-instance allocations | Generated bytes |
+|---|---:|---:|---:|---:|---:|---:|
+| Namae | 1.194x | 0.994x | **0.849x** | 1.185x | 1.208x | **0.862x** |
+| BCDice command | 1.204x | 0.995x | **0.666x** | 1.258x | 1.129x | **0.981x** |
+| Nokogiri CSS | 1.187x | **0.953x** | **0.662x** | 1.167x | 1.075x | **0.937x** |
+
+Result values and ordered result sequences are equivalent in every runtime
+row. The measured parser-time medians in milliseconds per parse were
+0.038783/0.039015 (Ibex/Racc) for Namae reuse,
+0.019542/0.019646 for BCDice reuse, and
+0.024074/0.025269 for Nokogiri reuse. The corresponding new-instance medians
+were 0.047493/0.040076, 0.025937/0.020618, and 0.031041/0.026609.
+
+The remaining runtime gap is isolated to parser-instance construction and the
+per-instance mutation tracker; the reused hot loop's point estimates are at
+practical parity, although Namae and BCDice still need narrower passing
+intervals. Cold generation remains 18.7–20.4% slower and is dominated by
+process/load, frontend, and code-generation setup. These are evidence for the
+release decision, not portable scores.
+
+## Superseded native-backend diagnostic
+
+The previous artifact was collected from clean revision `53aa4ab` with ten
+alternating isolated runs per implementation, 50 warm-up workloads, 250
+measured workloads, and 10,000 bootstrap samples. Each workload parses the
+five fixed public inputs. Ruby 4.0.0 ran with YJIT disabled; racc 1.8.1 was
+verified to use its native backend. The formal release contract now compares
+the two Pure Ruby runtimes, so these stricter native-backend measurements are
+retained only as diagnostic context and do not decide the KPI. Ratios below
+are Ibex divided by racc.
+
+| Project | Cold generation time | Reuse time | Reuse allocations | New-instance time | New-instance allocations | Generated bytes |
+|---|---:|---:|---:|---:|---:|---:|
+| Namae | 1.210x | 1.177x | **0.970x** | 1.408x | 1.376x | **0.862x** |
+| BCDice command | 1.211x | 1.943x | **0.852x** | 2.445x | 1.430x | **0.981x** |
+| Nokogiri CSS | 1.236x | 1.562x | **0.881x** | 1.993x | 1.418x | **0.937x** |
+
+Result values and ordered result sequences are equivalent in every runtime
+row. The measured parser-time medians in milliseconds per parse were
+0.038938/0.033094 (Ibex/racc) for Namae reuse,
+0.020118/0.010354 for BCDice reuse, and
+0.024413/0.015626 for Nokogiri reuse. The corresponding new-instance medians
+were 0.046953/0.033340, 0.026933/0.011016, and 0.030744/0.015426.
+
+Profiles attribute the remaining reuse gap to the Pure Ruby direct LR loop
+after lexer and semantic-action costs. New-instance rows additionally include
+the per-instance tracker that preserves runtime hook mutation semantics.
+Cold-generation profiles are dominated by process/load cost plus frontend and
+code-generation work. These are diagnostic optimization evidence, not
+portable scores or formal release evidence.
+
+## Scale evidence
+
+Run:
+
+```sh
+benchmark/scale.rb --rules 500 --iterations 3
+```
+
+The reproducible synthetic chain shares its grammar shape with the 500-rule
+black-box compatibility test. The measured report contains 501 productions,
+503 direct-LALR construction states, 503 final states, no conflicts, and
+41,332 bytes of compact generated Ruby. Three complete parse-to-codegen builds
+averaged 125.008 ms with a 122.742–127.002 ms range. The Automaton IR digest
+was `fe0e2f0c9bf16e7aa22a7ebcea40b229fedbb573f83af37596b76e0b989c3338`.
+
+This synthetic result establishes the requested scale bound. The 139-production
+representative grammar remains the realistic workload, so neither result is
+presented as a substitute for an application-specific benchmark.
+
+## Error UX and type evidence
+
+[`error-ux.md`](error-ux.md) and its versioned JSON publish the same ten
+malformed JSON inputs against the public racc callback. Eight selected repairs
+were assessed useful. A maintainer assessment is not independent review; at
+least one external reviewer must record a review of the cases and usefulness
+labels before this KPI passes.
+
+The current whole-library `steep stats` result is 18,115 typed calls and 2,097
+untyped calls out of 20,212, or 89.6% typed. Generated parser RBS refines
+declared terminal, nonterminal, RHS tuple, and reduction result types. Untyped
+values remain explicit at undeclared grammar symbols, decoded JSON, dynamic
+table cells, and opaque application Ruby boundaries.
+
+## Actions required to release v1.0
+
+1. Optimize cold generation and parser-instance construction, then repeat the
+   formal Pure Ruby measurements until no row is slower than Racc.
+2. Obtain and link an independent review of the ten error cases and usefulness
+   judgments; revise the versioned assessment if the review disagrees.
+3. Re-run every compatibility, IR, type, benchmark, and site gate on the exact
+   release revision.

data/docs/stability.md

@@ -0,0 +1,124 @@
+# Stability, compatibility, and deprecation
+
+Ibex separates support level from activation. Opt-in controls do not determine
+maturity: a documented opt-in API can be Stable, Preview, or Experimental.
+
+## Maturity ladder
+
+| Level | Activation | Guarantee |
+|---|---|---|
+| Stable | Default compatible mode or a documented stable API | Semantic versioning; compatible-mode behavior remains unchanged |
+| Preview | Explicit pragma, mode, command, or algorithm | Breaking changes require notice one minor release in advance |
+| Experimental | Explicit policy/object or research entry point | May change without notice; budgets and failure modes are part of the experiment |
+
+After v1.0, promotion requires representative use in a shadow or gallery
+grammar, feature-specific invariant/property tests, two released versions
+without a specification change, dependent-tool support where applicable, and
+complete public documentation. For the initial v1.0 contract, accepted ADRs,
+versioned benchmark evidence, invariant/property coverage, dependent-tool
+support where applicable, and complete public documentation replace the
+impossible two-prior-release requirement. Only Stable features may be adopted
+by the production self-hosted grammar.
+
+## v1 inventory
+
+Stable:
+
+- racc-compatible grammar input and generated `do_parse`/`yyparse` runtime;
+- default direct LALR construction, parser tables, recovery callbacks,
+  observation events, resource limits, migration checks, and bounded
+  counterexample/ambiguity analysis;
+- versioned core Grammar IR, Automaton IR, Lexer IR, table formats, report
+  schemas, and their validators;
+- format-v6 Red/Green batch CST parsing, typed syntax views, persistent editing
+  and diffing, and the closed `ibex_cst` schema v1 serialization contract.
+
+Preview:
+
+- `pragma extended` / `--mode=extended`, including EBNF groups, parameterized
+  and inline rules, middle actions, multiple entries, canonical imports,
+  generated lexers, semantic locations/types, AST generation, grammar tests,
+  and documentation tooling;
+- the conservative `--algorithm=ielr` backend;
+- LSP, watch, debug, coverage, browser playground, and static action-shadow
+  integration.
+
+Experimental:
+
+- opt-in bounded insertion/deletion/replacement repair through
+  `Runtime::RepairPolicy`;
+- syntax-only incremental CST sessions and conservative Blender subtree reuse.
+
+The batch CST contract is selected for the initial v1 API under the
+initial-major evidence rule above. The remaining Preview features have not
+completed the normal two-release field period. Repair passed its usefulness
+spike but remains Experimental because two of ten baseline plans were not
+useful and inserted semantic values may be nil.
+
+## Research decisions
+
+- IELR passed its correctness/state-bound spike and remains preview while it
+  gains field experience. Direct LALR remains the default.
+- Bounded repair passed SP-4 with 8/10 useful plans and remains experimental.
+- GLR and `%dprec`/`%merge` did not enter the product: no delayed-action spike
+  met the ≤5% deterministic-overhead and ambiguity-policy gates.
+- Syntax-only incremental parsing entered as experimental after fixed-seed
+  structural edits matched fresh Green trees and the representative benchmark
+  measured 1.04–2.83x over Stage A and 1.66–4.48x over fresh syntax sessions.
+  It remains experimental until it completes the two-release field period.
+
+The supported alternatives are canonical LR(1) or IELR for LALR inadequacy,
+bounded ambiguity/counterexample analysis for ambiguous grammars, and
+`--watch` with full deterministic reparsing for editing workflows.
+
+## Core IR freeze
+
+The required fields, meanings, identity rules, ordering, and validation
+semantics of the published core Grammar IR and Automaton IR versions are
+frozen. Existing version-1 and version-2 documents retain byte-stable
+round-trips and one-generation read compatibility.
+
+Additive optional core fields require a minor release. Meaning changes,
+required-field changes, or removals require a new major schema version and
+continued reading of the immediately preceding major version.
+
+The `x-` namespace is reserved for future experimental data and is not frozen.
+Current closed schemas intentionally reject unknown fields and do not emit
+`x-` data. Introducing an experimental envelope therefore requires a new
+additive schema version; it cannot silently place fields into an existing
+closed document. Promotion moves reviewed data into a documented core field in
+a later schema version.
+
+## Compatibility policy
+
+Compatible mode is the permanent default. Opt-in extensions, exact lookahead
+defaults, repair, and research algorithms do not silently replace compatible
+behavior. Public migration evidence uses public commands and black-box
+behavior; implementation and generated-source layouts are not compatibility
+surfaces.
+
+An undeclared invalid token intentionally calls `on_error` before ordinary yacc
+recovery. This is the documented recommended behavior and is not changed by
+the freeze.
+
+Parser-table formats v1 through v6 remain readable for non-CST parsers. CST
+tables must use the current format v6 structured metadata; older CST tables
+and the boolean `cst: true` shape fail before token consumption with a
+regeneration instruction. Format v6 is the only writer and does not change
+Grammar IR v2. The closed `ibex_cst` schema v1 is a versioned interchange
+contract. Additive meaning requires a new schema version; readers do not
+accept unknown fields.
+
+## Deprecation policy
+
+After v1.0, a Stable API or syntax first emits a migration warning for at least
+two minor releases. The release notes and documentation must name the first
+warning release, replacement, migration command or procedure, and earliest
+removal release. Automated migration is supplied when practical; policy does
+not promise a command that the product does not provide. Preview features
+receive at least one minor release of notice, and Experimental features may
+change without notice.
+
+The pre-v1 mixed semantic/syntax CST was a Preview contract and is removed
+while selecting the initial stable API. Its parser tables are rejected with a
+regeneration instruction. There are no Stable removals scheduled.

data/examples/README.md

@@ -0,0 +1,60 @@
+# Ibex examples
+
+These examples pair an Ibex grammar with either the declarative lexer DSL or a
+small handwritten lexer. Generate any example from the repository
+root, then run the generated file:
+
+```sh
+bundle exec ruby -Ilib exe/ibex examples/calculator.y
+bundle exec ruby -Ilib examples/calculator.rb "2 + 3 * (4 - 1)"
+```
+
+The generated `.rb` file is disposable. The `.y` file is the maintained
+source.
+
+Each grammar also keeps accept/reject source examples beside its rules. Run all
+of them without generating repository files. The task also requires 100%
+production coverage from the declared cases:
+
+```sh
+bundle exec rake grammar:test
+```
+
+## Included grammars
+
+- `calculator.y` evaluates integer arithmetic with parentheses.
+- `csv.y` parses rows and quoted fields with a generated lexer.
+- `json.y` parses JSON values into Ruby Hashes, Arrays, Strings, numbers,
+  booleans, and `nil`. It demonstrates the generated lexer, including conversion
+  actions and punctuation emission.
+- `ini.y` parses sections and key/value entries into a nested Hash.
+- `tiny_language.y` parses assignments, arithmetic, and `print` statements,
+  then executes the resulting small AST.
+
+The JSON, INI, and tiny-language runners read standard input:
+
+```sh
+printf '%s\n' '{"name":"Ibex","values":[1,true,null]}' |
+  bundle exec ruby -Ilib examples/json.rb
+```
+
+## Lexer integration patterns
+
+The JSON example declares `lexer ... end`; generated `parse` accepts String,
+IO, and Fiber input. Rules are anchored, choose the longest match, and use
+declaration order for ties.
+
+Handwritten lexers remain useful when an application already owns tokenization.
+They implement `next_token`, returning `[token, semantic_value]`; `false` or
+`nil` marks EOF. The remaining examples demonstrate two `StringScanner`
+patterns:
+
+1. calculator and tiny language scan the source incrementally and
+   return one token per `next_token` call;
+2. INI tokenizes line-oriented records into a small queue before parsing.
+
+Quoted grammar terminals such as `'+'` are returned as Strings. Declared bare
+tokens such as `NUMBER` are returned as Symbols. A production lexer can retain
+offset or line information alongside its own semantic values and use that
+information in application diagnostics; these examples keep the contract
+small enough to read in one file.

data/examples/calculator.y

@@ -0,0 +1,44 @@
+class Examples::CalculatorParser
+pragma extended
+token NUMBER
+%test accept "1+2*3"
+%test accept "8-3"
+%test accept "8/2"
+%test accept "(1)"
+%test reject "1+"
+rule
+  expression : expression '+' term { result = val[0] + val[2] }
+             | expression '-' term { result = val[0] - val[2] }
+             | term { result = val[0] }
+  term       : term '*' factor { result = val[0] * val[2] }
+             | term '/' factor { result = val[0] / val[2] }
+             | factor { result = val[0] }
+  factor     : NUMBER { result = val[0] }
+             | '(' expression ')' { result = val[1] }
+end
+---- header
+require "strscan"
+---- inner
+def parse(source)
+  @scanner = StringScanner.new(source)
+  do_parse
+end
+
+def next_token
+  @scanner.skip(/\s+/)
+  return false if @scanner.eos?
+
+  if (number = @scanner.scan(/\d+/))
+    [:NUMBER, Integer(number, 10)]
+  elsif (operator = @scanner.scan(/[()+*\/-]/))
+    [operator, nil]
+  else
+    raise ArgumentError, "unexpected calculator input at offset #{@scanner.pos}"
+  end
+end
+---- footer
+if $PROGRAM_NAME == __FILE__
+  abort "usage: ruby calculator.rb EXPRESSION" if ARGV.empty?
+
+  puts Examples::CalculatorParser.new.parse(ARGV.join(" "))
+end

data/examples/csv.y

@@ -0,0 +1,32 @@
+class Examples::CSVParser
+pragma extended
+token FIELD NEWLINE
+%test accept "name,age\n\"Doe, Jane\",36\n"
+%test reject "name,age"
+lexer
+  FIELD /"(?:[^"]|"")*"|[^,\r\n]+/ { |source| decode_csv_field(source) }
+  NEWLINE /\r\n|\n|\r/
+  on /,/ { |source| emit source, nil }
+end
+rule
+  document : rows { result = val[0] }
+  rows     : rows row { result = val[0] + [val[1]] }
+           | row { result = [val[0]] }
+  row      : fields NEWLINE { result = val[0] }
+  fields   : fields ',' field { result = val[0] + [val[2]] }
+           | field { result = [val[0]] }
+  field    : FIELD { result = val[0] }
+end
+---- inner
+def parse(source, file: "(csv)") = super
+
+def decode_csv_field(source)
+  return source unless source.start_with?('"')
+
+  source[1...-1].gsub('""', '"')
+end
+---- footer
+if $PROGRAM_NAME == __FILE__
+  require "json"
+  puts JSON.generate(Examples::CSVParser.new.parse(ARGF.read))
+end

data/examples/ini.y

@@ -0,0 +1,70 @@
+class Examples::INIParser
+pragma extended
+token SECTION KEY VALUE NEWLINE
+%test accept "[app]\nname=ibex\n"
+%test accept "\n"
+rule
+  document : lines { result = build_document(val[0]) }
+  lines    : lines line { result = val[1] ? val[0] + [val[1]] : val[0] }
+           | { result = [] }
+  line     : SECTION NEWLINE { result = [:section, val[0]] }
+           | KEY '=' VALUE NEWLINE { result = [:entry, val[0], val[2]] }
+           | NEWLINE { result = nil }
+end
+---- header
+require "strscan"
+---- inner
+def parse(source)
+  @tokens = tokenize_ini(source)
+  do_parse
+end
+
+def next_token
+  @tokens.shift || false
+end
+
+def tokenize_ini(source)
+  source.lines.flat_map.with_index(1) do |line, line_number|
+    scanner = StringScanner.new(line.chomp)
+    scanner.skip(/\s*/)
+    if scanner.eos? || scanner.peek(1).match?(/[;#]/)
+      [[:NEWLINE, nil]]
+    elsif scanner.scan(/\[/)
+      name = scanner.scan(/[^\]]+/)&.strip
+      closer = scanner.scan(/\]/)
+      scanner.skip(/\s*/)
+      invalid_ini_line!(line_number) unless name && !name.empty? && closer && scanner.eos?
+      [[:SECTION, name], [:NEWLINE, nil]]
+    else
+      key = scanner.scan(/[A-Za-z0-9_.-]+/)
+      scanner.skip(/\s*/)
+      separator = scanner.scan(/=/)
+      scanner.skip(/\s*/)
+      value = scanner.rest.strip
+      invalid_ini_line!(line_number) unless key && separator
+      [[:KEY, key], ["=", nil], [:VALUE, value], [:NEWLINE, nil]]
+    end
+  end
+end
+
+def invalid_ini_line!(line_number)
+  raise ArgumentError, "invalid INI input on line #{line_number}"
+end
+
+def build_document(records)
+  document = {}
+  current = document
+  records.each do |record|
+    if record[0] == :section
+      current = (document[record[1]] ||= {})
+    else
+      current[record[1]] = record[2]
+    end
+  end
+  document
+end
+---- footer
+if $PROGRAM_NAME == __FILE__
+  require "json"
+  puts JSON.generate(Examples::INIParser.new.parse(ARGF.read))
+end

data/examples/json.y

@@ -0,0 +1,48 @@
+class Examples::JSONParser
+pragma extended
+token STRING NUMBER TRUE FALSE NULL
+%test accept "{\"ok\":[true,null]}"
+%test accept "{\"object\":{},\"array\":[],\"string\":\"ibex\",\"number\":1,\"false\":false,\"many\":[1,2]}"
+%test reject "{\"ok\":}"
+lexer
+  skip /\s+/
+  STRING /"(?:\\(?:["\\\/bfnrt]|u[0-9a-fA-F]{4})|[^"\\])*"/ { |source| decode_json_string(source) }
+  NUMBER /-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?/ {
+    |source| source.match?(/[.eE]/) ? Float(source) : Integer(source, 10)
+  }
+  TRUE /true/ { true }
+  FALSE /false/ { false }
+  NULL /null/ { nil }
+  on /[{}\[\],:]/ { |source| emit source, nil }
+end
+rule
+  document : value { result = val[0] }
+  value    : object { result = val[0] }
+           | array { result = val[0] }
+           | STRING { result = val[0] }
+           | NUMBER { result = val[0] }
+           | TRUE { result = true }
+           | FALSE { result = false }
+           | NULL { result = nil }
+  object   : '{' '}' { result = {} }
+           | '{' members '}' { result = val[1] }
+  members  : pair { result = { val[0][0] => val[0][1] } }
+           | members ',' pair { result = val[0].merge(val[2][0] => val[2][1]) }
+  pair     : STRING ':' value { result = [val[0], val[2]] }
+  array    : '[' ']' { result = [] }
+           | '[' elements ']' { result = val[1] }
+  elements : value { result = [val[0]] }
+           | elements ',' value { result = val[0] + [val[2]] }
+end
+---- header
+require "json"
+---- inner
+def parse(source, file: "(json)") = super
+
+def decode_json_string(source)
+  JSON.parse(source)
+end
+---- footer
+if $PROGRAM_NAME == __FILE__
+  puts JSON.generate(Examples::JSONParser.new.parse(ARGF.read))
+end