flexr 1.0.0

data/docs/explanation/runtime-vs-generated.md

@@ -0,0 +1,22 @@
+# Runtime versus generated
+
+Both modes begin with the same DSL concepts and compile to the same IR-shaped
+rule and state model, but they serve different operational needs.
+
+| Concern | Runtime | Generated |
+|---|---|---|
+| Specification | Evaluated by Ruby | Read statically with Prism by default |
+| DFA construction | First class use | Build time |
+| Dynamic pattern methods | Available | Requires trusted `--eval` |
+| Startup cost | Includes compilation | Loads compiled payload |
+| Action code | Ruby block | Ruby source preserved in artifact |
+| Prism at runtime | Not needed | Not needed after generation |
+
+Keeping actions as Ruby is deliberate: semantic work belongs in the user's
+action, while flexr owns rule selection and token boundaries. It also means
+that generated files are executable code and must be treated as trusted build
+artifacts.
+
+The generated header and payload digest make artifact provenance visible. The
+repository's mode-equivalence checks execute each example in both modes and
+compare token streams.

data/docs/explanation/security-model.md

@@ -0,0 +1,18 @@
+# Security model
+
+flexr is a code generator, not a sandbox. A specification is ordinary Ruby and
+its actions can perform any operation available to the Ruby process. Generated
+files retain those actions as Ruby code.
+
+Static generation parses the specification and evaluates only a constrained set
+of literal and constant forms. This reduces accidental build-time execution but
+does not make the generated artifact safe to load if its source is untrusted.
+
+`--eval` is an explicit execution boundary: it evaluates the full specification
+to discover dynamic patterns. Use it only with trusted files and a trusted
+environment. The same rule applies to generated output: review or sign the
+artifact before loading it in a process with sensitive access.
+
+The runtime's `max_token_size` and `max_state_stack` limits provide resource
+guards for input processing. They are operational controls, not a security
+sandbox and do not protect against arbitrary Ruby actions.

data/docs/explanation/unicode-and-encoding.md

@@ -0,0 +1,20 @@
+# Unicode and encoding
+
+The automaton consumes bytes, but UTF-8 patterns are expanded into byte ranges
+that respect codepoint boundaries. Binary specifications operate directly on
+all 256 byte values. This separation keeps the hot path predictable while
+allowing Unicode-aware patterns.
+
+The vendored Unicode Character Database snapshot, currently 15.1.0, is the
+compatibility oracle for `\p{...}`, POSIX properties where a Unicode property
+is defined, Unicode shorthand behavior with `option :unicode`, and simple case
+folding. Host Ruby Unicode tables are not used as a cross-version oracle.
+
+Use `encoding Encoding::BINARY` for protocol or byte-oriented formats. Use the
+default UTF-8 encoding for text. A pattern with invalid UTF-8 or an input that
+breaks a required boundary is rejected or reported as an unmatched byte rather
+than normalized.
+
+`rake unicode:verify` checks range ordering, scalar expansions, and deterministic
+random range cases. Updating Unicode data requires regenerating generated golden
+files and is documented as a minor-version compatibility change.

data/docs/how-to/deploy-a-standalone-lexer.md

@@ -0,0 +1,23 @@
+# Deploy a standalone lexer
+
+Use `--standalone` to embed the flexr runtime into the generated file:
+
+```sh
+flexr lexer.flexr.rb --standalone -o lexer_standalone.rb
+```
+
+The generated file removes `require "flexr"` and includes the runtime sources
+needed to execute the compiled tables. It can be loaded by a Ruby process that
+does not have the flexr gem installed:
+
+```sh
+ruby -e 'load ARGV.fetch(0); p Lexer.new("12").tokens' lexer_standalone.rb
+```
+
+Generation still needs the flexr gem and Prism on Ruby 3.3 or newer. Runtime
+execution needs the generated file and Ruby's standard library only.
+
+Standalone output is larger and duplicates the runtime for every artifact.
+Prefer ordinary generated output when several lexers share one application and
+can depend on the flexr gem. The standalone behavior is covered by the CLI
+end-to-end test.

data/docs/how-to/generate-a-lexer.md

@@ -0,0 +1,39 @@
+# Generate a lexer
+
+Generate a Ruby artifact from a `.flexr.rb` specification:
+
+```sh
+flexr path/to/lexer.flexr.rb -o path/to/lexer.rb
+```
+
+Generation is static by default. Prism reads the source, resolves supported
+constant and expression forms, compiles the same DFA model used by runtime
+mode, and replaces DSL calls with a compiled installation payload. Ordinary
+Ruby constants, comments, requires, and non-DSL code remain in the file.
+
+## Static expressions
+
+The default path accepts regexp and string literals, arrays of supported
+values, constants, ranges, interpolation whose values are static, `.freeze`,
+and `Regexp.union`. A call whose value depends on runtime state raises
+`FLEXR-E017`. See the [static-expression reference](../reference/generated-artifacts.md)
+for the exact boundary.
+
+## Dynamic specifications
+
+```sh
+flexr path/to/lexer.flexr.rb --eval -o path/to/lexer.rb
+```
+
+`--eval` executes the specification and is therefore limited to trusted build
+inputs. It is useful when a pattern is assembled by a method or external
+configuration, but it removes the static-analysis guarantee and can make the
+artifact depend on the build environment.
+
+## Reproducible builds
+
+Commit generated files when downstream consumers should not need Prism, or
+generate them in the build when the source is the only artifact you distribute.
+The generated header records the source path, payload digest, Unicode snapshot,
+backend, compilation mode, and standalone setting. Use the digest and
+`generated:verify` in CI to detect stale committed output.

data/docs/how-to/handle-errors.md

@@ -0,0 +1,32 @@
+# Handle errors
+
+There are two error layers: specification diagnostics and input-time lexical
+errors.
+
+## Validate the specification
+
+```sh
+flexr check lexer.flexr.rb --format json --warn all
+```
+
+The command returns status 0 when the specification compiles, even when it has
+warnings. Add `--warn-as-error` in CI to make warnings fail the check. Syntax,
+unsupported regexp, and compile-limit failures return status 1; malformed CLI
+usage returns status 2.
+
+## Choose input error behavior
+
+```ruby
+Lexer.new(input, error_mode: :raise)
+Lexer.new(input, error_mode: :token)
+Lexer.new(input, error_mode: :panic)
+```
+
+`:raise` raises `Flexr::LexError`, `:token` emits `[:error, text]`, and `:panic`
+skips the unmatched byte and continues. You can override the decision for one
+instance with `lexer.on_error = ->(error) { :skip }`, `:raise`, `:token`, or
+`:halt`. The callback receives the `LexError` and may inspect its filename,
+byte position, line, and text.
+
+`error!` creates the same error from an action. `max_token_size` protects the
+runtime from unexpectedly large tokens and raises `FLEXR-E012` when exceeded.

data/docs/how-to/integrate-with-lrama.md

@@ -0,0 +1,21 @@
+# Integrate with Lrama
+
+Lrama-generated parsers use the same two-element token protocol as Racc.
+Implement the lexer with `racc_next_token`, declare the grammar tokens with
+`emits`, and generate the lexer as part of the parser build.
+
+```ruby
+class Lexer < Flexr::Lexer
+  emits :INTEGER, :MINUS
+  rule(/[ \t\r\n]+/, skip: true)
+  rule(/[0-9]+/, emit: :INTEGER)
+  rule(/-/, emit: :MINUS)
+end
+```
+
+`racc_next_token` returns `[type, value]` for a token and `[false, "$end"]` at
+EOF. `emits` is a documentation and diagnostic contract; it does not alter
+the value returned by the lexer.
+
+See the runnable [Lrama example](../../examples/with_lrama/README.md) and the
+[token contract](../reference/tokens-and-locations.md).

data/docs/how-to/integrate-with-racc.md

@@ -0,0 +1,25 @@
+# Integrate with Racc
+
+flexr exposes the conventional `racc_next_token` method, so a parser can use a
+lexer without an adapter:
+
+```ruby
+class Lexer < Flexr::Lexer
+  emits :INTEGER, :PLUS
+  rule(/[ \t\r\n]+/, skip: true)
+  rule(/[0-9]+/, emit: :INTEGER)
+  rule(/\+/, emit: :PLUS)
+end
+
+lexer = Lexer.new("12 + 3")
+lexer.racc_next_token # => [:INTEGER, "12"]
+```
+
+At EOF the method returns `[false, "$end"]`. Keep the names in `emits` aligned
+with the grammar's `%token` declarations. `flexr tokens SPEC.flexr.rb` prints
+the declared names for a build-time comparison.
+
+The complete executable lexer is in
+[examples/with_racc](../../examples/with_racc/README.md). Generate the lexer
+before running a parser build if the parser should consume a committed Ruby
+artifact.

data/docs/how-to/migrate-from-flex.md

@@ -0,0 +1,21 @@
+# Migrate from flex
+
+Use the importer to translate a flex `.l` file into a Ruby specification:
+
+```sh
+flexr import scanner.l -o lexer.flexr.rb
+```
+
+Common rules and actions are translated, including `return`, `BEGIN`,
+`yyless`, `yymore`, and `ECHO`. The importer preserves header and footer text as
+Ruby comments where possible. Run the result through `flexr check` before
+loading it.
+
+Untranslated C actions are emitted as `FLEXR-TODO`; the command returns status 1
+and does not create or overwrite the requested output file. Complete the Ruby
+action manually and repeat the check. Also review rule overlaps: flexr uses
+longest-match semantics, which can differ from a flex specification that relies
+on rule order.
+
+The [CLI reference](../reference/cli.md) documents importer exit statuses and
+the [errors reference](../reference/errors.md) documents `FLEXR-TODO` behavior.

data/docs/how-to/migrate-from-rexical.md

@@ -0,0 +1,23 @@
+# Migrate from Rexical
+
+Import a Rexical specification with:
+
+```sh
+flexr import scanner.rex -o lexer.flexr.rb
+```
+
+Rexical uses first-match semantics while flexr uses longest-match semantics.
+The importer reports that semantic boundary and searches for concrete
+counterexamples when it finds overlapping rules. Treat every reported witness
+as a migration review item.
+
+After rewriting actions and reviewing overlaps:
+
+```sh
+flexr check lexer.flexr.rb --warn all
+flexr lexer.flexr.rb -o lexer.rb
+```
+
+Do not enable `backend :firstmatch` merely to hide a migration difference. It
+is experimental, requires `option :experimental`, and generation performs a
+differential check against the table matcher.

data/docs/how-to/run-a-lexer-at-runtime.md

@@ -0,0 +1,29 @@
+# Run a lexer at runtime
+
+Runtime mode loads the Ruby specification and builds its automaton when the
+lexer class is first used. It is the simplest mode for development and tests.
+
+```ruby
+require "flexr"
+
+class Lexer < Flexr::Lexer
+  rule(/[ \t\n]+/, skip: true)
+  rule(/[0-9]+/) { emit :INTEGER, text.to_i }
+end
+
+lexer = Lexer.new(StringIO.new("12 34"))
+lexer.each_token { |token| p token }
+```
+
+The input may be a `String` or an object responding to `read`. IO input is read
+in chunks; `chunk_size:` controls the buffer size. `next_token` returns one
+token or `nil` at EOF, `tokens` drains the lexer into an array, and
+`each_token` returns an enumerator when no block is given.
+
+Use runtime mode when the specification deliberately depends on ordinary Ruby
+execution. Static generation is a better fit when startup cost, deployment
+reproducibility, or a generator-free runtime matters.
+
+See the [runtime reference](../reference/runtime.md) for initialization options
+and the [action reference](../reference/actions.md) for methods available in a
+rule block.

data/docs/how-to/track-token-locations.md

@@ -0,0 +1,27 @@
+# Track token locations
+
+Use `token_kind :struct` when consumers need a value object with a location:
+
+```ruby
+class Lexer < Flexr::Lexer
+  token_kind :struct
+  option :eager_columns
+
+  rule(/[0-9]+/) { emit :INTEGER, text.to_i }
+end
+
+token = Lexer.new("12\n34").next_token
+token.type
+token.value
+token.location.line_begin
+token.location.column_begin
+```
+
+Locations use byte offsets for `byte_begin` and `byte_end`, with an exclusive
+end offset. Lines and columns are one-based. UTF-8 columns count characters;
+binary columns count bytes. Column values are computed lazily unless
+`option :eager_columns` is set. `filename:` on the lexer is copied into each
+location and lexical error.
+
+`more` joins successive matches into one `text` value, so `last_location` spans
+the complete assembled token. See the [token and location contract](../reference/tokens-and-locations.md).

data/docs/how-to/tune-performance.md

@@ -0,0 +1,23 @@
+# Tune performance
+
+Measure a representative input before changing the backend:
+
+```sh
+ruby -Ilib benchmark/run.rb --spec examples/json/lexer.flexr.rb
+bundle exec rake bench:regression
+```
+
+Use `token_kind :yield` when the consumer streams tokens and does not need an
+intermediate token array. Use generated mode to move DFA construction out of
+application startup. `backend :auto` chooses `:direct` for large transition
+tables and `:table` otherwise; `:table` and `:direct` preserve the same
+longest-match semantics.
+
+Acceleration is an optimization over the DFA, not a second matcher. `accel
+:auto` selects an available safe path, while `:strscan`, `:regexp`, and `:none`
+make the choice explicit. Trailing context disables region acceleration for the
+affected rule and emits `FLEXR-W012` unless acceleration is disabled.
+
+The [performance log](../perf-log.md) records reproducible local observations,
+and the [backend explanation](../explanation/backends.md) describes the memory
+and compatibility trade-offs.

data/docs/how-to/use-states.md

@@ -0,0 +1,36 @@
+# Use states
+
+States model lexical contexts such as strings, comments, or interpolation.
+The initial state is inclusive; a named state is exclusive by default.
+
+```ruby
+class Lexer < Flexr::Lexer
+  emits :TEXT, :STRING
+
+  rule(/[^"<]+/) { emit :TEXT }
+  rule(/"/) { push :string; more; skip }
+
+  state :string do
+    rule(/[^"\\]+/) { more; skip }
+    rule(/"/) { pop; emit :STRING }
+  end
+end
+```
+
+`push(name)` saves the current state, `pop` restores it, and `begin_state(name)`
+switches without changing the stack. `push_state`, `pop_state`, and `state=`
+are aliases. An undefined state raises `FLEXR-E003`; exceeding
+`max_state_stack` raises `StateStackOverflowError`.
+
+Use `inclusive: true` when a state should also use rules declared for the
+initial state:
+
+```ruby
+state :interpolation, inclusive: true do
+  rule(/\}/) { pop; skip }
+end
+```
+
+`all_states` applies a rule block to every state known at that point. Keep
+state transitions in actions so the rule set remains easy to inspect with
+`flexr explain`.

data/docs/how-to/use-trailing-context.md

@@ -0,0 +1,22 @@
+# Use trailing context
+
+Use `followed_by:` when a rule should be selected only if another expression
+starts immediately after it, but that expression must remain in the input for
+the next token:
+
+```ruby
+class Lexer < Flexr::Lexer
+  rule(/name/, followed_by: /\(/) { emit :CALL }
+  rule(/[a-z_][a-z0-9_]*/) { emit :IDENT }
+  rule(/\(/) { emit :LPAREN }
+end
+```
+
+The input `name(` produces `CALL` for `name`, then `LPAREN` for `(`. A string
+can be used for literal trailing context. The body and trailing expression
+should be fixed length when possible; variable-length context produces
+`FLEXR-W003` and can prevent region acceleration (`FLEXR-W012`).
+
+Trailing context is not the same as regexp lookahead. Lookahead syntax is
+rejected with `FLEXR-E014`; `followed_by:` is the supported replacement.
+Disable acceleration with `accel :none` only when the trade-off is understood.

data/docs/internals/README.md

@@ -0,0 +1,14 @@
+# Internals
+
+This is maintainer-facing material. The public contract is in
+[reference/public-api.md](../reference/public-api.md); implementation choices
+are explained in the [architecture decisions](../adr/) and
+[explanation pages](../explanation/).
+
+Both runtime and generated modes compile the same byte-oriented model: regexp
+AST, Thompson NFA, byte-class DFA, deterministic minimization, and last-accept
+tracking. Generated output embeds compiled tables and generated dispatch code;
+runtime mode uses the interpreter and optional acceleration paths.
+
+The internal names `Flexr::IR`, `Flexr::Automaton`, `Flexr::Codegen`, and
+`__flexr_*` are not compatibility-stable.

data/docs/perf-log.md

@@ -0,0 +1,56 @@
+# Performance log
+
+The benchmark harness reports bytes/s, tokens/s, and allocations per token.
+Numbers are intentionally recorded only after running the harness on the
+target Ruby and hardware; no unmeasured figures are presented as guarantees.
+
+Run:
+
+```sh
+ruby -Ilib benchmark/run.rb
+```
+
+The JSON result includes `mb_per_s`, `tokens_per_s`, and cumulative Ruby
+`allocations_per_token` for runtime, generated, and handwritten modes. JSON
+also records each Flexr mode as a ratio against the handwritten baseline.
+The large deterministic corpus can be materialized when needed:
+
+```sh
+ruby benchmark/corpora/generate_json.rb > benchmark/corpora/json_10mb.json
+ruby -Ilib benchmark/run.rb --spec examples/json/lexer.flexr.rb \
+  --input-file benchmark/corpora/json_10mb.json
+```
+
+## 2026-08-10
+
+On the local Ruby 4.0.0 environment, the default 140,000-byte JSON input and
+three measured iterations produced the following sample. These are recorded
+observations, not cross-machine guarantees.
+
+| mode | MB/s | tokens/s | allocations/token | handwritten ratio |
+|---|---:|---:|---:|---:|
+| runtime | 1.671 | 596,789 | 5.201 | 0.194x |
+| generated | 1.604 | 572,954 | 4.201 | 0.187x |
+| handwritten | 8.597 | 3,070,499 | 3.4 | 1.000x |
+
+This sample is above the v1.0 measured regression floor of 0.18x, but below the
+0.7x handwritten stretch target. The measured ratios are recorded explicitly
+rather than presented as a claim that the stretch target has been reached.
+Runtime is 1.042x generated in this sample, above the 0.2x runtime-mode risk
+threshold. See ADR 0019 for the target revision.
+
+### Byte/token cost estimate
+
+The same 10 MB JSON corpus was measured with compact records and with 100
+padding spaces after each numeric value. The compact sample had 3,448,271
+tokens; the padded sample had 775,191 tokens. Solving
+`elapsed = c_byte * bytes + c_token * tokens` from the two measurements gives
+the following local estimates (one iteration, `backend :direct`):
+
+| mode | c_byte | c_token |
+|---|---:|---:|
+| runtime | 619 ns/byte | 1.090 us/token |
+| generated | 611 ns/byte | 0.983 us/token |
+
+The padded corpus can be reproduced with
+`FLEXR_JSON_PADDING_BYTES=100 ruby benchmark/corpora/generate_json.rb`.

data/docs/reference/README.md

@@ -0,0 +1,23 @@
+# Reference
+
+This section describes behavior rather than teaching a sequence. Start with
+the [DSL](dsl.md) when writing a specification, the [runtime](runtime.md) when
+consuming tokens, or the [CLI](cli.md) when validating and generating files.
+
+## Reference pages
+
+- [DSL](dsl.md) — class-level specification methods and defaults.
+- [Runtime](runtime.md) — initialization, token iteration, input sources, and
+  limits.
+- [Actions](actions.md) — methods available inside rule and EOF actions.
+- [Tokens and locations](tokens-and-locations.md) — exact return shapes and
+  offsets.
+- [Errors](errors.md) — exception and input-recovery behavior.
+- [Regexp compatibility](regexp.md) — supported syntax and alternatives.
+- [CLI](cli.md) — commands, options, output, and exit statuses.
+- [Diagnostics](diagnostics.md) — diagnostic code catalog.
+- [Generated artifacts](generated-artifacts.md) — static evaluation, output
+  headers, dependencies, and standalone files.
+- [Public API](public-api.md) — stable, experimental, and internal boundaries.
+- [Compatibility](compatibility.md) — Ruby, Unicode, backend, and artifact
+  compatibility commitments.

data/docs/reference/actions.md

@@ -0,0 +1,47 @@
+# Action context reference
+
+Rule blocks, `on_eof` blocks, and generated action bodies run with the lexer
+instance as `self`.
+
+## Token methods
+
+| Method | Contract |
+|---|---|
+| `emit(type, value = text)` | Queue one token; `type` may be `nil` |
+| `skip` | Queue no token for the current match |
+| `echo` | Equivalent to `emit(nil, text)` |
+| `error!(message)` | Apply the configured input error policy to a `LexError` |
+
+## Match and position methods
+
+| Method | Contract |
+|---|---|
+| `text` | Matched text, including a `more` prefix when present |
+| `text_bytesize` | Byte size of `text` |
+| `last_location` | A `Runtime::Location` for the current text span |
+| `lineno` / `line` | One-based current line number |
+| `byte_pos` | Current byte offset |
+| `beginning_of_line?` | Whether the current position is at line start |
+| `state` | Current state symbol |
+| `binary_input` | Input as a binary string |
+
+`text` is memoized for the current action. `last_location` uses the same span;
+`more` makes the span include all joined matches.
+
+## State methods
+
+| Method | Contract |
+|---|---|
+| `push(name)` / `push_state(name)` | Save the current state and enter `name` |
+| `pop` / `pop_state` | Restore the previous state, or `:initial` when the stack is empty |
+| `begin_state(name)` / `state=` | Enter `name` without changing the stack |
+
+## Flex-compatible methods
+
+| Method | Contract |
+|---|---|
+| `less(count)` | Keep only the first `count` bytes of the current match |
+| `more` | Join the next match into the current token text |
+
+`less` cannot exceed the current matched byte count. `more` is finalized after
+the action; call `emit` only after the complete text has been assembled.

data/docs/reference/cli.md

@@ -0,0 +1,80 @@
+# CLI reference
+
+The executable is `flexr`, with usage:
+
+```text
+flexr [COMMAND] SPEC.rb [options]
+```
+
+Without a command, `flexr SPEC.rb` generates Ruby. `SPEC.rb` is normally a
+`.flexr.rb` file, except for `import`, which accepts flex `.l` and Rexical
+`.rex` input.
+
+## Commands
+
+| Command | Purpose | Main output |
+|---|---|---|
+| `check SPEC` | Compile and render diagnostics | Human or JSON diagnostics |
+| `stats SPEC` | Report DFA states, byte classes, and table sizes | JSON |
+| `tokens SPEC` | Print names declared by `emits` | Space-separated names |
+| `dot SPEC` | Render the DFA as Graphviz DOT | DOT on stdout |
+| `trace SPEC` | Print DFA states, accepts, and transitions | Text on stdout |
+| `explain SPEC` | List parsed rules and patterns | Text on stdout |
+| `bench SPEC` | Run the benchmark harness | Benchmark output |
+| `import INPUT` | Translate flex/Rexical input | Ruby on stdout or `--output` |
+
+`trace` describes the automaton; it does not trace a particular input string.
+`dot` marks accepting states and accelerated regions for visual inspection.
+
+## Options
+
+| Option | Values / argument | Applies to |
+|---|---|---|
+| `-o, --output PATH` | File path | Generation and import |
+| `-b, --backend NAME` | `table`, `direct`, `firstmatch`, `auto` | Compilation |
+| `--token-kind KIND` | `array`, `struct`, `yield` | Generated/runtime contract |
+| `--accel MODE` | `auto`, `strscan`, `regexp`, `none` | Runtime optimization |
+| `--standalone` | Flag | Generation |
+| `--eval` | Flag | Generation and check |
+| `--table-compression VALUE` | `none`, `rows`, `full` | Generation |
+| `--table-format VALUE` | `literal`, `packed` | Generation |
+| `--max-dfa-states N` | Positive integer | Compilation |
+| `-W, --warn LEVEL` | `all`, `default`, `none` | Check |
+| `--warn-as-error` | Flag | Check and generation |
+| `--color WHEN` | `auto`, `always`, `never` | Human diagnostics |
+| `--format FMT` | `human`, `json` | Check and stats |
+| `--rule N` | Non-negative integer | Explain |
+
+The complete help output is kept as a checked snapshot:
+
+<!-- flexr-help:start -->
+Usage: flexr [COMMAND] SPEC.rb [options]
+
+Commands: check, stats, tokens, dot, explain, trace, bench, import
+Options:
+  -o, --output PATH          generated output path
+  -b, --backend NAME         table | direct | firstmatch | auto
+      --token-kind KIND      array | struct | yield
+      --accel MODE            auto | strscan | regexp | none
+      --standalone
+      --eval
+      --table-compression VALUE  none | rows | full
+      --table-format VALUE       literal | packed
+      --max-dfa-states N
+  -W, --warn LEVEL           all | default | none
+      --warn-as-error
+      --color WHEN            auto | always | never
+      --format FMT            human | json
+<!-- flexr-help:end -->
+
+## Exit statuses
+
+| Status | Meaning |
+|---:|---|
+| `0` | Command completed; warnings do not fail by default |
+| `1` | Specification, input, import, or generation failure |
+| `2` | Invalid command-line usage |
+
+`check --format json` writes one JSON array to stdout. Human diagnostics go to
+stdout for `check` and errors go to stderr for command failures. Use
+`--warn-as-error` when a warning should fail CI.

data/docs/reference/compatibility.md

@@ -0,0 +1,38 @@
+# Compatibility
+
+## Ruby and dependencies
+
+| Surface | Requirement |
+|---|---|
+| Runtime gem | Ruby 3.1+ |
+| Generator | Ruby 3.3+ with Prism available |
+| Standalone runtime | Compatible Ruby plus the generated file |
+| Optional acceleration | Ruby `strscan` when available; regexp fallback remains valid |
+
+Runtime mode does not require Prism. Generated mode uses Prism only while
+reading the source; normal generated output does not need the generator.
+
+## Matching and encoding
+
+UTF-8 and binary are the supported specification encodings. The automaton is
+byte-oriented, but UTF-8 boundaries and Unicode properties are validated using
+the vendored data snapshot. Invalid UTF-8 is not silently repaired.
+
+## Unicode snapshot
+
+The current Unicode Character Database snapshot is 15.1.0. It supplies Unicode
+properties, POSIX behavior where applicable, Unicode shorthand behavior under
+`option :unicode`, and case folding. Updating the snapshot can change generated
+output and is therefore a minor-version compatibility change. Run
+`bundle exec rake unicode:verify` and regenerate golden artifacts after an
+update.
+
+## Generated files and SemVer
+
+Runtime and generated-file compatibility is preserved within a major version
+unless a migration is explicitly documented. Generated artifacts include a
+payload digest and Unicode version so stale or cross-version output can be
+detected. Do not assume that internal table layouts are stable.
+
+Stable backends are `table`, `direct`, and `auto`. `firstmatch` is experimental
+and does not share the longest-match promise.