flexr 1.0.0

data/docs/reference/diagnostics.md

@@ -0,0 +1,41 @@
+# Diagnostics catalog
+
+Diagnostics have a code, severity, message, optional location, help, and note.
+Use `flexr check SPEC --format json` to consume the structured form.
+
+## Errors
+
+| Code | Meaning | First fix to try |
+|---|---|---|
+| `FLEXR-E001` | Regexp syntax is malformed | Correct the regexp syntax |
+| `FLEXR-E003` | A state name is undefined | Declare the state before entering it |
+| `FLEXR-E005` | A rule can match an empty string | Make it consuming or opt into `allow_empty_match` |
+| `FLEXR-E006` | DFA state limit exceeded | Raise `max_dfa_states` or split rules |
+| `FLEXR-E007` | Bounded repetition exceeds 1000 | Use a smaller bound or split the rule |
+| `FLEXR-E009` | Anchor is not at an outermost boundary | Move `^` / `$` or split alternatives |
+| `FLEXR-E010` | Prism reported source syntax errors | Fix Ruby syntax in the specification |
+| `FLEXR-E011` | Encoding is not UTF-8 or binary | Change `encoding` |
+| `FLEXR-E012` | Runtime token exceeded the size limit | Raise `max_token_size` or split the token |
+| `FLEXR-E013` | `reject` is not supported | Use states and `less` |
+| `FLEXR-E014` | Regexp construct is not DFA-compatible | Follow the diagnostic alternative |
+| `FLEXR-E017` | Pattern is not statically resolvable | Rewrite as a static expression or use trusted `--eval` |
+| `FLEXR-E018` | Rule pattern or trailing context has an invalid type | Use a regexp, string, or valid array |
+| `FLEXR-E019` | Prism is unavailable for generation | Install the generator dependencies |
+
+## Warnings
+
+| Code | Meaning | First fix to try |
+|---|---|---|
+| `FLEXR-W001` | Rule is unreachable or shadowed | Remove, reorder, or distinguish it |
+| `FLEXR-W002` | Named state has no rules | Add rules or remove the state |
+| `FLEXR-W003` | Trailing context has variable length | Make both parts fixed length |
+| `FLEXR-W010` | `firstmatch` overlap may change semantics | Use the stable table backend |
+| `FLEXR-W011` | Transition table is large | Use direct backend, compression, or split rules |
+| `FLEXR-W012` | Trailing context cannot use region acceleration | Remove context or set `accel :none` |
+| `FLEXR-W013` | Capturing group is treated as non-capturing | Use `(?:...)` |
+| `FLEXR-W014` | Emitted token is not declared | Add it to `emits` or accept the warning |
+| `FLEXR-W016` | DFA construction took more than 0.5 seconds | Use generated mode for startup |
+
+Warnings are reported by `check` according to `--warn`. The default omits the
+wall-clock `FLEXR-W016`; `--warn all` includes it, and `--warn none` suppresses
+warnings. `--warn-as-error` turns selected warnings into a failed command.

data/docs/reference/dsl.md

@@ -0,0 +1,81 @@
+# DSL reference
+
+The DSL is extended onto a subclass of `Flexr::Lexer`. Class bodies are normal
+Ruby and may contain constants, methods, requires, comments, and ordinary
+expressions.
+
+## `rule(pattern, skip: false, emit: nil, followed_by: nil, &action)`
+
+Registers one rule. `pattern` is a `Regexp`, `String`, or an array containing
+those values. A string is matched literally. Array members are alternatives of
+one rule and share its action.
+
+`skip: true` discards the match and takes precedence over `emit` and the block.
+`emit: :TOKEN` installs a simple action that emits the token with the matched
+text as its value. Without either option, the block runs in the lexer instance;
+if no block is given, the default action emits `[nil, text]`.
+
+`followed_by:` accepts a `Regexp` or `String`. It participates in rule
+selection but is not consumed. The method returns `nil`. Rules are numbered in
+source order starting at zero.
+
+## `state(*names, inclusive: false, &block)`
+
+Creates one or more named states and registers rules in the block for those
+states. The block is required. Named states are exclusive by default. An
+inclusive state also sees rules registered for `:initial`.
+
+## `all_states(&block)`
+
+Registers the block for every state known when the call is evaluated, including
+`:initial`. It has the same rule-registration behavior as `state`.
+
+## `on_eof(&action)`
+
+Installs an action for EOF in the current state. If called outside a state block,
+the action belongs to `:initial`. The action can emit a final token, change
+state, or do nothing. It runs at most once per state for a lexer instance.
+
+## `emits(*tokens)`
+
+Declares token names for diagnostics and parser integration. Arguments are
+flattened and converted to symbols. Duplicate names are removed. The method
+does not restrict `emit` at runtime.
+
+## `backend(name)`
+
+Selects `:table`, `:direct`, `:auto`, or the experimental `:firstmatch`
+backend. `:table`, `:direct`, and `:auto` preserve the stable longest-match
+contract. `:firstmatch` requires `option :experimental` and can change the
+meaning of overlapping rules.
+
+## `token_kind(name)`
+
+Selects `:array`, `:struct`, or `:yield`. The default is `:array`. See the
+[token contract](tokens-and-locations.md) for the exact shapes.
+
+## `encoding(value)`
+
+Accepts `Encoding::UTF_8`, `Encoding::BINARY`, or a name resolving to one of
+them. Other encodings raise `FLEXR-E011`. The regexp encoding can override the
+specification encoding for an individual pattern only when the resulting byte
+model is valid.
+
+## `option(*values)`
+
+Adds boolean options to the specification. Stable options include
+`:unicode`, `:eager_columns`, and `:allow_empty_match`. `:experimental` is a
+capability opt-in for experimental behavior such as `:firstmatch`. Unknown
+options are retained in generated configuration but may have no effect.
+
+## `accel(value)`
+
+Selects `:auto`, `:strscan`, `:regexp`, or `:none` for runtime region
+acceleration. Acceleration is an optimization; the DFA remains the semantic
+source of truth. Affected trailing-context rules cannot use region acceleration.
+
+## Compilation helpers
+
+`compile!` and `dfa` exist for diagnostics and integration internals. They are
+not required for normal use and are not compatibility-stable; use the lexer
+constructor and token methods instead. See [public API](public-api.md).

data/docs/reference/errors.md

@@ -0,0 +1,27 @@
+# Errors and recovery
+
+## Specification errors
+
+Compile-time failures raise `Flexr::CompileError` or its
+`Flexr::UnsupportedRegexpError` subclass. Both expose a structured
+`diagnostic`. Static generation can additionally raise
+`Flexr::StaticResolutionError` with `FLEXR-E017`.
+
+## Input errors
+
+`Flexr::LexError` records the message, filename, byte position, line, matched
+text, and optional diagnostic. `Flexr::Runtime::TokenTooLargeError` is a
+`LexError` with code `FLEXR-E012`. `StateStackOverflowError` is raised when a
+`push` would exceed `max_state_stack`.
+
+The default `error_mode: :raise` raises the lexical error. `:token` queues
+`[:error, text]`; `:panic` discards the unmatched byte and continues. An
+`on_error` callback can return `:skip`, `:raise`, `:token`, or `:halt` for
+instance-specific control.
+
+## Import failures
+
+`flexr import` reports untranslated actions as `FLEXR-TODO`, returns status 1,
+and never writes incomplete output. This is intentionally stricter than a
+best-effort migration because a partially translated lexer can silently change
+language behavior.

data/docs/reference/generated-artifacts.md

@@ -0,0 +1,50 @@
+# Generated artifacts
+
+## Dependency matrix
+
+| Use | Build-time requirement | Runtime requirement |
+|---|---|---|
+| Runtime mode | None beyond the flexr gem | `flexr` gem |
+| Normal generated mode | flexr gem and Prism on Ruby 3.3+ | `flexr` gem |
+| Standalone generated mode | flexr gem and Prism on Ruby 3.3+ | Generated file and Ruby standard library |
+
+The runtime gem supports Ruby 3.1 and newer. The generator's Prism path is
+supported on Ruby 3.3 and newer. A generated file can be loaded by a compatible
+runtime without the generator installed.
+
+## Static evaluation boundary
+
+The default generator resolves:
+
+- regexp, string, numeric, symbol, and range literals;
+- arrays and hashes made from supported values;
+- constants in lexical and qualified scope;
+- interpolation whose expressions resolve statically;
+- `.freeze` on a static value; and
+- `Regexp.union` with static arguments.
+
+Methods with runtime-dependent results, `Time`, IO, environment reads, and
+arbitrary dynamic calls produce `FLEXR-E017`. `--eval` is the explicit escape
+hatch and executes the complete specification.
+
+## Source transformation
+
+The generator removes DSL call spans, inserts `Flexr::Generated.install_compiled!`,
+and adds generated scanner methods. User actions remain Ruby source. A generated
+header records:
+
+- source path;
+- SHA-256 digest of the generated payload;
+- vendored Unicode version;
+- effective backend;
+- whether compilation used eval; and
+- whether the runtime is standalone.
+
+Do not edit generated files by hand. Regenerate them when the specification,
+generator, backend, table format, or Unicode snapshot changes.
+
+## Table formats
+
+`--table-compression none|rows|full` controls row packing. `--table-format
+literal|packed` controls whether packed arrays are emitted as Ruby literals or
+Base64. These affect artifact size and loading cost, not matching semantics.

data/docs/reference/public-api.md

@@ -0,0 +1,42 @@
+# Public API and stability
+
+Compatibility guarantees apply only to the stable surface below. Everything
+else may change between releases without a deprecation cycle.
+
+## Public and stable
+
+- `Flexr::Lexer` and the DSL methods in [dsl.md](dsl.md).
+- Runtime consumption: `initialize`, `next_token`, `tokens`, `each_token`, and
+  `racc_next_token`.
+- Action context methods in [actions.md](actions.md).
+- `Flexr::Runtime::Token`, `Flexr::Runtime::Location`, `Flexr::LexError`,
+  `Flexr::CompileError`, `Flexr::UnsupportedRegexpError`, and
+  `Flexr::Runtime::TokenTooLargeError`.
+- `Flexr::Diagnostic`, `Flexr::DiagnosticSet`, and `Flexr::Diagnostics`.
+- `Flexr::Generator` for source generation, `Flexr::RakeTask` for Rake builds,
+  and `Flexr::CLI.run` for programmatic CLI invocation.
+- `Flexr.compile_pattern` and `Flexr.parse_pattern` for regexp tooling.
+
+Method arguments, return shapes, and diagnostics for this surface are described
+in the reference pages and are covered by the test suite.
+
+## Experimental
+
+- `backend :firstmatch` and `--backend firstmatch`.
+- `option :experimental` as the opt-in for experimental behavior.
+- Any future backend or option explicitly marked experimental in its reference.
+
+Experimental behavior may change semantics or output between minor releases.
+`firstmatch` requires the opt-in and generation performs a deterministic
+differential check against the table matcher.
+
+## Internal; compatibility is not guaranteed
+
+- `Flexr::IR`, `Flexr::Automaton`, `Flexr::Codegen`, and their members.
+- `Flexr::Runtime::Buffer` and `Flexr::Runtime::Interpreter` internals.
+- `Flexr::Generated` installation helpers.
+- `__flexr_*` methods and instance variables.
+- `Flexr::Lexer.compile!` and `.dfa` as implementation inspection hooks.
+
+Use stable token and diagnostic APIs instead of depending on internal compiled
+tables or generated installation methods.

data/docs/reference/regexp.md

@@ -0,0 +1,39 @@
+# Regexp compatibility
+
+flexr accepts the regular-expression subset that can be represented by its
+byte-oriented automaton. All matches start at the current lexer position.
+
+| Ruby construct | Status | Notes and alternative |
+|---|---|---|
+| Literals, concatenation, alternation | Supported | |
+| Groups and `(?:...)` | Supported | Captures are treated as non-capturing and warn |
+| Character classes and ranges | Supported | POSIX classes are supported; see encoding notes |
+| Unicode `\p{...}` / `\P{...}` | Supported | Uses the vendored UCD snapshot |
+| `*`, `+`, `?` | Supported | Greedy forms only |
+| `{n}`, `{n,m}` | Supported | The upper bound is 1000 |
+| `{n,}` | Unsupported | Split the rule or use states |
+| `^` / `$` | Restricted | Only at outermost pattern boundaries; otherwise `FLEXR-E009` |
+| `/i`, `/m`, `/x`, `/n` | Supported | Inline forms are supported where Ruby permits them |
+| `\d`, `\w`, `\s` | Supported | Unicode-aware with `option :unicode` in UTF-8 |
+| Lookahead / lookbehind | Unsupported | Use `followed_by:` or a state; `FLEXR-E014` |
+| Backreferences | Unsupported | Split the language into states or actions |
+| `\b`, `\B`, `\A`, `\z`, `\G`, `\K` | Unsupported | Use rule boundaries or state/action logic |
+| Lazy or possessive quantifiers | Unsupported | Use a negated character class or separate rule |
+| Conditional, atomic, and capture-dependent groups | Unsupported | Rewrite as DFA-compatible rules |
+
+Malformed syntax produces `FLEXR-E001`. Unsupported constructs produce
+`FLEXR-E014`; the diagnostic help normally names the supported replacement.
+Empty matches are rejected with `FLEXR-E005` unless `option :allow_empty_match`
+is explicitly enabled, in which case flexr advances to guarantee progress.
+
+## Bounded repetition
+
+`{n,m}` expands into a finite automaton. `m` must be at least `n` and no greater
+than 1000. Open repetition is rejected because it is easy to express with `+`
+or a separate rule and otherwise gives a misleading compatibility promise.
+
+## Captures
+
+Capturing parentheses do not expose match groups to actions. They are treated
+as grouping syntax and produce `FLEXR-W013`; use `(?:...)` and inspect `text`
+in the action instead.

data/docs/reference/runtime.md

@@ -0,0 +1,49 @@
+# Runtime reference
+
+`Flexr::Lexer` includes the runtime methods below. Runtime mode and generated
+mode expose the same consumer-facing methods.
+
+## Initialization
+
+```ruby
+Lexer.new(input,
+  filename: nil,
+  error_mode: :raise,
+  max_token_size: 16 * 1024 * 1024,
+  max_state_stack: 1024,
+  chunk_size: 64 * 1024)
+```
+
+`input` must be a `String` or an object responding to `read`. `filename` is
+copied into locations and lexical errors. `error_mode` is `:raise`, `:token`,
+or `:panic`; unknown values fall back to raising when an unmatched byte is
+handled. Size limits are non-negative, and `chunk_size` must be positive.
+
+## Consumption
+
+| Method | Result |
+|---|---|
+| `next_token` | One token, or `nil` at EOF |
+| `tokens` | Array containing every token until EOF |
+| `each_token` | Enumerator without a block; otherwise yields tokens and returns `self` |
+| `racc_next_token` | `[type, value]`, or `[false, "$end"]` at EOF |
+| `input` | The buffered input string |
+| `buffer` | Internal buffered input object; use only for integrations that need it |
+| `filename` | The configured filename |
+| `error_mode` | The configured default input error mode |
+| `max_token_size` | The configured token-size limit |
+
+`each_token` yields two arguments instead of one when `token_kind :yield` is
+selected. `tokens` always returns an array of token values.
+
+## State and action context
+
+The public action methods are collected in the [actions reference](actions.md).
+State transitions are instance-local; changing one lexer does not change the
+class or another lexer instance.
+
+## Streaming input
+
+IO input is read lazily in chunks. UTF-8 input is checked at codepoint
+boundaries, while binary input is consumed byte by byte. Invalid UTF-8 becomes
+an unmatched-byte error rather than being silently normalized.

data/docs/reference/tokens-and-locations.md

@@ -0,0 +1,33 @@
+# Tokens and locations
+
+## Token kinds
+
+| `token_kind` | `next_token` / `tokens` | `each_token` |
+|---|---|---|
+| `:array` (default) | `[type, value]` | Yields one array |
+| `:struct` | `Flexr::Runtime::Token` | Yields one `Token` |
+| `:yield` | `[type, value]` | Yields `type, value` as separate arguments |
+
+For `:struct`, `Token` has `type`, `value`, and `location` members. `emit(nil,
+value)` is valid and produces a token whose type is `nil`. `skip` produces no
+token.
+
+## Locations
+
+`Flexr::Runtime::Location` has these public members:
+
+| Member | Meaning |
+|---|---|
+| `filename` | The `filename:` value passed to the lexer |
+| `byte_begin` | Inclusive byte offset of the token |
+| `byte_end` | Exclusive byte offset of the token |
+| `line_begin` / `line_end` | One-based line range |
+| `column_begin` / `column_end` | One-based character columns for UTF-8, byte columns for binary input |
+
+Columns are computed lazily by default. `option :eager_columns` fills them while
+the location is created. The end line and column identify the position after
+the token's last byte, so a token ending at the start of the next line may have
+an end column of 1.
+
+`more` extends the location from the first match to the final match. Trailing
+context is not part of the location because it is not consumed.

data/docs/tutorial/build-a-calculator-lexer.md

@@ -0,0 +1,96 @@
+# Build a calculator lexer
+
+This tutorial takes one specification through runtime execution, diagnostics,
+generation, and generated execution. The complete executable source is
+[`examples/calculator/lexer.flexr.rb`](../../examples/calculator/lexer.flexr.rb).
+
+## 1. Install flexr
+
+```sh
+gem install flexr
+```
+
+For a checkout, use the bundle and add `lib` to Ruby's load path:
+
+```sh
+bundle install
+bundle exec ruby -Ilib examples/calculator/lexer.flexr.rb
+```
+
+## 2. Read the specification
+
+```ruby
+class Lexer < Flexr::Lexer
+  emits :EQ, :ASSIGN, :IF, :IDENT, :INTEGER, :PLUS
+
+  rule(/[ \t\r\n]+/, skip: true)
+  rule(/==/) { emit :EQ }
+  rule(/=/) { emit :ASSIGN }
+  rule(/if/) { emit :IF }
+  rule(/[a-z_][a-z0-9_]*/) { emit :IDENT }
+  rule(/[0-9]+/) { emit :INTEGER, text.to_i }
+  rule(/\+/) { emit :PLUS }
+end
+```
+
+`emits` is a declaration used by diagnostics and parser integration. It does
+not create constants and is not required for `emit` to work.
+
+The `==` rule is longer than `=`, so it wins even though both rules are active.
+The input `if` matches both the keyword and identifier rules with equal length;
+the keyword rule appears first, so it wins. The input `ifx` is longer under the
+identifier rule and therefore becomes one identifier token.
+
+## 3. Run runtime mode
+
+```sh
+ruby -Ilib -e 'require "json"; load "examples/calculator/lexer.flexr.rb"; puts JSON.generate(CalculatorExample::Lexer.new("if ifx == = 12 + 3").tokens)'
+```
+
+Expected output:
+
+```json
+[["IF","if"],["IDENT","ifx"],["EQ","=="],["ASSIGN","="],["INTEGER",12],["PLUS","+"],["INTEGER",3]]
+```
+
+Runtime mode evaluates the Ruby specification normally. The DFA is compiled
+when the lexer class is first used, and actions run in the lexer instance.
+
+## 4. Validate the specification
+
+```sh
+flexr check examples/calculator/lexer.flexr.rb --format json
+```
+
+An empty JSON array means that the specification produced no errors or selected
+warnings. Use `--warn all` to include the full warning catalog.
+
+## 5. Generate Ruby
+
+```sh
+flexr examples/calculator/lexer.flexr.rb -o /tmp/calculator_lexer.rb
+```
+
+The default generator statically resolves patterns and constants with Prism.
+The generated file keeps ordinary Ruby surrounding the DSL, embeds the
+compiled tables, and keeps actions as Ruby code.
+
+## 6. Run the generated lexer
+
+```sh
+ruby -Ilib -e 'require "json"; load ARGV.fetch(0); puts JSON.generate(CalculatorExample::Lexer.new("if ifx == = 12 + 3").tokens)' /tmp/calculator_lexer.rb
+```
+
+The output must match runtime mode. `rake modes:equivalence` checks this
+relationship for every committed example.
+
+## 7. Inspect the contract
+
+```sh
+flexr tokens examples/calculator/lexer.flexr.rb
+flexr stats examples/calculator/lexer.flexr.rb
+flexr explain examples/calculator/lexer.flexr.rb --rule 1
+```
+
+Use the [DSL reference](../reference/dsl.md) for method arguments and the
+[CLI reference](../reference/cli.md) for output and exit-status behavior.

data/examples/calculator/README.md

@@ -0,0 +1,27 @@
+# Calculator lexer example
+
+This is the executable source for the first-time user tutorial. It keeps the
+rules deliberately small while showing two important matching contracts:
+
+- `==` wins over `=` because the longest match wins.
+- `if` wins over the identifier rule for the input `if` because equal-length
+  matches use source order.
+
+Run it directly:
+
+```sh
+ruby -Ilib -e 'require "json"; load "examples/calculator/lexer.flexr.rb"; puts JSON.generate(CalculatorExample::Lexer.new("if ifx == = 12 + 3").tokens)'
+```
+
+Expected output:
+
+```json
+[["IF","if"],["IDENT","ifx"],["EQ","=="],["ASSIGN","="],["INTEGER",12],["PLUS","+"],["INTEGER",3]]
+```
+
+Validate and generate it with:
+
+```sh
+flexr check examples/calculator/lexer.flexr.rb --format json
+flexr examples/calculator/lexer.flexr.rb -o /tmp/calculator_lexer.rb
+```

data/examples/calculator/lexer.flexr.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require "flexr"
+
+module CalculatorExample
+  class Lexer < Flexr::Lexer
+    emits :EQ, :ASSIGN, :IF, :IDENT, :INTEGER, :PLUS
+
+    rule(/[ \t\r\n]+/, skip: true)
+    rule(/==/) { emit :EQ }
+    rule(/=/) { emit :ASSIGN }
+    rule(/if/) { emit :IF }
+    rule(/[a-z_][a-z0-9_]*/) { emit :IDENT }
+    rule(/[0-9]+/) { emit :INTEGER, text.to_i }
+    rule(/\+/) { emit :PLUS }
+  end
+end

data/examples/json/README.md

@@ -0,0 +1,30 @@
+# JSON lexer example
+
+## What this example demonstrates
+
+- a complete JSON token set with value conversion;
+- the stable `direct` backend; and
+- runtime/generated equivalence for a small practical lexer.
+
+## Run in runtime mode
+
+```sh
+ruby -Ilib -e 'require "json"; load "examples/json/lexer.flexr.rb"; puts JSON.generate(JsonExample::Lexer.new(%q({"answer":42})).tokens)'
+```
+
+Expected output:
+
+```json
+[["LBRACE","{"],["STRING","answer"],["COLON",":"],["NUMBER",42.0],["RBRACE","}"]]
+```
+
+## Validate and generate
+
+```sh
+flexr check examples/json/lexer.flexr.rb --format json
+flexr examples/json/lexer.flexr.rb -o /tmp/json_lexer.rb
+ruby -Ilib -e 'require "json"; load ARGV.fetch(0); puts JSON.generate(JsonExample::Lexer.new(%q({"answer":42})).tokens)' /tmp/json_lexer.rb
+```
+
+The generated output should produce the same JSON token array. Use
+`--standalone` when the runtime artifact must not require the flexr gem.

data/examples/json/lexer.flexr.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+require "flexr"
+
+module JsonExample
+  class Lexer < Flexr::Lexer
+    backend :direct
+    emits :LBRACE, :RBRACE, :LBRACKET, :RBRACKET, :COLON, :COMMA,
+      :STRING, :NUMBER, :TRUE, :FALSE, :NULL
+
+    rule(/[ \t\r\n]+/, skip: true)
+    rule(/\{/) { emit :LBRACE }
+    rule(/\}/) { emit :RBRACE }
+    rule(/\[/) { emit :LBRACKET }
+    rule(/\]/) { emit :RBRACKET }
+    rule(/:/) { emit :COLON }
+    rule(/,/) { emit :COMMA }
+    rule(/true/) { emit :TRUE, true }
+    rule(/false/) { emit :FALSE, false }
+    rule(/null/) { emit :NULL, nil }
+    rule(/-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?/) { emit :NUMBER, text.to_f }
+    rule(/"(?:\\.|[^"\\])*"/) { emit :STRING, text.byteslice(1...-1) }
+  end
+end

data/examples/ruby_subset/README.md

@@ -0,0 +1,17 @@
+# Ruby subset lexer
+
+## What this example demonstrates
+
+The specification contains constants, a `%w` array, a heredoc inside an action,
+and ordinary Ruby interpolation. The generator removes only DSL calls and
+preserves the surrounding Ruby.
+
+## Run and validate
+
+```sh
+ruby -Ilib -e 'require "json"; load "examples/ruby_subset/lexer.flexr.rb"; puts JSON.generate(RubySubset::Lexer.new(%q(class Foo "ok" end)).tokens)'
+flexr check examples/ruby_subset/lexer.flexr.rb --format json
+flexr examples/ruby_subset/lexer.flexr.rb -o /tmp/ruby_subset_lexer.rb
+```
+
+Inspect the generated file to see that the constants and action body remain.

data/examples/ruby_subset/lexer.flexr.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "flexr"
+
+module RubySubset
+  class Lexer < Flexr::Lexer
+    KEYWORDS = %w[class def end].freeze
+    IDENT = /[A-Za-z_][A-Za-z_0-9]*/
+
+    emits :KEYWORD, :IDENT, :STRING
+
+    rule(/[ \t\r\n]+/, skip: true)
+    rule(KEYWORDS) { emit :KEYWORD, text.to_sym }
+    rule(IDENT) { emit :IDENT, "#{text}:identifier" }
+    rule(/"(?:\\.|[^"\\])*"/) do
+      value = <<~TOKEN
+        #{text[1...-1]}
+      TOKEN
+      emit :STRING, value.chomp
+    end
+  end
+end

data/examples/toy_lang/README.md

@@ -0,0 +1,17 @@
+# Toy language lexer
+
+## What this example demonstrates
+
+This lexer keeps `DIGIT` and `IDENT` as ordinary Ruby constants and interpolates
+them into regexp literals. It therefore exercises static constant resolution
+without introducing a second lexer language.
+
+## Run and validate
+
+```sh
+ruby -Ilib -e 'require "json"; load "examples/toy_lang/lexer.flexr.rb"; puts JSON.generate(ToyLang::Lexer.new("answer + 12").tokens)'
+flexr check examples/toy_lang/lexer.flexr.rb --format json
+flexr examples/toy_lang/lexer.flexr.rb -o /tmp/toy_lang_lexer.rb
+```
+
+The generated file preserves the constants and produces the same token stream.

data/examples/toy_lang/lexer.flexr.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require "flexr"
+
+module ToyLang
+  class Lexer < Flexr::Lexer
+    DIGIT = /[0-9]/
+    IDENT = /[A-Za-z_]/
+
+    emits :INT, :IDENT, :PLUS, :STRING
+
+    rule(/[ \t\r\n]+/, skip: true)
+    rule(/#{DIGIT}+/) { emit :INT, text.to_i }
+    rule(/#{IDENT}[A-Za-z_0-9]*/) { emit :IDENT, text }
+    rule(/\+/) { emit :PLUS }
+    rule(/"(?:\\.|[^"\\])*"/) { emit :STRING, text[1...-1] }
+  end
+end

data/examples/with_lrama/README.md

@@ -0,0 +1,17 @@
+# flexr with Lrama
+
+## What this example demonstrates
+
+Lrama consumes the same `racc_next_token` protocol as Racc. The lexer declares
+grammar-facing names with `emits`, and the dependency on Lrama remains optional.
+
+## Run and validate
+
+```sh
+ruby -Ilib -e 'load "examples/with_lrama/lexer.flexr.rb"; lexer = WithLrama::LramaLexer.new("12 - 3"); p lexer.tokens'
+flexr tokens examples/with_lrama/lexer.flexr.rb
+flexr check examples/with_lrama/lexer.flexr.rb --format json
+```
+
+See the [Lrama integration guide](../../docs/how-to/integrate-with-lrama.md)
+for the EOF protocol and build flow.