flexr 1.0.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.rubocop.yml +2 -1
- data/CHANGELOG.md +116 -0
- data/CONTRIBUTING.md +46 -3
- data/README.md +5 -4
- data/Rakefile +241 -72
- data/Steepfile +7 -0
- data/benchmark/baselines/json.json +2 -2
- data/benchmark/golden/calculator_lexer.rb +276 -0
- data/benchmark/golden/graphql_lexer.rb +310 -0
- data/benchmark/golden/json_lexer.rb +292 -0
- data/benchmark/golden/regexp_tokenizer.rb +293 -0
- data/benchmark/golden/ruby_subset_lexer.rb +280 -0
- data/benchmark/golden/sql_lexer.rb +293 -0
- data/benchmark/golden/toml_lexer.rb +292 -0
- data/benchmark/golden/toy_lang_lexer.rb +273 -0
- data/benchmark/golden/with_lrama_lexer.rb +264 -0
- data/benchmark/golden/with_racc_lexer.rb +264 -0
- data/benchmark/lrama_lexer.rb +165 -0
- data/benchmark/matrix.rb +79 -0
- data/benchmark/profile.rb +109 -0
- data/benchmark/run.rb +4 -1
- data/benchmark/state_dispatch.rb +78 -0
- data/docs/RELEASING.md +3 -2
- data/docs/explanation/backends.md +30 -10
- data/docs/explanation/matching-semantics.md +4 -2
- data/docs/explanation/security-model.md +10 -3
- data/docs/explanation/unicode-and-encoding.md +3 -1
- data/docs/how-to/deploy-a-standalone-lexer.md +6 -4
- data/docs/how-to/generate-a-lexer.md +3 -3
- data/docs/how-to/handle-errors.md +4 -1
- data/docs/how-to/run-a-lexer-at-runtime.md +5 -0
- data/docs/how-to/track-token-locations.md +2 -1
- data/docs/how-to/tune-performance.md +15 -3
- data/docs/migration-v2.md +74 -0
- data/docs/perf-log.md +212 -0
- data/docs/reference/README.md +2 -0
- data/docs/reference/actions.md +5 -2
- data/docs/reference/cli.md +71 -5
- data/docs/reference/compatibility.md +10 -4
- data/docs/reference/diagnostics.md +346 -0
- data/docs/reference/dsl.md +30 -9
- data/docs/reference/errors.md +34 -3
- data/docs/reference/generated-artifacts.md +71 -6
- data/docs/reference/public-api.md +37 -7
- data/docs/reference/regexp.md +27 -13
- data/docs/reference/ruby-lsp.md +34 -0
- data/docs/reference/runtime.md +21 -4
- data/docs/reference/testing.md +57 -0
- data/docs/reference/tokens-and-locations.md +14 -4
- data/examples/graphql/README.md +14 -0
- data/examples/graphql/lexer.flexr.rb +29 -0
- data/examples/json/lexer.flexr.rb +6 -6
- data/examples/sql/README.md +16 -0
- data/examples/sql/lexer.flexr.rb +23 -0
- data/examples/toml/README.md +14 -0
- data/examples/toml/lexer.flexr.rb +23 -0
- data/gemfiles/ruby_lsp.gemfile +4 -0
- data/gemfiles/types.gemfile +7 -0
- data/lib/flexr/action_resolver.rb +15 -0
- data/lib/flexr/artifact_writer.rb +61 -0
- data/lib/flexr/automaton/accel.rb +78 -5
- data/lib/flexr/automaton/analysis.rb +202 -7
- data/lib/flexr/automaton/backend_cost_model.rb +40 -0
- data/lib/flexr/automaton/compiler.rb +229 -96
- data/lib/flexr/automaton/dfa.rb +106 -19
- data/lib/flexr/automaton/minimizer.rb +79 -31
- data/lib/flexr/automaton/nfa.rb +45 -13
- data/lib/flexr/automaton/types.rb +14 -0
- data/lib/flexr/cli.rb +128 -22
- data/lib/flexr/codegen/direct.rb +5 -41
- data/lib/flexr/codegen/firstmatch.rb +1 -7
- data/lib/flexr/codegen/table.rb +181 -69
- data/lib/flexr/codegen.rb +8 -0
- data/lib/flexr/configuration.rb +45 -0
- data/lib/flexr/diagnostics.rb +39 -3
- data/lib/flexr/dsl.rb +116 -47
- data/lib/flexr/errors.rb +7 -6
- data/lib/flexr/generated.rb +120 -46
- data/lib/flexr/generator.rb +102 -109
- data/lib/flexr/grammar.rb +41 -0
- data/lib/flexr/importer.rb +4 -40
- data/lib/flexr/ir.rb +2 -2
- data/lib/flexr/migration.rb +31 -0
- data/lib/flexr/minitest.rb +24 -0
- data/lib/flexr/options.rb +9 -5
- data/lib/flexr/rake_task.rb +6 -1
- data/lib/flexr/regexp/ast.rb +5 -2
- data/lib/flexr/regexp/matcher.rb +26 -0
- data/lib/flexr/regexp/normalizer.rb +63 -21
- data/lib/flexr/regexp/parser.rb +241 -60
- data/lib/flexr/regexp/tokenizer.rb +172 -47
- data/lib/flexr/rspec.rb +51 -0
- data/lib/flexr/runtime/buffer.rb +115 -9
- data/lib/flexr/runtime/core.rb +248 -51
- data/lib/flexr/runtime/errors.rb +66 -4
- data/lib/flexr/runtime/interpreter.rb +213 -97
- data/lib/flexr/runtime/location.rb +35 -12
- data/lib/flexr/runtime/record.rb +14 -0
- data/lib/flexr/runtime/token.rb +7 -1
- data/lib/flexr/runtime.rb +75 -0
- data/lib/flexr/source/passthrough.rb +70 -14
- data/lib/flexr/source/prism_reader.rb +217 -51
- data/lib/flexr/source/static_eval.rb +40 -4
- data/lib/flexr/source.rb +6 -0
- data/lib/flexr/testing.rb +47 -0
- data/lib/flexr/trace.rb +197 -0
- data/lib/flexr/unicode/data/properties.rb +1 -1
- data/lib/flexr/unicode/data.rb +11 -0
- data/lib/flexr/unicode/property.rb +4 -8
- data/lib/flexr/unicode/reference_regexp.rb +11 -9
- data/lib/flexr/unicode/version.rb +7 -0
- data/lib/flexr/version.rb +1 -1
- data/lib/flexr.rb +10 -76
- data/lib/ruby_lsp/flexr/addon.rb +114 -0
- data/sig/flexr.rbs +191 -0
- data/site/README.md +54 -3
- data/site/astro.config.mjs +3 -1
- data/site/package.json +12 -5
- data/site/playwright.config.mjs +16 -0
- data/site/pnpm-lock.yaml +218 -88
- data/site/ruby/playground.rb +57 -0
- data/site/scripts/prepare-playground.mjs +35 -0
- data/site/scripts/verify-site.mjs +5 -3
- data/site/src/content/docs/concepts/matching-semantics.md +4 -2
- data/site/src/content/docs/concepts/security-model.md +9 -1
- data/site/src/content/docs/learn/generation.md +1 -1
- data/site/src/content/docs/reference/dsl.md +2 -2
- data/site/src/content/docs/reference/public-api.md +4 -2
- data/site/src/content/docs/reference/tokens-and-locations.md +3 -0
- data/site/src/layouts/SiteLayout.astro +11 -3
- data/site/src/pages/index.astro +69 -155
- data/site/src/pages/playground.astro +40 -38
- data/site/src/playground/client.js +211 -0
- data/site/src/playground/graph-worker.mjs +11 -0
- data/site/src/playground/presets.mjs +33 -0
- data/site/src/playground/runtime.mjs +34 -0
- data/site/src/playground/share.mjs +30 -0
- data/site/src/playground/worker.mjs +29 -0
- data/site/src/styles/custom.css +12 -287
- data/site/src/styles/landing.css +239 -0
- data/site/tests/browser/playground.spec.mjs +99 -0
- data/site/tests/playground-runtime.test.mjs +85 -0
- data/test/ruby_lsp_addon_spec.rb +135 -0
- data/tools/coverage.rb +6 -1
- data/tools/docs_verify.rb +3 -2
- data/tools/golden.rb +54 -0
- data/tools/verification_failure.rb +119 -0
- metadata +66 -12
- data/benchmark/golden/calculator_lexer.sha256 +0 -1
- data/benchmark/golden/json_lexer.sha256 +0 -1
- data/benchmark/golden/regexp_tokenizer.sha256 +0 -1
- data/benchmark/golden/ruby_subset_lexer.sha256 +0 -1
- data/benchmark/golden/toy_lang_lexer.sha256 +0 -1
- data/benchmark/golden/with_lrama_lexer.sha256 +0 -1
- data/benchmark/golden/with_racc_lexer.sha256 +0 -1
- data/site/public/playground.js +0 -189
|
@@ -3,8 +3,132 @@
|
|
|
3
3
|
Diagnostics have a code, severity, message, optional location, help, and note.
|
|
4
4
|
Use `flexr check SPEC --format json` to consume the structured form.
|
|
5
5
|
|
|
6
|
+
## Source context in human output
|
|
7
|
+
|
|
8
|
+
When a diagnostic has a source range, human output includes the file, line,
|
|
9
|
+
column, source line, and carets. For example, given `lexer.flexr.rb`:
|
|
10
|
+
|
|
11
|
+
```ruby
|
|
12
|
+
class Lexer < Flexr::Lexer
|
|
13
|
+
rule(/[a-z]+/, skip: true)
|
|
14
|
+
rule(/if/, skip: true)
|
|
15
|
+
end
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
`flexr check lexer.flexr.rb --color never` prints:
|
|
19
|
+
|
|
20
|
+
```text
|
|
21
|
+
warning[FLEXR-W001]: rule /if/ is unreachable (shadowed by rule /[a-z]+/)
|
|
22
|
+
--> lexer.flexr.rb:3:3
|
|
23
|
+
|
|
|
24
|
+
3 | rule(/if/, skip: true)
|
|
25
|
+
| ^^^^^^^^^^^^^^^^^^^^^^
|
|
26
|
+
help: remove it, reorder the rules, or make its language distinct
|
|
27
|
+
note: example: "if" selects rule /[a-z]+/ (same length, earlier rule wins)
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Line and column numbers are one-based. Columns count characters rather than
|
|
31
|
+
UTF-8 bytes. A multiline range shows its first source line; an empty range
|
|
32
|
+
gets one caret. Tabs are preserved, but wide or combining characters can make
|
|
33
|
+
caret alignment differ from terminal display widths.
|
|
34
|
+
|
|
35
|
+
Source ranges are available for:
|
|
36
|
+
|
|
37
|
+
- Ruby syntax errors (`FLEXR-E010`).
|
|
38
|
+
- Static DSL errors: unsupported encoding (`FLEXR-E011`), `reject` actions
|
|
39
|
+
(`FLEXR-E013`), unresolved expressions or unsupported DSL structure
|
|
40
|
+
(`FLEXR-E017`), and invalid pattern or trailing-context types (`FLEXR-E018`).
|
|
41
|
+
- Rule compilation errors, including empty matches (`FLEXR-E005`), repetition
|
|
42
|
+
limits (`FLEXR-E007`), misplaced anchors (`FLEXR-E009`), and unsupported regexp
|
|
43
|
+
constructs (`FLEXR-E014`), including errors in trailing context.
|
|
44
|
+
- Rule warnings: `FLEXR-W001`, `FLEXR-W003`, `FLEXR-W010`, `FLEXR-W012`,
|
|
45
|
+
`FLEXR-W013`, and `FLEXR-W014`.
|
|
46
|
+
- Rule overlap information: `FLEXR-I001`, pointing to the later rule.
|
|
47
|
+
- State and specification diagnostics: DFA state limits (`FLEXR-E006`), empty
|
|
48
|
+
states (`FLEXR-W002`), large transition tables (`FLEXR-W011`), and slow DFA
|
|
49
|
+
construction (`FLEXR-W016`).
|
|
50
|
+
|
|
51
|
+
Rule diagnostics highlight the complete rule declaration. `FLEXR-W010` points
|
|
52
|
+
to the later rule in the reported pair. `FLEXR-W014` points to the first rule
|
|
53
|
+
emitting the undeclared token, with one warning per token.
|
|
54
|
+
|
|
55
|
+
Compiler diagnostics identify rules by their patterns rather than numeric
|
|
56
|
+
indices. Regexp options, quoted strings, arrays of patterns, and `followed_by:`
|
|
57
|
+
context are preserved in these labels. For longest-match specifications without
|
|
58
|
+
trailing context, `FLEXR-W001` also names the patterns that shadow the unreachable
|
|
59
|
+
rule; otherwise, its example identifies the verified winner. `FLEXR-E006` lists
|
|
60
|
+
the patterns in the state being compiled. String trailing context is shown as
|
|
61
|
+
its compiled literal regexp. These labels also appear in JSON messages and when
|
|
62
|
+
source ranges are unavailable.
|
|
63
|
+
|
|
64
|
+
`FLEXR-W001` notes include a shortest nonempty input, measured in bytes, and
|
|
65
|
+
the rule selected for that input. Examples for named lexical states
|
|
66
|
+
include the state name. Control characters and binary bytes are escaped, and
|
|
67
|
+
the same note is preserved in JSON and generated artifacts.
|
|
68
|
+
|
|
69
|
+
For `firstmatch`, the candidate is checked with the scanner's regexp matcher
|
|
70
|
+
without running actions. The note identifies the first rule selected and its
|
|
71
|
+
matched byte count, which can be shorter than the example input. For example,
|
|
72
|
+
`"ab"` can select an earlier `/a/` rule with a one-byte match.
|
|
73
|
+
|
|
74
|
+
For states using `followed_by:`, the candidate includes the shortest trailing
|
|
75
|
+
input and is checked using the active rules. Longest-match backends compare
|
|
76
|
+
the combined body and lookahead length; `firstmatch` keeps rule-order priority.
|
|
77
|
+
The note separates consumed bytes from lookahead bytes. For example, two rules
|
|
78
|
+
with body `/a/` and trailing context `/b+/` produce this note with backend `table`:
|
|
79
|
+
|
|
80
|
+
```text
|
|
81
|
+
example: "ab" selects rule /a/, followed_by: /b+/ (longest total match, then rule order; matched 1 byte, looked ahead 1 byte)
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
If a checked candidate selects the supposedly unreachable rule in any lexical
|
|
85
|
+
state, W001 is omitted for that rule. No rule actions run during these checks.
|
|
86
|
+
An example is omitted when the candidate is empty, invalid in the specification's
|
|
87
|
+
encoding, or cannot be confirmed by the matcher. This includes trailing patterns
|
|
88
|
+
that the runtime cannot compile. Candidate generation does not search for an
|
|
89
|
+
alternative if anchor or encoding checks reject the shortest candidate.
|
|
90
|
+
|
|
91
|
+
Static DSL errors highlight the offending DSL call. Unresolved expressions
|
|
92
|
+
point to the expression instead; ambiguous lexer selection points to the
|
|
93
|
+
second lexer class containing DSL calls.
|
|
94
|
+
|
|
95
|
+
State diagnostics point to the first declaration of the affected state.
|
|
96
|
+
`FLEXR-E006` identifies the state being compiled, while `FLEXR-W011` identifies
|
|
97
|
+
the state with the largest transition table. The implicit `initial` state
|
|
98
|
+
and the total construction time (`FLEXR-W016`) point to the lexer class.
|
|
99
|
+
|
|
100
|
+
Rules and states created directly through the runtime DSL do not carry source
|
|
101
|
+
ranges. Diagnostics without ranges, or whose source cannot be read, retain
|
|
102
|
+
the message, help, and note without a source excerpt.
|
|
103
|
+
|
|
104
|
+
For direct rendering, pass the matching source text and optional file path to
|
|
105
|
+
`DiagnosticSet#render(source: source, path: path)`. Without a path, the label is
|
|
106
|
+
`(source)`. JSON output keeps `location` as a pair of byte offsets with an
|
|
107
|
+
exclusive end; source excerpts are not added to JSON.
|
|
108
|
+
|
|
109
|
+
Runtime input diagnostics (`FLEXR-E003`, `FLEXR-E012`, `FLEXR-E013`,
|
|
110
|
+
and `FLEXR-E021` through `FLEXR-E027`)
|
|
111
|
+
use `[byte_pos, byte_pos]` to mark their reported position in the original
|
|
112
|
+
input. Guard notes identify the affected rule by its pattern when available,
|
|
113
|
+
using the same labels as compiler diagnostics. Pass the complete input to
|
|
114
|
+
`DiagnosticSet#render` to show its line and caret; see
|
|
115
|
+
[errors and recovery](errors.md). These offsets remain absolute
|
|
116
|
+
when a streaming buffer discards consumed input. If the CLI reports a runtime
|
|
117
|
+
input failure, it reads the error's `filename` for context, preserving the
|
|
118
|
+
input's original line endings. To bound memory use while reporting input
|
|
119
|
+
failures, the CLI omits excerpts for input files larger than 64 KiB. An absent
|
|
120
|
+
or unreadable input file also produces the diagnostic without an excerpt.
|
|
121
|
+
|
|
6
122
|
## Errors
|
|
7
123
|
|
|
124
|
+
If Prism cannot be loaded (`FLEXR-E019`), the diagnostic points to the start of
|
|
125
|
+
the specification and suggests installing the dependency. This is a file-level
|
|
126
|
+
failure; no Ruby syntax has been parsed yet.
|
|
127
|
+
|
|
128
|
+
Incompatible generated artifacts (`FLEXR-E020`) identify the start of the file
|
|
129
|
+
that installs the artifact, when that file exists. Direct installation calls
|
|
130
|
+
without a readable caller file still report the regeneration guidance.
|
|
131
|
+
|
|
8
132
|
| Code | Meaning | First fix to try |
|
|
9
133
|
|---|---|---|
|
|
10
134
|
| `FLEXR-E001` | Regexp syntax is malformed | Correct the regexp syntax |
|
|
@@ -21,6 +145,170 @@ Use `flexr check SPEC --format json` to consume the structured form.
|
|
|
21
145
|
| `FLEXR-E017` | Pattern is not statically resolvable | Rewrite as a static expression or use trusted `--eval` |
|
|
22
146
|
| `FLEXR-E018` | Rule pattern or trailing context has an invalid type | Use a regexp, string, or valid array |
|
|
23
147
|
| `FLEXR-E019` | Prism is unavailable for generation | Install the generator dependencies |
|
|
148
|
+
| `FLEXR-E020` | Generated artifact metadata is missing or incompatible | Regenerate with the installed flexr version |
|
|
149
|
+
| `FLEXR-E021` | An action or state cycle made no input progress | Consume input or break the state cycle |
|
|
150
|
+
| `FLEXR-E022` | The retained streaming buffer is too large | Shorten tokens, raise the limit, or disable retention |
|
|
151
|
+
| `FLEXR-E023` | Runtime work exceeded `max_steps` | Raise the limit or simplify the specification |
|
|
152
|
+
| `FLEXR-E024` | Trailing context exceeded its lookahead limit | Bound the context or raise the limit |
|
|
153
|
+
| `FLEXR-E025` | The lexical state stack overflowed | Remove recursive pushes or raise the limit |
|
|
154
|
+
| `FLEXR-E026` | A cancellation callback stopped lexing | Resume with a new lexer when ready |
|
|
155
|
+
| `FLEXR-E027` | An error callback returned an unknown action | Return `:skip`, `:raise`, `:token`, or `:halt` |
|
|
156
|
+
|
|
157
|
+
### Regexp rewrite help
|
|
158
|
+
|
|
159
|
+
For positive lookahead, `FLEXR-E014` includes a concrete example:
|
|
160
|
+
|
|
161
|
+
```text
|
|
162
|
+
error[FLEXR-E014]: look-around is not supported by flexr
|
|
163
|
+
help: for a trailing positive lookahead, rewrite rule(/a(?=b)/) as rule(/a/, followed_by: /b/); keep the action
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
Keep the action or `emit:` / `skip:` options on the rewritten rule. The example
|
|
167
|
+
consumes `a` only when `b` follows, leaving `b` for the next token. The combined
|
|
168
|
+
body and trailing length participates in longest-match selection.
|
|
169
|
+
|
|
170
|
+
Adapt this example to positive lookahead at the end of a rule's pattern,
|
|
171
|
+
preserving the regexp options needed by each part. Lookahead inside a pattern
|
|
172
|
+
needs a separate rewrite. Negative lookahead and lookbehind receive separate
|
|
173
|
+
guidance to use rules or states because `followed_by:` cannot express those
|
|
174
|
+
assertions.
|
|
175
|
+
|
|
176
|
+
For lazy and possessive quantifiers, `FLEXR-E014` shows a negated character
|
|
177
|
+
class for unescaped, single-line quoted text:
|
|
178
|
+
|
|
179
|
+
```text
|
|
180
|
+
error[FLEXR-E014]: lazy or possessive quantifier is not supported by flexr
|
|
181
|
+
help: for unescaped single-line quoted text, use a negated character class: rule(/"[^"\n]*"/); keep the action; removing lazy or possessive modifiers can change matches
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
For this form of quoted text, `/".*?"/` and `/"[^"\n]*+"/` can both be
|
|
185
|
+
rewritten as `/"[^"\n]*"/`. Excluding the closing quote makes the match stop
|
|
186
|
+
at the first delimiter, so adjacent strings remain separate tokens. Excluding
|
|
187
|
+
`\n` preserves the line-feed restriction of `.` without `/m`. Escaped quotes
|
|
188
|
+
and multiline strings need a different pattern. Simply dropping a lazy or
|
|
189
|
+
possessive modifier can change which inputs match or how much text is consumed.
|
|
190
|
+
|
|
191
|
+
For atomic groups, the same quoted-text rule can replace `/"(?>[^"\n]*)"/`.
|
|
192
|
+
The repeated character class cannot consume the closing quote, so this example
|
|
193
|
+
does not need backtracking to find it. The diagnostic warns against removing
|
|
194
|
+
atomic groups in general: [Ruby's atomic grouping](https://docs.ruby-lang.org/en/4.0/Regexp.html)
|
|
195
|
+
can change whether the rest of a pattern matches. The example still excludes
|
|
196
|
+
escaped quotes and line feeds.
|
|
197
|
+
|
|
198
|
+
For numbered backreferences, the help uses a capture with only two possible
|
|
199
|
+
values. Rewrite `rule(/(a|b)\1/)` or `rule(/(a|b)\k<1>/)` as
|
|
200
|
+
`rule(/(?:aa|bb)/)` and keep the action. This accepts `aa` and `bb`, while
|
|
201
|
+
rejecting `ab` and `ba`. It reproduces the matched text; flexr does not expose
|
|
202
|
+
capture values to actions. If the repeated text cannot be enumerated, split
|
|
203
|
+
the rule into states and compare the matched text in actions instead.
|
|
204
|
+
|
|
205
|
+
For a non-recursive subexpression call, the help shows
|
|
206
|
+
`rule(/(a|b)\g<1>/)` becoming `rule(/[ab]{2}/)` with the same action.
|
|
207
|
+
A call reuses the expression, not the previously captured text: this example
|
|
208
|
+
accepts `aa`, `ab`, `ba`, and `bb`. Recursive calls need a parser or a state
|
|
209
|
+
stack; they cannot be expanded by this example.
|
|
210
|
+
|
|
211
|
+
For named captures used only for grouping, rewrite `rule(/(?<word>ab)/)` or
|
|
212
|
+
`rule(/(?'word'ab)/)` as `rule(/(?:ab)/)` and keep the action. Backreferences
|
|
213
|
+
and conditions that depend on the capture need a separate rewrite; removing
|
|
214
|
+
the name alone does not preserve them.
|
|
215
|
+
|
|
216
|
+
For the conditional example `rule(/(a)?(?(1)b|c)/)`, use `rule(/(?:ab|c)/)`
|
|
217
|
+
with the same action. The original matches `ab` when the optional `a` was
|
|
218
|
+
captured, or `c` otherwise. Other conditions need their own alternatives,
|
|
219
|
+
separate rules, or state checks.
|
|
220
|
+
|
|
221
|
+
For an absence group excluding a single literal, rewrite `rule(/x(?~a)/)`
|
|
222
|
+
as `rule(/x[^a]*/)` and keep the action. Both consume `x` and the following
|
|
223
|
+
characters up to the next `a` or end of input, including line feeds. The `x`
|
|
224
|
+
keeps the rule from matching empty input. Larger absence expressions need a
|
|
225
|
+
separate rewrite: `(?~ab)` does not mean `[^ab]*`.
|
|
226
|
+
|
|
227
|
+
For word boundaries, the help uses examples where the surrounding literals
|
|
228
|
+
already determine the assertion. Rewrite `rule(/foo\b:/)` as `rule(/foo:/)`:
|
|
229
|
+
`o` is a word character and `:` is not. Similarly, `rule(/foo\Bbar/)` becomes
|
|
230
|
+
`rule(/foobar/)` because `o` and `b` are both word characters. Keep the action.
|
|
231
|
+
These examples do not cover boundaries at token edges, such as `/if\b/`;
|
|
232
|
+
those need separate rules or state checks. Inside a character class, `[\b]`
|
|
233
|
+
remains a supported backspace escape. The escapes `\A`, `\B`, `\G`, `\z`,
|
|
234
|
+
`\Z`, `\K`, `\g`, and `\X` inside character classes receive literal-character examples,
|
|
235
|
+
such as rewriting `rule(/[\A]/)` as `rule(/[A]/)` with the same action.
|
|
236
|
+
|
|
237
|
+
For `\G` at the start of a rule, rewrite `rule(/\Ga/)` as `rule(/a/)` and
|
|
238
|
+
keep the action. Every flexr rule already starts matching at the current
|
|
239
|
+
lexer position, including after earlier tokens. A `\G` inside a larger
|
|
240
|
+
pattern needs a separate rewrite.
|
|
241
|
+
|
|
242
|
+
For input without line feeds, `rule(/\Aa/)` can become `rule(/^a/)`, and
|
|
243
|
+
`rule(/a\z/)` or `rule(/a\Z/)` can become `rule(/a$/)`. Keep the action.
|
|
244
|
+
This requires a guarantee about the entire input, not just the token pattern.
|
|
245
|
+
[Ruby's line and string anchors](https://docs.ruby-lang.org/en/4.0/Regexp.html)
|
|
246
|
+
differ when line feeds occur: `^` can match after a line feed, `$` can match
|
|
247
|
+
before one, and `\Z` also permits a final line feed. Do not apply these
|
|
248
|
+
examples to multiline input.
|
|
249
|
+
|
|
250
|
+
For the match-reset example `rule(/a\Kb/) { emit :B }` in `:initial`, the help
|
|
251
|
+
splits the prefix and suffix into separate rules:
|
|
252
|
+
|
|
253
|
+
```ruby
|
|
254
|
+
rule(/a/, followed_by: /b/) { begin_state :suffix }
|
|
255
|
+
state(:suffix) do
|
|
256
|
+
rule(/b/) { begin_state :initial; emit :B }
|
|
257
|
+
end
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
The prefix is consumed only when `b` follows. The suffix rule returns to
|
|
261
|
+
`:initial` before running the original action, so `text` and the token location
|
|
262
|
+
cover only `b`. Keep the original action in place of `emit :B`, choose an unused
|
|
263
|
+
state name, and return to the original state when adapting the example.
|
|
264
|
+
Larger patterns and competing rules need a separate review; this is not an
|
|
265
|
+
automatic replacement for every `\K`.
|
|
266
|
+
|
|
267
|
+
Escape diagnostics also include concrete examples. For an octal tab,
|
|
268
|
+
rewrite `rule(/\011/)` as `rule(/\x09/)`. For an unknown escape such as `\q`,
|
|
269
|
+
use `rule(/q/)` only if `q` is the intended literal character. Keep the original
|
|
270
|
+
action and regexp options when adapting either example.
|
|
271
|
+
|
|
272
|
+
Inside a character class, `\1` is a character code. Its diagnostic recommends
|
|
273
|
+
an explicit hexadecimal escape such as `rule(/[\x01]/)`, which matches the same
|
|
274
|
+
control character as `rule(/[\1]/)`.
|
|
275
|
+
|
|
276
|
+
When raw regexp sources contain control or meta escapes, their help gives
|
|
277
|
+
hexadecimal alternatives: `rule(/\cA/)` and `rule(/\C-A/)` become
|
|
278
|
+
`rule(/\x01/)`, while `rule(/\M-a/n)` becomes `rule(/\xE1/n)` for binary input.
|
|
279
|
+
Keep the action and use the corresponding hexadecimal value for other
|
|
280
|
+
characters. Ruby may already normalize these escapes in regexp literals;
|
|
281
|
+
the diagnostics apply when the escape reaches the parser unchanged.
|
|
282
|
+
|
|
283
|
+
For a standalone `\X` token with ASCII-only input, use:
|
|
284
|
+
|
|
285
|
+
```ruby
|
|
286
|
+
rule(/\r\n|[\x00-\x7f]/) { emit :GRAPHEME }
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
This keeps CRLF together and consumes other ASCII characters individually.
|
|
290
|
+
Keep the original action; the CRLF alternative must come first for `firstmatch`.
|
|
291
|
+
The example does not cover Unicode graphemes or `\X` inside larger patterns.
|
|
292
|
+
Unicode graphemes need separate segmentation, such as Ruby's
|
|
293
|
+
[`String#each_grapheme_cluster`](https://docs.ruby-lang.org/en/4.0/String.html#method-i-each_grapheme_cluster).
|
|
294
|
+
|
|
295
|
+
For `\R`, the help chooses a standalone line-break rule for the specification's
|
|
296
|
+
encoding. In UTF-8, use:
|
|
297
|
+
|
|
298
|
+
```ruby
|
|
299
|
+
rule(/\r\n|[\n\v\f\r\u0085\u2028\u2029]/) { emit :NEWLINE }
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
This covers the [line breaks recognized by Ruby's `\R`](https://docs.ruby-lang.org/en/4.0/Regexp.html#class-Regexp-label-Shorthand+Character+Classes).
|
|
303
|
+
With `encoding Encoding::BINARY`, the example uses only ASCII line breaks:
|
|
304
|
+
|
|
305
|
+
```ruby
|
|
306
|
+
rule(/\r\n|[\n\v\f\r]/n) { emit :NEWLINE }
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
CRLF comes first so that `firstmatch` also consumes the pair as one token.
|
|
310
|
+
Apply these examples as separate token rules. Replacing `\R` inside a larger
|
|
311
|
+
expression can change how the surrounding pattern matches.
|
|
24
312
|
|
|
25
313
|
## Warnings
|
|
26
314
|
|
|
@@ -35,7 +323,65 @@ Use `flexr check SPEC --format json` to consume the structured form.
|
|
|
35
323
|
| `FLEXR-W013` | Capturing group is treated as non-capturing | Use `(?:...)` |
|
|
36
324
|
| `FLEXR-W014` | Emitted token is not declared | Add it to `emits` or accept the warning |
|
|
37
325
|
| `FLEXR-W016` | DFA construction took more than 0.5 seconds | Use generated mode for startup |
|
|
326
|
+
| `FLEXR-W017` | Grammar token is missing from `emits` | Declare and emit it, or remove it from the grammar |
|
|
327
|
+
| `FLEXR-W018` | Token in `emits` is missing from the grammar | Add a `%token` declaration or remove it from `emits` |
|
|
328
|
+
| `FLEXR-W020` | Structured records become immutable in v2.0 | Review consumer mutations; use `record.with(...)` in v2 |
|
|
38
329
|
|
|
39
330
|
Warnings are reported by `check` according to `--warn`. The default omits the
|
|
40
331
|
wall-clock `FLEXR-W016`; `--warn all` includes it, and `--warn none` suppresses
|
|
41
332
|
warnings. `--warn-as-error` turns selected warnings into a failed command.
|
|
333
|
+
|
|
334
|
+
## Informational diagnostics
|
|
335
|
+
|
|
336
|
+
`flexr check --target 2.0` includes `FLEXR-I002` with Ruby and artifact
|
|
337
|
+
requirements. See the [v2 migration guide](../migration-v2.md) for the static
|
|
338
|
+
check's limits and opt-in runtime deprecation warnings.
|
|
339
|
+
|
|
340
|
+
`check --warn all` also reports `FLEXR-I001` when multiple rules match the same
|
|
341
|
+
input at the same length. This can happen even when both rules are reachable.
|
|
342
|
+
For example, `rule(%w[a c], emit: :FIRST)` followed by
|
|
343
|
+
`rule(%w[b c], emit: :SECOND)` produces:
|
|
344
|
+
|
|
345
|
+
```text
|
|
346
|
+
info[FLEXR-I001]: rule ["b", "c"] overlaps rule ["a", "c"] at the same length
|
|
347
|
+
help: reorder the rules if the later rule should take priority
|
|
348
|
+
note: example: "c" selects rule ["a", "c"] (1 byte, earlier rule wins)
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
Source context highlights the later rule. At most one shortest nonempty example
|
|
352
|
+
is reported for each later rule in each lexical state; this is not an exhaustive
|
|
353
|
+
list of overlapping rule pairs. Named states appear in the note, and lengths
|
|
354
|
+
are measured in bytes. Invalid input for the specification's encoding is omitted.
|
|
355
|
+
|
|
356
|
+
For `firstmatch`, the candidate is checked with the scanner's regexp matcher.
|
|
357
|
+
The diagnostic is emitted only if the first selected rule and the later rule
|
|
358
|
+
match the same number of bytes. For example, `/a|ab/` followed by `/ab/` does not
|
|
359
|
+
produce a tie for `"ab"`: the first rule matches one byte and the second matches
|
|
360
|
+
two. The note reports the actual matched byte count, including zero when empty
|
|
361
|
+
matches are allowed. No actions run during the check. If the shortest candidate
|
|
362
|
+
does not confirm a tie, no alternative candidate is searched.
|
|
363
|
+
|
|
364
|
+
For states using `followed_by:`, the candidate combines the shortest body with
|
|
365
|
+
the shortest trailing input. The check compares the total length, so rules can
|
|
366
|
+
overlap even when they consume different numbers of bytes. For example,
|
|
367
|
+
`rule(/a/, followed_by: /bc/, emit: :FIRST)` followed by
|
|
368
|
+
`rule(/ab/, followed_by: /c/, emit: :SECOND)` produces:
|
|
369
|
+
|
|
370
|
+
```text
|
|
371
|
+
info[FLEXR-I001]: rule /ab/, followed_by: /c/ overlaps rule /a/, followed_by: /bc/ at the same total length
|
|
372
|
+
help: reorder the rules if the later rule should take priority
|
|
373
|
+
note: example: "abc" selects rule /a/, followed_by: /bc/ (same total length, earlier rule wins; matched 1 byte, looked ahead 2 bytes)
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
The winner is checked against the full candidate, including trailing matches
|
|
377
|
+
and end anchors. The note separates the bytes consumed from those looked ahead.
|
|
378
|
+
For `firstmatch`, it says `first match wins` and requires the selected rule and
|
|
379
|
+
the later rule to have equal total lengths. Empty input, invalid character
|
|
380
|
+
boundaries, and candidates the runtime's trailing matcher cannot verify are
|
|
381
|
+
omitted. A rejected shortest candidate does not trigger a search for alternatives.
|
|
382
|
+
|
|
383
|
+
This analysis covers `table`, `direct`, `auto`, and `firstmatch` states.
|
|
384
|
+
`--warn default` and `--warn none` omit this information, and
|
|
385
|
+
`--warn-as-error` does not turn informational diagnostics into failures.
|
|
386
|
+
Generation with `--warn all` also preserves the information in generated and
|
|
387
|
+
standalone artifacts.
|
data/docs/reference/dsl.md
CHANGED
|
@@ -15,6 +15,10 @@ one rule and share its action.
|
|
|
15
15
|
text as its value. Without either option, the block runs in the lexer instance;
|
|
16
16
|
if no block is given, the default action emits `[nil, text]`.
|
|
17
17
|
|
|
18
|
+
Prefer `emit:` when the value is the unchanged matched text, such as
|
|
19
|
+
`rule(/\+/, emit: :PLUS)`. This avoids executing a Ruby action block for each
|
|
20
|
+
token. Use a block when converting a value or changing lexer state.
|
|
21
|
+
|
|
18
22
|
`followed_by:` accepts a `Regexp` or `String`. It participates in rule
|
|
19
23
|
selection but is not consumed. The method returns `nil`. Rules are numbered in
|
|
20
24
|
source order starting at zero.
|
|
@@ -25,6 +29,11 @@ Creates one or more named states and registers rules in the block for those
|
|
|
25
29
|
states. The block is required. Named states are exclusive by default. An
|
|
26
30
|
inclusive state also sees rules registered for `:initial`.
|
|
27
31
|
|
|
32
|
+
Nested state blocks register their rules for every enclosing state, matching
|
|
33
|
+
Ruby runtime evaluation. Repeating a state declaration with a different
|
|
34
|
+
`inclusive:` value raises immediately instead of silently retaining or replacing
|
|
35
|
+
one declaration.
|
|
36
|
+
|
|
28
37
|
## `all_states(&block)`
|
|
29
38
|
|
|
30
39
|
Registers the block for every state known when the call is evaluated, including
|
|
@@ -44,10 +53,13 @@ does not restrict `emit` at runtime.
|
|
|
44
53
|
|
|
45
54
|
## `backend(name)`
|
|
46
55
|
|
|
47
|
-
Selects `:table`, `:direct`, `:auto`, or
|
|
56
|
+
Selects `:table`, `:direct`, `:auto`, or `:firstmatch`
|
|
48
57
|
backend. `:table`, `:direct`, and `:auto` preserve the stable longest-match
|
|
49
|
-
contract. `:firstmatch`
|
|
50
|
-
|
|
58
|
+
contract. `:firstmatch` selects the first matching rule in source order,
|
|
59
|
+
even when a later rule could consume more text. Within a rule, alternation
|
|
60
|
+
also follows Ruby regexp order. Choose it when rule priority is intentional;
|
|
61
|
+
the compiler reports possible differences from longest-match as `FLEXR-W010`.
|
|
62
|
+
The default remains `:table`.
|
|
51
63
|
|
|
52
64
|
## `token_kind(name)`
|
|
53
65
|
|
|
@@ -64,9 +76,10 @@ model is valid.
|
|
|
64
76
|
## `option(*values)`
|
|
65
77
|
|
|
66
78
|
Adds boolean options to the specification. Stable options include
|
|
67
|
-
`:unicode`, `:eager_columns`, and `:allow_empty_match`. `:experimental`
|
|
68
|
-
|
|
69
|
-
|
|
79
|
+
`:unicode`, `:eager_columns`, and `:allow_empty_match`. `:experimental` remains
|
|
80
|
+
accepted for compatibility; it is no longer required by `:firstmatch`.
|
|
81
|
+
`:standalone` is build metadata used by standalone generation. Unknown option
|
|
82
|
+
names raise immediately so misspellings cannot silently change behavior.
|
|
70
83
|
|
|
71
84
|
## `accel(value)`
|
|
72
85
|
|
|
@@ -76,6 +89,14 @@ source of truth. Affected trailing-context rules cannot use region acceleration.
|
|
|
76
89
|
|
|
77
90
|
## Compilation helpers
|
|
78
91
|
|
|
79
|
-
`compile!` and `dfa` exist for diagnostics and integration internals.
|
|
80
|
-
|
|
81
|
-
|
|
92
|
+
`compile!` and `dfa` exist for diagnostics and integration internals. The first
|
|
93
|
+
compilation makes the specification immutable; later calls to `rule`, `state`,
|
|
94
|
+
`backend`, `option`, and other declaration methods raise
|
|
95
|
+
`Flexr::FrozenSpecificationError`. This keeps the compiled DFA and its DSL
|
|
96
|
+
definition from diverging. These helpers are not required for normal use and
|
|
97
|
+
are not compatibility-stable; use the lexer constructor and token methods
|
|
98
|
+
instead. See [public API](public-api.md).
|
|
99
|
+
|
|
100
|
+
Subclassing a custom lexer class starts a new, empty specification; rules and
|
|
101
|
+
states are not inherited. This avoids silently changing rule order or state
|
|
102
|
+
membership in the child class.
|
data/docs/reference/errors.md
CHANGED
|
@@ -11,13 +11,44 @@ Compile-time failures raise `Flexr::CompileError` or its
|
|
|
11
11
|
|
|
12
12
|
`Flexr::LexError` records the message, filename, byte position, line, matched
|
|
13
13
|
text, and optional diagnostic. `Flexr::Runtime::TokenTooLargeError` is a
|
|
14
|
-
`LexError` with code `FLEXR-E012`.
|
|
15
|
-
`
|
|
14
|
+
`LexError` with code `FLEXR-E012`. Other runtime guard failures use stable codes
|
|
15
|
+
`FLEXR-E021` through `FLEXR-E027`; see the
|
|
16
|
+
[diagnostics catalog](diagnostics.md). `StateStackOverflowError` is raised when
|
|
17
|
+
a `push` would exceed `max_state_stack`.
|
|
18
|
+
|
|
19
|
+
Undefined state transitions (`FLEXR-E003`) and `reject` calls (`FLEXR-E013`)
|
|
20
|
+
at runtime continue to raise `Flexr::CompileError`. These errors also expose
|
|
21
|
+
the input `filename` and `byte_pos`, with a diagnostic point range at the
|
|
22
|
+
current cursor. During a rule action this is after the matched input, unless
|
|
23
|
+
the action moved the cursor with `less`. EOF actions point to the end of input.
|
|
24
|
+
Static `reject` diagnostics still highlight the rule in the specification.
|
|
25
|
+
|
|
26
|
+
Guard diagnostics include a point range at `byte_pos` when it is known. This
|
|
27
|
+
is the guard's reported input position, which may be the start of a token or
|
|
28
|
+
the end of the loaded buffer. In a rescue handler, render it with the complete
|
|
29
|
+
original input:
|
|
30
|
+
|
|
31
|
+
```ruby
|
|
32
|
+
diagnostics = Flexr::DiagnosticSet.new << error.diagnostic
|
|
33
|
+
warn diagnostics.render(source: original_input, path: error.filename)
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
With `retain_input: false`, `lexer.input` contains only the retained buffer;
|
|
37
|
+
keep the original input separately if a source excerpt is needed. JSON
|
|
38
|
+
diagnostics retain absolute byte offsets even after input has been discarded.
|
|
39
|
+
|
|
40
|
+
When a guard knows the affected rule, its diagnostic note shows the pattern,
|
|
41
|
+
including regexp options and trailing context, for example `rule /a+/i at
|
|
42
|
+
byte 0`. The error's `rule` attribute still returns the numeric rule index.
|
|
43
|
+
Errors constructed with only a numeric index retain a numeric label; guards
|
|
44
|
+
without an identified rule omit the note. The diagnostic location continues
|
|
45
|
+
to refer to the input, not the rule declaration.
|
|
16
46
|
|
|
17
47
|
The default `error_mode: :raise` raises the lexical error. `:token` queues
|
|
18
48
|
`[:error, text]`; `:panic` discards the unmatched byte and continues. An
|
|
19
49
|
`on_error` callback can return `:skip`, `:raise`, `:token`, or `:halt` for
|
|
20
|
-
instance-specific control.
|
|
50
|
+
instance-specific control. Any other result raises
|
|
51
|
+
`InvalidRecoveryActionError` instead of silently applying the default policy.
|
|
21
52
|
|
|
22
53
|
## Import failures
|
|
23
54
|
|
|
@@ -5,13 +5,40 @@
|
|
|
5
5
|
| Use | Build-time requirement | Runtime requirement |
|
|
6
6
|
|---|---|---|
|
|
7
7
|
| Runtime mode | None beyond the flexr gem | `flexr` gem |
|
|
8
|
-
| Normal generated mode | flexr gem and Prism on Ruby 3.
|
|
9
|
-
| Standalone generated mode | flexr gem and Prism on Ruby 3.
|
|
8
|
+
| Normal generated mode | flexr gem and Prism on Ruby 3.4+ | `flexr` gem and Ruby 3.4+ |
|
|
9
|
+
| Standalone generated mode | flexr gem and Prism on Ruby 3.4+ | Generated file and Ruby 3.4+ standard library |
|
|
10
10
|
|
|
11
|
-
The runtime gem
|
|
12
|
-
supported on Ruby 3.3 and newer. A generated file can be loaded by a compatible
|
|
11
|
+
The runtime gem and generator support Ruby 3.4 and newer. A generated file can be loaded by a compatible
|
|
13
12
|
runtime without the generator installed.
|
|
14
13
|
|
|
14
|
+
Version 2 artifacts use schema 2 and runtime ABI 2. Regenerate v1 generated
|
|
15
|
+
files with the v2 generator. Old metadata is rejected with `FLEXR-E020` when
|
|
16
|
+
loading an artifact into the new runtime.
|
|
17
|
+
|
|
18
|
+
## Ractor sharing
|
|
19
|
+
|
|
20
|
+
Load a generated class in the main Ractor, then explicitly prepare its data:
|
|
21
|
+
|
|
22
|
+
```ruby
|
|
23
|
+
load "lexer.generated.rb"
|
|
24
|
+
Flexr::Generated.shareable!(Lexer)
|
|
25
|
+
worker = Ractor.new(Lexer) { |klass| klass.new("input").tokens }
|
|
26
|
+
tokens = worker.respond_to?(:value) ? worker.value : worker.take
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
This eagerly materializes transition rows and calls `Ractor.make_shareable` on
|
|
30
|
+
the compiled data and configuration. Each Ractor creates its own lexer and
|
|
31
|
+
input. Simple `emit` blocks, `emit:` rules, and `skip:` rules use generated
|
|
32
|
+
methods. Captured closures, runtime specifications, and EOF blocks are rejected
|
|
33
|
+
by `shareable!`; arbitrary user methods must also obey Ruby's Ractor rules.
|
|
34
|
+
`firstmatch` patterns that need the Unicode reference matcher are also rejected
|
|
35
|
+
before preparation, because its shared caches are not Ractor-safe. Use `table`,
|
|
36
|
+
`direct`, or `auto` for those patterns. Local variables that shadow action
|
|
37
|
+
methods such as `text`, `line`, or `lineno` keep their closure bindings and are
|
|
38
|
+
not converted to shareable method actions.
|
|
39
|
+
Preparation may use more memory than lazy packed tables. Call it before
|
|
40
|
+
starting workers; it does not make an existing lexer instance shareable.
|
|
41
|
+
|
|
15
42
|
## Static evaluation boundary
|
|
16
43
|
|
|
17
44
|
The default generator resolves:
|
|
@@ -27,14 +54,24 @@ Methods with runtime-dependent results, `Time`, IO, environment reads, and
|
|
|
27
54
|
arbitrary dynamic calls produce `FLEXR-E017`. `--eval` is the explicit escape
|
|
28
55
|
hatch and executes the complete specification.
|
|
29
56
|
|
|
57
|
+
Constants are resolved in source order, array splats are flattened with Ruby
|
|
58
|
+
semantics, and regexp interpolation retains the embedded regexp's options.
|
|
59
|
+
Receiver-qualified DSL calls, DSL hidden in conditionals or other unsupported
|
|
60
|
+
control flow, unknown DSL keywords, and files containing multiple candidate
|
|
61
|
+
lexer classes fail closed with `FLEXR-E017`. A file intended for static
|
|
62
|
+
generation should contain one lexer class and place DSL statements directly in
|
|
63
|
+
that class or a `state`/`all_states` body.
|
|
64
|
+
|
|
30
65
|
## Source transformation
|
|
31
66
|
|
|
32
67
|
The generator removes DSL call spans, inserts `Flexr::Generated.install_compiled!`,
|
|
33
|
-
and adds generated scanner methods.
|
|
68
|
+
and adds generated scanner methods. Source edits use Prism spans, so heredocs and
|
|
69
|
+
the `__END__` data section are not rewritten. User actions remain Ruby source. A generated
|
|
34
70
|
header records:
|
|
35
71
|
|
|
36
72
|
- source path;
|
|
37
73
|
- SHA-256 digest of the generated payload;
|
|
74
|
+
- artifact schema and runtime ABI versions;
|
|
38
75
|
- vendored Unicode version;
|
|
39
76
|
- effective backend;
|
|
40
77
|
- whether compilation used eval; and
|
|
@@ -43,8 +80,36 @@ header records:
|
|
|
43
80
|
Do not edit generated files by hand. Regenerate them when the specification,
|
|
44
81
|
generator, backend, table format, or Unicode snapshot changes.
|
|
45
82
|
|
|
83
|
+
Compiled payloads also record the artifact schema, compiler version, runtime ABI,
|
|
84
|
+
and Unicode version. Loading fails with `FLEXR-E020` when those contracts are
|
|
85
|
+
missing or incompatible. This prevents a stale artifact from silently running
|
|
86
|
+
against a runtime with different table or Unicode semantics.
|
|
87
|
+
|
|
88
|
+
Generation refuses to write through the input path, including aliases made with
|
|
89
|
+
symbolic or hard links. Outputs are written to a file in the destination directory,
|
|
90
|
+
synced, and atomically renamed. An unchanged output is left untouched so build
|
|
91
|
+
timestamps remain stable. For input names other than `*.flexr.rb`, the default
|
|
92
|
+
output is `*.generated.rb`.
|
|
93
|
+
|
|
46
94
|
## Table formats
|
|
47
95
|
|
|
48
96
|
`--table-compression none|rows|full` controls row packing. `--table-format
|
|
49
97
|
literal|packed` controls whether packed arrays are emitted as Ruby literals or
|
|
50
|
-
Base64.
|
|
98
|
+
Base64. Packed artifacts retain and query their packed representation; the
|
|
99
|
+
dense row view is populated lazily only when an inspection API requests a row.
|
|
100
|
+
These settings affect artifact size and loading cost, not matching semantics.
|
|
101
|
+
|
|
102
|
+
Standalone output embeds a runtime core, not the compiler or generator. The
|
|
103
|
+
regexp reference matcher and vendored Unicode tables are added only for a
|
|
104
|
+
`firstmatch` backend or trailing-context rules that can use that matcher.
|
|
105
|
+
|
|
106
|
+
The generator compiles `followed_by:` expressions into read-only DFA tables
|
|
107
|
+
stored with their rules. These tables use the selected compression and table
|
|
108
|
+
format, so scanning trailing context does not require the compiler at runtime.
|
|
109
|
+
Any errors found while compiling them are reported at the rule declaration
|
|
110
|
+
before an existing output file is replaced.
|
|
111
|
+
|
|
112
|
+
The optional `trailing_dfa` rule field is compatible with schema 1. The runtime
|
|
113
|
+
continues to compile trailing context lazily for older generated files without
|
|
114
|
+
this field. Regenerate older standalone lexers that use `followed_by:` to fix
|
|
115
|
+
their missing `Flexr.compile_pattern` error.
|
|
@@ -20,15 +20,45 @@ else may change between releases without a deprecation cycle.
|
|
|
20
20
|
Method arguments, return shapes, and diagnostics for this surface are described
|
|
21
21
|
in the reference pages and are covered by the test suite.
|
|
22
22
|
|
|
23
|
-
|
|
23
|
+
`require "flexr"` keeps the build tools lazy while preserving all public
|
|
24
|
+
constants through autoload. Applications that want an explicit load boundary
|
|
25
|
+
can require `flexr/runtime`, `flexr/generator`, or `flexr/cli` directly.
|
|
26
|
+
|
|
27
|
+
Runtime and generated `firstmatch` modes are stable and use the same rule-order
|
|
28
|
+
semantics. The former `option :experimental` opt-in remains accepted but is
|
|
29
|
+
unnecessary. Differences from longest-match behavior are reported as
|
|
30
|
+
`FLEXR-W010` rather than making generation fail.
|
|
31
|
+
## Type signatures
|
|
32
|
+
|
|
33
|
+
The gem includes RBS signatures in `sig/flexr.rbs` for the stable API above.
|
|
34
|
+
Steep can discover them through an application's RBS collection or an explicit
|
|
35
|
+
`library "flexr"` entry. User-defined token values and internal regexp ASTs
|
|
36
|
+
remain `untyped`; the signatures do not make internal tables stable APIs.
|
|
37
|
+
Token and location records inherit from `Data`; fields are read-only and `with`
|
|
38
|
+
returns a replacement. Their fields are nullable because their public constructors
|
|
39
|
+
allow omitted keywords. The token representation remains a union of arrays and
|
|
40
|
+
structured tokens because `token_kind` is selected dynamically.
|
|
41
|
+
Cancellation signatures accept Procs, Methods, and callable objects taking zero
|
|
42
|
+
arguments or the lexer. `on_error=` accepts a callable returning `:skip`,
|
|
43
|
+
`:raise`, `:token`, or `:halt`; assigning nil clears the handler.
|
|
44
|
+
|
|
45
|
+
Validate the signatures and type-check the consumer example with:
|
|
24
46
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
47
|
+
```sh
|
|
48
|
+
BUNDLE_GEMFILE=gemfiles/types.gemfile bundle install
|
|
49
|
+
BUNDLE_GEMFILE=gemfiles/types.gemfile bundle exec rbs -I sig validate
|
|
50
|
+
BUNDLE_GEMFILE=gemfiles/types.gemfile bundle exec steep check
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
The type-checking tools run in a separate CI matrix for Ruby 3.4 and 4.0,
|
|
54
|
+
matching the runtime's supported Ruby versions. They add no runtime dependency to flexr.
|
|
55
|
+
This checks the public signatures and a typed consumer, not the entire internal
|
|
56
|
+
implementation.
|
|
57
|
+
|
|
58
|
+
## Experimental
|
|
28
59
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
differential check against the table matcher.
|
|
60
|
+
Future backends or options explicitly marked experimental in their reference
|
|
61
|
+
may change semantics or output between minor releases.
|
|
32
62
|
|
|
33
63
|
## Internal; compatibility is not guaranteed
|
|
34
64
|
|