ibex 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (437) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +95 -272
  3. data/docs/architecture.md +187 -35
  4. data/docs/bison-import.md +103 -0
  5. data/docs/comparison-policy.md +127 -0
  6. data/docs/configuration-model.md +59 -0
  7. data/docs/configuration-report.md +28 -0
  8. data/docs/conflict-explanation-reviews/v1/records/README.md +28 -0
  9. data/docs/conflict-explanation-study.md +62 -0
  10. data/docs/construction-profiling.md +157 -0
  11. data/docs/cst-migration.md +3 -4
  12. data/docs/cst.md +24 -0
  13. data/docs/decisions/0000-template.md +24 -0
  14. data/docs/decisions/0001-separate-ir-pipeline.md +34 -0
  15. data/docs/decisions/0002-opaque-user-code-boundary.md +32 -0
  16. data/docs/decisions/0003-self-hosted-grammar-frontend.md +32 -0
  17. data/docs/decisions/0004-shared-semantic-and-lossless-source-model.md +33 -0
  18. data/docs/decisions/0005-contained-grammar-composition.md +34 -0
  19. data/docs/decisions/0006-bounded-structural-grammar-lowering.md +32 -0
  20. data/docs/decisions/0007-shared-parser-construction-pipeline.md +33 -0
  21. data/docs/decisions/0008-versioned-runtime-package-boundary.md +36 -0
  22. data/docs/decisions/0009-isolated-parser-sessions.md +34 -0
  23. data/docs/decisions/0010-committed-runtime-observation.md +34 -0
  24. data/docs/decisions/0011-versioned-semantic-action-boundary.md +37 -0
  25. data/docs/decisions/0012-bounded-nonexecuting-analysis.md +37 -0
  26. data/docs/decisions/0013-transactional-generation-publication.md +37 -0
  27. data/docs/decisions/0014-versioned-generated-lexer.md +35 -0
  28. data/docs/decisions/0015-worker-isolated-browser-analysis.md +32 -0
  29. data/docs/decisions/0016-red-green-concrete-syntax.md +34 -0
  30. data/docs/decisions/0017-persistent-syntax-artifacts.md +35 -0
  31. data/docs/decisions/0018-conservative-incremental-syntax-reuse.md +38 -0
  32. data/docs/decisions/0019-runtime-syntax-session-boundary.md +51 -0
  33. data/docs/decisions/0020-grammar-ir-parser-contract.md +65 -0
  34. data/docs/decisions/0021-data-only-parser-table-sidecar.md +50 -0
  35. data/docs/decisions/0022-manifest-bound-verification-report.md +77 -0
  36. data/docs/decisions/0023-syntax-only-repair-results.md +48 -0
  37. data/docs/decisions/0024-direct-ielr-construction.md +42 -0
  38. data/docs/decisions/README.md +74 -0
  39. data/docs/declarative-configuration.md +262 -0
  40. data/docs/development.md +179 -2
  41. data/docs/direct-ielr-decision.md +120 -0
  42. data/docs/direct-multi-entry-decision.md +82 -0
  43. data/docs/editor-setup.md +2 -1
  44. data/docs/error-ux-review-rubric-v1.md +117 -0
  45. data/docs/error-ux-reviews/v1/records/README.md +22 -0
  46. data/docs/error-ux-round2-review-status-v1.json +27 -0
  47. data/docs/error-ux-round2-reviews/v1/records/README.md +30 -0
  48. data/docs/error-ux-round2-v1.json +1581 -0
  49. data/docs/error-ux-round2.md +111 -0
  50. data/docs/error-ux.md +54 -0
  51. data/docs/getting-started.md +80 -0
  52. data/docs/grammar-reference.md +184 -13
  53. data/docs/ielr-design.md +43 -0
  54. data/docs/ielr.md +92 -0
  55. data/docs/ja/bison-import.md +51 -0
  56. data/docs/ja/stable-api.md +53 -0
  57. data/docs/lexer-construction-profile.md +109 -0
  58. data/docs/lexer-migration.md +5 -0
  59. data/docs/maturity.md +128 -0
  60. data/docs/project-site-strategy.md +54 -0
  61. data/docs/racc-migration-evidence.md +45 -0
  62. data/docs/racc-migration.md +11 -0
  63. data/docs/release-readiness.md +126 -68
  64. data/docs/repair-semantics.md +123 -0
  65. data/docs/runtime-abi-evolution.md +307 -0
  66. data/docs/stability.md +144 -34
  67. data/docs/status.md +43 -0
  68. data/docs/syntax-sessions.md +183 -0
  69. data/docs/table-artifact.md +125 -0
  70. data/docs/test-interactions.md +219 -0
  71. data/docs/verification-report.md +137 -0
  72. data/docs/verifier-trust-boundary.md +199 -0
  73. data/docs/workloads.md +132 -0
  74. data/lib/ibex/analysis/digraph.rb +128 -0
  75. data/lib/ibex/analysis.rb +1 -0
  76. data/lib/ibex/bison_import/importer.rb +516 -0
  77. data/lib/ibex/bison_import/tokenizer.rb +260 -0
  78. data/lib/ibex/bison_import.rb +200 -0
  79. data/lib/ibex/bounded_subprocess.rb +171 -0
  80. data/lib/ibex/cli/ambiguity.rb +14 -5
  81. data/lib/ibex/cli/analysis.rb +134 -0
  82. data/lib/ibex/cli/bison_import.rb +121 -0
  83. data/lib/ibex/cli/config.rb +128 -0
  84. data/lib/ibex/cli/diagnostics.rb +31 -11
  85. data/lib/ibex/cli/documentation.rb +11 -8
  86. data/lib/ibex/cli/equiv.rb +169 -0
  87. data/lib/ibex/cli/error_messages.rb +11 -4
  88. data/lib/ibex/cli/explain.rb +14 -5
  89. data/lib/ibex/cli/fix.rb +203 -0
  90. data/lib/ibex/cli/formatting.rb +11 -10
  91. data/lib/ibex/cli/fuzz.rb +200 -0
  92. data/lib/ibex/cli/fuzz_regressions.rb +145 -0
  93. data/lib/ibex/cli/generation_artifacts.rb +53 -2
  94. data/lib/ibex/cli/generation_error_messages.rb +1 -1
  95. data/lib/ibex/cli/grammar_tests.rb +32 -14
  96. data/lib/ibex/cli/ir_tools.rb +5 -61
  97. data/lib/ibex/cli/outputs.rb +60 -33
  98. data/lib/ibex/cli/reduce.rb +202 -0
  99. data/lib/ibex/cli/reduce_reporting.rb +71 -0
  100. data/lib/ibex/cli/samples.rb +49 -23
  101. data/lib/ibex/cli/verify.rb +116 -0
  102. data/lib/ibex/cli/watch.rb +6 -5
  103. data/lib/ibex/cli.rb +357 -56
  104. data/lib/ibex/codegen/action_locations.rb +1 -1
  105. data/lib/ibex/codegen/action_method_source.rb +5 -6
  106. data/lib/ibex/codegen/action_source.rb +3 -2
  107. data/lib/ibex/codegen/ambiguity.rb +5 -4
  108. data/lib/ibex/codegen/explain.rb +123 -55
  109. data/lib/ibex/codegen/generated_action_abi.rb +10 -5
  110. data/lib/ibex/codegen/rbs.rb +15 -7
  111. data/lib/ibex/codegen/report.rb +25 -13
  112. data/lib/ibex/codegen/ruby.rb +28 -64
  113. data/lib/ibex/codegen/ruby_actions.rb +9 -6
  114. data/lib/ibex/codegen/ruby_syntax.rb +15 -9
  115. data/lib/ibex/configuration/analysis_grammar.rb +35 -0
  116. data/lib/ibex/configuration/explanation.rb +421 -0
  117. data/lib/ibex/configuration/inspector.rb +232 -0
  118. data/lib/ibex/configuration.rb +580 -0
  119. data/lib/ibex/coverage/collector.rb +21 -11
  120. data/lib/ibex/coverage/event_stream.rb +13 -7
  121. data/lib/ibex/coverage/report.rb +26 -18
  122. data/lib/ibex/coverage/runtime_event_validator.rb +30 -21
  123. data/lib/ibex/delta_reducer.rb +99 -0
  124. data/lib/ibex/diff.rb +140 -0
  125. data/lib/ibex/equiv/machine.rb +135 -0
  126. data/lib/ibex/equiv.rb +373 -0
  127. data/lib/ibex/fix.rb +557 -0
  128. data/lib/ibex/frontend/ast.rb +22 -2
  129. data/lib/ibex/frontend/bootstrap_parser.rb +6 -13
  130. data/lib/ibex/frontend/diagnostic.rb +1 -1
  131. data/lib/ibex/frontend/formatter.rb +72 -26
  132. data/lib/ibex/frontend/generated_parser.rb +147 -117
  133. data/lib/ibex/frontend/generated_parser_base.rb +7 -3
  134. data/lib/ibex/frontend/generated_parser_includes.rb +1 -11
  135. data/lib/ibex/frontend/generation.rb +1 -1
  136. data/lib/ibex/frontend/parser/declarations.rb +17 -2
  137. data/lib/ibex/frontend/parser_configuration_support.rb +56 -0
  138. data/lib/ibex/frontend/regenerator.rb +1 -0
  139. data/lib/ibex/frontend/resolution.rb +2 -1
  140. data/lib/ibex/frontend/resolver.rb +1 -1
  141. data/lib/ibex/frontend/rule_documentation.rb +2 -1
  142. data/lib/ibex/frontend/source_cursor.rb +2 -2
  143. data/lib/ibex/frontend/source_span.rb +9 -1
  144. data/lib/ibex/frontend/token_adapter/declaration_state.rb +38 -6
  145. data/lib/ibex/frontend/token_adapter.rb +10 -0
  146. data/lib/ibex/frontend.rb +1 -0
  147. data/lib/ibex/fuzz.rb +219 -0
  148. data/lib/ibex/generation_input.rb +1 -1
  149. data/lib/ibex/generation_manifest.rb +36 -25
  150. data/lib/ibex/generation_transaction.rb +1 -1
  151. data/lib/ibex/generation_transaction_recovery.rb +4 -4
  152. data/lib/ibex/generation_transaction_validation.rb +2 -2
  153. data/lib/ibex/grammar_tests.rb +1 -1
  154. data/lib/ibex/ir/automaton_ir.rb +61 -19
  155. data/lib/ibex/ir/grammar_ir.rb +76 -45
  156. data/lib/ibex/ir/lexer_ir.rb +5 -5
  157. data/lib/ibex/ir/parser_contract.rb +135 -0
  158. data/lib/ibex/ir/serialize.rb +203 -76
  159. data/lib/ibex/ir/validator/automaton.rb +64 -34
  160. data/lib/ibex/ir/validator/base.rb +51 -33
  161. data/lib/ibex/ir/validator/grammar.rb +113 -83
  162. data/lib/ibex/ir/validator/lexer.rb +2 -2
  163. data/lib/ibex/ir/validator.rb +2 -1
  164. data/lib/ibex/ir.rb +16 -3
  165. data/lib/ibex/lalr/build_metrics.rb +50 -2
  166. data/lib/ibex/lalr/builder.rb +191 -56
  167. data/lib/ibex/lalr/conflict_search.rb +15 -4
  168. data/lib/ibex/lalr/counterexample.rb +19 -14
  169. data/lib/ibex/lalr/direct_lookaheads.rb +110 -57
  170. data/lib/ibex/lalr/goto_follows.rb +229 -0
  171. data/lib/ibex/lalr/ielr/annotator.rb +214 -0
  172. data/lib/ibex/lalr/ielr/bits.rb +28 -0
  173. data/lib/ibex/lalr/ielr/inadequacy.rb +47 -0
  174. data/lib/ibex/lalr/ielr/item_lookaheads.rb +78 -0
  175. data/lib/ibex/lalr/ielr/pipeline.rb +75 -0
  176. data/lib/ibex/lalr/ielr/split_stability.rb +78 -0
  177. data/lib/ibex/lalr/ielr/split_state.rb +20 -0
  178. data/lib/ibex/lalr/ielr/state_splitter.rb +258 -0
  179. data/lib/ibex/lalr/ielr_partition.rb +22 -8
  180. data/lib/ibex/lalr/inadequacy_report.rb +50 -0
  181. data/lib/ibex/lalr/lookahead_propagation.rb +111 -0
  182. data/lib/ibex/lalr/lr0_collection.rb +121 -0
  183. data/lib/ibex/lalr/unreachable_states.rb +80 -0
  184. data/lib/ibex/lalr.rb +52 -1
  185. data/lib/ibex/location.rb +3 -3
  186. data/lib/ibex/lsp/document_handlers.rb +9 -7
  187. data/lib/ibex/lsp/initialization_handlers.rb +9 -7
  188. data/lib/ibex/lsp/navigation_handlers.rb +25 -11
  189. data/lib/ibex/lsp/parser_configuration_assistance.rb +149 -0
  190. data/lib/ibex/lsp/position_codec.rb +6 -2
  191. data/lib/ibex/lsp/request_handlers.rb +3 -2
  192. data/lib/ibex/lsp/request_support.rb +14 -7
  193. data/lib/ibex/lsp/server.rb +13 -11
  194. data/lib/ibex/lsp/symbol_index.rb +13 -13
  195. data/lib/ibex/lsp/symbol_index_builder.rb +30 -20
  196. data/lib/ibex/lsp/symbol_occurrence.rb +8 -2
  197. data/lib/ibex/lsp/transport.rb +13 -3
  198. data/lib/ibex/lsp/workspace.rb +5 -4
  199. data/lib/ibex/lsp/workspace_analyzer.rb +1 -1
  200. data/lib/ibex/lsp.rb +1 -0
  201. data/lib/ibex/messages/en.yml +28 -0
  202. data/lib/ibex/messages/ja.yml +28 -0
  203. data/lib/ibex/messages.rb +68 -0
  204. data/lib/ibex/metrics.rb +175 -0
  205. data/lib/ibex/normalize/declarations.rb +2 -0
  206. data/lib/ibex/normalize/diagnostics.rb +63 -38
  207. data/lib/ibex/normalize/expander.rb +5 -4
  208. data/lib/ibex/normalize/expression.rb +8 -6
  209. data/lib/ibex/normalize/grammar_builder.rb +26 -0
  210. data/lib/ibex/normalize/inline_expansion.rb +100 -49
  211. data/lib/ibex/normalize/lexer.rb +3 -3
  212. data/lib/ibex/normalize/parameter_ebnf_lowering.rb +11 -10
  213. data/lib/ibex/normalize/parameter_lowering.rb +40 -23
  214. data/lib/ibex/normalize/parameters.rb +31 -4
  215. data/lib/ibex/normalize/parser_configuration.rb +70 -0
  216. data/lib/ibex/normalize.rb +10 -20
  217. data/lib/ibex/racc_migration/report.rb +6 -3
  218. data/lib/ibex/rake_task.rb +1 -1
  219. data/lib/ibex/samples.rb +43 -13
  220. data/lib/ibex/table_artifact/builder.rb +266 -0
  221. data/lib/ibex/table_artifact/cst_projection.rb +70 -0
  222. data/lib/ibex/table_artifact/document.rb +39 -0
  223. data/lib/ibex/table_artifact/executor.rb +187 -0
  224. data/lib/ibex/table_artifact/serializer.rb +57 -0
  225. data/lib/ibex/table_artifact/validator/metadata.rb +179 -0
  226. data/lib/ibex/table_artifact/validator/support.rb +86 -0
  227. data/lib/ibex/table_artifact/validator/tables.rb +238 -0
  228. data/lib/ibex/table_artifact/validator.rb +253 -0
  229. data/lib/ibex/table_artifact.rb +88 -0
  230. data/lib/ibex/table_simulation/result.rb +6 -2
  231. data/lib/ibex/table_simulation/step.rb +3 -1
  232. data/lib/ibex/tables.rb +14 -0
  233. data/lib/ibex/verifiable_generation_bundle.rb +86 -0
  234. data/lib/ibex/verification_report/builder.rb +159 -0
  235. data/lib/ibex/verification_report/canonical_ir.rb +123 -0
  236. data/lib/ibex/verification_report/logical_path.rb +68 -0
  237. data/lib/ibex/verification_report/validator.rb +329 -0
  238. data/lib/ibex/verification_report.rb +69 -0
  239. data/lib/ibex/verify/action_correspondence.rb +138 -0
  240. data/lib/ibex/verify/language_witness.rb +295 -0
  241. data/lib/ibex/verify/reference_collection.rb +190 -0
  242. data/lib/ibex/verify/result.rb +73 -0
  243. data/lib/ibex/verify/verifier.rb +598 -0
  244. data/lib/ibex/verify.rb +15 -0
  245. data/lib/ibex/version.rb +1 -1
  246. data/lib/ibex/watch/runner.rb +10 -1
  247. data/lib/ibex/watch/source_snapshot.rb +13 -8
  248. data/lib/ibex.rb +10 -0
  249. data/schema/{automaton-ir-v1.schema.json → automaton-ir-definitions.schema.json} +6 -6
  250. data/schema/{automaton-ir-v2.schema.json → automaton-ir.schema.json} +14 -7
  251. data/schema/bison-import-v1.schema.json +106 -0
  252. data/schema/conflict-explanation-review-v1.schema.json +196 -0
  253. data/schema/conflict-explanation-study-v1.schema.json +528 -0
  254. data/schema/construction-profile-v1.schema.json +779 -0
  255. data/schema/diff-v1.schema.json +41 -0
  256. data/schema/direct-ielr-decision-v1.schema.json +200 -0
  257. data/schema/equiv-v1.schema.json +64 -0
  258. data/schema/error-ux-review-v1.schema.json +609 -0
  259. data/schema/error-ux-round2-review-v1.schema.json +191 -0
  260. data/schema/error-ux-round2-v1.schema.json +519 -0
  261. data/schema/explain-v1.schema.json +63 -1
  262. data/schema/fix-v1.schema.json +64 -0
  263. data/schema/fix-v2.schema.json +66 -0
  264. data/schema/fix-v3.schema.json +101 -0
  265. data/schema/fuzz-regression-v1.schema.json +79 -0
  266. data/schema/fuzz-v1.schema.json +114 -0
  267. data/schema/{grammar-ir-v2.schema.json → grammar-ir-extensions.schema.json} +20 -65
  268. data/schema/{grammar-ir-v1.schema.json → grammar-ir-foundation.schema.json} +3 -3
  269. data/schema/grammar-ir.schema.json +271 -0
  270. data/schema/ielr-benchmark-v1.schema.json +85 -0
  271. data/schema/lexer-profile-v1.schema.json +491 -0
  272. data/schema/metrics-v1.schema.json +60 -0
  273. data/schema/reduce-v1.schema.json +22 -0
  274. data/schema/reduce-v2.schema.json +63 -0
  275. data/schema/table-artifact-v1.schema.json +377 -0
  276. data/schema/verification-report-v1.schema.json +222 -0
  277. data/schema/verify-v1.schema.json +42 -0
  278. data/sig/ibex/analysis/digraph.rbs +19 -0
  279. data/sig/ibex/bison_import/importer.rbs +140 -0
  280. data/sig/ibex/bison_import/tokenizer.rbs +91 -0
  281. data/sig/ibex/bison_import.rbs +95 -0
  282. data/sig/ibex/bounded_subprocess.rbs +76 -0
  283. data/sig/ibex/cli/ambiguity.rbs +6 -0
  284. data/sig/ibex/cli/analysis.rbs +43 -0
  285. data/sig/ibex/cli/bison_import.rbs +32 -0
  286. data/sig/ibex/cli/config.rbs +35 -0
  287. data/sig/ibex/cli/diagnostics.rbs +16 -3
  288. data/sig/ibex/cli/documentation.rbs +5 -1
  289. data/sig/ibex/cli/equiv.rbs +49 -0
  290. data/sig/ibex/cli/error_messages.rbs +6 -0
  291. data/sig/ibex/cli/explain.rbs +6 -0
  292. data/sig/ibex/cli/fix.rbs +53 -0
  293. data/sig/ibex/cli/formatting.rbs +5 -1
  294. data/sig/ibex/cli/fuzz.rbs +47 -0
  295. data/sig/ibex/cli/fuzz_regressions.rbs +40 -0
  296. data/sig/ibex/cli/generation_artifacts.rbs +21 -2
  297. data/sig/ibex/cli/generation_error_messages.rbs +2 -2
  298. data/sig/ibex/cli/grammar_tests.rbs +15 -3
  299. data/sig/ibex/cli/ir_tools.rbs +7 -19
  300. data/sig/ibex/cli/outputs.rbs +9 -3
  301. data/sig/ibex/cli/reduce.rbs +60 -0
  302. data/sig/ibex/cli/reduce_reporting.rbs +27 -0
  303. data/sig/ibex/cli/samples.rbs +8 -0
  304. data/sig/ibex/cli/verify.rbs +28 -0
  305. data/sig/ibex/cli/watch.rbs +2 -2
  306. data/sig/ibex/cli.rbs +79 -5
  307. data/sig/ibex/codegen/action_method_source.rbs +4 -4
  308. data/sig/ibex/codegen/explain.rbs +45 -25
  309. data/sig/ibex/codegen/generated_action_abi.rbs +14 -10
  310. data/sig/ibex/codegen/rbs.rbs +12 -10
  311. data/sig/ibex/codegen/report.rbs +12 -12
  312. data/sig/ibex/codegen/ruby.rbs +18 -15
  313. data/sig/ibex/codegen/ruby_actions.rbs +6 -6
  314. data/sig/ibex/codegen/ruby_syntax.rbs +10 -8
  315. data/sig/ibex/configuration/analysis_grammar.rbs +13 -0
  316. data/sig/ibex/configuration/explanation.rbs +195 -0
  317. data/sig/ibex/configuration/inspector.rbs +53 -0
  318. data/sig/ibex/configuration.rbs +242 -0
  319. data/sig/ibex/coverage/collector.rbs +16 -14
  320. data/sig/ibex/coverage/event_stream.rbs +12 -10
  321. data/sig/ibex/coverage/report.rbs +20 -18
  322. data/sig/ibex/coverage/runtime_event_validator.rbs +38 -36
  323. data/sig/ibex/delta_reducer.rbs +38 -0
  324. data/sig/ibex/diff.rbs +47 -0
  325. data/sig/ibex/equiv/machine.rbs +54 -0
  326. data/sig/ibex/equiv.rbs +129 -0
  327. data/sig/ibex/fix.rbs +138 -0
  328. data/sig/ibex/frontend/ast.rbs +33 -7
  329. data/sig/ibex/frontend/bootstrap_parser.rbs +2 -0
  330. data/sig/ibex/frontend/diagnostic.rbs +2 -2
  331. data/sig/ibex/frontend/formatter.rbs +38 -30
  332. data/sig/ibex/frontend/generated_parser.rbs +87 -77
  333. data/sig/ibex/frontend/generated_parser_base.rbs +4 -2
  334. data/sig/ibex/frontend/parser/declarations.rbs +3 -0
  335. data/sig/ibex/frontend/parser_configuration_support.rbs +27 -0
  336. data/sig/ibex/frontend/resolution.rbs +3 -2
  337. data/sig/ibex/frontend/source_cursor.rbs +2 -2
  338. data/sig/ibex/frontend/source_span.rbs +5 -2
  339. data/sig/ibex/frontend/token_adapter/declaration_state.rbs +13 -1
  340. data/sig/ibex/frontend/token_adapter.rbs +6 -0
  341. data/sig/ibex/fuzz.rbs +71 -0
  342. data/sig/ibex/generation_input.rbs +2 -2
  343. data/sig/ibex/generation_manifest.rbs +35 -32
  344. data/sig/ibex/generation_transaction_recovery.rbs +2 -2
  345. data/sig/ibex/generation_transaction_validation.rbs +2 -2
  346. data/sig/ibex/grammar_tests.rbs +2 -2
  347. data/sig/ibex/ir/automaton_ir.rbs +30 -11
  348. data/sig/ibex/ir/grammar_ir.rbs +41 -27
  349. data/sig/ibex/ir/lexer_ir.rbs +8 -8
  350. data/sig/ibex/ir/parser_contract.rbs +69 -0
  351. data/sig/ibex/ir/serialize.rbs +63 -28
  352. data/sig/ibex/ir/validator/automaton.rbs +47 -44
  353. data/sig/ibex/ir/validator/base.rbs +35 -31
  354. data/sig/ibex/ir/validator/grammar.rbs +78 -67
  355. data/sig/ibex/ir/validator/lexer.rbs +4 -4
  356. data/sig/ibex/ir.rbs +9 -3
  357. data/sig/ibex/lalr/build_metrics.rbs +42 -2
  358. data/sig/ibex/lalr/builder.rbs +75 -24
  359. data/sig/ibex/lalr/conflict_search.rbs +5 -2
  360. data/sig/ibex/lalr/counterexample.rbs +17 -14
  361. data/sig/ibex/lalr/direct_lookaheads.rbs +35 -28
  362. data/sig/ibex/lalr/goto_follows.rbs +89 -0
  363. data/sig/ibex/lalr/ielr/annotator.rbs +67 -0
  364. data/sig/ibex/lalr/ielr/bits.rbs +14 -0
  365. data/sig/ibex/lalr/ielr/inadequacy.rbs +24 -0
  366. data/sig/ibex/lalr/ielr/item_lookaheads.rbs +35 -0
  367. data/sig/ibex/lalr/ielr/pipeline.rbs +17 -0
  368. data/sig/ibex/lalr/ielr/split_stability.rbs +27 -0
  369. data/sig/ibex/lalr/ielr/split_state.rbs +14 -0
  370. data/sig/ibex/lalr/ielr/state_splitter.rbs +76 -0
  371. data/sig/ibex/lalr/ielr_partition.rbs +17 -5
  372. data/sig/ibex/lalr/inadequacy_report.rbs +21 -0
  373. data/sig/ibex/lalr/lookahead_propagation.rbs +38 -0
  374. data/sig/ibex/lalr/lr0_collection.rbs +50 -0
  375. data/sig/ibex/lalr/unreachable_states.rbs +25 -0
  376. data/sig/ibex/lalr.rbs +11 -1
  377. data/sig/ibex/location.rbs +6 -6
  378. data/sig/ibex/lsp/document_handlers.rbs +8 -8
  379. data/sig/ibex/lsp/initialization_handlers.rbs +12 -12
  380. data/sig/ibex/lsp/navigation_handlers.rbs +15 -12
  381. data/sig/ibex/lsp/parser_configuration_assistance.rbs +49 -0
  382. data/sig/ibex/lsp/position_codec.rbs +8 -4
  383. data/sig/ibex/lsp/request_handlers.rbs +4 -4
  384. data/sig/ibex/lsp/request_support.rbs +8 -8
  385. data/sig/ibex/lsp/server.rbs +16 -16
  386. data/sig/ibex/lsp/symbol_index.rbs +20 -21
  387. data/sig/ibex/lsp/symbol_index_builder.rbs +9 -8
  388. data/sig/ibex/lsp/symbol_occurrence.rbs +11 -5
  389. data/sig/ibex/lsp/transport.rbs +18 -6
  390. data/sig/ibex/lsp/workspace.rbs +3 -2
  391. data/sig/ibex/lsp/workspace_analyzer.rbs +1 -1
  392. data/sig/ibex/messages.rbs +26 -0
  393. data/sig/ibex/metrics.rbs +50 -0
  394. data/sig/ibex/normalize/diagnostics.rbs +4 -4
  395. data/sig/ibex/normalize/expander.rbs +2 -2
  396. data/sig/ibex/normalize/expression.rbs +10 -8
  397. data/sig/ibex/normalize/grammar_builder.rbs +11 -0
  398. data/sig/ibex/normalize/inline_expansion.rbs +59 -50
  399. data/sig/ibex/normalize/lexer.rbs +4 -4
  400. data/sig/ibex/normalize/parameter_ebnf_lowering.rbs +15 -14
  401. data/sig/ibex/normalize/parameter_lowering.rbs +18 -18
  402. data/sig/ibex/normalize/parameters.rbs +13 -8
  403. data/sig/ibex/normalize/parser_configuration.rbs +23 -0
  404. data/sig/ibex/normalize.rbs +11 -5
  405. data/sig/ibex/racc_migration/report.rbs +8 -6
  406. data/sig/ibex/rake_task.rbs +1 -1
  407. data/sig/ibex/samples.rbs +13 -4
  408. data/sig/ibex/table_artifact/builder.rbs +93 -0
  409. data/sig/ibex/table_artifact/cst_projection.rbs +20 -0
  410. data/sig/ibex/table_artifact/document.rbs +27 -0
  411. data/sig/ibex/table_artifact/executor.rbs +66 -0
  412. data/sig/ibex/table_artifact/serializer.rbs +25 -0
  413. data/sig/ibex/table_artifact/validator/metadata.rbs +52 -0
  414. data/sig/ibex/table_artifact/validator/support.rbs +46 -0
  415. data/sig/ibex/table_artifact/validator/tables.rbs +63 -0
  416. data/sig/ibex/table_artifact/validator.rbs +66 -0
  417. data/sig/ibex/table_artifact.rbs +32 -0
  418. data/sig/ibex/table_simulation/result.rbs +8 -4
  419. data/sig/ibex/table_simulation/step.rbs +4 -2
  420. data/sig/ibex/tables.rbs +10 -9
  421. data/sig/ibex/verifiable_generation_bundle.rbs +24 -0
  422. data/sig/ibex/verification_report/builder.rbs +53 -0
  423. data/sig/ibex/verification_report/canonical_ir.rbs +41 -0
  424. data/sig/ibex/verification_report/logical_path.rbs +34 -0
  425. data/sig/ibex/verification_report/validator.rbs +114 -0
  426. data/sig/ibex/verification_report.rbs +44 -0
  427. data/sig/ibex/verify/action_correspondence.rbs +50 -0
  428. data/sig/ibex/verify/language_witness.rbs +112 -0
  429. data/sig/ibex/verify/reference_collection.rbs +63 -0
  430. data/sig/ibex/verify/result.rbs +52 -0
  431. data/sig/ibex/verify/verifier.rbs +170 -0
  432. data/sig/ibex/verify.rbs +6 -0
  433. data/sig/ibex/watch/runner.rbs +12 -2
  434. data/sig/ibex/watch/source_snapshot.rbs +13 -9
  435. metadata +232 -12
  436. data/lib/ibex/ir/migration.rb +0 -120
  437. data/sig/ibex/ir/migration.rbs +0 -34
data/docs/ielr.md ADDED
@@ -0,0 +1,92 @@
1
+ ---
2
+ title: IELR(1) construction
3
+ description: The opt-in IELR construction strategies and their verification boundary.
4
+ ---
5
+
6
+ # IELR(1) construction
7
+
8
+ Ibex exposes IELR(1) as an opt-in parser-construction algorithm. It keeps the
9
+ same Grammar IR, Automaton IR, conflict resolver, table generator, and runtime
10
+ as SLR, LALR, and canonical LR(1). The default remains the compatible LALR
11
+ path; selecting IELR does not change generated parser APIs.
12
+
13
+ ## Strategies
14
+
15
+ `--algorithm=ielr` accepts two construction strategies:
16
+
17
+ - `partition` (the default) constructs canonical LR(1) states and partitions
18
+ compatible isocores. It is the conservative reference strategy and remains
19
+ the release baseline.
20
+ - `direct` starts with LR(0), computes LALR lookaheads and goto-follow
21
+ relations, traces grammar-relative inadequacies, splits only affected
22
+ isocores, and recalculates lookaheads on the split graph. It does not build a
23
+ canonical collection in the normal construction path.
24
+
25
+ The direct strategy is deliberately not a claim of globally minimal state
26
+ count. Its benefit is bounded construction work and phase-level evidence, not
27
+ an unconditional reduction in the final table. `--remove-unreachable` is
28
+ available as an explicit post-resolution compaction; it is off by default so
29
+ state numbers used by diagnostics and generated artifacts remain stable.
30
+
31
+ ```sh
32
+ ibex --algorithm=ielr --ielr-strategy=direct grammar.y
33
+ ibex --algorithm=ielr --ielr-report grammar.y
34
+ ibex --algorithm=ielr --ielr-strategy=direct --remove-unreachable grammar.y
35
+ ```
36
+
37
+ `--ielr-report` is diagnostic-only. It constructs a canonical reference and
38
+ reports action differences; it must not be confused with the direct build
39
+ path. The report is useful for review and regression tests, but it is not a
40
+ proof that every viable prefix has a unique state correspondence.
41
+
42
+ ## Construction phases
43
+
44
+ The direct path is organized as the following explicit boundaries:
45
+
46
+ 1. `LR0Collection`, `LookaheadPropagation`, and `GotoFollows` build the LR(0)
47
+ collection, direct reads, successor/internal/includes relations, and LALR
48
+ lookaheads. `DirectLookaheads` remains the existing compatibility-path
49
+ cross-check.
50
+ 2. `IELR::Annotator` and `ItemLookaheads` classify conflicts and propagate
51
+ annotations over predecessor lanes. `SplitStability` discards annotations
52
+ whose resolved action cannot change under a split.
53
+ 3. `IELR::StateSplitter` copies transition tables for incompatible isocores and
54
+ preserves deterministic construction order.
55
+ 4. `LookaheadPropagation` reruns item-level propagation over the split graph.
56
+ 5. The existing `Builder` phase resolves conflicts and applies error/default
57
+ reductions, preserving the common Automaton IR contract.
58
+
59
+ All lookahead sets are integer bitsets. `Analysis::Digraph` computes the
60
+ successor, always, and includes closures with an iterative worklist so deep
61
+ grammars do not depend on Ruby recursion depth.
62
+
63
+ ## Metrics and reports
64
+
65
+ Profiled builds expose `ielr_annotations`, `ielr_annotated_states`,
66
+ `ielr_inadequacies`, `ielr_split_stable_discarded`, `ielr_lalr_states`,
67
+ `ielr_split_states`, `ielr_unreachable_removed`,
68
+ `ielr_remergeable_candidates`, and the existing partition counts. Use
69
+ `tool/construction_profile.rb --ielr-strategy=direct` to capture the direct
70
+ measurements. Wall-clock values remain observations, not release gates.
71
+
72
+ The phase implementation is covered by the paper-derived fixtures in
73
+ `test/fixtures/ielr` and the structural tests in
74
+ `test/lalr/ielr_direct_test.rb`. The direct strategy remains experimental
75
+ until representative real-workload evidence and an independent adequacy
76
+ review satisfy the direct-IELR decision dossier.
77
+
78
+ ## Verification boundary
79
+
80
+ `Verify::ActionCorrespondence` and `LALR::InadequacyReport` are explicit
81
+ diagnostic tools. They compare resolved actions and preserve a bounded search
82
+ budget, but canonical-vs-target state pairing is not assumed to be a function.
83
+ `Verify::LanguageWitness` supplies the strict IELR V9 bounded acceptance
84
+ comparison; it reports token-level acceptance witnesses and fails closed when
85
+ its finite case budget is exhausted. It does not prove unbounded equivalence,
86
+ conflict preservation, or split correctness. Normal parser construction never
87
+ invokes these diagnostics. Existing canonical-core verification remains the
88
+ trusted boundary for release decisions.
89
+
90
+ The implementation is independently derived from the published IELR paper and
91
+ the repository's parser specifications. GNU Bison or another GPL
92
+ implementation is not a design input.
@@ -0,0 +1,51 @@
1
+ # Bison 文法の解析用インポート
2
+
3
+ `ibex import bison` は Bison 形式の文法構造を、解析専用の extended Ibex
4
+ ソースへ一方向に変換します。互換パーサを生成するモードではありません。C
5
+ アクションは不透明なまま保持され、解析中には実行されず、Ruby 生成はコードを
6
+ 出力する前に拒否されます。
7
+
8
+ ```sh
9
+ ibex import bison parser.y -o parser.analysis.y
10
+ ibex import bison --format=json parser.y > import-report.json
11
+ ibex explain --state=STATE parser.y
12
+ ibex metrics parser.y
13
+ ```
14
+
15
+ 2 個の `%%` 区切りを持つファイルは `explain` や `metrics` から直接解析
16
+ できます。修復差分は変換後ソースのバイト位置に対するものなので、修復には
17
+ 明示的に出力した canonical 解析ファイルが必要です。
18
+
19
+ 対応指令は次の3種類です。
20
+
21
+ - `%token`、結合規則、`%start`、`%expect`、`%empty`、`%prec` は既存の
22
+ Ibex 構文へ変換します。
23
+ - `%type`、`%union`、printer/destructor、parameter、`%define`、`%code`
24
+ は位置付き metadata として報告します。
25
+ - GLR、`%dprec`、`%merge`、生成制御、未知の指令は、出現ごとに元の行・列を
26
+ 付けて未対応として報告します。
27
+
28
+ `$$`、`$1`、型付き参照、`@$` は機械的に変換されますが、C の意味を Ruby
29
+ として解釈しません。大文字の非終端、予約語、正規化後の名前衝突も、
30
+ `bison_nt_*` / `BISON_T_*` と決定的な suffix で分離します。
31
+
32
+ レポートと生成ソースは、単なる未対応と「生成規則を完全には復元できない」
33
+ 構造的不完全を区別します。後者には
34
+ `ibex-bison-structural-status: incomplete` が付き、`ibex fix` は安全な
35
+ 同値修復を主張せず拒否します。
36
+
37
+ 既定上限は入力 20 MiB、構造 token 1,000,000、rule group 50,000、action
38
+ 100,000 です。`--max-*` は正の整数だけを受け付け、上限超過は JSON で明示し
39
+ 終了コード 2 になります。`-o` は原子的に書き込み、入出力 alias、symlink、
40
+ 複数 hard link を拒否します。
41
+
42
+ 外部 CI は5個の checksum 固定 GNU Bison 文法を一時取得し、成果物には含め
43
+ ません。Bison 時代の CRuby `parse.y` では production 781 と未解決競合 0 が
44
+ 一致し、Bison 1,304 state と Ibex 1,303 state の差は、Bison が EOF shift
45
+ 専用 accept state を持つためです。
46
+
47
+ 現行 CRuby `parse.y` は Lrama 拡張を含み、GNU Bison の生入力ではありません。
48
+ Ibex は `%rule` などを構造的不完全として全列挙し、限定した `explain` は
49
+ 近似文法の証拠としてのみ提示します。この状態で `fix` を提案しないことが安全
50
+ 契約です。固定 revision、license、取得元は
51
+ [`gallery/EXTERNAL.md`](../../gallery/EXTERNAL.md) にあります。
@@ -0,0 +1,53 @@
1
+ # v1 安定 API
2
+
3
+ この文書は v1 で Stable と判定された公開契約の日本語案内です。詳細な
4
+ 構文表や全メソッド一覧はリンク先の英語リファレンスを正本としますが、
5
+ 下記の契約、互換性、失敗条件は日本語でも同じ意味を持ちます。
6
+
7
+ ## racc 互換パーサ生成
8
+
9
+ <!-- stable:compatible-parser:v2 -->
10
+
11
+ 既定モードは racc 互換の文法を読み、Pure Ruby の LR パーサを生成します。
12
+ 生成クラスは `do_parse` と `yyparse` を提供します。既定の直接 LALR 構築、
13
+ plain/compact 表、yacc `error` 回復、コールバック、観測イベント、
14
+ `ResourceLimits`、移行検査、境界付きの競合反例解析が Stable です。
15
+ 互換モードを明示せずに拡張構文へ切り替えることはありません。構文と CLI
16
+ の詳細は [`grammar-reference.md`](../grammar-reference.md) を参照してください。
17
+
18
+ `ibex fix` が検証済み修復として出すのは、対象競合を除去し、独立検証と
19
+ 有界同値検査を通った提案だけです。`%expect` と回復品質の設定は競合を
20
+ 除去しないため、修復や `--apply` 対象ではなく未検証の助言として分離して
21
+ 表示します。
22
+
23
+ ## Current IR と表
24
+
25
+ <!-- stable:current-ir:v1 -->
26
+
27
+ Grammar IR と Automaton IR は一つの current closed schema を使い、Lexer IR は
28
+ 独立した schema を持ちます。検証器は未知フィールド、不正な参照、schema の
29
+ 不一致を拒否します。dump → load → dump は決定的です。v1.0 前の互換性のない
30
+ 変更は writer、reader、schema、fixture、利用側を同じ変更で更新します。詳しい境界は
31
+ [`architecture.md`](../architecture.md) にあります。
32
+
33
+ ## バッチ Red/Green CST
34
+
35
+ <!-- stable:batch-cst:v1 -->
36
+
37
+ format v6 のバッチ CST は、損失のない Green 値と位置・親参照を持つ Red
38
+ ビューを分離します。typed syntax view、永続編集、差分、`ibex_cst` schema
39
+ v1 の直列化が Stable です。エラー入力も `Error` / `Missing` 要素として木に
40
+ 残ります。syntax-only incremental session は Experimental であり、この
41
+ Stable 契約には含まれません。API 例は [`cst.md`](../cst.md) を参照してください。
42
+
43
+ ## 互換性と非推奨化
44
+
45
+ <!-- stable:compatibility-policy:v1 -->
46
+
47
+ Stable API は semantic versioning に従います。v1.0 以降の削除や意味変更は、
48
+ 少なくとも2回の minor release にわたって移行警告、代替手段、最短削除版を
49
+ 文書化します。v1.0 前の Preview は固定された後方互換性契約ではなく、変更時に
50
+ release note、移行メモ、回帰テスト、maturity evidence を同じ変更で更新します。
51
+ v1.0 以降の Preview は少なくとも1回の minor release で予告し、Experimental は
52
+ 予告なく変わる可能性があります。正確な在庫と昇格条件は
53
+ [`stability.md`](../stability.md) にあります。
@@ -0,0 +1,109 @@
1
+ # Generated lexer behavior and automaton decision
2
+
3
+ H006 profiles the generated lexer that exists today. It does not replace that
4
+ lexer, add an automaton implementation, or claim that the two lexical models
5
+ are interchangeable. The machine-readable authority is
6
+ [`lexer-profile-v1.json`](../tool/profile/evidence/lexer-profile-v1.json),
7
+ validated by
8
+ [`lexer-profile-v1.schema.json`](../schema/lexer-profile-v1.schema.json).
9
+
10
+ Run and verify the profile with:
11
+
12
+ ```sh
13
+ bundle exec ruby tool/lexer_profile.rb \
14
+ --output=tool/profile/evidence/lexer-profile-v1.json
15
+ bundle exec rake quality:lexer_profile
16
+ ```
17
+
18
+ The profiler generates parsers from committed grammar sources and executes
19
+ their lexer and parser actions. It is therefore limited to trusted repository
20
+ sources. It never downloads or executes the registered third-party grammars.
21
+
22
+ ## Semantics are different
23
+
24
+ The current `regexp` profile anchors one Ruby `Regexp` per declared rule. Ruby
25
+ chooses alternatives left-to-right inside that rule. Ibex then compares the
26
+ lexemes returned by every active rule, chooses the longest byte length, and
27
+ uses declaration order for a tie. These are two separate selection steps.
28
+
29
+ For example, `/a|ab/` returns `a` for input `ab` inside its one Ruby rule. A
30
+ regular-language automaton using maximal munch would select `ab`. Calling the
31
+ second result a faster implementation of the first would hide a tokenization
32
+ change.
33
+
34
+ The conditional `automaton` profile described by the design has maximal munch,
35
+ a declaration-order tie, declarative transitions, and bounded construction. It
36
+ also excludes arbitrary Ruby actions and unspecified parser-to-lexer feedback.
37
+ Those exclusions are a semantic boundary, not an implementation detail.
38
+
39
+ ## Workloads and provenance
40
+
41
+ The evidence keeps source classes separate:
42
+
43
+ - `synthetic` contains all eight required adversarial fixtures plus the three
44
+ registered gallery grammars. Every grammar and input has an exact Git
45
+ revision, path, byte digest, and derivation. The chunk fixture expands its
46
+ committed 128-byte seed 129 times, producing a 16,512-byte single token that
47
+ crosses the 16KiB streaming boundary.
48
+ - `public_real` contains the exact BCDice, Namae, and Nokogiri CSS registry
49
+ identities. They use external application lexers and have no verified Ibex
50
+ generated-lexer port, so their status is `not_run`, their result is `null`,
51
+ and they contribute no inferred measurements.
52
+
53
+ Synthetic success is not evidence that public-real lexers are suitable for an
54
+ automaton migration. Evidence is generated from a clean exact revision, and
55
+ that revision must be the first parent of the commit that records the evidence.
56
+ The capture identity covers that revision, clean/status provenance, the complete
57
+ bound execution-source closure, implementation subset, deterministic report
58
+ input, measurement policy, and heuristic policy. The verifier locates the same
59
+ identity in committed evidence history instead of accepting any ancestor with
60
+ rewritten derived fields. A shallow checkout that lacks the evidence commit's
61
+ parent or pinned fixture revisions fails closed; CI must fetch full history
62
+ rather than treating missing provenance as a match.
63
+
64
+ The capture command deliberately loads only the frontend, Grammar/Automaton IR,
65
+ normalizer, analysis/LALR construction, Ruby code generator, and generated
66
+ runtime graph. It does not load the public `ibex` aggregate, CLI, or configuration
67
+ graph. Every repository Ruby source reachable from that measurement graph is a
68
+ bound path, and capture aborts if another repository source is loaded. The public
69
+ schema and this document are explicit generator-gem packaging expectations.
70
+
71
+ ## Measurements and limits
72
+
73
+ Each measured workload records states, rules per state, token byte lengths,
74
+ action-bearing rules, direct lexer/parser state mutation, parser feedback,
75
+ Regexp warnings, streaming peak buffer bytes, and incremental bytes read from
76
+ offset zero. Elapsed time and `GC.total_allocated_objects` deltas are retained
77
+ only as host-bound diagnostics. Both the schema and semantic validator require
78
+ `release_gate: false`; the quality comparison removes their values.
79
+
80
+ The alternation, lazy-quantifier, and direct `lexer_state =` inventories are
81
+ source-text heuristics over normalized rules and actions. They do not form a
82
+ complete Ruby or Regexp semantic analysis. Aliases, helper calls,
83
+ metaprogramming, indirect mutation, or dynamically composed behavior can be
84
+ missed. A zero count is not proof of absence.
85
+
86
+ The Unicode-property fixture uses String input. The current incremental
87
+ `SourceText` path stores binary bytes, so this profile does not claim that
88
+ Unicode-property matching works through that incremental path. The
89
+ parser-feedback fixture is semantically parsed successfully, but its
90
+ syntax-only incremental scan is `not_measured`: suppressing the parser action
91
+ also suppresses the state change needed before the next token.
92
+
93
+ Peak buffer bytes cover only `LexerInput`, not whole-process memory. The
94
+ incremental full-scan share covers generated-lexer bytes read after a committed
95
+ identity edit; it does not measure CST subtree reuse.
96
+
97
+ ## Decision
98
+
99
+ | Scope | Decision | Reason |
100
+ | --- | --- | --- |
101
+ | Automatic replacement of the current Regexp lexer | **NO-GO** | `/a|ab/` proves different token boundaries; arbitrary actions and parser-driven state changes have no specified semantic translation. |
102
+ | A separately named automaton lexer profile | **MORE DATA** | All adversarial fixtures are measured, but there are zero verified public-real generated-lexer workloads, no standalone semantic contract, and no verified adoption trigger. |
103
+
104
+ Timing and allocation observations are not thresholds for either decision.
105
+ Reconsidering automatic replacement requires an explicit compatibility model,
106
+ an action/feedback contract, and differential evidence on verified real
107
+ generated lexers. Reconsidering the separate profile requires at least two such
108
+ public-real workloads, a bounded declarative semantics, and demonstrated user
109
+ pressure that the design addresses.
@@ -1,3 +1,8 @@
1
+ ---
2
+ title: Lexer migration
3
+ description: Move a handwritten lexer or adopt the explicit generated lexer contract.
4
+ ---
5
+
1
6
  # Migrating a handwritten lexer
2
7
 
3
8
  The generated lexer uses the same `next_token` contract as a handwritten pull
data/docs/maturity.md ADDED
@@ -0,0 +1,128 @@
1
+ # Preview and Experimental maturity audit
2
+
3
+ [`maturity.yml`](maturity.yml) and the validator-owned commit authorities in
4
+ [`maturity_authority.rb`](../tool/quality/maturity_authority.rb) form the joint
5
+ human-reviewed trust boundary for H001. The registry binds the current 18
6
+ Preview and two Experimental features to the reviewed repository revision,
7
+ source digests, activation, external-use status, specification-history
8
+ confidence, issue audit, documentation and tooling gaps, performance and safety
9
+ limits, decision, next trigger, and release gates. Neither file authenticates
10
+ the other; changing a classification requires reviewing and updating both.
11
+
12
+ ## Result
13
+
14
+ <!-- maturity-summary:start -->
15
+ Inventory: **18 Preview, 2 Experimental**. Active new Preview tracks: **1/3** (grammar syntax: **1/1**). Experimental product features: **2/5**.
16
+ Release dependency state: R001 **hold_external**; R002 **pending_exact_revision**; no feature is promoted by this audit.
17
+
18
+ | Stable ID | Feature | Current maturity | Decision | External use | Release gate |
19
+ | --- | --- | --- | --- | --- | --- |
20
+ | `ebnf-groups` | EBNF groups | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
21
+ | `parameterized-rules` | parameterized rules | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
22
+ | `inline-rules` | inline rules | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
23
+ | `middle-actions` | middle actions | Preview | Redesign Preview | not demonstrated | Blocked: R001, R002 |
24
+ | `multiple-entries` | multiple entries | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
25
+ | `canonical-imports` | canonical imports | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
26
+ | `generated-lexers` | generated lexers | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
27
+ | `semantic-locations-types` | semantic locations/types | Preview | Redesign Preview | not demonstrated | Blocked: R001, R002 |
28
+ | `ast-generation` | AST generation | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
29
+ | `grammar-tests` | grammar tests | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
30
+ | `documentation-tooling` | documentation tooling | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
31
+ | `ielr` | IELR | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
32
+ | `lsp` | LSP | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
33
+ | `watch` | watch | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
34
+ | `debug` | debug | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
35
+ | `coverage` | coverage | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
36
+ | `browser-playground` | browser playground | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
37
+ | `action-shadow` | action-shadow | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
38
+ | `bounded-repair` | bounded repair | Experimental | Keep Experimental | not demonstrated | Blocked: R001, R002 |
39
+ | `incremental-cst` | incremental CST | Experimental | Keep Experimental | not demonstrated | Blocked: R001, R002 |
40
+ <!-- maturity-summary:end -->
41
+
42
+ Every keep decision names unmet promotion evidence. Two rows record redesigns.
43
+ Compatible/default Racc middle actions are already governed by the Stable
44
+ compatibility contract, and no separable Preview activation currently exists;
45
+ the redundant Preview classification must be split or removed at the next
46
+ reviewed release unless a distinct opt-in extension is defined. Compatible-mode
47
+ location APIs and extended-only type declarations also require separate
48
+ inventory records and promotion gates. Passing repository tests is necessary,
49
+ but cannot establish external field compatibility, complete R001's independent
50
+ review, or replace R002 on the exact release revision.
51
+
52
+ Activation and maturity are independent. Middle actions are accepted in
53
+ compatible/default grammars when an action is embedded before the end of a
54
+ production, so the Stable guarantee takes precedence over Preview notice.
55
+ Compatible semantic actions can consume lexer locations; only grammar `type`
56
+ declarations require extended mode. Preview therefore does not imply that every
57
+ audited surface is disabled by default or may break under Preview policy.
58
+
59
+ The active declarative-parser-construction track changes how the already
60
+ Preview IELR and multiple-entry contracts can be activated: an extended root
61
+ grammar may now own those settings. It consumes the grammar-syntax budget but
62
+ does not add or promote a maturity-inventory row. Its source history and field
63
+ experience must be incorporated into the next reviewed maturity audit.
64
+
65
+ ## Evidence boundaries
66
+
67
+ External use is evaluated against the stable IDs and classifications in the
68
+ [public workload registry](workloads.md). `repository_synthetic` gallery use
69
+ supports correctness testing only. `external_real` Bison inputs support only
70
+ their registered diagnostic scope. Neither class is reported as third-party
71
+ Ibex adoption. The current `public_real` records exercise compatible-mode
72
+ generation and runtime, not one of the audited Preview or Experimental
73
+ features, so every external-use result is `not_demonstrated`.
74
+
75
+ The issue audit ran the exact GitHub Search API query and command recorded in
76
+ the YAML on 2026-08-04. It returned zero open issues with a complete result.
77
+ That result expires after 30 days and means only that the public tracker had no
78
+ open report matching the query; it is not a correctness proof. The validator
79
+ fails after the recorded freshness deadline until the exact audit is rerun and
80
+ reviewed.
81
+
82
+ Each feature records its validator-owned first Git pickaxe introduction, first
83
+ containing release, canonical-blob presence, and integrity snapshots at v0.1.0,
84
+ v0.2.0, and the exact reviewed revision. Each release boundary separately
85
+ records one assessment for every path-relevant commit. An assessment binds the
86
+ full revision, exact Git subject, classification, and commit-specific public
87
+ contract effect. The validator reconstructs the reviewed commits from Git,
88
+ checks ancestry and changed-path relevance, and derives both the boundary's
89
+ reviewed and semantic commit sets from those exhaustive assessments. The
90
+ path match determines which commits require assessment, not whether they are
91
+ semantic. `semantic_change` is reserved for an observable syntax, API, output,
92
+ or runtime change when that inventory row's feature configuration is active;
93
+ changes behind mutually exclusive guards or confined to adjacent features are
94
+ classified as nonsemantic and name the relevant guard or code path. The
95
+ remaining `unknowns` state what repository history cannot establish, such as
96
+ downstream compatibility and field use; source age or a digest change alone
97
+ receives no semantic or promotion credit.
98
+
99
+ The introduction method is exactly
100
+ `git log --reverse --format=%H -SQUERY REVIEWED_REVISION -- PATH`; the reviewed
101
+ revision bound prevents later history from rewriting the audit result.
102
+ Boundary review uses `git log --reverse --format=%H FROM..TO -- PATH...` over
103
+ the validator-owned introduction and canonical paths. The first range starts at
104
+ the introduction commit's parent so the introduction itself is reviewed through
105
+ its first containing release; subsequent ranges join exact release and reviewed
106
+ revisions. Each source-tree digest hashes sorted canonical paths as path, NUL,
107
+ exact Git blob bytes (or the literal `<absent>`), NUL. Digests are integrity
108
+ evidence only: they neither discover nor classify semantic changes. Tags
109
+ resolve to full commit IDs, and introductions, releases, and reviewed commits
110
+ must have the recorded ancestry and order.
111
+
112
+ ## Updating the audit
113
+
114
+ Run:
115
+
116
+ ```sh
117
+ bundle exec rake quality:maturity
118
+ bundle exec ruby -Itest test/quality/maturity_test.rb
119
+ ```
120
+
121
+ For history changes, update the exhaustive assessments in `maturity.yml` and
122
+ the explicit per-feature/per-boundary semantic authority together after human
123
+ review. Update source SHA-256 values only as integrity records; never use them
124
+ to infer semantics. A promotion also requires the feature's next-review
125
+ evidence, completed R001 and exact-revision R002 states, and a synchronized
126
+ generated summary in this file and [`stability.md`](stability.md). A redesign
127
+ or removal is visible in the same summary and still follows the applicable
128
+ Preview or Experimental compatibility policy.
@@ -0,0 +1,54 @@
1
+ ---
2
+ title: Project site strategy
3
+ description: Information architecture and implementation boundaries for the Ibex public site.
4
+ ---
5
+
6
+ # Project site strategy
7
+
8
+ The public site is an adoption path rather than a catalogue of every internal
9
+ feature. The intended sequence is:
10
+
11
+ ```text
12
+ understand → try → install → investigate → verify → contribute
13
+ ```
14
+
15
+ ## Information architecture
16
+
17
+ - The README answers the installation and first-parser decision in roughly 90
18
+ seconds.
19
+ - Pages provides the task-oriented documentation hub, with Getting Started,
20
+ Guides, Reference, Concepts, Examples, Playground, API, and Project areas.
21
+ - Compatibility, Extensions, Experimental, Gallery, Playground, and API keep
22
+ their existing URLs. They are canonical landing pages or redirects, not
23
+ discarded history.
24
+ - Stable, Preview, and Experimental are feature metadata. They do not become
25
+ separate product versions or imply multiple persisted IR versions.
26
+
27
+ ## Content rules
28
+
29
+ Public copy must distinguish racc migration compatibility from a claim of
30
+ byte-for-byte replacement, bounded evidence from proof, and trusted generated
31
+ Ruby from sandboxed execution. Comparative claims link to their complete
32
+ evidence record. Maturity is read from `docs/maturity.yml`; prose does not
33
+ duplicate a second feature registry.
34
+
35
+ ## Build and safety rules
36
+
37
+ `tool/build_site.rb` is the canonical static builder. It reads docs source,
38
+ renders the selected documentation pages, copies the canonical landing pages,
39
+ and emits the Playground bundle into `tmp/site`. The builder must remain
40
+ reproducible, self-hosted, and free of runtime gem dependencies. Every page
41
+ uses a strict same-origin CSP, has a skip link and canonical metadata, and
42
+ does not send grammar source to a third party. Playground actions remain
43
+ non-executing, bounded, and worker-isolated.
44
+
45
+ The implementation is intentionally framework-free. A framework migration is
46
+ only worth revisiting if versioned documentation, multilingual content, large
47
+ faceted search, or builder maintenance becomes a demonstrated cost.
48
+
49
+ ## Operational review
50
+
51
+ Review the site with keyboard navigation, narrow viewports, zoom, reduced
52
+ motion, link/anchor checks, and a clean rebuild. New feature claims must point
53
+ to the maturity registry and evidence rather than being inferred from a
54
+ landing-page badge.
@@ -0,0 +1,45 @@
1
+ # Racc migration evidence
2
+
3
+ This page is the H002 evidence index for the compatibility claims in
4
+ [`racc-migration.md`](racc-migration.md). It records observable behavior only;
5
+ it does not claim that a third party has adopted Ibex, and it is not a
6
+ performance result.
7
+
8
+ ## Evidence boundary
9
+
10
+ The three public migration suites are pinned in
11
+ [`release-readiness.md`](release-readiness.md): Namae, BCDice, and Nokogiri.
12
+ Their revisions, adapters, and observed test counts are release evidence.
13
+ The rows below are repository-owned black-box probes that target compatibility
14
+ risks which are easy to miss in a single arithmetic grammar. They run against
15
+ the installed `racc` executable when it is available and skip otherwise; a
16
+ skip is not reported as a pass.
17
+
18
+ | Risk | Probe and observable contract | Reproduction |
19
+ |---|---|---|
20
+ | precedence and conflict accounting | arithmetic results match; resolved conflicts remain visible in Automaton IR but do not inflate the CLI conflict count or `expect` | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
21
+ | empty productions and string tokens | empty reductions and literal-token paths produce the same public result values | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
22
+ | `convert` and `no_result_var` | token conversion and result-variable suppression preserve the generated parser's public action behavior | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
23
+ | inline actions | action placement and returned semantic values are equivalent at the public parser boundary | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
24
+ | `expect` and dangling-else behavior | the expected conflict count and parse result remain stable for the ambiguous grammar | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
25
+ | error recovery and `on_error` | callback arguments and recovered result values are compared; the documented undeclared-token difference remains explicit | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
26
+ | source locations and line conversion | semantic action line mapping follows the documented default and `--line-convert-all` modes | `bundle exec ruby -Itest test/racc_migration_test.rb` |
27
+ | runtime initialization and value-stack reads | an initializer that omits `super` retains application state while historical value-stack reads remain compatible | `bundle exec ruby -Itest test/racc_migration_test.rb` |
28
+ | unqualified `ParseError` | parser methods and actions can retain the explicit rejection path through `Runtime::Parser` lookup | `bundle exec ruby -Itest test/racc_migration_test.rb` |
29
+ | large grammar construction | a 500-production grammar completes with the recorded production/state bound and no conflicts | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
30
+ | push-parser runtime contract | push-session lifecycle, token feeding, acceptance, and cleanup remain stable within Ibex's public runtime API | `bundle exec ruby -Itest test/runtime/push_parser_test.rb` |
31
+
32
+ The compatibility test files are the executable source of truth. The commands
33
+ above were last run at the evidence refresh and produced zero failures and zero
34
+ errors (black-box: 6 runs/33 assertions; migration: 5 runs/20 assertions;
35
+ push-parser: 46 runs/183 assertions). The black-box suite reported no skip in
36
+ that environment.
37
+
38
+ ## Interpretation
39
+
40
+ These probes establish coverage of distinct compatibility risks, not a claim
41
+ that every racc grammar is interchangeable. Generated source layout, internal
42
+ table arrays, `require "racc/parser"` replacement, and previously generated
43
+ racc table loading remain explicitly out of scope. Formal cross-project
44
+ performance comparison is tracked separately in the release-readiness report
45
+ and remains incomplete until a clean, exact-revision artifact is published.
@@ -1,3 +1,8 @@
1
+ ---
2
+ title: Migrating from racc
3
+ description: Move a racc grammar to Ibex with explicit compatibility checks and runtime boundaries.
4
+ ---
5
+
1
6
  # Migrating from racc
2
7
 
3
8
  Ibex targets grammar-file compatibility, generated parser public API compatibility, and the main racc CLI options. It does not
@@ -49,6 +54,12 @@ generated parsers in bounded child processes. This is not a sandbox; use a conta
49
54
 
50
55
  ## Compatibility baseline
51
56
 
57
+ The repository probes and their exact reproduction commands are recorded in
58
+ [`racc-migration-evidence.md`](racc-migration-evidence.md). That matrix is an
59
+ internal evidence index: it distinguishes self-authored black-box coverage
60
+ from the separately pinned public-gem migration suites and does not turn either
61
+ set into a claim of third-party adoption or performance parity.
62
+
52
63
  The compatibility claims above were checked on 2026-07-22 against racc 1.8.1 using only its public documentation, `racc --help`,
53
64
  and black-box execution. Ibex does not inspect racc implementation files or generated source. Self-authored probes compare
54
65
  observable results for arithmetic precedence, empty rules, string tokens, `convert`, `no_result_var`, inline actions,