ibex 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +95 -272
- data/docs/architecture.md +187 -35
- data/docs/bison-import.md +103 -0
- data/docs/comparison-policy.md +127 -0
- data/docs/configuration-model.md +59 -0
- data/docs/configuration-report.md +28 -0
- data/docs/conflict-explanation-reviews/v1/records/README.md +28 -0
- data/docs/conflict-explanation-study.md +62 -0
- data/docs/construction-profiling.md +157 -0
- data/docs/cst-migration.md +3 -4
- data/docs/cst.md +24 -0
- data/docs/decisions/0000-template.md +24 -0
- data/docs/decisions/0001-separate-ir-pipeline.md +34 -0
- data/docs/decisions/0002-opaque-user-code-boundary.md +32 -0
- data/docs/decisions/0003-self-hosted-grammar-frontend.md +32 -0
- data/docs/decisions/0004-shared-semantic-and-lossless-source-model.md +33 -0
- data/docs/decisions/0005-contained-grammar-composition.md +34 -0
- data/docs/decisions/0006-bounded-structural-grammar-lowering.md +32 -0
- data/docs/decisions/0007-shared-parser-construction-pipeline.md +33 -0
- data/docs/decisions/0008-versioned-runtime-package-boundary.md +36 -0
- data/docs/decisions/0009-isolated-parser-sessions.md +34 -0
- data/docs/decisions/0010-committed-runtime-observation.md +34 -0
- data/docs/decisions/0011-versioned-semantic-action-boundary.md +37 -0
- data/docs/decisions/0012-bounded-nonexecuting-analysis.md +37 -0
- data/docs/decisions/0013-transactional-generation-publication.md +37 -0
- data/docs/decisions/0014-versioned-generated-lexer.md +35 -0
- data/docs/decisions/0015-worker-isolated-browser-analysis.md +32 -0
- data/docs/decisions/0016-red-green-concrete-syntax.md +34 -0
- data/docs/decisions/0017-persistent-syntax-artifacts.md +35 -0
- data/docs/decisions/0018-conservative-incremental-syntax-reuse.md +38 -0
- data/docs/decisions/0019-runtime-syntax-session-boundary.md +51 -0
- data/docs/decisions/0020-grammar-ir-parser-contract.md +65 -0
- data/docs/decisions/0021-data-only-parser-table-sidecar.md +50 -0
- data/docs/decisions/0022-manifest-bound-verification-report.md +77 -0
- data/docs/decisions/0023-syntax-only-repair-results.md +48 -0
- data/docs/decisions/0024-direct-ielr-construction.md +42 -0
- data/docs/decisions/README.md +74 -0
- data/docs/declarative-configuration.md +262 -0
- data/docs/development.md +179 -2
- data/docs/direct-ielr-decision.md +120 -0
- data/docs/direct-multi-entry-decision.md +82 -0
- data/docs/editor-setup.md +2 -1
- data/docs/error-ux-review-rubric-v1.md +117 -0
- data/docs/error-ux-reviews/v1/records/README.md +22 -0
- data/docs/error-ux-round2-review-status-v1.json +27 -0
- data/docs/error-ux-round2-reviews/v1/records/README.md +30 -0
- data/docs/error-ux-round2-v1.json +1581 -0
- data/docs/error-ux-round2.md +111 -0
- data/docs/error-ux.md +54 -0
- data/docs/getting-started.md +80 -0
- data/docs/grammar-reference.md +184 -13
- data/docs/ielr-design.md +43 -0
- data/docs/ielr.md +92 -0
- data/docs/ja/bison-import.md +51 -0
- data/docs/ja/stable-api.md +53 -0
- data/docs/lexer-construction-profile.md +109 -0
- data/docs/lexer-migration.md +5 -0
- data/docs/maturity.md +128 -0
- data/docs/project-site-strategy.md +54 -0
- data/docs/racc-migration-evidence.md +45 -0
- data/docs/racc-migration.md +11 -0
- data/docs/release-readiness.md +126 -68
- data/docs/repair-semantics.md +123 -0
- data/docs/runtime-abi-evolution.md +307 -0
- data/docs/stability.md +144 -34
- data/docs/status.md +43 -0
- data/docs/syntax-sessions.md +183 -0
- data/docs/table-artifact.md +125 -0
- data/docs/test-interactions.md +219 -0
- data/docs/verification-report.md +137 -0
- data/docs/verifier-trust-boundary.md +199 -0
- data/docs/workloads.md +132 -0
- data/lib/ibex/analysis/digraph.rb +128 -0
- data/lib/ibex/analysis.rb +1 -0
- data/lib/ibex/bison_import/importer.rb +516 -0
- data/lib/ibex/bison_import/tokenizer.rb +260 -0
- data/lib/ibex/bison_import.rb +200 -0
- data/lib/ibex/bounded_subprocess.rb +171 -0
- data/lib/ibex/cli/ambiguity.rb +14 -5
- data/lib/ibex/cli/analysis.rb +134 -0
- data/lib/ibex/cli/bison_import.rb +121 -0
- data/lib/ibex/cli/config.rb +128 -0
- data/lib/ibex/cli/diagnostics.rb +31 -11
- data/lib/ibex/cli/documentation.rb +11 -8
- data/lib/ibex/cli/equiv.rb +169 -0
- data/lib/ibex/cli/error_messages.rb +11 -4
- data/lib/ibex/cli/explain.rb +14 -5
- data/lib/ibex/cli/fix.rb +203 -0
- data/lib/ibex/cli/formatting.rb +11 -10
- data/lib/ibex/cli/fuzz.rb +200 -0
- data/lib/ibex/cli/fuzz_regressions.rb +145 -0
- data/lib/ibex/cli/generation_artifacts.rb +53 -2
- data/lib/ibex/cli/generation_error_messages.rb +1 -1
- data/lib/ibex/cli/grammar_tests.rb +32 -14
- data/lib/ibex/cli/ir_tools.rb +5 -61
- data/lib/ibex/cli/outputs.rb +60 -33
- data/lib/ibex/cli/reduce.rb +202 -0
- data/lib/ibex/cli/reduce_reporting.rb +71 -0
- data/lib/ibex/cli/samples.rb +49 -23
- data/lib/ibex/cli/verify.rb +116 -0
- data/lib/ibex/cli/watch.rb +6 -5
- data/lib/ibex/cli.rb +357 -56
- data/lib/ibex/codegen/action_locations.rb +1 -1
- data/lib/ibex/codegen/action_method_source.rb +5 -6
- data/lib/ibex/codegen/action_source.rb +3 -2
- data/lib/ibex/codegen/ambiguity.rb +5 -4
- data/lib/ibex/codegen/explain.rb +123 -55
- data/lib/ibex/codegen/generated_action_abi.rb +10 -5
- data/lib/ibex/codegen/rbs.rb +15 -7
- data/lib/ibex/codegen/report.rb +25 -13
- data/lib/ibex/codegen/ruby.rb +28 -64
- data/lib/ibex/codegen/ruby_actions.rb +9 -6
- data/lib/ibex/codegen/ruby_syntax.rb +15 -9
- data/lib/ibex/configuration/analysis_grammar.rb +35 -0
- data/lib/ibex/configuration/explanation.rb +421 -0
- data/lib/ibex/configuration/inspector.rb +232 -0
- data/lib/ibex/configuration.rb +580 -0
- data/lib/ibex/coverage/collector.rb +21 -11
- data/lib/ibex/coverage/event_stream.rb +13 -7
- data/lib/ibex/coverage/report.rb +26 -18
- data/lib/ibex/coverage/runtime_event_validator.rb +30 -21
- data/lib/ibex/delta_reducer.rb +99 -0
- data/lib/ibex/diff.rb +140 -0
- data/lib/ibex/equiv/machine.rb +135 -0
- data/lib/ibex/equiv.rb +373 -0
- data/lib/ibex/fix.rb +557 -0
- data/lib/ibex/frontend/ast.rb +22 -2
- data/lib/ibex/frontend/bootstrap_parser.rb +6 -13
- data/lib/ibex/frontend/diagnostic.rb +1 -1
- data/lib/ibex/frontend/formatter.rb +72 -26
- data/lib/ibex/frontend/generated_parser.rb +147 -117
- data/lib/ibex/frontend/generated_parser_base.rb +7 -3
- data/lib/ibex/frontend/generated_parser_includes.rb +1 -11
- data/lib/ibex/frontend/generation.rb +1 -1
- data/lib/ibex/frontend/parser/declarations.rb +17 -2
- data/lib/ibex/frontend/parser_configuration_support.rb +56 -0
- data/lib/ibex/frontend/regenerator.rb +1 -0
- data/lib/ibex/frontend/resolution.rb +2 -1
- data/lib/ibex/frontend/resolver.rb +1 -1
- data/lib/ibex/frontend/rule_documentation.rb +2 -1
- data/lib/ibex/frontend/source_cursor.rb +2 -2
- data/lib/ibex/frontend/source_span.rb +9 -1
- data/lib/ibex/frontend/token_adapter/declaration_state.rb +38 -6
- data/lib/ibex/frontend/token_adapter.rb +10 -0
- data/lib/ibex/frontend.rb +1 -0
- data/lib/ibex/fuzz.rb +219 -0
- data/lib/ibex/generation_input.rb +1 -1
- data/lib/ibex/generation_manifest.rb +36 -25
- data/lib/ibex/generation_transaction.rb +1 -1
- data/lib/ibex/generation_transaction_recovery.rb +4 -4
- data/lib/ibex/generation_transaction_validation.rb +2 -2
- data/lib/ibex/grammar_tests.rb +1 -1
- data/lib/ibex/ir/automaton_ir.rb +61 -19
- data/lib/ibex/ir/grammar_ir.rb +76 -45
- data/lib/ibex/ir/lexer_ir.rb +5 -5
- data/lib/ibex/ir/parser_contract.rb +135 -0
- data/lib/ibex/ir/serialize.rb +203 -76
- data/lib/ibex/ir/validator/automaton.rb +64 -34
- data/lib/ibex/ir/validator/base.rb +51 -33
- data/lib/ibex/ir/validator/grammar.rb +113 -83
- data/lib/ibex/ir/validator/lexer.rb +2 -2
- data/lib/ibex/ir/validator.rb +2 -1
- data/lib/ibex/ir.rb +16 -3
- data/lib/ibex/lalr/build_metrics.rb +50 -2
- data/lib/ibex/lalr/builder.rb +191 -56
- data/lib/ibex/lalr/conflict_search.rb +15 -4
- data/lib/ibex/lalr/counterexample.rb +19 -14
- data/lib/ibex/lalr/direct_lookaheads.rb +110 -57
- data/lib/ibex/lalr/goto_follows.rb +229 -0
- data/lib/ibex/lalr/ielr/annotator.rb +214 -0
- data/lib/ibex/lalr/ielr/bits.rb +28 -0
- data/lib/ibex/lalr/ielr/inadequacy.rb +47 -0
- data/lib/ibex/lalr/ielr/item_lookaheads.rb +78 -0
- data/lib/ibex/lalr/ielr/pipeline.rb +75 -0
- data/lib/ibex/lalr/ielr/split_stability.rb +78 -0
- data/lib/ibex/lalr/ielr/split_state.rb +20 -0
- data/lib/ibex/lalr/ielr/state_splitter.rb +258 -0
- data/lib/ibex/lalr/ielr_partition.rb +22 -8
- data/lib/ibex/lalr/inadequacy_report.rb +50 -0
- data/lib/ibex/lalr/lookahead_propagation.rb +111 -0
- data/lib/ibex/lalr/lr0_collection.rb +121 -0
- data/lib/ibex/lalr/unreachable_states.rb +80 -0
- data/lib/ibex/lalr.rb +52 -1
- data/lib/ibex/location.rb +3 -3
- data/lib/ibex/lsp/document_handlers.rb +9 -7
- data/lib/ibex/lsp/initialization_handlers.rb +9 -7
- data/lib/ibex/lsp/navigation_handlers.rb +25 -11
- data/lib/ibex/lsp/parser_configuration_assistance.rb +149 -0
- data/lib/ibex/lsp/position_codec.rb +6 -2
- data/lib/ibex/lsp/request_handlers.rb +3 -2
- data/lib/ibex/lsp/request_support.rb +14 -7
- data/lib/ibex/lsp/server.rb +13 -11
- data/lib/ibex/lsp/symbol_index.rb +13 -13
- data/lib/ibex/lsp/symbol_index_builder.rb +30 -20
- data/lib/ibex/lsp/symbol_occurrence.rb +8 -2
- data/lib/ibex/lsp/transport.rb +13 -3
- data/lib/ibex/lsp/workspace.rb +5 -4
- data/lib/ibex/lsp/workspace_analyzer.rb +1 -1
- data/lib/ibex/lsp.rb +1 -0
- data/lib/ibex/messages/en.yml +28 -0
- data/lib/ibex/messages/ja.yml +28 -0
- data/lib/ibex/messages.rb +68 -0
- data/lib/ibex/metrics.rb +175 -0
- data/lib/ibex/normalize/declarations.rb +2 -0
- data/lib/ibex/normalize/diagnostics.rb +63 -38
- data/lib/ibex/normalize/expander.rb +5 -4
- data/lib/ibex/normalize/expression.rb +8 -6
- data/lib/ibex/normalize/grammar_builder.rb +26 -0
- data/lib/ibex/normalize/inline_expansion.rb +100 -49
- data/lib/ibex/normalize/lexer.rb +3 -3
- data/lib/ibex/normalize/parameter_ebnf_lowering.rb +11 -10
- data/lib/ibex/normalize/parameter_lowering.rb +40 -23
- data/lib/ibex/normalize/parameters.rb +31 -4
- data/lib/ibex/normalize/parser_configuration.rb +70 -0
- data/lib/ibex/normalize.rb +10 -20
- data/lib/ibex/racc_migration/report.rb +6 -3
- data/lib/ibex/rake_task.rb +1 -1
- data/lib/ibex/samples.rb +43 -13
- data/lib/ibex/table_artifact/builder.rb +266 -0
- data/lib/ibex/table_artifact/cst_projection.rb +70 -0
- data/lib/ibex/table_artifact/document.rb +39 -0
- data/lib/ibex/table_artifact/executor.rb +187 -0
- data/lib/ibex/table_artifact/serializer.rb +57 -0
- data/lib/ibex/table_artifact/validator/metadata.rb +179 -0
- data/lib/ibex/table_artifact/validator/support.rb +86 -0
- data/lib/ibex/table_artifact/validator/tables.rb +238 -0
- data/lib/ibex/table_artifact/validator.rb +253 -0
- data/lib/ibex/table_artifact.rb +88 -0
- data/lib/ibex/table_simulation/result.rb +6 -2
- data/lib/ibex/table_simulation/step.rb +3 -1
- data/lib/ibex/tables.rb +14 -0
- data/lib/ibex/verifiable_generation_bundle.rb +86 -0
- data/lib/ibex/verification_report/builder.rb +159 -0
- data/lib/ibex/verification_report/canonical_ir.rb +123 -0
- data/lib/ibex/verification_report/logical_path.rb +68 -0
- data/lib/ibex/verification_report/validator.rb +329 -0
- data/lib/ibex/verification_report.rb +69 -0
- data/lib/ibex/verify/action_correspondence.rb +138 -0
- data/lib/ibex/verify/language_witness.rb +295 -0
- data/lib/ibex/verify/reference_collection.rb +190 -0
- data/lib/ibex/verify/result.rb +73 -0
- data/lib/ibex/verify/verifier.rb +598 -0
- data/lib/ibex/verify.rb +15 -0
- data/lib/ibex/version.rb +1 -1
- data/lib/ibex/watch/runner.rb +10 -1
- data/lib/ibex/watch/source_snapshot.rb +13 -8
- data/lib/ibex.rb +10 -0
- data/schema/{automaton-ir-v1.schema.json → automaton-ir-definitions.schema.json} +6 -6
- data/schema/{automaton-ir-v2.schema.json → automaton-ir.schema.json} +14 -7
- data/schema/bison-import-v1.schema.json +106 -0
- data/schema/conflict-explanation-review-v1.schema.json +196 -0
- data/schema/conflict-explanation-study-v1.schema.json +528 -0
- data/schema/construction-profile-v1.schema.json +779 -0
- data/schema/diff-v1.schema.json +41 -0
- data/schema/direct-ielr-decision-v1.schema.json +200 -0
- data/schema/equiv-v1.schema.json +64 -0
- data/schema/error-ux-review-v1.schema.json +609 -0
- data/schema/error-ux-round2-review-v1.schema.json +191 -0
- data/schema/error-ux-round2-v1.schema.json +519 -0
- data/schema/explain-v1.schema.json +63 -1
- data/schema/fix-v1.schema.json +64 -0
- data/schema/fix-v2.schema.json +66 -0
- data/schema/fix-v3.schema.json +101 -0
- data/schema/fuzz-regression-v1.schema.json +79 -0
- data/schema/fuzz-v1.schema.json +114 -0
- data/schema/{grammar-ir-v2.schema.json → grammar-ir-extensions.schema.json} +20 -65
- data/schema/{grammar-ir-v1.schema.json → grammar-ir-foundation.schema.json} +3 -3
- data/schema/grammar-ir.schema.json +271 -0
- data/schema/ielr-benchmark-v1.schema.json +85 -0
- data/schema/lexer-profile-v1.schema.json +491 -0
- data/schema/metrics-v1.schema.json +60 -0
- data/schema/reduce-v1.schema.json +22 -0
- data/schema/reduce-v2.schema.json +63 -0
- data/schema/table-artifact-v1.schema.json +377 -0
- data/schema/verification-report-v1.schema.json +222 -0
- data/schema/verify-v1.schema.json +42 -0
- data/sig/ibex/analysis/digraph.rbs +19 -0
- data/sig/ibex/bison_import/importer.rbs +140 -0
- data/sig/ibex/bison_import/tokenizer.rbs +91 -0
- data/sig/ibex/bison_import.rbs +95 -0
- data/sig/ibex/bounded_subprocess.rbs +76 -0
- data/sig/ibex/cli/ambiguity.rbs +6 -0
- data/sig/ibex/cli/analysis.rbs +43 -0
- data/sig/ibex/cli/bison_import.rbs +32 -0
- data/sig/ibex/cli/config.rbs +35 -0
- data/sig/ibex/cli/diagnostics.rbs +16 -3
- data/sig/ibex/cli/documentation.rbs +5 -1
- data/sig/ibex/cli/equiv.rbs +49 -0
- data/sig/ibex/cli/error_messages.rbs +6 -0
- data/sig/ibex/cli/explain.rbs +6 -0
- data/sig/ibex/cli/fix.rbs +53 -0
- data/sig/ibex/cli/formatting.rbs +5 -1
- data/sig/ibex/cli/fuzz.rbs +47 -0
- data/sig/ibex/cli/fuzz_regressions.rbs +40 -0
- data/sig/ibex/cli/generation_artifacts.rbs +21 -2
- data/sig/ibex/cli/generation_error_messages.rbs +2 -2
- data/sig/ibex/cli/grammar_tests.rbs +15 -3
- data/sig/ibex/cli/ir_tools.rbs +7 -19
- data/sig/ibex/cli/outputs.rbs +9 -3
- data/sig/ibex/cli/reduce.rbs +60 -0
- data/sig/ibex/cli/reduce_reporting.rbs +27 -0
- data/sig/ibex/cli/samples.rbs +8 -0
- data/sig/ibex/cli/verify.rbs +28 -0
- data/sig/ibex/cli/watch.rbs +2 -2
- data/sig/ibex/cli.rbs +79 -5
- data/sig/ibex/codegen/action_method_source.rbs +4 -4
- data/sig/ibex/codegen/explain.rbs +45 -25
- data/sig/ibex/codegen/generated_action_abi.rbs +14 -10
- data/sig/ibex/codegen/rbs.rbs +12 -10
- data/sig/ibex/codegen/report.rbs +12 -12
- data/sig/ibex/codegen/ruby.rbs +18 -15
- data/sig/ibex/codegen/ruby_actions.rbs +6 -6
- data/sig/ibex/codegen/ruby_syntax.rbs +10 -8
- data/sig/ibex/configuration/analysis_grammar.rbs +13 -0
- data/sig/ibex/configuration/explanation.rbs +195 -0
- data/sig/ibex/configuration/inspector.rbs +53 -0
- data/sig/ibex/configuration.rbs +242 -0
- data/sig/ibex/coverage/collector.rbs +16 -14
- data/sig/ibex/coverage/event_stream.rbs +12 -10
- data/sig/ibex/coverage/report.rbs +20 -18
- data/sig/ibex/coverage/runtime_event_validator.rbs +38 -36
- data/sig/ibex/delta_reducer.rbs +38 -0
- data/sig/ibex/diff.rbs +47 -0
- data/sig/ibex/equiv/machine.rbs +54 -0
- data/sig/ibex/equiv.rbs +129 -0
- data/sig/ibex/fix.rbs +138 -0
- data/sig/ibex/frontend/ast.rbs +33 -7
- data/sig/ibex/frontend/bootstrap_parser.rbs +2 -0
- data/sig/ibex/frontend/diagnostic.rbs +2 -2
- data/sig/ibex/frontend/formatter.rbs +38 -30
- data/sig/ibex/frontend/generated_parser.rbs +87 -77
- data/sig/ibex/frontend/generated_parser_base.rbs +4 -2
- data/sig/ibex/frontend/parser/declarations.rbs +3 -0
- data/sig/ibex/frontend/parser_configuration_support.rbs +27 -0
- data/sig/ibex/frontend/resolution.rbs +3 -2
- data/sig/ibex/frontend/source_cursor.rbs +2 -2
- data/sig/ibex/frontend/source_span.rbs +5 -2
- data/sig/ibex/frontend/token_adapter/declaration_state.rbs +13 -1
- data/sig/ibex/frontend/token_adapter.rbs +6 -0
- data/sig/ibex/fuzz.rbs +71 -0
- data/sig/ibex/generation_input.rbs +2 -2
- data/sig/ibex/generation_manifest.rbs +35 -32
- data/sig/ibex/generation_transaction_recovery.rbs +2 -2
- data/sig/ibex/generation_transaction_validation.rbs +2 -2
- data/sig/ibex/grammar_tests.rbs +2 -2
- data/sig/ibex/ir/automaton_ir.rbs +30 -11
- data/sig/ibex/ir/grammar_ir.rbs +41 -27
- data/sig/ibex/ir/lexer_ir.rbs +8 -8
- data/sig/ibex/ir/parser_contract.rbs +69 -0
- data/sig/ibex/ir/serialize.rbs +63 -28
- data/sig/ibex/ir/validator/automaton.rbs +47 -44
- data/sig/ibex/ir/validator/base.rbs +35 -31
- data/sig/ibex/ir/validator/grammar.rbs +78 -67
- data/sig/ibex/ir/validator/lexer.rbs +4 -4
- data/sig/ibex/ir.rbs +9 -3
- data/sig/ibex/lalr/build_metrics.rbs +42 -2
- data/sig/ibex/lalr/builder.rbs +75 -24
- data/sig/ibex/lalr/conflict_search.rbs +5 -2
- data/sig/ibex/lalr/counterexample.rbs +17 -14
- data/sig/ibex/lalr/direct_lookaheads.rbs +35 -28
- data/sig/ibex/lalr/goto_follows.rbs +89 -0
- data/sig/ibex/lalr/ielr/annotator.rbs +67 -0
- data/sig/ibex/lalr/ielr/bits.rbs +14 -0
- data/sig/ibex/lalr/ielr/inadequacy.rbs +24 -0
- data/sig/ibex/lalr/ielr/item_lookaheads.rbs +35 -0
- data/sig/ibex/lalr/ielr/pipeline.rbs +17 -0
- data/sig/ibex/lalr/ielr/split_stability.rbs +27 -0
- data/sig/ibex/lalr/ielr/split_state.rbs +14 -0
- data/sig/ibex/lalr/ielr/state_splitter.rbs +76 -0
- data/sig/ibex/lalr/ielr_partition.rbs +17 -5
- data/sig/ibex/lalr/inadequacy_report.rbs +21 -0
- data/sig/ibex/lalr/lookahead_propagation.rbs +38 -0
- data/sig/ibex/lalr/lr0_collection.rbs +50 -0
- data/sig/ibex/lalr/unreachable_states.rbs +25 -0
- data/sig/ibex/lalr.rbs +11 -1
- data/sig/ibex/location.rbs +6 -6
- data/sig/ibex/lsp/document_handlers.rbs +8 -8
- data/sig/ibex/lsp/initialization_handlers.rbs +12 -12
- data/sig/ibex/lsp/navigation_handlers.rbs +15 -12
- data/sig/ibex/lsp/parser_configuration_assistance.rbs +49 -0
- data/sig/ibex/lsp/position_codec.rbs +8 -4
- data/sig/ibex/lsp/request_handlers.rbs +4 -4
- data/sig/ibex/lsp/request_support.rbs +8 -8
- data/sig/ibex/lsp/server.rbs +16 -16
- data/sig/ibex/lsp/symbol_index.rbs +20 -21
- data/sig/ibex/lsp/symbol_index_builder.rbs +9 -8
- data/sig/ibex/lsp/symbol_occurrence.rbs +11 -5
- data/sig/ibex/lsp/transport.rbs +18 -6
- data/sig/ibex/lsp/workspace.rbs +3 -2
- data/sig/ibex/lsp/workspace_analyzer.rbs +1 -1
- data/sig/ibex/messages.rbs +26 -0
- data/sig/ibex/metrics.rbs +50 -0
- data/sig/ibex/normalize/diagnostics.rbs +4 -4
- data/sig/ibex/normalize/expander.rbs +2 -2
- data/sig/ibex/normalize/expression.rbs +10 -8
- data/sig/ibex/normalize/grammar_builder.rbs +11 -0
- data/sig/ibex/normalize/inline_expansion.rbs +59 -50
- data/sig/ibex/normalize/lexer.rbs +4 -4
- data/sig/ibex/normalize/parameter_ebnf_lowering.rbs +15 -14
- data/sig/ibex/normalize/parameter_lowering.rbs +18 -18
- data/sig/ibex/normalize/parameters.rbs +13 -8
- data/sig/ibex/normalize/parser_configuration.rbs +23 -0
- data/sig/ibex/normalize.rbs +11 -5
- data/sig/ibex/racc_migration/report.rbs +8 -6
- data/sig/ibex/rake_task.rbs +1 -1
- data/sig/ibex/samples.rbs +13 -4
- data/sig/ibex/table_artifact/builder.rbs +93 -0
- data/sig/ibex/table_artifact/cst_projection.rbs +20 -0
- data/sig/ibex/table_artifact/document.rbs +27 -0
- data/sig/ibex/table_artifact/executor.rbs +66 -0
- data/sig/ibex/table_artifact/serializer.rbs +25 -0
- data/sig/ibex/table_artifact/validator/metadata.rbs +52 -0
- data/sig/ibex/table_artifact/validator/support.rbs +46 -0
- data/sig/ibex/table_artifact/validator/tables.rbs +63 -0
- data/sig/ibex/table_artifact/validator.rbs +66 -0
- data/sig/ibex/table_artifact.rbs +32 -0
- data/sig/ibex/table_simulation/result.rbs +8 -4
- data/sig/ibex/table_simulation/step.rbs +4 -2
- data/sig/ibex/tables.rbs +10 -9
- data/sig/ibex/verifiable_generation_bundle.rbs +24 -0
- data/sig/ibex/verification_report/builder.rbs +53 -0
- data/sig/ibex/verification_report/canonical_ir.rbs +41 -0
- data/sig/ibex/verification_report/logical_path.rbs +34 -0
- data/sig/ibex/verification_report/validator.rbs +114 -0
- data/sig/ibex/verification_report.rbs +44 -0
- data/sig/ibex/verify/action_correspondence.rbs +50 -0
- data/sig/ibex/verify/language_witness.rbs +112 -0
- data/sig/ibex/verify/reference_collection.rbs +63 -0
- data/sig/ibex/verify/result.rbs +52 -0
- data/sig/ibex/verify/verifier.rbs +170 -0
- data/sig/ibex/verify.rbs +6 -0
- data/sig/ibex/watch/runner.rbs +12 -2
- data/sig/ibex/watch/source_snapshot.rbs +13 -9
- metadata +232 -12
- data/lib/ibex/ir/migration.rb +0 -120
- data/sig/ibex/ir/migration.rbs +0 -34
data/docs/ielr.md
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: IELR(1) construction
|
|
3
|
+
description: The opt-in IELR construction strategies and their verification boundary.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# IELR(1) construction
|
|
7
|
+
|
|
8
|
+
Ibex exposes IELR(1) as an opt-in parser-construction algorithm. It keeps the
|
|
9
|
+
same Grammar IR, Automaton IR, conflict resolver, table generator, and runtime
|
|
10
|
+
as SLR, LALR, and canonical LR(1). The default remains the compatible LALR
|
|
11
|
+
path; selecting IELR does not change generated parser APIs.
|
|
12
|
+
|
|
13
|
+
## Strategies
|
|
14
|
+
|
|
15
|
+
`--algorithm=ielr` accepts two construction strategies:
|
|
16
|
+
|
|
17
|
+
- `partition` (the default) constructs canonical LR(1) states and partitions
|
|
18
|
+
compatible isocores. It is the conservative reference strategy and remains
|
|
19
|
+
the release baseline.
|
|
20
|
+
- `direct` starts with LR(0), computes LALR lookaheads and goto-follow
|
|
21
|
+
relations, traces grammar-relative inadequacies, splits only affected
|
|
22
|
+
isocores, and recalculates lookaheads on the split graph. It does not build a
|
|
23
|
+
canonical collection in the normal construction path.
|
|
24
|
+
|
|
25
|
+
The direct strategy is deliberately not a claim of globally minimal state
|
|
26
|
+
count. Its benefit is bounded construction work and phase-level evidence, not
|
|
27
|
+
an unconditional reduction in the final table. `--remove-unreachable` is
|
|
28
|
+
available as an explicit post-resolution compaction; it is off by default so
|
|
29
|
+
state numbers used by diagnostics and generated artifacts remain stable.
|
|
30
|
+
|
|
31
|
+
```sh
|
|
32
|
+
ibex --algorithm=ielr --ielr-strategy=direct grammar.y
|
|
33
|
+
ibex --algorithm=ielr --ielr-report grammar.y
|
|
34
|
+
ibex --algorithm=ielr --ielr-strategy=direct --remove-unreachable grammar.y
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
`--ielr-report` is diagnostic-only. It constructs a canonical reference and
|
|
38
|
+
reports action differences; it must not be confused with the direct build
|
|
39
|
+
path. The report is useful for review and regression tests, but it is not a
|
|
40
|
+
proof that every viable prefix has a unique state correspondence.
|
|
41
|
+
|
|
42
|
+
## Construction phases
|
|
43
|
+
|
|
44
|
+
The direct path is organized as the following explicit boundaries:
|
|
45
|
+
|
|
46
|
+
1. `LR0Collection`, `LookaheadPropagation`, and `GotoFollows` build the LR(0)
|
|
47
|
+
collection, direct reads, successor/internal/includes relations, and LALR
|
|
48
|
+
lookaheads. `DirectLookaheads` remains the existing compatibility-path
|
|
49
|
+
cross-check.
|
|
50
|
+
2. `IELR::Annotator` and `ItemLookaheads` classify conflicts and propagate
|
|
51
|
+
annotations over predecessor lanes. `SplitStability` discards annotations
|
|
52
|
+
whose resolved action cannot change under a split.
|
|
53
|
+
3. `IELR::StateSplitter` copies transition tables for incompatible isocores and
|
|
54
|
+
preserves deterministic construction order.
|
|
55
|
+
4. `LookaheadPropagation` reruns item-level propagation over the split graph.
|
|
56
|
+
5. The existing `Builder` phase resolves conflicts and applies error/default
|
|
57
|
+
reductions, preserving the common Automaton IR contract.
|
|
58
|
+
|
|
59
|
+
All lookahead sets are integer bitsets. `Analysis::Digraph` computes the
|
|
60
|
+
successor, always, and includes closures with an iterative worklist so deep
|
|
61
|
+
grammars do not depend on Ruby recursion depth.
|
|
62
|
+
|
|
63
|
+
## Metrics and reports
|
|
64
|
+
|
|
65
|
+
Profiled builds expose `ielr_annotations`, `ielr_annotated_states`,
|
|
66
|
+
`ielr_inadequacies`, `ielr_split_stable_discarded`, `ielr_lalr_states`,
|
|
67
|
+
`ielr_split_states`, `ielr_unreachable_removed`,
|
|
68
|
+
`ielr_remergeable_candidates`, and the existing partition counts. Use
|
|
69
|
+
`tool/construction_profile.rb --ielr-strategy=direct` to capture the direct
|
|
70
|
+
measurements. Wall-clock values remain observations, not release gates.
|
|
71
|
+
|
|
72
|
+
The phase implementation is covered by the paper-derived fixtures in
|
|
73
|
+
`test/fixtures/ielr` and the structural tests in
|
|
74
|
+
`test/lalr/ielr_direct_test.rb`. The direct strategy remains experimental
|
|
75
|
+
until representative real-workload evidence and an independent adequacy
|
|
76
|
+
review satisfy the direct-IELR decision dossier.
|
|
77
|
+
|
|
78
|
+
## Verification boundary
|
|
79
|
+
|
|
80
|
+
`Verify::ActionCorrespondence` and `LALR::InadequacyReport` are explicit
|
|
81
|
+
diagnostic tools. They compare resolved actions and preserve a bounded search
|
|
82
|
+
budget, but canonical-vs-target state pairing is not assumed to be a function.
|
|
83
|
+
`Verify::LanguageWitness` supplies the strict IELR V9 bounded acceptance
|
|
84
|
+
comparison; it reports token-level acceptance witnesses and fails closed when
|
|
85
|
+
its finite case budget is exhausted. It does not prove unbounded equivalence,
|
|
86
|
+
conflict preservation, or split correctness. Normal parser construction never
|
|
87
|
+
invokes these diagnostics. Existing canonical-core verification remains the
|
|
88
|
+
trusted boundary for release decisions.
|
|
89
|
+
|
|
90
|
+
The implementation is independently derived from the published IELR paper and
|
|
91
|
+
the repository's parser specifications. GNU Bison or another GPL
|
|
92
|
+
implementation is not a design input.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Bison 文法の解析用インポート
|
|
2
|
+
|
|
3
|
+
`ibex import bison` は Bison 形式の文法構造を、解析専用の extended Ibex
|
|
4
|
+
ソースへ一方向に変換します。互換パーサを生成するモードではありません。C
|
|
5
|
+
アクションは不透明なまま保持され、解析中には実行されず、Ruby 生成はコードを
|
|
6
|
+
出力する前に拒否されます。
|
|
7
|
+
|
|
8
|
+
```sh
|
|
9
|
+
ibex import bison parser.y -o parser.analysis.y
|
|
10
|
+
ibex import bison --format=json parser.y > import-report.json
|
|
11
|
+
ibex explain --state=STATE parser.y
|
|
12
|
+
ibex metrics parser.y
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
2 個の `%%` 区切りを持つファイルは `explain` や `metrics` から直接解析
|
|
16
|
+
できます。修復差分は変換後ソースのバイト位置に対するものなので、修復には
|
|
17
|
+
明示的に出力した canonical 解析ファイルが必要です。
|
|
18
|
+
|
|
19
|
+
対応指令は次の3種類です。
|
|
20
|
+
|
|
21
|
+
- `%token`、結合規則、`%start`、`%expect`、`%empty`、`%prec` は既存の
|
|
22
|
+
Ibex 構文へ変換します。
|
|
23
|
+
- `%type`、`%union`、printer/destructor、parameter、`%define`、`%code`
|
|
24
|
+
は位置付き metadata として報告します。
|
|
25
|
+
- GLR、`%dprec`、`%merge`、生成制御、未知の指令は、出現ごとに元の行・列を
|
|
26
|
+
付けて未対応として報告します。
|
|
27
|
+
|
|
28
|
+
`$$`、`$1`、型付き参照、`@$` は機械的に変換されますが、C の意味を Ruby
|
|
29
|
+
として解釈しません。大文字の非終端、予約語、正規化後の名前衝突も、
|
|
30
|
+
`bison_nt_*` / `BISON_T_*` と決定的な suffix で分離します。
|
|
31
|
+
|
|
32
|
+
レポートと生成ソースは、単なる未対応と「生成規則を完全には復元できない」
|
|
33
|
+
構造的不完全を区別します。後者には
|
|
34
|
+
`ibex-bison-structural-status: incomplete` が付き、`ibex fix` は安全な
|
|
35
|
+
同値修復を主張せず拒否します。
|
|
36
|
+
|
|
37
|
+
既定上限は入力 20 MiB、構造 token 1,000,000、rule group 50,000、action
|
|
38
|
+
100,000 です。`--max-*` は正の整数だけを受け付け、上限超過は JSON で明示し
|
|
39
|
+
終了コード 2 になります。`-o` は原子的に書き込み、入出力 alias、symlink、
|
|
40
|
+
複数 hard link を拒否します。
|
|
41
|
+
|
|
42
|
+
外部 CI は5個の checksum 固定 GNU Bison 文法を一時取得し、成果物には含め
|
|
43
|
+
ません。Bison 時代の CRuby `parse.y` では production 781 と未解決競合 0 が
|
|
44
|
+
一致し、Bison 1,304 state と Ibex 1,303 state の差は、Bison が EOF shift
|
|
45
|
+
専用 accept state を持つためです。
|
|
46
|
+
|
|
47
|
+
現行 CRuby `parse.y` は Lrama 拡張を含み、GNU Bison の生入力ではありません。
|
|
48
|
+
Ibex は `%rule` などを構造的不完全として全列挙し、限定した `explain` は
|
|
49
|
+
近似文法の証拠としてのみ提示します。この状態で `fix` を提案しないことが安全
|
|
50
|
+
契約です。固定 revision、license、取得元は
|
|
51
|
+
[`gallery/EXTERNAL.md`](../../gallery/EXTERNAL.md) にあります。
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# v1 安定 API
|
|
2
|
+
|
|
3
|
+
この文書は v1 で Stable と判定された公開契約の日本語案内です。詳細な
|
|
4
|
+
構文表や全メソッド一覧はリンク先の英語リファレンスを正本としますが、
|
|
5
|
+
下記の契約、互換性、失敗条件は日本語でも同じ意味を持ちます。
|
|
6
|
+
|
|
7
|
+
## racc 互換パーサ生成
|
|
8
|
+
|
|
9
|
+
<!-- stable:compatible-parser:v2 -->
|
|
10
|
+
|
|
11
|
+
既定モードは racc 互換の文法を読み、Pure Ruby の LR パーサを生成します。
|
|
12
|
+
生成クラスは `do_parse` と `yyparse` を提供します。既定の直接 LALR 構築、
|
|
13
|
+
plain/compact 表、yacc `error` 回復、コールバック、観測イベント、
|
|
14
|
+
`ResourceLimits`、移行検査、境界付きの競合反例解析が Stable です。
|
|
15
|
+
互換モードを明示せずに拡張構文へ切り替えることはありません。構文と CLI
|
|
16
|
+
の詳細は [`grammar-reference.md`](../grammar-reference.md) を参照してください。
|
|
17
|
+
|
|
18
|
+
`ibex fix` が検証済み修復として出すのは、対象競合を除去し、独立検証と
|
|
19
|
+
有界同値検査を通った提案だけです。`%expect` と回復品質の設定は競合を
|
|
20
|
+
除去しないため、修復や `--apply` 対象ではなく未検証の助言として分離して
|
|
21
|
+
表示します。
|
|
22
|
+
|
|
23
|
+
## Current IR と表
|
|
24
|
+
|
|
25
|
+
<!-- stable:current-ir:v1 -->
|
|
26
|
+
|
|
27
|
+
Grammar IR と Automaton IR は一つの current closed schema を使い、Lexer IR は
|
|
28
|
+
独立した schema を持ちます。検証器は未知フィールド、不正な参照、schema の
|
|
29
|
+
不一致を拒否します。dump → load → dump は決定的です。v1.0 前の互換性のない
|
|
30
|
+
変更は writer、reader、schema、fixture、利用側を同じ変更で更新します。詳しい境界は
|
|
31
|
+
[`architecture.md`](../architecture.md) にあります。
|
|
32
|
+
|
|
33
|
+
## バッチ Red/Green CST
|
|
34
|
+
|
|
35
|
+
<!-- stable:batch-cst:v1 -->
|
|
36
|
+
|
|
37
|
+
format v6 のバッチ CST は、損失のない Green 値と位置・親参照を持つ Red
|
|
38
|
+
ビューを分離します。typed syntax view、永続編集、差分、`ibex_cst` schema
|
|
39
|
+
v1 の直列化が Stable です。エラー入力も `Error` / `Missing` 要素として木に
|
|
40
|
+
残ります。syntax-only incremental session は Experimental であり、この
|
|
41
|
+
Stable 契約には含まれません。API 例は [`cst.md`](../cst.md) を参照してください。
|
|
42
|
+
|
|
43
|
+
## 互換性と非推奨化
|
|
44
|
+
|
|
45
|
+
<!-- stable:compatibility-policy:v1 -->
|
|
46
|
+
|
|
47
|
+
Stable API は semantic versioning に従います。v1.0 以降の削除や意味変更は、
|
|
48
|
+
少なくとも2回の minor release にわたって移行警告、代替手段、最短削除版を
|
|
49
|
+
文書化します。v1.0 前の Preview は固定された後方互換性契約ではなく、変更時に
|
|
50
|
+
release note、移行メモ、回帰テスト、maturity evidence を同じ変更で更新します。
|
|
51
|
+
v1.0 以降の Preview は少なくとも1回の minor release で予告し、Experimental は
|
|
52
|
+
予告なく変わる可能性があります。正確な在庫と昇格条件は
|
|
53
|
+
[`stability.md`](../stability.md) にあります。
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
# Generated lexer behavior and automaton decision
|
|
2
|
+
|
|
3
|
+
H006 profiles the generated lexer that exists today. It does not replace that
|
|
4
|
+
lexer, add an automaton implementation, or claim that the two lexical models
|
|
5
|
+
are interchangeable. The machine-readable authority is
|
|
6
|
+
[`lexer-profile-v1.json`](../tool/profile/evidence/lexer-profile-v1.json),
|
|
7
|
+
validated by
|
|
8
|
+
[`lexer-profile-v1.schema.json`](../schema/lexer-profile-v1.schema.json).
|
|
9
|
+
|
|
10
|
+
Run and verify the profile with:
|
|
11
|
+
|
|
12
|
+
```sh
|
|
13
|
+
bundle exec ruby tool/lexer_profile.rb \
|
|
14
|
+
--output=tool/profile/evidence/lexer-profile-v1.json
|
|
15
|
+
bundle exec rake quality:lexer_profile
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The profiler generates parsers from committed grammar sources and executes
|
|
19
|
+
their lexer and parser actions. It is therefore limited to trusted repository
|
|
20
|
+
sources. It never downloads or executes the registered third-party grammars.
|
|
21
|
+
|
|
22
|
+
## Semantics are different
|
|
23
|
+
|
|
24
|
+
The current `regexp` profile anchors one Ruby `Regexp` per declared rule. Ruby
|
|
25
|
+
chooses alternatives left-to-right inside that rule. Ibex then compares the
|
|
26
|
+
lexemes returned by every active rule, chooses the longest byte length, and
|
|
27
|
+
uses declaration order for a tie. These are two separate selection steps.
|
|
28
|
+
|
|
29
|
+
For example, `/a|ab/` returns `a` for input `ab` inside its one Ruby rule. A
|
|
30
|
+
regular-language automaton using maximal munch would select `ab`. Calling the
|
|
31
|
+
second result a faster implementation of the first would hide a tokenization
|
|
32
|
+
change.
|
|
33
|
+
|
|
34
|
+
The conditional `automaton` profile described by the design has maximal munch,
|
|
35
|
+
a declaration-order tie, declarative transitions, and bounded construction. It
|
|
36
|
+
also excludes arbitrary Ruby actions and unspecified parser-to-lexer feedback.
|
|
37
|
+
Those exclusions are a semantic boundary, not an implementation detail.
|
|
38
|
+
|
|
39
|
+
## Workloads and provenance
|
|
40
|
+
|
|
41
|
+
The evidence keeps source classes separate:
|
|
42
|
+
|
|
43
|
+
- `synthetic` contains all eight required adversarial fixtures plus the three
|
|
44
|
+
registered gallery grammars. Every grammar and input has an exact Git
|
|
45
|
+
revision, path, byte digest, and derivation. The chunk fixture expands its
|
|
46
|
+
committed 128-byte seed 129 times, producing a 16,512-byte single token that
|
|
47
|
+
crosses the 16KiB streaming boundary.
|
|
48
|
+
- `public_real` contains the exact BCDice, Namae, and Nokogiri CSS registry
|
|
49
|
+
identities. They use external application lexers and have no verified Ibex
|
|
50
|
+
generated-lexer port, so their status is `not_run`, their result is `null`,
|
|
51
|
+
and they contribute no inferred measurements.
|
|
52
|
+
|
|
53
|
+
Synthetic success is not evidence that public-real lexers are suitable for an
|
|
54
|
+
automaton migration. Evidence is generated from a clean exact revision, and
|
|
55
|
+
that revision must be the first parent of the commit that records the evidence.
|
|
56
|
+
The capture identity covers that revision, clean/status provenance, the complete
|
|
57
|
+
bound execution-source closure, implementation subset, deterministic report
|
|
58
|
+
input, measurement policy, and heuristic policy. The verifier locates the same
|
|
59
|
+
identity in committed evidence history instead of accepting any ancestor with
|
|
60
|
+
rewritten derived fields. A shallow checkout that lacks the evidence commit's
|
|
61
|
+
parent or pinned fixture revisions fails closed; CI must fetch full history
|
|
62
|
+
rather than treating missing provenance as a match.
|
|
63
|
+
|
|
64
|
+
The capture command deliberately loads only the frontend, Grammar/Automaton IR,
|
|
65
|
+
normalizer, analysis/LALR construction, Ruby code generator, and generated
|
|
66
|
+
runtime graph. It does not load the public `ibex` aggregate, CLI, or configuration
|
|
67
|
+
graph. Every repository Ruby source reachable from that measurement graph is a
|
|
68
|
+
bound path, and capture aborts if another repository source is loaded. The public
|
|
69
|
+
schema and this document are explicit generator-gem packaging expectations.
|
|
70
|
+
|
|
71
|
+
## Measurements and limits
|
|
72
|
+
|
|
73
|
+
Each measured workload records states, rules per state, token byte lengths,
|
|
74
|
+
action-bearing rules, direct lexer/parser state mutation, parser feedback,
|
|
75
|
+
Regexp warnings, streaming peak buffer bytes, and incremental bytes read from
|
|
76
|
+
offset zero. Elapsed time and `GC.total_allocated_objects` deltas are retained
|
|
77
|
+
only as host-bound diagnostics. Both the schema and semantic validator require
|
|
78
|
+
`release_gate: false`; the quality comparison removes their values.
|
|
79
|
+
|
|
80
|
+
The alternation, lazy-quantifier, and direct `lexer_state =` inventories are
|
|
81
|
+
source-text heuristics over normalized rules and actions. They do not form a
|
|
82
|
+
complete Ruby or Regexp semantic analysis. Aliases, helper calls,
|
|
83
|
+
metaprogramming, indirect mutation, or dynamically composed behavior can be
|
|
84
|
+
missed. A zero count is not proof of absence.
|
|
85
|
+
|
|
86
|
+
The Unicode-property fixture uses String input. The current incremental
|
|
87
|
+
`SourceText` path stores binary bytes, so this profile does not claim that
|
|
88
|
+
Unicode-property matching works through that incremental path. The
|
|
89
|
+
parser-feedback fixture is semantically parsed successfully, but its
|
|
90
|
+
syntax-only incremental scan is `not_measured`: suppressing the parser action
|
|
91
|
+
also suppresses the state change needed before the next token.
|
|
92
|
+
|
|
93
|
+
Peak buffer bytes cover only `LexerInput`, not whole-process memory. The
|
|
94
|
+
incremental full-scan share covers generated-lexer bytes read after a committed
|
|
95
|
+
identity edit; it does not measure CST subtree reuse.
|
|
96
|
+
|
|
97
|
+
## Decision
|
|
98
|
+
|
|
99
|
+
| Scope | Decision | Reason |
|
|
100
|
+
| --- | --- | --- |
|
|
101
|
+
| Automatic replacement of the current Regexp lexer | **NO-GO** | `/a|ab/` proves different token boundaries; arbitrary actions and parser-driven state changes have no specified semantic translation. |
|
|
102
|
+
| A separately named automaton lexer profile | **MORE DATA** | All adversarial fixtures are measured, but there are zero verified public-real generated-lexer workloads, no standalone semantic contract, and no verified adoption trigger. |
|
|
103
|
+
|
|
104
|
+
Timing and allocation observations are not thresholds for either decision.
|
|
105
|
+
Reconsidering automatic replacement requires an explicit compatibility model,
|
|
106
|
+
an action/feedback contract, and differential evidence on verified real
|
|
107
|
+
generated lexers. Reconsidering the separate profile requires at least two such
|
|
108
|
+
public-real workloads, a bounded declarative semantics, and demonstrated user
|
|
109
|
+
pressure that the design addresses.
|
data/docs/lexer-migration.md
CHANGED
data/docs/maturity.md
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# Preview and Experimental maturity audit
|
|
2
|
+
|
|
3
|
+
[`maturity.yml`](maturity.yml) and the validator-owned commit authorities in
|
|
4
|
+
[`maturity_authority.rb`](../tool/quality/maturity_authority.rb) form the joint
|
|
5
|
+
human-reviewed trust boundary for H001. The registry binds the current 18
|
|
6
|
+
Preview and two Experimental features to the reviewed repository revision,
|
|
7
|
+
source digests, activation, external-use status, specification-history
|
|
8
|
+
confidence, issue audit, documentation and tooling gaps, performance and safety
|
|
9
|
+
limits, decision, next trigger, and release gates. Neither file authenticates
|
|
10
|
+
the other; changing a classification requires reviewing and updating both.
|
|
11
|
+
|
|
12
|
+
## Result
|
|
13
|
+
|
|
14
|
+
<!-- maturity-summary:start -->
|
|
15
|
+
Inventory: **18 Preview, 2 Experimental**. Active new Preview tracks: **1/3** (grammar syntax: **1/1**). Experimental product features: **2/5**.
|
|
16
|
+
Release dependency state: R001 **hold_external**; R002 **pending_exact_revision**; no feature is promoted by this audit.
|
|
17
|
+
|
|
18
|
+
| Stable ID | Feature | Current maturity | Decision | External use | Release gate |
|
|
19
|
+
| --- | --- | --- | --- | --- | --- |
|
|
20
|
+
| `ebnf-groups` | EBNF groups | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
21
|
+
| `parameterized-rules` | parameterized rules | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
22
|
+
| `inline-rules` | inline rules | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
23
|
+
| `middle-actions` | middle actions | Preview | Redesign Preview | not demonstrated | Blocked: R001, R002 |
|
|
24
|
+
| `multiple-entries` | multiple entries | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
25
|
+
| `canonical-imports` | canonical imports | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
26
|
+
| `generated-lexers` | generated lexers | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
27
|
+
| `semantic-locations-types` | semantic locations/types | Preview | Redesign Preview | not demonstrated | Blocked: R001, R002 |
|
|
28
|
+
| `ast-generation` | AST generation | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
29
|
+
| `grammar-tests` | grammar tests | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
30
|
+
| `documentation-tooling` | documentation tooling | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
31
|
+
| `ielr` | IELR | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
32
|
+
| `lsp` | LSP | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
33
|
+
| `watch` | watch | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
34
|
+
| `debug` | debug | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
35
|
+
| `coverage` | coverage | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
36
|
+
| `browser-playground` | browser playground | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
37
|
+
| `action-shadow` | action-shadow | Preview | Keep Preview | not demonstrated | Blocked: R001, R002 |
|
|
38
|
+
| `bounded-repair` | bounded repair | Experimental | Keep Experimental | not demonstrated | Blocked: R001, R002 |
|
|
39
|
+
| `incremental-cst` | incremental CST | Experimental | Keep Experimental | not demonstrated | Blocked: R001, R002 |
|
|
40
|
+
<!-- maturity-summary:end -->
|
|
41
|
+
|
|
42
|
+
Every keep decision names unmet promotion evidence. Two rows record redesigns.
|
|
43
|
+
Compatible/default Racc middle actions are already governed by the Stable
|
|
44
|
+
compatibility contract, and no separable Preview activation currently exists;
|
|
45
|
+
the redundant Preview classification must be split or removed at the next
|
|
46
|
+
reviewed release unless a distinct opt-in extension is defined. Compatible-mode
|
|
47
|
+
location APIs and extended-only type declarations also require separate
|
|
48
|
+
inventory records and promotion gates. Passing repository tests is necessary,
|
|
49
|
+
but cannot establish external field compatibility, complete R001's independent
|
|
50
|
+
review, or replace R002 on the exact release revision.
|
|
51
|
+
|
|
52
|
+
Activation and maturity are independent. Middle actions are accepted in
|
|
53
|
+
compatible/default grammars when an action is embedded before the end of a
|
|
54
|
+
production, so the Stable guarantee takes precedence over Preview notice.
|
|
55
|
+
Compatible semantic actions can consume lexer locations; only grammar `type`
|
|
56
|
+
declarations require extended mode. Preview therefore does not imply that every
|
|
57
|
+
audited surface is disabled by default or may break under Preview policy.
|
|
58
|
+
|
|
59
|
+
The active declarative-parser-construction track changes how the already
|
|
60
|
+
Preview IELR and multiple-entry contracts can be activated: an extended root
|
|
61
|
+
grammar may now own those settings. It consumes the grammar-syntax budget but
|
|
62
|
+
does not add or promote a maturity-inventory row. Its source history and field
|
|
63
|
+
experience must be incorporated into the next reviewed maturity audit.
|
|
64
|
+
|
|
65
|
+
## Evidence boundaries
|
|
66
|
+
|
|
67
|
+
External use is evaluated against the stable IDs and classifications in the
|
|
68
|
+
[public workload registry](workloads.md). `repository_synthetic` gallery use
|
|
69
|
+
supports correctness testing only. `external_real` Bison inputs support only
|
|
70
|
+
their registered diagnostic scope. Neither class is reported as third-party
|
|
71
|
+
Ibex adoption. The current `public_real` records exercise compatible-mode
|
|
72
|
+
generation and runtime, not one of the audited Preview or Experimental
|
|
73
|
+
features, so every external-use result is `not_demonstrated`.
|
|
74
|
+
|
|
75
|
+
The issue audit ran the exact GitHub Search API query and command recorded in
|
|
76
|
+
the YAML on 2026-08-04. It returned zero open issues with a complete result.
|
|
77
|
+
That result expires after 30 days and means only that the public tracker had no
|
|
78
|
+
open report matching the query; it is not a correctness proof. The validator
|
|
79
|
+
fails after the recorded freshness deadline until the exact audit is rerun and
|
|
80
|
+
reviewed.
|
|
81
|
+
|
|
82
|
+
Each feature records its validator-owned first Git pickaxe introduction, first
|
|
83
|
+
containing release, canonical-blob presence, and integrity snapshots at v0.1.0,
|
|
84
|
+
v0.2.0, and the exact reviewed revision. Each release boundary separately
|
|
85
|
+
records one assessment for every path-relevant commit. An assessment binds the
|
|
86
|
+
full revision, exact Git subject, classification, and commit-specific public
|
|
87
|
+
contract effect. The validator reconstructs the reviewed commits from Git,
|
|
88
|
+
checks ancestry and changed-path relevance, and derives both the boundary's
|
|
89
|
+
reviewed and semantic commit sets from those exhaustive assessments. The
|
|
90
|
+
path match determines which commits require assessment, not whether they are
|
|
91
|
+
semantic. `semantic_change` is reserved for an observable syntax, API, output,
|
|
92
|
+
or runtime change when that inventory row's feature configuration is active;
|
|
93
|
+
changes behind mutually exclusive guards or confined to adjacent features are
|
|
94
|
+
classified as nonsemantic and name the relevant guard or code path. The
|
|
95
|
+
remaining `unknowns` state what repository history cannot establish, such as
|
|
96
|
+
downstream compatibility and field use; source age or a digest change alone
|
|
97
|
+
receives no semantic or promotion credit.
|
|
98
|
+
|
|
99
|
+
The introduction method is exactly
|
|
100
|
+
`git log --reverse --format=%H -SQUERY REVIEWED_REVISION -- PATH`; the reviewed
|
|
101
|
+
revision bound prevents later history from rewriting the audit result.
|
|
102
|
+
Boundary review uses `git log --reverse --format=%H FROM..TO -- PATH...` over
|
|
103
|
+
the validator-owned introduction and canonical paths. The first range starts at
|
|
104
|
+
the introduction commit's parent so the introduction itself is reviewed through
|
|
105
|
+
its first containing release; subsequent ranges join exact release and reviewed
|
|
106
|
+
revisions. Each source-tree digest hashes sorted canonical paths as path, NUL,
|
|
107
|
+
exact Git blob bytes (or the literal `<absent>`), NUL. Digests are integrity
|
|
108
|
+
evidence only: they neither discover nor classify semantic changes. Tags
|
|
109
|
+
resolve to full commit IDs, and introductions, releases, and reviewed commits
|
|
110
|
+
must have the recorded ancestry and order.
|
|
111
|
+
|
|
112
|
+
## Updating the audit
|
|
113
|
+
|
|
114
|
+
Run:
|
|
115
|
+
|
|
116
|
+
```sh
|
|
117
|
+
bundle exec rake quality:maturity
|
|
118
|
+
bundle exec ruby -Itest test/quality/maturity_test.rb
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
For history changes, update the exhaustive assessments in `maturity.yml` and
|
|
122
|
+
the explicit per-feature/per-boundary semantic authority together after human
|
|
123
|
+
review. Update source SHA-256 values only as integrity records; never use them
|
|
124
|
+
to infer semantics. A promotion also requires the feature's next-review
|
|
125
|
+
evidence, completed R001 and exact-revision R002 states, and a synchronized
|
|
126
|
+
generated summary in this file and [`stability.md`](stability.md). A redesign
|
|
127
|
+
or removal is visible in the same summary and still follows the applicable
|
|
128
|
+
Preview or Experimental compatibility policy.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Project site strategy
|
|
3
|
+
description: Information architecture and implementation boundaries for the Ibex public site.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Project site strategy
|
|
7
|
+
|
|
8
|
+
The public site is an adoption path rather than a catalogue of every internal
|
|
9
|
+
feature. The intended sequence is:
|
|
10
|
+
|
|
11
|
+
```text
|
|
12
|
+
understand → try → install → investigate → verify → contribute
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Information architecture
|
|
16
|
+
|
|
17
|
+
- The README answers the installation and first-parser decision in roughly 90
|
|
18
|
+
seconds.
|
|
19
|
+
- Pages provides the task-oriented documentation hub, with Getting Started,
|
|
20
|
+
Guides, Reference, Concepts, Examples, Playground, API, and Project areas.
|
|
21
|
+
- Compatibility, Extensions, Experimental, Gallery, Playground, and API keep
|
|
22
|
+
their existing URLs. They are canonical landing pages or redirects, not
|
|
23
|
+
discarded history.
|
|
24
|
+
- Stable, Preview, and Experimental are feature metadata. They do not become
|
|
25
|
+
separate product versions or imply multiple persisted IR versions.
|
|
26
|
+
|
|
27
|
+
## Content rules
|
|
28
|
+
|
|
29
|
+
Public copy must distinguish racc migration compatibility from a claim of
|
|
30
|
+
byte-for-byte replacement, bounded evidence from proof, and trusted generated
|
|
31
|
+
Ruby from sandboxed execution. Comparative claims link to their complete
|
|
32
|
+
evidence record. Maturity is read from `docs/maturity.yml`; prose does not
|
|
33
|
+
duplicate a second feature registry.
|
|
34
|
+
|
|
35
|
+
## Build and safety rules
|
|
36
|
+
|
|
37
|
+
`tool/build_site.rb` is the canonical static builder. It reads docs source,
|
|
38
|
+
renders the selected documentation pages, copies the canonical landing pages,
|
|
39
|
+
and emits the Playground bundle into `tmp/site`. The builder must remain
|
|
40
|
+
reproducible, self-hosted, and free of runtime gem dependencies. Every page
|
|
41
|
+
uses a strict same-origin CSP, has a skip link and canonical metadata, and
|
|
42
|
+
does not send grammar source to a third party. Playground actions remain
|
|
43
|
+
non-executing, bounded, and worker-isolated.
|
|
44
|
+
|
|
45
|
+
The implementation is intentionally framework-free. A framework migration is
|
|
46
|
+
only worth revisiting if versioned documentation, multilingual content, large
|
|
47
|
+
faceted search, or builder maintenance becomes a demonstrated cost.
|
|
48
|
+
|
|
49
|
+
## Operational review
|
|
50
|
+
|
|
51
|
+
Review the site with keyboard navigation, narrow viewports, zoom, reduced
|
|
52
|
+
motion, link/anchor checks, and a clean rebuild. New feature claims must point
|
|
53
|
+
to the maturity registry and evidence rather than being inferred from a
|
|
54
|
+
landing-page badge.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Racc migration evidence
|
|
2
|
+
|
|
3
|
+
This page is the H002 evidence index for the compatibility claims in
|
|
4
|
+
[`racc-migration.md`](racc-migration.md). It records observable behavior only;
|
|
5
|
+
it does not claim that a third party has adopted Ibex, and it is not a
|
|
6
|
+
performance result.
|
|
7
|
+
|
|
8
|
+
## Evidence boundary
|
|
9
|
+
|
|
10
|
+
The three public migration suites are pinned in
|
|
11
|
+
[`release-readiness.md`](release-readiness.md): Namae, BCDice, and Nokogiri.
|
|
12
|
+
Their revisions, adapters, and observed test counts are release evidence.
|
|
13
|
+
The rows below are repository-owned black-box probes that target compatibility
|
|
14
|
+
risks which are easy to miss in a single arithmetic grammar. They run against
|
|
15
|
+
the installed `racc` executable when it is available and skip otherwise; a
|
|
16
|
+
skip is not reported as a pass.
|
|
17
|
+
|
|
18
|
+
| Risk | Probe and observable contract | Reproduction |
|
|
19
|
+
|---|---|---|
|
|
20
|
+
| precedence and conflict accounting | arithmetic results match; resolved conflicts remain visible in Automaton IR but do not inflate the CLI conflict count or `expect` | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
|
|
21
|
+
| empty productions and string tokens | empty reductions and literal-token paths produce the same public result values | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
|
|
22
|
+
| `convert` and `no_result_var` | token conversion and result-variable suppression preserve the generated parser's public action behavior | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
|
|
23
|
+
| inline actions | action placement and returned semantic values are equivalent at the public parser boundary | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
|
|
24
|
+
| `expect` and dangling-else behavior | the expected conflict count and parse result remain stable for the ambiguous grammar | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
|
|
25
|
+
| error recovery and `on_error` | callback arguments and recovered result values are compared; the documented undeclared-token difference remains explicit | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
|
|
26
|
+
| source locations and line conversion | semantic action line mapping follows the documented default and `--line-convert-all` modes | `bundle exec ruby -Itest test/racc_migration_test.rb` |
|
|
27
|
+
| runtime initialization and value-stack reads | an initializer that omits `super` retains application state while historical value-stack reads remain compatible | `bundle exec ruby -Itest test/racc_migration_test.rb` |
|
|
28
|
+
| unqualified `ParseError` | parser methods and actions can retain the explicit rejection path through `Runtime::Parser` lookup | `bundle exec ruby -Itest test/racc_migration_test.rb` |
|
|
29
|
+
| large grammar construction | a 500-production grammar completes with the recorded production/state bound and no conflicts | `bundle exec ruby -Itest test/compat/black_box_test.rb` |
|
|
30
|
+
| push-parser runtime contract | push-session lifecycle, token feeding, acceptance, and cleanup remain stable within Ibex's public runtime API | `bundle exec ruby -Itest test/runtime/push_parser_test.rb` |
|
|
31
|
+
|
|
32
|
+
The compatibility test files are the executable source of truth. The commands
|
|
33
|
+
above were last run at the evidence refresh and produced zero failures and zero
|
|
34
|
+
errors (black-box: 6 runs/33 assertions; migration: 5 runs/20 assertions;
|
|
35
|
+
push-parser: 46 runs/183 assertions). The black-box suite reported no skip in
|
|
36
|
+
that environment.
|
|
37
|
+
|
|
38
|
+
## Interpretation
|
|
39
|
+
|
|
40
|
+
These probes establish coverage of distinct compatibility risks, not a claim
|
|
41
|
+
that every racc grammar is interchangeable. Generated source layout, internal
|
|
42
|
+
table arrays, `require "racc/parser"` replacement, and previously generated
|
|
43
|
+
racc table loading remain explicitly out of scope. Formal cross-project
|
|
44
|
+
performance comparison is tracked separately in the release-readiness report
|
|
45
|
+
and remains incomplete until a clean, exact-revision artifact is published.
|
data/docs/racc-migration.md
CHANGED
|
@@ -1,3 +1,8 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Migrating from racc
|
|
3
|
+
description: Move a racc grammar to Ibex with explicit compatibility checks and runtime boundaries.
|
|
4
|
+
---
|
|
5
|
+
|
|
1
6
|
# Migrating from racc
|
|
2
7
|
|
|
3
8
|
Ibex targets grammar-file compatibility, generated parser public API compatibility, and the main racc CLI options. It does not
|
|
@@ -49,6 +54,12 @@ generated parsers in bounded child processes. This is not a sandbox; use a conta
|
|
|
49
54
|
|
|
50
55
|
## Compatibility baseline
|
|
51
56
|
|
|
57
|
+
The repository probes and their exact reproduction commands are recorded in
|
|
58
|
+
[`racc-migration-evidence.md`](racc-migration-evidence.md). That matrix is an
|
|
59
|
+
internal evidence index: it distinguishes self-authored black-box coverage
|
|
60
|
+
from the separately pinned public-gem migration suites and does not turn either
|
|
61
|
+
set into a claim of third-party adoption or performance parity.
|
|
62
|
+
|
|
52
63
|
The compatibility claims above were checked on 2026-07-22 against racc 1.8.1 using only its public documentation, `racc --help`,
|
|
53
64
|
and black-box execution. Ibex does not inspect racc implementation files or generated source. Self-authored probes compare
|
|
54
65
|
observable results for arithmetic precedence, empty rules, string tokens, `convert`, `no_result_var`, inline actions,
|