expressir 1.2.10-aarch64-linux → 2.4.29-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (467) hide show
  1. checksums.yaml +4 -4
  2. data/.cargo/config.toml +3 -0
  3. data/.github/scripts/test_installed_gem.rb +34 -0
  4. data/.github/workflows/codeql.yml +35 -0
  5. data/.github/workflows/docs.yml +99 -0
  6. data/.github/workflows/links.yml +100 -0
  7. data/.github/workflows/native-gems.yml +201 -0
  8. data/.github/workflows/rake.yml +10 -296
  9. data/.github/workflows/release.yml +26 -114
  10. data/.github/workflows/rust-ext.yml +41 -0
  11. data/.github/workflows/stress.yml +36 -0
  12. data/.github/workflows/validate_schemas.yml +48 -0
  13. data/.github/workflows/verify_remarks.yml +33 -0
  14. data/.gitignore +33 -1
  15. data/.rubocop.yml +11 -11
  16. data/.rubocop_todo.yml +297 -0
  17. data/CHANGELOG.md +159 -0
  18. data/Gemfile +15 -1
  19. data/README.adoc +2675 -47
  20. data/Rakefile +34 -10
  21. data/TODO.bugs/01-stale-transformer-autoload.md +39 -0
  22. data/TODO.bugs/02-parser-class-instance-vars.md +36 -0
  23. data/TODO.bugs/03-builder-mutable-state.md +43 -0
  24. data/TODO.bugs/04-formatter-public-send-dispatch.md +53 -0
  25. data/TODO.bugs/05-anonymous-formatter-subclass.md +45 -0
  26. data/TODO.bugs/06-collection-registry-single-source.md +53 -0
  27. data/TODO.bugs/07-require-relative-cleanup.md +42 -0
  28. data/TODO.bugs/08-require-expressir-in-commands.md +34 -0
  29. data/TODO.bugs/09-parser-split.md +53 -0
  30. data/TODO.bugs/10-to-s-override.md +42 -0
  31. data/TODO.bugs/11-parser-class-variables.md +39 -0
  32. data/TODO.bugs/12-marker-modules-vs-registry.md +64 -0
  33. data/TODO.bugs/13-string-literal-scanner-limitation.md +52 -0
  34. data/TODO.bugs/14-model-formatting-leak.md +30 -0
  35. data/TODO.bugs/15-expression-children-macro.md +27 -0
  36. data/TODO.bugs/16-pretty-formatter-duplication.md +28 -0
  37. data/TODO.bugs/17-snake-case-cache-mutable-constant.md +28 -0
  38. data/TODO.bugs/18-const-get-private-constants.md +30 -0
  39. data/TODO.bugs/19-format-methods-public.md +22 -0
  40. data/TODO.bugs/20-coverage-nested-entities-dedup.md +20 -0
  41. data/TODO.bugs/21-operator-tokens-secondary-dispatch.md +21 -0
  42. data/TODO.bugs/22-builder-fast-path-wrappers.md +32 -0
  43. data/TODO.bugs/23-coverage-inverse-maps.md +21 -0
  44. data/TODO.bugs/24-streaming-builder-complexity.md +19 -0
  45. data/TODO.bugs/25-debug-puts-in-production.md +21 -0
  46. data/TODO.bugs/26-generic-entity-children-misplaced.md +21 -0
  47. data/TODO.bugs/27-package-build-god-method.md +19 -0
  48. data/TODO.bugs/28-package-god-class.md +30 -0
  49. data/TODO.bugs/29-validate-ascii-god-class.md +24 -0
  50. data/TODO.bugs/30-unicode-map-extraction.md +19 -0
  51. data/TODO.bugs/README.md +43 -0
  52. data/TODO.max-perf/01-restore-ci-green.md +36 -0
  53. data/TODO.max-perf/02-streaming-parse-path.md +31 -0
  54. data/TODO.max-perf/03-cli-parallel-opt-in.md +27 -0
  55. data/TODO.max-perf/04-benchmark-harness.md +28 -0
  56. data/TODO.max-perf/05-parallel-fidelity-specs.md +22 -0
  57. data/TODO.max-perf/06-builder-cpu-audit.md +41 -0
  58. data/TODO.max-perf/07-upstream-parsanol-roadmap.md +27 -0
  59. data/TODO.max-perf/08-builder-build-perf.md +45 -0
  60. data/TODO.max-perf/09-grammar-cold-start.md +25 -0
  61. data/TODO.max-perf/10-parser-facade-hygiene.md +23 -0
  62. data/TODO.max-perf/11-ci-green-closeout.md +25 -0
  63. data/TODO.max-perf/12-require-boot-profile.md +25 -0
  64. data/TODO.max-perf/13-key-conversion-specs.md +26 -0
  65. data/TODO.max-perf/14-builder-call-handler-audit.md +28 -0
  66. data/TODO.parity-ee/00-overview.md +76 -0
  67. data/TODO.parity-ee/01-annex-g-rule-extraction.md +41 -0
  68. data/TODO.parity-ee/02-eeng-algorithm-inventory.md +8 -0
  69. data/TODO.parity-ee/03-eeng-oracle-harness.md +33 -0
  70. data/TODO.parity-ee/04-shtolo-concatenate.md +44 -0
  71. data/TODO.parity-ee/05-shtolo-longform-flatten.md +53 -0
  72. data/TODO.parity-ee/06-interface-scheduling-parity.md +47 -0
  73. data/TODO.parity-ee/07-semantic-checks-port.md +35 -0
  74. data/TODO.parity-ee/08-pretty-roundtrip-gate.md +27 -0
  75. data/TODO.parity-ee/09-smrl-index-and-listing.md +22 -0
  76. data/TODO.parity-ee/10-interface-dot-graph.md +21 -0
  77. data/TODO.parity-ee/11-import-eeng-tests.md +79 -0
  78. data/TODO.parity-ee/12-full-check-catalog.md +67 -0
  79. data/TODO.parity-ee/13-population-p21.md +54 -0
  80. data/TODO.parity-ee/14-shtolo-completion.md +44 -0
  81. data/TODO.parity-ee/15-part28-xml.md +51 -0
  82. data/TODO.parity-ee/16-compare-patch.md +48 -0
  83. data/TODO.parity-ee/17-official-tests.md +42 -0
  84. data/TODO.parity-ee/18-mim-mapping.md +47 -0
  85. data/TODO.parity-ee/19-architecture.md +97 -0
  86. data/TODO.parity-ee/parity-matrix.md +108 -0
  87. data/TODO.refpath-v2/00-overview.md +64 -0
  88. data/TODO.suma-improvements/00-overview.md +23 -0
  89. data/TODO.suma-improvements/01-prebuilt-platform-gems.md +33 -0
  90. data/TODO.suma-improvements/02-suma-e2e.md +39 -0
  91. data/TODO.suma-improvements/03-item-graph.md +22 -0
  92. data/TODO.suma-improvements/04-compiled-set-default.md +16 -0
  93. data/TODO.suma-improvements/05-validation-in-docs.md +17 -0
  94. data/TODO.suma-improvements/06-memory-remeasure.md +36 -0
  95. data/TODO.suma-improvements/07-part28-xml.md +15 -0
  96. data/TODO.suma-improvements/08-windows-gnu-toolchain.md +15 -0
  97. data/TODO.suma-improvements/09-checker-tranche3.md +13 -0
  98. data/TODO.suma-improvements/10-expressdoc.md +8 -0
  99. data/benchmark/srl_benchmark.rb +458 -0
  100. data/benchmark/srl_native_benchmark.rb +146 -0
  101. data/benchmark/srl_ruby_benchmark.rb +132 -0
  102. data/bin/rspec +2 -1
  103. data/docs/Gemfile +12 -0
  104. data/docs/_config.yml +141 -0
  105. data/docs/_guides/changes/changes-format.adoc +778 -0
  106. data/docs/_guides/changes/importing-eengine.adoc +898 -0
  107. data/docs/_guides/changes/index.adoc +396 -0
  108. data/docs/_guides/changes/programmatic-usage.adoc +1038 -0
  109. data/docs/_guides/changes/validating-changes.adoc +681 -0
  110. data/docs/_guides/cli/benchmark-performance.adoc +834 -0
  111. data/docs/_guides/cli/coverage-analysis.adoc +921 -0
  112. data/docs/_guides/cli/format-schemas.adoc +547 -0
  113. data/docs/_guides/cli/index.adoc +8 -0
  114. data/docs/_guides/cli/managing-changes.adoc +927 -0
  115. data/docs/_guides/cli/validate-ascii.adoc +645 -0
  116. data/docs/_guides/cli/validate-schemas.adoc +534 -0
  117. data/docs/_guides/formatter/formatter-architecture.adoc +401 -0
  118. data/docs/_guides/index.adoc +165 -0
  119. data/docs/_guides/ler/creating-packages.adoc +664 -0
  120. data/docs/_guides/ler/index.adoc +305 -0
  121. data/docs/_guides/ler/loading-packages.adoc +707 -0
  122. data/docs/_guides/ler/package-formats.adoc +748 -0
  123. data/docs/_guides/ler/querying-packages.adoc +826 -0
  124. data/docs/_guides/ler/step-packages.adoc +385 -0
  125. data/docs/_guides/ler/validating-packages.adoc +750 -0
  126. data/docs/_guides/liquid/basic-templates.adoc +813 -0
  127. data/docs/_guides/liquid/documentation-generation.adoc +1042 -0
  128. data/docs/_guides/liquid/drops-reference.adoc +829 -0
  129. data/docs/_guides/liquid/filters-and-tags.adoc +912 -0
  130. data/docs/_guides/liquid/index.adoc +468 -0
  131. data/docs/_guides/manifests/creating-manifests.adoc +483 -0
  132. data/docs/_guides/manifests/index.adoc +307 -0
  133. data/docs/_guides/manifests/resolving-manifests.adoc +557 -0
  134. data/docs/_guides/manifests/validating-manifests.adoc +713 -0
  135. data/docs/_guides/ruby-api/formatting-schemas.adoc +605 -0
  136. data/docs/_guides/ruby-api/index.adoc +257 -0
  137. data/docs/_guides/ruby-api/parsing-files.adoc +421 -0
  138. data/docs/_guides/ruby-api/search-engine.adoc +609 -0
  139. data/docs/_guides/ruby-api/working-with-repository.adoc +577 -0
  140. data/docs/_pages/data-model.adoc +665 -0
  141. data/docs/_pages/express-language.adoc +506 -0
  142. data/docs/_pages/getting-started.adoc +414 -0
  143. data/docs/_pages/index.adoc +116 -0
  144. data/docs/_pages/introduction.adoc +256 -0
  145. data/docs/_pages/ler-packages.adoc +837 -0
  146. data/docs/_pages/parsers.adoc +709 -0
  147. data/docs/_pages/schema-manifests.adoc +431 -0
  148. data/docs/_references/index.adoc +228 -0
  149. data/docs/_tutorials/creating-ler-package.adoc +735 -0
  150. data/docs/_tutorials/documentation-coverage.adoc +795 -0
  151. data/docs/_tutorials/formatting-schemas.adoc +89 -0
  152. data/docs/_tutorials/index.adoc +231 -0
  153. data/docs/_tutorials/liquid-templates.adoc +806 -0
  154. data/docs/_tutorials/parsing-your-first-schema.adoc +522 -0
  155. data/docs/_tutorials/querying-schemas.adoc +751 -0
  156. data/docs/_tutorials/working-with-multiple-schemas.adoc +676 -0
  157. data/docs/index.adoc +242 -0
  158. data/docs/lychee.toml +90 -0
  159. data/examples/demo_ler_usage.sh +86 -0
  160. data/examples/ler/README.md +111 -0
  161. data/examples/ler/simple_example.ler +0 -0
  162. data/examples/ler/simple_schema.exp +33 -0
  163. data/examples/ler_build.rb +75 -0
  164. data/examples/ler_cli.rb +79 -0
  165. data/examples/ler_demo_complete.rb +276 -0
  166. data/examples/ler_query.rb +91 -0
  167. data/examples/ler_query_examples.rb +305 -0
  168. data/examples/ler_stats.rb +81 -0
  169. data/examples/phase3_demo.rb +159 -0
  170. data/examples/query_demo_simple.rb +131 -0
  171. data/exe/expressir +6 -17
  172. data/exe/expressir-format +14 -0
  173. data/exe/{format-test → expressir-format-test} +8 -7
  174. data/expressir.gemspec +50 -37
  175. data/lib/expressir/3.2/expressir_core.so +0 -0
  176. data/lib/expressir/3.3/expressir_core.so +0 -0
  177. data/lib/expressir/3.4/expressir_core.so +0 -0
  178. data/lib/expressir/4.0/expressir_core.so +0 -0
  179. data/lib/expressir/benchmark.rb +310 -0
  180. data/lib/expressir/changes/item_change.rb +20 -0
  181. data/lib/expressir/changes/mapping_change.rb +16 -0
  182. data/lib/expressir/changes/schema_change.rb +85 -0
  183. data/lib/expressir/changes/version_change.rb +26 -0
  184. data/lib/expressir/changes.rb +9 -0
  185. data/lib/expressir/cli.rb +138 -11
  186. data/lib/expressir/commands/base.rb +25 -0
  187. data/lib/expressir/commands/benchmark.rb +59 -0
  188. data/lib/expressir/commands/benchmark_cache.rb +80 -0
  189. data/lib/expressir/commands/changes.rb +34 -0
  190. data/lib/expressir/commands/changes_import_eengine.rb +146 -0
  191. data/lib/expressir/commands/changes_validate.rb +63 -0
  192. data/lib/expressir/commands/check.rb +56 -0
  193. data/lib/expressir/commands/clean.rb +20 -0
  194. data/lib/expressir/commands/coverage.rb +458 -0
  195. data/lib/expressir/commands/expand.rb +26 -0
  196. data/lib/expressir/commands/file_violations.rb +70 -0
  197. data/lib/expressir/commands/fix.rb +33 -0
  198. data/lib/expressir/commands/flatten.rb +29 -0
  199. data/lib/expressir/commands/format.rb +43 -0
  200. data/lib/expressir/commands/manifest.rb +420 -0
  201. data/lib/expressir/commands/mapping_validate.rb +68 -0
  202. data/lib/expressir/commands/non_ascii_character.rb +49 -0
  203. data/lib/expressir/commands/non_ascii_violation_collection.rb +301 -0
  204. data/lib/expressir/commands/package.rb +1222 -0
  205. data/lib/expressir/commands/parity_inputs.rb +160 -0
  206. data/lib/expressir/commands/validate.rb +106 -0
  207. data/lib/expressir/commands/validate_ascii.rb +94 -0
  208. data/lib/expressir/commands/validate_load.rb +91 -0
  209. data/lib/expressir/commands/version.rb +9 -0
  210. data/lib/expressir/commands/xsd.rb +23 -0
  211. data/lib/expressir/commands.rb +30 -0
  212. data/lib/expressir/config.rb +40 -3
  213. data/lib/expressir/coverage.rb +510 -0
  214. data/lib/expressir/eengine/arm_compare_report.rb +23 -0
  215. data/lib/expressir/eengine/changes_section.rb +16 -0
  216. data/lib/expressir/eengine/compare_report.rb +69 -0
  217. data/lib/expressir/eengine/mim_compare_report.rb +23 -0
  218. data/lib/expressir/eengine/modified_object.rb +21 -0
  219. data/lib/expressir/eengine.rb +9 -0
  220. data/lib/expressir/errors.rb +113 -0
  221. data/lib/expressir/express/adoc_hyperlink_formatter.rb +30 -0
  222. data/lib/expressir/express/adoc_source_formatter.rb +12 -0
  223. data/lib/expressir/express/ast_key_converter.rb +114 -0
  224. data/lib/expressir/express/builder.rb +277 -0
  225. data/lib/expressir/express/builder_context.rb +22 -0
  226. data/lib/expressir/express/builder_registry.rb +397 -0
  227. data/lib/expressir/express/builders/attribute_decl_builder.rb +32 -0
  228. data/lib/expressir/express/builders/built_in_builder.rb +75 -0
  229. data/lib/expressir/express/builders/constant_builder.rb +108 -0
  230. data/lib/expressir/express/builders/declaration_builder.rb +20 -0
  231. data/lib/expressir/express/builders/derive_clause_builder.rb +14 -0
  232. data/lib/expressir/express/builders/derived_attr_builder.rb +28 -0
  233. data/lib/expressir/express/builders/domain_rule_builder.rb +21 -0
  234. data/lib/expressir/express/builders/entity_decl_builder.rb +116 -0
  235. data/lib/expressir/express/builders/explicit_attr_builder.rb +49 -0
  236. data/lib/expressir/express/builders/expression_builder.rb +416 -0
  237. data/lib/expressir/express/builders/function_decl_builder.rb +86 -0
  238. data/lib/expressir/express/builders/helpers.rb +148 -0
  239. data/lib/expressir/express/builders/interface_builder.rb +151 -0
  240. data/lib/expressir/express/builders/inverse_attr_builder.rb +43 -0
  241. data/lib/expressir/express/builders/inverse_attr_type_builder.rb +30 -0
  242. data/lib/expressir/express/builders/inverse_clause_builder.rb +14 -0
  243. data/lib/expressir/express/builders/literal_builder.rb +93 -0
  244. data/lib/expressir/express/builders/procedure_decl_builder.rb +82 -0
  245. data/lib/expressir/express/builders/qualifier_builder.rb +102 -0
  246. data/lib/expressir/express/builders/reference_builder.rb +18 -0
  247. data/lib/expressir/express/builders/rule_decl_builder.rb +97 -0
  248. data/lib/expressir/express/builders/schema_body_decl_builder.rb +18 -0
  249. data/lib/expressir/express/builders/schema_decl_builder.rb +56 -0
  250. data/lib/expressir/express/builders/schema_version_builder.rb +34 -0
  251. data/lib/expressir/express/builders/simple_id_builder.rb +17 -0
  252. data/lib/expressir/express/builders/statement_builder.rb +243 -0
  253. data/lib/expressir/express/builders/subtype_constraint_builder.rb +166 -0
  254. data/lib/expressir/express/builders/syntax_builder.rb +30 -0
  255. data/lib/expressir/express/builders/type_builder.rb +238 -0
  256. data/lib/expressir/express/builders/type_decl_builder.rb +26 -0
  257. data/lib/expressir/express/builders/unique_clause_builder.rb +20 -0
  258. data/lib/expressir/express/builders/unique_rule_builder.rb +41 -0
  259. data/lib/expressir/express/builders/where_clause_builder.rb +20 -0
  260. data/lib/expressir/express/builders.rb +55 -0
  261. data/lib/expressir/express/cache.rb +43 -18
  262. data/lib/expressir/express/checker.rb +812 -0
  263. data/lib/expressir/express/concatenator.rb +79 -0
  264. data/lib/expressir/express/core.rb +95 -0
  265. data/lib/expressir/express/error.rb +124 -0
  266. data/lib/expressir/express/formatter.rb +89 -1548
  267. data/lib/expressir/express/formatters/data_types_formatter.rb +362 -0
  268. data/lib/expressir/express/formatters/declarations_formatter.rb +753 -0
  269. data/lib/expressir/express/formatters/expressions_formatter.rb +180 -0
  270. data/lib/expressir/express/formatters/literals_formatter.rb +55 -0
  271. data/lib/expressir/express/formatters/references_formatter.rb +53 -0
  272. data/lib/expressir/express/formatters/remark_formatter.rb +270 -0
  273. data/lib/expressir/express/formatters/remark_item_formatter.rb +28 -0
  274. data/lib/expressir/express/formatters/statements_formatter.rb +272 -0
  275. data/lib/expressir/express/formatters/supertype_expressions_formatter.rb +54 -0
  276. data/lib/expressir/express/formatters.rb +22 -0
  277. data/lib/expressir/express/grammar/parser.rb +712 -0
  278. data/lib/expressir/express/grammar.rb +11 -0
  279. data/lib/expressir/express/hyperlink_formatter.rb +4 -5
  280. data/lib/expressir/express/interface_dot.rb +105 -0
  281. data/lib/expressir/express/lazy_repository.rb +211 -0
  282. data/lib/expressir/express/line_map.rb +48 -0
  283. data/lib/expressir/express/listing.rb +174 -0
  284. data/lib/expressir/express/model_traversal.rb +42 -0
  285. data/lib/expressir/express/model_visitor.rb +11 -10
  286. data/lib/expressir/express/node_position_index.rb +342 -0
  287. data/lib/expressir/express/parallel_files.rb +229 -0
  288. data/lib/expressir/express/parser.rb +501 -62
  289. data/lib/expressir/express/pretty_formatter.rb +586 -0
  290. data/lib/expressir/express/pretty_gate.rb +61 -0
  291. data/lib/expressir/express/refs_overlay.rb +47 -0
  292. data/lib/expressir/express/remark_attacher.rb +1098 -0
  293. data/lib/expressir/express/remark_overlay.rb +86 -0
  294. data/lib/expressir/express/remark_scanner.rb +245 -0
  295. data/lib/expressir/express/resolve_references_model_visitor.rb +7 -9
  296. data/lib/expressir/express/schema_block_scanner.rb +137 -0
  297. data/lib/expressir/express/schema_head_formatter.rb +4 -5
  298. data/lib/expressir/express/schema_plain_source_formatter.rb +11 -0
  299. data/lib/expressir/express/schema_source_formatter.rb +15 -0
  300. data/lib/expressir/express/scope_resolver.rb +223 -0
  301. data/lib/expressir/express/self_schema_reference.rb +126 -0
  302. data/lib/expressir/express/shtolo.rb +679 -0
  303. data/lib/expressir/express/smrl_xml.rb +125 -0
  304. data/lib/expressir/express/source_formatter.rb +15 -0
  305. data/lib/expressir/express/streaming_builder.rb +435 -0
  306. data/lib/expressir/express/xsd.rb +259 -0
  307. data/lib/expressir/express.rb +53 -0
  308. data/lib/expressir/liquid.rb +1 -0
  309. data/lib/expressir/manifest/resolver.rb +210 -0
  310. data/lib/expressir/manifest/validator.rb +192 -0
  311. data/lib/expressir/manifest.rb +6 -0
  312. data/lib/expressir/mapping/refpath.rb +857 -0
  313. data/lib/expressir/mapping.rb +253 -0
  314. data/lib/expressir/model/cache.rb +10 -9
  315. data/lib/expressir/model/concerns.rb +42 -0
  316. data/lib/expressir/model/data_types/aggregate.rb +8 -13
  317. data/lib/expressir/model/data_types/array.rb +15 -21
  318. data/lib/expressir/model/data_types/bag.rb +11 -15
  319. data/lib/expressir/model/data_types/binary.rb +9 -12
  320. data/lib/expressir/model/data_types/boolean.rb +7 -2
  321. data/lib/expressir/model/data_types/enumeration.rb +11 -15
  322. data/lib/expressir/model/data_types/enumeration_item.rb +5 -7
  323. data/lib/expressir/model/data_types/generic.rb +6 -8
  324. data/lib/expressir/model/data_types/generic_entity.rb +11 -13
  325. data/lib/expressir/model/data_types/integer.rb +7 -2
  326. data/lib/expressir/model/data_types/list.rb +13 -18
  327. data/lib/expressir/model/data_types/logical.rb +7 -2
  328. data/lib/expressir/model/data_types/number.rb +7 -2
  329. data/lib/expressir/model/data_types/real.rb +7 -9
  330. data/lib/expressir/model/data_types/select.rb +13 -18
  331. data/lib/expressir/model/data_types/set.rb +11 -15
  332. data/lib/expressir/model/data_types/string.rb +9 -12
  333. data/lib/expressir/model/data_types.rb +25 -0
  334. data/lib/expressir/model/declarations/attribute.rb +19 -28
  335. data/lib/expressir/model/declarations/constant.rb +10 -16
  336. data/lib/expressir/model/declarations/derived_attribute.rb +25 -0
  337. data/lib/expressir/model/declarations/entity.rb +39 -30
  338. data/lib/expressir/model/declarations/function.rb +32 -40
  339. data/lib/expressir/model/declarations/informal_proposition_rule.rb +25 -0
  340. data/lib/expressir/model/declarations/interface.rb +25 -16
  341. data/lib/expressir/model/declarations/interface_item.rb +9 -11
  342. data/lib/expressir/model/declarations/interfaced_item.rb +15 -19
  343. data/lib/expressir/model/declarations/inverse_attribute.rb +25 -0
  344. data/lib/expressir/model/declarations/parameter.rb +10 -16
  345. data/lib/expressir/model/declarations/procedure.rb +30 -37
  346. data/lib/expressir/model/declarations/remark_item.rb +11 -11
  347. data/lib/expressir/model/declarations/rule.rb +38 -43
  348. data/lib/expressir/model/declarations/schema.rb +162 -64
  349. data/lib/expressir/model/declarations/schema_version.rb +8 -11
  350. data/lib/expressir/model/declarations/schema_version_item.rb +8 -11
  351. data/lib/expressir/model/declarations/subtype_constraint.rb +14 -22
  352. data/lib/expressir/model/declarations/type.rb +18 -18
  353. data/lib/expressir/model/declarations/unique_rule.rb +8 -13
  354. data/lib/expressir/model/declarations/variable.rb +10 -16
  355. data/lib/expressir/model/declarations/where_rule.rb +8 -13
  356. data/lib/expressir/model/declarations.rb +31 -0
  357. data/lib/expressir/model/dependency_resolver.rb +268 -0
  358. data/lib/expressir/model/exp_file.rb +42 -0
  359. data/lib/expressir/model/expressions/aggregate_initializer.rb +8 -9
  360. data/lib/expressir/model/expressions/aggregate_initializer_item.rb +10 -12
  361. data/lib/expressir/model/expressions/binary_expression.rb +42 -36
  362. data/lib/expressir/model/expressions/entity_constructor.rb +10 -12
  363. data/lib/expressir/model/expressions/function_call.rb +10 -12
  364. data/lib/expressir/model/expressions/interval.rb +18 -23
  365. data/lib/expressir/model/expressions/query_expression.rb +11 -15
  366. data/lib/expressir/model/expressions/unary_expression.rb +13 -15
  367. data/lib/expressir/model/expressions.rb +18 -0
  368. data/lib/expressir/model/identifier.rb +18 -26
  369. data/lib/expressir/model/indexes/entity_index.rb +103 -0
  370. data/lib/expressir/model/indexes/item_graph.rb +185 -0
  371. data/lib/expressir/model/indexes/reference_index.rb +148 -0
  372. data/lib/expressir/model/indexes/type_index.rb +149 -0
  373. data/lib/expressir/model/indexes.rb +12 -0
  374. data/lib/expressir/model/interface_validator.rb +384 -0
  375. data/lib/expressir/model/literals/binary.rb +7 -9
  376. data/lib/expressir/model/literals/integer.rb +7 -9
  377. data/lib/expressir/model/literals/logical.rb +10 -12
  378. data/lib/expressir/model/literals/real.rb +7 -9
  379. data/lib/expressir/model/literals/string.rb +10 -12
  380. data/lib/expressir/model/literals.rb +13 -0
  381. data/lib/expressir/model/model_element.rb +286 -117
  382. data/lib/expressir/model/references/attribute_reference.rb +9 -12
  383. data/lib/expressir/model/references/group_reference.rb +9 -12
  384. data/lib/expressir/model/references/index_reference.rb +11 -15
  385. data/lib/expressir/model/references/simple_reference.rb +11 -14
  386. data/lib/expressir/model/references.rb +12 -0
  387. data/lib/expressir/model/remark_format.rb +17 -0
  388. data/lib/expressir/model/remark_info.rb +92 -0
  389. data/lib/expressir/model/remark_placement.rb +42 -0
  390. data/lib/expressir/model/repository.rb +472 -12
  391. data/lib/expressir/model/repository_validator.rb +293 -0
  392. data/lib/expressir/model/search_engine.rb +551 -0
  393. data/lib/expressir/model/statements/alias.rb +12 -16
  394. data/lib/expressir/model/statements/assignment.rb +12 -11
  395. data/lib/expressir/model/statements/case.rb +31 -14
  396. data/lib/expressir/model/statements/case_action.rb +9 -11
  397. data/lib/expressir/model/statements/compound.rb +10 -8
  398. data/lib/expressir/model/statements/escape.rb +10 -2
  399. data/lib/expressir/model/statements/if.rb +15 -14
  400. data/lib/expressir/model/statements/null.rb +10 -2
  401. data/lib/expressir/model/statements/procedure_call.rb +11 -11
  402. data/lib/expressir/model/statements/repeat.rb +21 -28
  403. data/lib/expressir/model/statements/return.rb +9 -8
  404. data/lib/expressir/model/statements/skip.rb +10 -2
  405. data/lib/expressir/model/statements.rb +20 -0
  406. data/lib/expressir/model/supertype_expressions/binary_supertype_expression.rb +13 -17
  407. data/lib/expressir/model/supertype_expressions/oneof_supertype_expression.rb +7 -9
  408. data/lib/expressir/model/supertype_expressions.rb +12 -0
  409. data/lib/expressir/model.rb +38 -78
  410. data/lib/expressir/package/builder.rb +231 -0
  411. data/lib/expressir/package/metadata.rb +79 -0
  412. data/lib/expressir/package/reader.rb +149 -0
  413. data/lib/expressir/package.rb +8 -0
  414. data/lib/expressir/schema_manifest.rb +159 -0
  415. data/lib/expressir/schema_manifest_entry.rb +15 -0
  416. data/lib/expressir/version.rb +6 -1
  417. data/lib/expressir.rb +79 -9
  418. data/lib/tasks/verify_remarks.rake +16 -0
  419. data/rakelib/native.rake +24 -0
  420. data/rakelib/oracle.rake +52 -0
  421. data/rakelib/verify_remarks.rake +16 -0
  422. metadata +431 -129
  423. data/.cross_rubies +0 -20
  424. data/.gitmodules +0 -6
  425. data/.yardopts +0 -11
  426. data/docs/development.md +0 -90
  427. data/exe/format +0 -18
  428. data/exe/generate-parser +0 -51
  429. data/lib/expressir/cli/ui.rb +0 -36
  430. data/lib/expressir/express/2.7/express_parser.so +0 -0
  431. data/lib/expressir/express/3.0/express_parser.so +0 -0
  432. data/lib/expressir/express/3.1/express_parser.so +0 -0
  433. data/lib/expressir/express/3.2/express_parser.so +0 -0
  434. data/lib/expressir/express/visitor.rb +0 -2588
  435. data/lib/expressir/model/data_type.rb +0 -9
  436. data/lib/expressir/model/declaration.rb +0 -9
  437. data/lib/expressir/model/expression.rb +0 -9
  438. data/lib/expressir/model/literal.rb +0 -9
  439. data/lib/expressir/model/reference.rb +0 -9
  440. data/lib/expressir/model/statement.rb +0 -9
  441. data/lib/expressir/model/supertype_expression.rb +0 -9
  442. data/rakelib/antlr4-native.rake +0 -161
  443. data/rakelib/cross-ruby.rake +0 -368
  444. data/spec/acceptance/version_spec.rb +0 -27
  445. data/spec/expressir/express/cache_spec.rb +0 -89
  446. data/spec/expressir/express/formatter_spec.rb +0 -173
  447. data/spec/expressir/express/parser_spec.rb +0 -141
  448. data/spec/expressir/model/model_element_spec.rb +0 -318
  449. data/spec/spec_helper.rb +0 -24
  450. data/spec/support/console_helper.rb +0 -29
  451. data/spec/syntax/multiple.exp +0 -23
  452. data/spec/syntax/multiple.yaml +0 -198
  453. data/spec/syntax/multiple_formatted.exp +0 -71
  454. data/spec/syntax/multiple_hyperlink_formatted.exp +0 -71
  455. data/spec/syntax/multiple_schema_head_hyperlink_formatted.exp +0 -13
  456. data/spec/syntax/remark.exp +0 -193
  457. data/spec/syntax/remark.yaml +0 -471
  458. data/spec/syntax/remark_formatted.exp +0 -228
  459. data/spec/syntax/single.exp +0 -4
  460. data/spec/syntax/single.yaml +0 -18
  461. data/spec/syntax/single_formatted.exp +0 -10
  462. data/spec/syntax/single_formatted.yaml +0 -36
  463. data/spec/syntax/syntax.exp +0 -333
  464. data/spec/syntax/syntax.yaml +0 -3509
  465. data/spec/syntax/syntax_formatted.exp +0 -902
  466. data/spec/syntax/syntax_hyperlink_formatted.exp +0 -902
  467. data/spec/syntax/syntax_schema_head_formatted.exp +0 -18
@@ -0,0 +1,22 @@
1
+ # 09 — SMRL index XML + declaration listing (wo-list, wo-smrl-xml)
2
+
3
+ **Status: NOT STARTED**
4
+
5
+ ## Goal
6
+
7
+ The SRL publication byproducts eeng generates: declaration listings
8
+ (`-list.txt`) and the SMRL index XML for longform/concatenated files
9
+ (`<concatenated_express_file_content_list>`), emitted from the same
10
+ model that powers 04/05.
11
+
12
+ ## Sources
13
+
14
+ `plugins/p11/wo-list.lisp`, `wo-smrl-xml.lisp` (element names:
15
+ concatenated_express_file_content_list, per-schema content entries),
16
+ top-level.lisp:748-793 (when the index is generated: XML output +
17
+ longform/concatenated modes).
18
+
19
+ ## Acceptance
20
+
21
+ Index/list output for the 03 golden set structurally identical
22
+ (element-for-element) to eeng's; wired into the same CLI as 04.
@@ -0,0 +1,21 @@
1
+ # 10 — Interface dependency dot graph (kernel/dot-graph.lisp)
2
+
3
+ **Status: NOT STARTED**
4
+
5
+ ## Goal
6
+
7
+ Emit GraphViz dot for schema interface dependencies exactly like eeng
8
+ (qualify.sh "Generate Dot File (interface)"), from our compiled
9
+ reference graph (already computes interface edges + closures over the
10
+ full SMRL).
11
+
12
+ ## Notes
13
+
14
+ eeng's dot generation honors the concatenated mode (root-schema
15
+ subgraph); replicate node/edge/styling choices so WG12 visual
16
+ diffing stays stable. Trivial once 02 documents their exact output
17
+ shape — deliberately last in the queue.
18
+
19
+ ## Acceptance
20
+
21
+ Byte-comparable dot output vs eeng for arm/mim of the golden modules.
@@ -0,0 +1,79 @@
1
+ # 11 — Import the Express Engine test suite
2
+
3
+ **Status: NOT STARTED**
4
+
5
+ ## Goal
6
+
7
+ Fully import eeng's tests — their release-suite scripts, the bug
8
+ corpus, and the reference documents that define pass/fail — as
9
+ executable RSpec feature specs and regression fixtures in expressir.
10
+ Nothing in their suite should remain unrepresented.
11
+
12
+ ## Inventory (verified 2026-09-21)
13
+
14
+ eeng has no formal test framework (`tests/` is a README describing an
15
+ unrealized Makefile suite); the suite is:
16
+
17
+ ### CLI drivers — `scripts/*.sh` (14)
18
+
19
+ | script | drives | our target operation |
20
+ |---|---|---|
21
+ | `qual-arm.sh`, `qual-mod.sh` | per-module qualification loop (arm/mim × every qualify.sh op) | 04, 05, 07-10 end-to-end specs |
22
+ | `qual-res.sh` | same over `qualify-res.txt` resources | idem, resource side |
23
+ | `pretty-test.sh` | `--pretty` ap209/ap210/ap242 × {arm,mim} × {shortform,longform} | 08 formatter specs |
24
+ | `ap210.sh` | the 2.5MB long-form giant, pretty × 4 modes | 08 scale spec |
25
+ | `test-tt.sh`, `test-tt1.sh` | `--xml` arm/mim shortform + Saxon HTM conversion + regenerate-exp compare | 09 (XML chain) |
26
+ | `build-xml.sh` | XML builds over the whole `modules.lst` + `resources.lst` | 09 corpus spec |
27
+ | `test.sh` | ad-hoc resource XML/HTM driver (kinematic schema) | 09 |
28
+ | `scan-res.lisp` | resource corpus scan | 07 check corpus |
29
+ | `kevin.sh`, `veronique.sh`, `copy-mod-xml.sh` | person-named scratch drivers | triage: keep as scenarios or drop with rationale |
30
+
31
+ Each script's eeng CLI matrix (`-mode {arm,mim}_{shortform,longform,
32
+ concatenated}` `-stepmod` `-schema` `--pretty/--xml/--flat/--dot
33
+ --check`) maps 1:1 onto our planned operations — the flag matrix
34
+ becomes our spec matrix.
35
+
36
+ ### Bug corpus — `docs/issues/`
37
+
38
+ - `bug39-Greedy-Parens.txt` — **shtolo fixture**: contains "results
39
+ of old shtolo and stepcode exppp" vs "results of new shtolo"
40
+ outputs; doubles as a differential vs stepcode exppp.
41
+ - `bug40-Pretty-Expressions.txt` — pretty-printer expression cases.
42
+ Both become regression specs with the literal inputs embedded.
43
+
44
+ ### Reference documents — `docs/` (define pass/fail)
45
+
46
+ - `shtolo_converting_STEP_short_listings_to_annotated_listings_libes.pdf`
47
+ — the SHTOLO paper (short listings → annotated listings); required
48
+ reading for 01/05 semantics.
49
+ - BNF grammars: `iso-10303-11--2004{,-raw}.bnf`, `iso-10303-14*.bnf`,
50
+ **`iso-10303-21--{1994,2002}.bnf`** (Part 21!), `xml.bnf`,
51
+ `patch_schema.bnf` — grammar cross-checks (05's 1994 profile; the
52
+ Part 21 BNFs feed the future p21 parser).
53
+ - `express_model.dtd`, `module.dtd` — the DTDs their XML emits (09's
54
+ structural contract).
55
+ - `qualify.txt`, `interfaces.txt`, `interface-notes.pdf` —
56
+ qualification + interface semantics notes.
57
+ - `00check.txt` (repo root) — check.lisp design notes → 07.
58
+
59
+ ## Steps
60
+
61
+ 1. Copy the durable assets into the repo: scripts verbatim under
62
+ `spec/fixtures/eeng/scripts/` (provenance header), bug corpus under
63
+ `spec/fixtures/eeng/issues/`, DTDs + BNFs under
64
+ `spec/fixtures/eeng/grammar/` (license is MIT-style per eeng
65
+ headers — verify and record attribution).
66
+ 2. Translate the CLI matrix into RSpec feature specs
67
+ (`spec/expressir/eeng_parity/`): one describe per driver family,
68
+ one example per (schema × mode × operation), corpus-scaled ones
69
+ tagged `smrl`.
70
+ 3. Port bug39/bug40 as literal regression specs now (they need only
71
+ parser + formatter, both of which exist).
72
+ 4. Extend 03's oracle harness to emit/refresh the expected outputs
73
+ these specs consume.
74
+
75
+ ## Acceptance
76
+
77
+ Every driver row and both bugs have a corresponding spec (or a
78
+ documented drop rationale); `smrl`-tagged parity suite green against
79
+ the 03 golden corpus.
@@ -0,0 +1,67 @@
1
+ # 12 — Full check-p11 catalog port (eeng `kernel/check.lisp`)
2
+
3
+ ## Goal
4
+
5
+ Bring `Expressir::Express::Checker` from the current documented subset
6
+ to the full eeng check-p11 catalog: every note eeng 5.1.0 emits over
7
+ the SRL/SMRL corpora, with note ids and severities aligned so oracle
8
+ differentials stay cross-referenceable.
9
+
10
+ ## Current subset (shipped)
11
+
12
+ check-schema-interface-redundant, check-schema-iface-resource-duplicate,
13
+ check-subtype-ref, check-subtype-cycle (path-tracked DFS),
14
+ check-duplicate-declaration, check-where-name-pattern,
15
+ check-unique-name-pattern, check-unresolved-ref,
16
+ check-select-extended-type, check-enumeration-extended-type,
17
+ check-self-schema-reference (#125), parse-failure → nonzero exit.
18
+
19
+ ## Sources
20
+
21
+ - `~/src/external/exp-engine-engine/kernel/check.lisp` (835 lines,
22
+ `check-p11` generic + per-node-type methods) and
23
+ `kernel/check-notes.lisp` (note catalog).
24
+ - ISO 10303-11:2004 clause text — `~/src/mn/iso-10303-11/sources/sections/`
25
+ (each ported check cites its clause).
26
+ - Existing plan: `07-semantic-checks-port.md`; current impl
27
+ `lib/expressir/express/checker.rb`.
28
+
29
+ ## Design
30
+
31
+ 1. **Catalog extraction**: enumerate every `post-message` site in
32
+ check.lisp/check-notes.lisp → (note-id, severity, node type,
33
+ message shape). Commit as `spec/fixtures/eeng/check-catalog.md`
34
+ with our coverage column; CI gate diffs the catalog against the
35
+ oracle so new eeng notes surface.
36
+ 2. **Port order** (by eeng's own walk): schema-level (interfaces,
37
+ constants) → entity (supertype/subtype-of, attributes, derives,
38
+ inverses, uniques, wheres) → type (select/enumeration/aggregate
39
+ bounds) → function/procedure/rule bodies (expression typing lite:
40
+ arity/name resolution of called functions, built-ins from BUILTINS,
41
+ variable scoping via ScopeResolver) → attribute-level
42
+ redeclaration consistency (redispatches, `SELF\` qualifiers).
43
+ 3. **Differential gate**: run eeng `--validate` and our `validate
44
+ check` over the SRL corpus + module arm/mim set; every corpus file
45
+ must produce the SAME note-id multiset (message text may differ).
46
+ Land as `checker_eeng_full_catalog_differential_spec`
47
+ (production_scale-tagged for the big corpus; small fixtures in the
48
+ default suite).
49
+ 4. **Severity mapping** table: eeng `:error/:warning` → ours verbatim.
50
+
51
+ ## Acceptance
52
+
53
+ - Catalog file lists 100% of eeng note ids with status columns.
54
+ - SRL differential: zero unexplained note-id mismatches.
55
+ - `validate check` docs updated (README table).
56
+
57
+ ## Test cases
58
+
59
+ - eeng `docs/issues/` corpus inputs; qualify loops
60
+ (`scripts/qual-*.sh`) over resources.lst/modules.lst.
61
+ - Per-clause unit fixtures citing ISO 10303-11 clause numbers.
62
+
63
+ ## Release checkpoint
64
+
65
+ PR + expressir patch release once the entity/type/schema tiers land
66
+ (expression-tier notes may follow in a second PR — keep the catalog
67
+ honest about which tier each note belongs to).
@@ -0,0 +1,54 @@
1
+ # 13 — Population / Part 21 instance validation
2
+
3
+ ## Goal
4
+
5
+ Validate STEP Part 21 instance data (.stp/.p21) against a loaded
6
+ schema — eeng's core strength (`--validate` with population), absent
7
+ from expressir today.
8
+
9
+ ## Sources
10
+
11
+ - ISO 10303-21: exchange structure grammar — BNF in
12
+ `~/src/mn/iso-10303-detached-docs/sources/iso-10303-21*/` (and
13
+ eeng's copy under `plugins/p21/`, incl. its parser + validation).
14
+ - eeng: `plugins/p21/` (p21 reader, population), `--validate`
15
+ (`run-op` population path), `wo-xml.lisp` instance-side references.
16
+ - eeng banner: population validation "not fully supported" upstream —
17
+ our target is the supported subset, documented precisely.
18
+
19
+ ## Design (scope ladder; each step ships)
20
+
21
+ 1. **P21 reader**: parsanol grammar for the exchange structure
22
+ (header entities + DATA section: complex entity `(`name(...)`)`,
23
+ entity instances, list/set values, typed refs `#1`, enums `.T.`,
24
+ binary `\"..."`, redefines `/`). Emit a Population model
25
+ (`Model::Population`): instances with entity name + attribute
26
+ values keyed by explicit attribute position.
27
+ 2. **Structural validation**: every instance maps to a known entity
28
+ in the schema closure (complex entities → all parts); attribute
29
+ arity matches; value kinds match declared types (aggregate bounds,
30
+ select members, string/binary widths, REAL precision) — the
31
+ aggregate of eeng's population notes.
32
+ 3. **Constraint validation**: UNIQUE rules (per-entity + inherited),
33
+ WHERE rules — evaluation needs an EXPRESS expression interpreter;
34
+ reuse/extend the `benchmark`/expression-walk machinery, evaluate
35
+ over instance bindings (scope: no INVERSE/derives first).
36
+ 4. **CLI**: `expressir validate population SCHEMA -p DATA.p21`,
37
+ JSON/plain reports aligned with Checker notes.
38
+
39
+ ## Acceptance
40
+
41
+ - Official Part 21 test corpora parse + validate (see tests).
42
+ - Differential vs eeng `--validate` on shared fixtures: same
43
+ accept/reject decision and note ids.
44
+
45
+ ## Test cases
46
+
47
+ - ISO 10303-21 validation suite files from detached-docs where
48
+ available; else eeng `tests/` + `scripts/test-tt*.sh` fixtures.
49
+ - Hand fixtures: arity mismatch, bad enum, UNIQUE collision, WHERE
50
+ failure, complex-entity mismatch.
51
+
52
+ ## Release checkpoint
53
+
54
+ Steps 1–2 = first release (structural validator); step 3 second.
@@ -0,0 +1,44 @@
1
+ # 14 — SHTOLO completion (expressir#32, Annex G)
2
+
3
+ ## Goal
4
+
5
+ Finish `Expressir::Express::Shtolo` to full Annex G conformance and
6
+ close the eeng differential over the whole corpus. Current state:
7
+ G.1 + G.2 stages implemented (522 lines), suite green (10 examples),
8
+ `extenders:` :all/:none matches the eeng-default split.
9
+
10
+ ## Remaining work
11
+
12
+ 1. **Rule audit**: walk `spec/fixtures/shtolo/annex-g-rules.md`
13
+ (G.NM.1-2, G.1.1-10, G.2.1-7) and mark every rule PASS/partial/✖
14
+ in the file; a fixture per rule. Known-thin areas from the audit:
15
+ - G.1.6 dependently-instantiable rule output (function + WHERE
16
+ wiring) — verify against Annex G text, not eeng (eeng ✖).
17
+ - name-munging policy (`_dot_` and friends) — eeng ✖; standard wins.
18
+ - prune pass reachability (unreachable subtype branches).
19
+ 2. **Corpus differential**: for every module arm/mim in modules.lst,
20
+ compare our flatten output to `eengine --flat` (eeng default =
21
+ `extenders: :none` shape); classify divergences
22
+ standard-conformant vs bug. Feed the mismatch corpus into specs.
23
+ 3. **Scale**: SMRL full set flatten over the compiled-set warm model;
24
+ memory ceiling documented.
25
+ 4. **CLI**: `expressir shtolo MANIFEST -o longform.exp` (root schema
26
+ + closure resolution reuses ParityInputs).
27
+
28
+ ## Acceptance
29
+
30
+ - Every Annex G rule has a green fixture.
31
+ - Corpus differential: zero unexplained diffs; every explained
32
+ divergence documented in `parity-matrix.md`.
33
+ - SMRL flatten completes; output parses under the 1994 profile.
34
+
35
+ ## Test cases
36
+
37
+ - annex-g-rules.md fixtures; eeng `docs/issues/bug39` shtolo outputs
38
+ (old-vs-new vs stepcode exppp); `scripts/qual-*.sh` loops;
39
+ `pretty-test.sh` ap209/210/242 × arm/mim shape checks.
40
+
41
+ ## Release checkpoint
42
+
43
+ One PR per coherent slice (rule audit + fixtures; differential gate;
44
+ CLI), patch releases as each merges.
@@ -0,0 +1,51 @@
1
+ # 15 — EXPRESS → XML (ISO 10303-28 / eeng --xml)
2
+
3
+ ## Goal
4
+
5
+ Generate XML Schema output for a schema (expressir#276): eeng's
6
+ `--xml` (`wo-xml-p11.lisp`) plus the SMRL index listing
7
+ (`wo-smrl-xml.lisp`, feeds the SMRL document set).
8
+
9
+ ## Sources
10
+
11
+ - ISO 10303-28:2007 ( EXPRESS-driven XML representation) — detached
12
+ docs `~/src/mn/iso-10303-detached-docs/sources/iso-10303-28*/`;
13
+ eeng `docs/` carries `express_model.dtd` + module.dtd (legacy
14
+ forms).
15
+ - eeng: `plugins/p11/wo-xml-p11.lisp` (schema → XSD),
16
+ `wo-smrl-xml.lisp` (SMRL index XML), `--xml` run-op.
17
+ - Consumer: iso-10303 repo's `schemas/data/*.xml` SMRL pages
18
+ (rendered by metanorma today) — our output must be drop-in for
19
+ that pipeline.
20
+
21
+ ## Design
22
+
23
+ 1. **SMRL index XML first** (small, high value for iso-10303): the
24
+ index of all schemas in a repository (name, file, interfaces,
25
+ counts) as the repo's data-layer source — replaces the
26
+ hand-maintained entries.
27
+ 2. **express2xsd**: schema → XSD via a declarative mapping table
28
+ (entity → element+complexType, type → simpleType/complexType,
29
+ select → choice, explicit attrs in order, supertype substitution
30
+ groups for SUBTYPE OF, aggregates → minOccurs/maxOccurs). Emit
31
+ through a Nokogiri-free XML writer (ROXML via lutaml-model where
32
+ it fits; hand-rolled tree otherwise — the mapping is the product,
33
+ not the serialization).
34
+ 3. **CLI**: `expressir xml SCHEMA -o schema.xsd`;
35
+ `expressir xml-index REPO -o smrl.xml`.
36
+
37
+ ## Acceptance
38
+
39
+ - Output validates against the Part 28 reference XSDs (detached
40
+ docs) for the fixture set; xsd diff vs eeng --xml on SRL corpus
41
+ classified standard-vs-eeng.
42
+ - iso-10303 SMRL pages render from our generated index unchanged.
43
+
44
+ ## Test cases
45
+
46
+ - eeng `scripts/build-xml.sh` full-corpus reference outputs;
47
+ `test-tt{,1}.sh` (XML + Saxon HTM chain) as end-to-end smoke.
48
+
49
+ ## Release checkpoint
50
+
51
+ SMRL index first (unblocks iso-10303 pipeline), express2xsd second.
@@ -0,0 +1,48 @@
1
+ # 16 — Schema compare & patch (eeng --compare / --patch)
2
+
3
+ ## Goal
4
+
5
+ Structural diff of two schemata with a machine-applicable patch
6
+ output — eeng `--compare` (+ `--concat_compare`, `--patch`,
7
+ `--patch_schema_name`), used by WG12 to track schema revisions.
8
+
9
+ ## Sources
10
+
11
+ - eeng `plugins/p11/wo-*.lisp` compare entry (`run-op (:compare)`
12
+ in top-level.lisp) and patch application paths; mode args:
13
+ `:trial_schema :reference_schema :xml-output :patch-output`.
14
+ - Semantics baseline: declaration-level matching by id within the
15
+ flattened/concatenated form (eeng compares post-concat shapes for
16
+ shortforms).
17
+
18
+ ## Design
19
+
20
+ 1. **Model diff** on the resolved repositories: declarations matched
21
+ by canonical path (`schema.item`); kinds of delta — added,
22
+ removed, changed (per-attribute structural diff: types,
23
+ attribute lists, where/unique bodies via formatted-text
24
+ comparison first, structural second).
25
+ 2. **Outputs**: unified-ish text report (human), XML
26
+ (`--xml-output` parity), and patch script (the eeng patch
27
+ format: rename/redefine vocabulary it defines — ported from
28
+ wo-patch output shapes).
29
+ 3. **CLI**: `expressir compare TRIAL.exp REFERENCE.exp [--xml|--patch]`;
30
+ `concat-compare` rides the Concatenator for shortform pairs.
31
+ 4. **Patch application**: apply a patch file to a model (rename
32
+ declarations, remap interfaces) — the write-back side eeng uses
33
+ for round-trip patching.
34
+
35
+ ## Acceptance
36
+
37
+ - SRL corpus self-compare = empty diff; mutated fixtures (rename,
38
+ add/remove attr, retype) produce the expected delta kinds.
39
+ - Differential vs eeng `--compare` XML on the same pairs.
40
+
41
+ ## Test cases
42
+
43
+ - eeng `docs/issues/` pairs; hand-mutated fixture ladder; the
44
+ `--concat_compare` module-vs-module pairs from scripts.
45
+
46
+ ## Release checkpoint
47
+
48
+ Text+XML diff first release; patch apply second.
@@ -0,0 +1,42 @@
1
+ # 17 — Official test-case incorporation
2
+
3
+ ## Goal
4
+
5
+ One place that maps every OFFICIAL/reference test corpus to our
6
+ specs, and CI gates that run the tractable tier on every push.
7
+
8
+ ## Corpora
9
+
10
+ 1. **eeng suite** (already imported, `spec/fixtures/eeng/`):
11
+ `scripts/*.sh` 14 drivers (qual-arm/mod/res loops, pretty-test.sh
12
+ ap209/210/242 × arm/mim × shortform/longform, ap210 giant file,
13
+ test-tt{,1}.sh XML chain, build-xml.sh), `docs/issues/` bug
14
+ corpus (bug39 shtolo old-vs-new vs stepcode exppp; bug40 pretty
15
+ expressions), `qualify.txt`, `interfaces.txt`, `00check.txt`.
16
+ 2. **ISO 10303-11** grammar/conformance sources:
17
+ `~/src/mn/iso-10303-11/sources/sections/` — Annex G rules
18
+ (fixtures live in `spec/fixtures/shtolo/annex-g-rules.md`),
19
+ syntax annexes for negative/positive grammar tests.
20
+ 3. **ISO detached docs** `~/src/mn/iso-10303-detached-docs/sources/
21
+ iso-10303-*`: Part 21 exchange-structure fixtures (population),
22
+ Part 28 reference XSDs (XML), Part 22 (SDAI vocabulary, later).
23
+ 4. **SRL/SMRL corpora** (`resources.lst`/`modules.lst` names match
24
+ our checkout): the differential tier.
25
+
26
+ ## Structure
27
+
28
+ - `spec/expressir/eeng_parity/` — one spec per capability matrix
29
+ row; each example tags its source corpus file in a comment.
30
+ - Tiering: `:fast` (fixtures, every push) / `:corpus`
31
+ (SRL differential, nightly or manual tag `production_scale`) /
32
+ `:smoke_eeng` (runs the local eeng oracle when the binary exists —
33
+ skipped otherwise).
34
+ - `rake parity:status` prints the capability × corpus matrix with
35
+ green/gray cells — the single progress view.
36
+
37
+ ## Rules
38
+
39
+ - No test is added without its provenance row in the fixture dir's
40
+ PROVENANCE.md (established practice, keep it).
41
+ - Oracle outputs are regenerated, never hand-edited; the regeneration
42
+ command lives next to each golden.
@@ -0,0 +1,47 @@
1
+ # 18 — EXPRESS MIM mapping parser (expressir#88)
2
+
3
+ ## Goal
4
+
5
+ Parse and validate MIM "reference path" mappings — the application ↔
6
+ interpretation mapping every STEP module carries — into a typed model,
7
+ with the mapping paths validated against the ARM/MIM schemas.
8
+
9
+ ## Sources
10
+
11
+ - This checkout's canonical form: `schemas/modules/*/mapping.yaml` —
12
+ structured ae/aimelt/aa records whose values use the
13
+ `<<express:SCHEMA.ITEM,ITEM>>` link markup (the same markup
14
+ `Suma::LinkValidation` validates today). Generated upstream from
15
+ module.xml.
16
+ - Upstream SMRL module.xml: `mapping`/`ae`/`aimelt` elements with
17
+ `reference_path` bodies — the ISO 10303-1x mapping methodology
18
+ grammar: entity chains `A -> B -> C`, qualifiers `[attr => val]`,
19
+ `=>` derived references, `->>` subtype traversal, `<>` select
20
+ member. Port the grammar from the Part 1x mapping annex text
21
+ (detached docs) + existing module corpus as the oracle.
22
+ - eeng has no counterpart — this is an expressir-first capability.
23
+
24
+ ## Design
25
+
26
+ 1. `Expressir::Model::Mapping` — MappingDocument →
27
+ ApplicationElements (entity, aimelt, attributes) and
28
+ ReferencePath nodes (path-expression AST: steps, qualifiers,
29
+ subtype traversals).
30
+ 2. `mapping.yaml` reader (lutaml-model typed) — yaml → MappingDocument.
31
+ 3. Reference-path expression parser — parsanol grammar over the path
32
+ syntax; AST reuses Model::References where shapes coincide.
33
+ 4. Validation: every path resolves against the resolved ARM+MIM
34
+ repository (rides SchemaIndex / ItemGraph); unknown step = error
35
+ note (Checker id `check-mapping-path`).
36
+ 5. CLI: `expressir mapping validate DIR|FILE` over module dirs.
37
+
38
+ ## Acceptance
39
+
40
+ - All module mapping.yaml files in the SMRL parse; path validation
41
+ reports zero unknown steps (or a classified, explained residue).
42
+ - Round-trip: MappingDocument → yaml byte-stable.
43
+
44
+ ## Test cases
45
+
46
+ - module corpus (603 modules); hand fixtures for each path grammar
47
+ production; invalid-path fixtures.
@@ -0,0 +1,97 @@
1
+ # 19 — End-state architecture: Expressir in Rust, bound to metanorma via lutaml-model
2
+
3
+ **Question (2026-09-23): should Expressir ultimately be in Rust, bound to
4
+ metanorma via lutaml-model?**
5
+
6
+ **Answer: yes as a direction, no as a rewrite.** The end state is
7
+ *Rust engine, Ruby presentation* — reached in three milestones, not by
8
+ porting everything at once. The binding to metanorma goes through
9
+ lutaml-model's Liquid Drop layer, which is exactly the seam that
10
+ already works (expressir#255, #319, #421, #426).
11
+
12
+ ## Current state (v2.4.9, measured)
13
+
14
+ ```
15
+ .exp ──► parsanol-rs 0.8.8 (parse, wire tree)
16
+ └─► expressir_core ext (batch, EXSCS1 artifact, warm start)
17
+ └─► instantiate → Expressir::Model (lutaml-model, Ruby)
18
+ └─► Checker / Shtolo / formatters (Ruby semantics)
19
+ └─► Liquid Drops ──► metanorma / suma
20
+ ```
21
+
22
+ - Parse is Rust and fast (SRL cold 36 s CPU, SMRL 558 s CPU / 1448
23
+ schemas); parsanol 0.8.8's selective memoization is adopted.
24
+ - The measured floor is now the **wire→Ruby-object hydration**
25
+ (71 % of warm CPU) and the overlays (29 %). Rendering (metanorma)
26
+ touches a small fraction of the model.
27
+ - Semantics (Checker subset, SHTOLO with the 50-module eengine
28
+ differential, PrettyFormatter, SMRL XML) live in Ruby, eeng-parity
29
+ validated, with Ruby CI harnesses.
30
+
31
+ ## Options considered
32
+
33
+ **A — Status quo+.** Rust keeps parse+artifact; all new semantics
34
+ (Checker catalog, mapping validation) land in Ruby. No rewrite risk,
35
+ parity harnesses stay, but the hydration floor remains and the Rust
36
+ surface stays private.
37
+
38
+ **B — Lazy binding (the recommended milestone).** Keep the Ruby model
39
+ as the canonical object model for feature work, and add a READ path
40
+ that serves metanorma directly from Rust-held compiled-set data:
41
+ expressir_core exposes accessors over the loaded set (schema by id,
42
+ kind listings, field reads), and a lutaml-model Drop proxies them on
43
+ demand — rendering never instantiates the full object graph.
44
+ lutaml-model keeps ownership of mappings/serialization/Drop
45
+ integration; the Drop becomes the literal "Expressir in Rust bound to
46
+ metanorma via lutaml-model".
47
+
48
+ **C — Full Rust Expressir.** Model + semantics in Rust; Ruby keeps
49
+ only Drops and CLI glue; publish expressir-rs on crates.io; unlock
50
+ non-Ruby consumers (standalone binary, WASM). Rejected for now: it
51
+ re-implements eeng-validated semantics with parity risk, doubles CI,
52
+ and metanorma still needs a Drop/FFI layer — the boundary cost moves,
53
+ it does not disappear.
54
+
55
+ **Decision: B now; C deferred behind explicit triggers**
56
+ (a non-Ruby consumer materializes; the Ruby semantic layers become a
57
+ measured bottleneck — they are not today; or crate publishing is
58
+ wanted for supply-chain reasons independent of consumers).
59
+
60
+ ## Milestones
61
+
62
+ ### M1 — rev-pinned core dependency (supply chain, one line)
63
+ The ext pins `expressir-rs = { git …, branch = "main" }`; a released
64
+ gem can pick up a shifted core. Pin `rev = "<sha>"` per expressir
65
+ release (2.4.8 ↔ today's expressir-rs main), bump per release.
66
+
67
+ ### M2 — lazy Drops for the metanorma hot path
68
+ 1. expressir-rs: read accessors over the loaded compiled set
69
+ (`schema(id)`, kind listings, per-declaration field reads,
70
+ remark/refs overlay application in place).
71
+ 2. expressir_core: magnus wrappers returning primitives.
72
+ 3. Ruby: `LazySchemaDrop` (lutaml-model #319 custom Drop, the same
73
+ seam as #255/#426) implementing id / declaration listings / source
74
+ faces via the accessors.
75
+ 4. Gates: suma-style schema-doc pages render output-comparable with
76
+ the eager path on a fixture corpus; warm page-render benchmark;
77
+ SMRL warm-start CPU target ≤ 30 % of today's 128 s for the
78
+ read-only path.
79
+
80
+ ### M3 — semantics placement rule (standing)
81
+ New semantic features land in Ruby **unless** they are
82
+ parse-adjacent or graph-scale (then Rust in expressir-rs). The
83
+ Checker catalog (12) and mapping validation (18) stay Ruby; SHTOLO
84
+ stays Ruby with its eengine differential as the gate.
85
+
86
+ ### M4 — trigger review for option C
87
+ Reassess on any of: a non-Ruby consumer request; hydrate/overlay CPU
88
+ re-emerging as the top cost after M2; a crates.io publication ask.
89
+ M2's accessors are exactly the seed of C's public API — nothing in B
90
+ is throwaway.
91
+
92
+ ## What lutaml-model owns in every option
93
+
94
+ The mapping DSL (yaml/json faces — incl. the #88 `mapping.yaml`
95
+ reader), Liquid integration (#319 Drops), artifact serialization —
96
+ the binding contract between the engine and metanorma. Rust never
97
+ duplicates it.