expressir 1.2.11-aarch64-linux → 2.4.29-aarch64-linux

data/TODO.parity-ee/parity-matrix.md

@@ -0,0 +1,108 @@
+# eeng ↔ expressir parity matrix (2026-09-21)
+
+Reference: `~/src/external/exp-engine-engine` (eengine-5.0.20-Beta1).
+Audited files: kernel/{find-declaration,find-entity,find-type,
+resolve-interface,resolve-object,resolve-ref,references,schema-search,
+interfaces,compute-inheritance,supertypes,derive-attrs,
+inverse-attrs,check,compare,dot-graph}.lisp; plugins/p11/{schedule-*
+,resolve-declaration,cleanup-interface,wo-*}.lisp.
+
+## 1. Name resolution ladder (kernel/find-declaration.lisp:125-172)
+
+eeng's schema-scope restriction ladder for `find-declaration`:
+
+```
+nil        = :use-from
+:local     → own declarations + constants only
+:use-from  → :local OR :use-only
+:use-only  → walk USE FROM interfaces only, transitively, each in
+             :use-from scope (so USE visibility is transitive)
+:all-iface → :local OR all interfaces (USE and REFERENCE), transitively
+cycle guard → *schema-path*: schema already on path ⇒ nil
+```
+
+Per-kind restrictions (`resolve-interface.lisp:30-66`):
+- USE FROM resolves **type + entity only**.
+- REFERENCE FROM resolves constant/entity/function/procedure/type.
+- Unresolved resource ⇒ hard **error** `"Resource '~A' not found"`.
+
+**expressir today:** `ModelElement#find` climbs parent scopes through
+`children_by_id`; `Schema#children` = `interfaced_items + safe_children`
+where `interfaced_items` is **single-hop** (own interfaces only, no
+transitivity), and the resolver never errors on unresolved resources.
+Our `find` also treats REFERENCE and USE identically (no visibility
+tiering), and has no cycle guard (schema interface cycles loop until
+the memo fills — the memo makes this safe in practice, not by design).
+
+**Gaps:** (a) transitive USE visibility — item visible through a chain
+of USEs is invisible to our single-hop lookup; (b) USE/REFERENCE
+visibility tiering absent; (c) no unresolved-resource diagnostics;
+(d) no cycle guard by construction. Fix tracked in TODO 06.
+
+## 2. Load scheduling (plugins/p11/schedule-*.lisp)
+
+eeng: `schedule-load` (two-pass parse), `schedule-interfaces`
+(one method per session kind: double-stepmod keeps trl/ref schema
+sets — schema1/schemata1 = trial, schemata2 = reference), recursing
+per interface symbol → find/load the foreign file via the stepmod
+directory layout, then `schedule-resolve` → `resolve-declaration` per
+schema with `*schema?*` dynamic scope (:trl-schema / :ref-schema).
+
+**expressir today:** `from_files` (batch + compiled set), repository
+flat over all files; no trl/ref duality (we never compare two
+specification universes); interface targets located by name at resolve
+time through the repository (single-hop, see §1).
+
+**Verdict:** structural difference, not a bug — our repository model
+subsumes their session model for everything except `--compare`
+(two-universe schema comparison, kernel/compare.lisp, unported; TODO
+07 candidate).
+
+## 3. Entity computations (kernel/compute-inheritance.lisp,
+derive-attrs.lisp, inverse-attrs.lisp, supertypes.lisp,
+explicit-attrs.lisp)
+
+eeng **materializes** inherited attributes onto entities:
+explicit attributes from the full supertype closure, derived attrs,
+inverse attrs, subtype/supertype links — used by `--validate`,
+population checks, and SHTOLO's generalized-attribute handling
+(Annex G.2.6 needs exactly these).
+
+**expressir today:** nothing materialized; consumers walk the
+supertype closure themselves (our item graph does this for edges, not
+attributes).
+
+**Gaps:** inherited-explicit-attribute flattening, derived/inverse
+closures. Needed by 05 (G.2.6 select synthesis) and 07 (checks).
+TODO 05 scope.
+
+## 4. Semantic checks (kernel/check.lisp, 835 lines)
+
+check-p11 methods per node type: schema-level (interface resource
+existence, per-kind resolution rules), entity (attribute redeclare
+compatibility, inverse target/entity/inverse-attr existence, derive
+aggregation rules), function/procedure/rule parameters, subtype
+constraints, types. Check severity/notes via check-notes.lisp.
+
+**expressir today:** `RepositoryValidator` (interface-level only) +
+`validate ascii` CLI. Gap tracked in TODO 07.
+
+## 5. Writers (plugins/p11/wo-*.lisp)
+
+| eeng | semantics | expressir |
+|---|---|---|
+| wo-pretty | normal encoding, 2-space indent | Formatter clean mode — parity specs landed (bug39/40) |
+| wo-flat | `:flat` encoding, downcased symbols, typeof-option banner | 05 (Annex G — note eeng's flat KEEPS 2003 constructs: partial) |
+| wo-list | declaration listing | 09 |
+| wo-smrl-xml | `<concatenated_express_file_content_list>` index for longform/concatenated modes | 09 |
+| wo-lexp | lisp S-expression dump | not planned (internal debugging) |
+
+## 6. Bugs found by this audit
+
+- **expressir#373** — Ruby parse path corrupts small USE FROM schemas
+  (<~70B, non-monotonic; core path correct). Found by the concatenator
+  fixtures; `dependency_resolver_spec` failures on main are the same
+  root cause. Fixed by the #371/#370 stack.
+- eeng partial Annex G coverage — documented in
+  `spec/fixtures/shtolo/annex-g-rules.md` (eeng is a partial oracle;
+  the annex is the contract).

data/TODO.refpath-v2/00-overview.md

@@ -0,0 +1,64 @@
+# 00 — RefPath v2: ELF 5006 conformance
+
+Goal: make `Expressir::Mapping::RefPath` a conforming validator for the
+EXPRESS mapping language specification (ELF 5006:2025,
+`~/src/expresslang/docs/sources/express-mapping/`), so that the WG12
+validation of annotated EXPRESS links (TRThurman PDES proposal,
+expressir 2.4.20+ capability, AP 242 ed5 DIS December 2026) finds
+expressir ready at every reference position.
+
+## Gap matrix (ELF 5006 §8 vs current RefPath)
+
+| §8 requirement | status |
+|---|---|
+| entity existence, case-insensitive | done |
+| `<=` / `=>` subtype/supertype, direct or transitive | done |
+| attribute existence for `->` / `<-` (inherited attrs) | done |
+| forward/inverse TYPE compatibility (attr type references target; SELECT membership) | v2 |
+| navigation continuity (running path context) | v2 |
+| constraint: entity in path context; attribute exists; not DERIVED/INVERSE | v2 |
+| constraint value typing (STRING/INTEGER/REAL/NUMBER/BOOLEAN/LOGICAL/ENUMERATION/SELECT table) | v2 |
+| start entity = ARM entity (relaxable for attribute mappings); end entity = MIM/aimelt | v2 |
+| `*>` / `<*` select extension semantics | v2 |
+| `!{...}` negation, `<...>` required path, `||...||` marker, `*...*` tree | v2 (tokenize + semantics) |
+| collect-all-errors | done |
+| structured output (text/JSON/YAML), per-path VALID/INVALID | v2 |
+| case-mismatch warnings (optional) | v2 optional |
+
+## Mapping-model gaps (ae-level fields in the wg12-step corpus)
+
+587 module mapping files; keys beyond the current model:
+
+- `rules` — 159 ae entries carry a rules list (a reference position in
+  the PDES proposal: "the rules list").
+- `alt_map` — 74 ae entries carry an alternative map ("an alternative
+  map" reference position).
+
+Both must be modeled with lutaml-model attributes (never hand-rolled
+parsing) and their link/reference positions validated like ae/sc.
+
+## Validation semantics (per §8 + §5/§6)
+
+- Running context: the entity a path is "at". `<=`/`=>` move to the
+  right-hand entity; `a.attr -> b` requires attr on the current entity
+  whose type references b (supertypes and SELECT options count), moves
+  to b; `b <- a.attr` same checks, moves to a; `[i]`/`[n]` keep the
+  element type as context; a bare name restates or continues the
+  context (§6 chaining).
+- Constraints `{e ... e.attr = v}`: e equals the current context entity
+  (§8: "at or before the constraint location"); attr exists on e and is
+  not DERIVED/INVERSE; v typed per the §8 table (STRING single-quoted,
+  INTEGER/REAL/NUMBER unquoted, BOOLEAN TRUE/FALSE, LOGICAL +UNKNOWN,
+  ENUMERATION item of the attribute's type, SELECT option match).
+  `!{...}` inverts the finding.
+- Start entity: compatible with the ae `entity` (relaxed for aa-level
+  paths). End entity: a MIM-declared entity (cross-check `aimelt`).
+
+## Test oracle
+
+- `spec/expressir/mapping/refpath_spec.rb` — extend with one example
+  per §8 rule (firing + clean negative).
+- Corpus: all 587 module mapping.yaml files; the suma MappingDrift
+  baseline (1541 entries) is the pre-v2 reference; v2 must not add
+  errors on paths the corpus proves valid, and must explain every new
+  error by an ELF 5006 rule.

data/TODO.suma-improvements/00-overview.md

@@ -0,0 +1,23 @@
+# Suma improvements — integration + expressir roadmap
+
+The metanorma ↔ expressir chain via suma: ISO 10303 SMRL schemas → expressir
+parse (compiled set EXSCS1) → LazyRepository + Liquid drops
+(metanorma-plugin-lutaml) → metanorma-iso schema documents. The chain is
+proven piecewise (1307 schemas, 0 failures, warm start 7.8s, live xrefs in
+rendered XML); the items below close the last mile and deepen expressir.
+
+| # | item | status |
+|---|------|--------|
+| 01 | prebuilt platform gems (oxidize-rb cross-gem, parsanol pattern) | in flight |
+| 02 | suma full-document E2E on current stack | pending |
+| 03 | ItemGraph — cross-schema subtype/interface graph API | pending |
+| 04 | compiled-set + lazy path as suma default build | pending |
+| 05 | corpus validation surfaced in suma builds | pending |
+| 06 | memory/perf re-measure with binding-by-default | pending |
+| 07 | #276 Part 28 XML — select mapping v2 | pending |
+| 08 | windows gnu toolchain for in-place ext builds | superseded by 01 for CI; kept for local dev |
+| 09 | checker tranche 3 (INVERSE, SELF\ qualifiers) | pending |
+| 10 | #91 EXPRESSdoc (upstream-gated) | parked |
+
+Evidence trail: #88/#461 (refpath validation), #462/#467 (ABSTRACT body
+form), v2.4.21–v2.4.24 (binding by default), plugin 0.7.53 + #305/#306.

data/TODO.suma-improvements/01-prebuilt-platform-gems.md

@@ -0,0 +1,33 @@
+# 01 — Prebuilt platform gems
+
+## Goal
+`gem install expressir` compiles nothing on machines with a prebuilt
+platform gem: the binding binary ships inside per-platform gems
+(`expressir-<ver>-arm64-darwin` etc.), following the parsanol-ruby
+pattern (oxidize-rb cross-gem + RbSys::ExtensionTask).
+
+## Pattern (from ~/src/parsanol/parsanol-ruby)
+- `oxidize-rb/actions/fetch-ci-data@v1` computes supported platforms
+- `oxidize-rb/actions/cross-gem@v1` matrix builds per-platform gems in
+  rake-compiler-dock containers (coherent gnu toolchain per target —
+  this is what fixes windows)
+- per-platform smoke jobs on REAL runners installing the gem and running
+  a smoke script (NATIVE true + parse)
+- source gem still builds via the gemspec (extensions stay declared);
+  rubygems prefers a matching platform gem automatically
+
+## expressir implementation
+- `rakelib/native.rake`: `RbSys::ExtensionTask.new("expressir_core")`,
+  `ext.lib_dir = "lib/expressir"`, `cross_compiling` callback stages
+  `expressir_core.{so,dylib,dll}` into `lib/expressir/` (loads via the
+  existing first `require "expressir/expressir_core"` in core.rb)
+- extconf: mingw skip becomes bypassable (`EXPRESSIR_CORE_FORCE=1` set in
+  the cross-gem container, where cargo gnu matches the mingw-ucrt ruby)
+- `.github/workflows/native-gems.yml`: ci-data + cross-gem matrix +
+  smoke jobs + source gem + `gem push` on tag (RUBYGEMS_API_KEY secret)
+- `.github/scripts/test_installed_gem.rb`: smoke script
+
+## Acceptance
+- workflow green across platforms with smoke jobs passing
+- a platform gem installs cargo-less and reports NATIVE true
+- source-gem install path unchanged (2.4.21 behavior)

data/TODO.suma-improvements/02-suma-e2e.md

@@ -0,0 +1,39 @@
+# 02 — Suma full-document E2E on the current stack
+
+## Goal
+One clean E2E run of an actual suma document build with expressir 2.4.24 +
+metanorma-plugin-lutaml 0.7.53 + current metanorma gems, rendering a real
+ISO 10303 module document to XML.
+
+## Why now
+The earlier full-render was blocked by the metanorma-document/relaton
+0.2.x skew; that ecosystem has moved (metanorma-document 0.5.1, relaton
+3.0 pre-alpha). The expressir side is proven current.
+
+## Steps
+- locate the suma build entry (~/src/mn/iso-10303 suma config or the
+  metanorma-iso document that renders schema pages)
+- bundle with current metanorma; render one module doc end to end
+- capture XML diffs vs the last good render; if the relaton skew
+  persists, name the exact unresolved constraint
+
+## Acceptance
+- a rendered module document, or a one-line statement of the precise
+  upstream blocker with the failing constraint
+
+## Result: DONE (2026-09-27)
+
+The upstream blocker has cleared. On the iso-10303 ladder (user's
+Gemfile WIP + suma from path + relaton prereleases with the rubyzip-3
+fix now on rubygems — the fix/rubyzip-3 git pins were dropped as
+obsolete):
+
+    bundle exec suma build srl-smol.yml   # one-part SMOL collection
+    # => Compiling schema collection... Compiling complete collection...
+    # => SMOL-EXIT=0
+
+Artifacts: 286 files in _site/ — real ISO document XML (flavor iso,
+schema content present), schema-doc pages, presentation XML. The stale
+sums pin was the rubygems suma 0.1.11 (expressir ~> 2.1, metanorma-cli
+0.0.3 → broken 2019 stack); suma from source (metanorma ~> 2.3 line)
+resolves cleanly.

data/TODO.suma-improvements/03-item-graph.md

@@ -0,0 +1,22 @@
+# 03 — ItemGraph
+
+## Goal
+Cross-schema graph API over a parsed repository: subtype edges, interface
+(USE FROM/REFERENCE FROM) edges, transitive queries — feeding suma doc
+generation (hyperlink coverage, "used by" sections, dependency diagrams).
+
+## Evidence
+SMRL full-set load proved the data: 37.7k subtype edges, 5.5k interface
+edges over 1307 schemas (TODO.memory SMRL notes). The class is the
+documented remaining gap.
+
+## Shape
+- `Expressir::Model::ItemGraph` built from a Repository (or a
+  LazyRepository hydrated once)
+- `subtypes(id)`, `supertypes(id)`, `used_by(id)`, `uses(id)`,
+  `transitive_subtypes(id)` — downcased-id keyed, cycle-safe
+- built once per repository; O(V+E)
+
+## Acceptance
+- specs over a multi-schema fixture (diamond subtyping, mutual USE FROM)
+- SMRL smoke: edge counts match the recorded 37.7k/5.5k within tolerance

data/TODO.suma-improvements/04-compiled-set-default.md

@@ -0,0 +1,16 @@
+# 04 — Compiled-set + lazy path as the suma default build
+
+## Goal
+suma builds parse the SMRL once into an EXSCS1 artifact and all document
+renders hydrate per schema from it (warm start 7.8s vs 12s cold), instead
+of eager-parsing on every build.
+
+## Steps
+- locate the suma/rake build entry that calls Parser.from_files
+- switch to `Parser.from_files(files, compiled_set:)` then
+  `LazyRepository.new(set)`
+- keep an escape hatch (env) for debugging with eager parse
+
+## Acceptance
+- a suma build log showing one artifact build + per-page hydration
+- rendered output byte-identical to the eager path

data/TODO.suma-improvements/05-validation-in-docs.md

@@ -0,0 +1,17 @@
+# 05 — Corpus validation surfaced in suma builds
+
+## Goal
+The expressir validation layers (mapping_validate incl. refpath
+resolution — which found 2 genuine corpus drifts in a 60-module sweep)
+run as part of suma builds, reporting per-module drift in the build log
+(and optionally in published output).
+
+## Steps
+- wire `expressir mapping_validate` over each module's mapping.yaml in
+  the suma build loop
+- aggregate a drift report artifact; non-zero exit only on NEW drift vs
+  a committed baseline file
+
+## Acceptance
+- baseline for the 2 known drifts; a build with a synthetic new drift
+  fails with the module/location in the message

data/TODO.suma-improvements/06-memory-remeasure.md

@@ -0,0 +1,36 @@
+# 06 — Memory/perf re-measure with binding-by-default
+
+## Goal
+The recorded profile ("50x memory ratio not achieved", pure-Ruby parse
+dominating) predates binding-by-default. Consumers now get the Rust
+parse path automatically; re-measure and refresh the story.
+
+## Steps
+- rerun the memory harness (TODO.memory methodology) with
+  NATIVE_AVAILABLE true on the same fixtures
+- record ratio vs the 50x target; note remaining hot spots
+- update TODO.memory notes + the perf memory file
+
+## Acceptance
+- fresh numbers committed alongside the old ones, with the parse
+  backend identified per measurement
+
+## Results (2026-09-27, ruby 3.4.8, 227KB synthetic / 4000 entities)
+
+| path | parse time | RSS delta | after GC |
+|------|-----------|-----------|----------|
+| Rust binding (NATIVE_AVAILABLE) | 7.7s | 348 MB | 387 MB |
+| pure-Ruby (use_native: false) | 7.5s | 307 MB | 353 MB |
+
+Both paths hydrate the full lutaml-model object tree, which dominates
+the delta — the native parse itself is not the differentiator at this
+size (arena overhead is offset by wire-hash materialization). The
+binding's memory advantage in the field comes from the compiled-set
+path (hydrate per file / warm start 7.8s), not from a lower per-parse
+floor. Ratio ~1.5 MB/KB of source on both paths — consistent with the
+older "~600x for large files" observation: the object MODEL is the
+cost driver, which is exactly what M2 laziness addresses.
+
+Old "50x memory ratio target" concern: RETIRED as the wrong metric for
+the binding; per-page hydration (LazyRepository) is the delivered
+mechanism for bounded memory at render time.

data/TODO.suma-improvements/07-part28-xml.md

@@ -0,0 +1,15 @@
+# 07 — Part 28 XML: select mapping v2 (#276)
+
+## Goal
+Full ISO 10303-28 EXPRESS-to-XML beyond the v1 skeleton
+(lib/expressir/express/xsd.rb): proper SELECT mapping (currently
+xs:anyType), aggregate/optional fidelity, schema-level structure.
+
+## Steps
+- ISO 10303-28 mapping rules for SELECT types (union/member semantics)
+- apply on top of xsd.rb's element/complexType generation
+- validate generated XSDs against the ISO 10303-28 conformance examples
+
+## Acceptance
+- a select-typed attribute renders per the standard (not xs:anyType)
+- the v1 examples still generate unchanged except where v2 improves them

data/TODO.suma-improvements/08-windows-gnu-toolchain.md

@@ -0,0 +1,15 @@
+# 08 — Windows gnu toolchain for in-place ext builds
+
+## Goal
+Local windows source installs and windows CI in-place builds compile the
+binding: rustup gnu target (`x86_64-pc-windows-gnu`) matching the mingw
+ruby, so extconf's guard can be lifted there.
+
+## Status
+Superseded for CI by 01 (cross-gem containers carry coherent toolchains;
+platform gems ship prebuilt). Kept for developers hacking on windows
+checkouts directly.
+
+## Acceptance
+- a windows machine with the gnu target builds the ext via extconf
+  without EXPRESSIR_CORE_FORCE

data/TODO.suma-improvements/09-checker-tranche3.md

@@ -0,0 +1,13 @@
+# 09 — Checker tranche 3 + SHTOLO completion
+
+## Goal
+Remaining eengine-parity checks: INVERSE-family rules and SELF\ qualifier
+validation (tranche 3), then the leftover SHTOLO longform items
+(TODO.parity-ee/14).
+
+## Gate
+Grammar support for INVERSE-scoped references and SELF\ qualified
+refs — verify against the current grammar after recent fixes.
+
+## Acceptance
+- each new check has a fixture that fires it + a clean-corpus negative

data/TODO.suma-improvements/10-expressdoc.md

@@ -0,0 +1,8 @@
+# 10 — EXPRESSdoc (#91, upstream-gated)
+
+## Goal
+Support the upcoming EXPRESSdoc format.
+
+## Status
+Parked on annotated-express#94 (upstream). Revisit when that lands;
+expressir's side is the remark/annotation model already in place.