expressir 2.4.3 → 2.4.4

data/TODO.parity-ee/11-import-eeng-tests.md

@@ -0,0 +1,79 @@
+# 11 — Import the Express Engine test suite
+
+**Status: NOT STARTED**
+
+## Goal
+
+Fully import eeng's tests — their release-suite scripts, the bug
+corpus, and the reference documents that define pass/fail — as
+executable RSpec feature specs and regression fixtures in expressir.
+Nothing in their suite should remain unrepresented.
+
+## Inventory (verified 2026-09-21)
+
+eeng has no formal test framework (`tests/` is a README describing an
+unrealized Makefile suite); the suite is:
+
+### CLI drivers — `scripts/*.sh` (14)
+
+| script | drives | our target operation |
+|---|---|---|
+| `qual-arm.sh`, `qual-mod.sh` | per-module qualification loop (arm/mim × every qualify.sh op) | 04, 05, 07-10 end-to-end specs |
+| `qual-res.sh` | same over `qualify-res.txt` resources | idem, resource side |
+| `pretty-test.sh` | `--pretty` ap209/ap210/ap242 × {arm,mim} × {shortform,longform} | 08 formatter specs |
+| `ap210.sh` | the 2.5MB long-form giant, pretty × 4 modes | 08 scale spec |
+| `test-tt.sh`, `test-tt1.sh` | `--xml` arm/mim shortform + Saxon HTM conversion + regenerate-exp compare | 09 (XML chain) |
+| `build-xml.sh` | XML builds over the whole `modules.lst` + `resources.lst` | 09 corpus spec |
+| `test.sh` | ad-hoc resource XML/HTM driver (kinematic schema) | 09 |
+| `scan-res.lisp` | resource corpus scan | 07 check corpus |
+| `kevin.sh`, `veronique.sh`, `copy-mod-xml.sh` | person-named scratch drivers | triage: keep as scenarios or drop with rationale |
+
+Each script's eeng CLI matrix (`-mode {arm,mim}_{shortform,longform,
+concatenated}` `-stepmod` `-schema` `--pretty/--xml/--flat/--dot
+--check`) maps 1:1 onto our planned operations — the flag matrix
+becomes our spec matrix.
+
+### Bug corpus — `docs/issues/`
+
+- `bug39-Greedy-Parens.txt` — **shtolo fixture**: contains "results
+  of old shtolo and stepcode exppp" vs "results of new shtolo"
+  outputs; doubles as a differential vs stepcode exppp.
+- `bug40-Pretty-Expressions.txt` — pretty-printer expression cases.
+Both become regression specs with the literal inputs embedded.
+
+### Reference documents — `docs/` (define pass/fail)
+
+- `shtolo_converting_STEP_short_listings_to_annotated_listings_libes.pdf`
+  — the SHTOLO paper (short listings → annotated listings); required
+  reading for 01/05 semantics.
+- BNF grammars: `iso-10303-11--2004{,-raw}.bnf`, `iso-10303-14*.bnf`,
+  **`iso-10303-21--{1994,2002}.bnf`** (Part 21!), `xml.bnf`,
+  `patch_schema.bnf` — grammar cross-checks (05's 1994 profile; the
+  Part 21 BNFs feed the future p21 parser).
+- `express_model.dtd`, `module.dtd` — the DTDs their XML emits (09's
+  structural contract).
+- `qualify.txt`, `interfaces.txt`, `interface-notes.pdf` —
+  qualification + interface semantics notes.
+- `00check.txt` (repo root) — check.lisp design notes → 07.
+
+## Steps
+
+1. Copy the durable assets into the repo: scripts verbatim under
+   `spec/fixtures/eeng/scripts/` (provenance header), bug corpus under
+   `spec/fixtures/eeng/issues/`, DTDs + BNFs under
+   `spec/fixtures/eeng/grammar/` (license is MIT-style per eeng
+   headers — verify and record attribution).
+2. Translate the CLI matrix into RSpec feature specs
+   (`spec/expressir/eeng_parity/`): one describe per driver family,
+   one example per (schema × mode × operation), corpus-scaled ones
+   tagged `smrl`.
+3. Port bug39/bug40 as literal regression specs now (they need only
+   parser + formatter, both of which exist).
+4. Extend 03's oracle harness to emit/refresh the expected outputs
+   these specs consume.
+
+## Acceptance
+
+Every driver row and both bugs have a corresponding spec (or a
+documented drop rationale); `smrl`-tagged parity suite green against
+the 03 golden corpus.

data/TODO.parity-ee/parity-matrix.md

@@ -0,0 +1,108 @@
+# eeng ↔ expressir parity matrix (2026-09-21)
+
+Reference: `~/src/external/exp-engine-engine` (eengine-5.0.20-Beta1).
+Audited files: kernel/{find-declaration,find-entity,find-type,
+resolve-interface,resolve-object,resolve-ref,references,schema-search,
+interfaces,compute-inheritance,supertypes,derive-attrs,
+inverse-attrs,check,compare,dot-graph}.lisp; plugins/p11/{schedule-*
+,resolve-declaration,cleanup-interface,wo-*}.lisp.
+
+## 1. Name resolution ladder (kernel/find-declaration.lisp:125-172)
+
+eeng's schema-scope restriction ladder for `find-declaration`:
+
+```
+nil        = :use-from
+:local     → own declarations + constants only
+:use-from  → :local OR :use-only
+:use-only  → walk USE FROM interfaces only, transitively, each in
+             :use-from scope (so USE visibility is transitive)
+:all-iface → :local OR all interfaces (USE and REFERENCE), transitively
+cycle guard → *schema-path*: schema already on path ⇒ nil
+```
+
+Per-kind restrictions (`resolve-interface.lisp:30-66`):
+- USE FROM resolves **type + entity only**.
+- REFERENCE FROM resolves constant/entity/function/procedure/type.
+- Unresolved resource ⇒ hard **error** `"Resource '~A' not found"`.
+
+**expressir today:** `ModelElement#find` climbs parent scopes through
+`children_by_id`; `Schema#children` = `interfaced_items + safe_children`
+where `interfaced_items` is **single-hop** (own interfaces only, no
+transitivity), and the resolver never errors on unresolved resources.
+Our `find` also treats REFERENCE and USE identically (no visibility
+tiering), and has no cycle guard (schema interface cycles loop until
+the memo fills — the memo makes this safe in practice, not by design).
+
+**Gaps:** (a) transitive USE visibility — item visible through a chain
+of USEs is invisible to our single-hop lookup; (b) USE/REFERENCE
+visibility tiering absent; (c) no unresolved-resource diagnostics;
+(d) no cycle guard by construction. Fix tracked in TODO 06.
+
+## 2. Load scheduling (plugins/p11/schedule-*.lisp)
+
+eeng: `schedule-load` (two-pass parse), `schedule-interfaces`
+(one method per session kind: double-stepmod keeps trl/ref schema
+sets — schema1/schemata1 = trial, schemata2 = reference), recursing
+per interface symbol → find/load the foreign file via the stepmod
+directory layout, then `schedule-resolve` → `resolve-declaration` per
+schema with `*schema?*` dynamic scope (:trl-schema / :ref-schema).
+
+**expressir today:** `from_files` (batch + compiled set), repository
+flat over all files; no trl/ref duality (we never compare two
+specification universes); interface targets located by name at resolve
+time through the repository (single-hop, see §1).
+
+**Verdict:** structural difference, not a bug — our repository model
+subsumes their session model for everything except `--compare`
+(two-universe schema comparison, kernel/compare.lisp, unported; TODO
+07 candidate).
+
+## 3. Entity computations (kernel/compute-inheritance.lisp,
+derive-attrs.lisp, inverse-attrs.lisp, supertypes.lisp,
+explicit-attrs.lisp)
+
+eeng **materializes** inherited attributes onto entities:
+explicit attributes from the full supertype closure, derived attrs,
+inverse attrs, subtype/supertype links — used by `--validate`,
+population checks, and SHTOLO's generalized-attribute handling
+(Annex G.2.6 needs exactly these).
+
+**expressir today:** nothing materialized; consumers walk the
+supertype closure themselves (our item graph does this for edges, not
+attributes).
+
+**Gaps:** inherited-explicit-attribute flattening, derived/inverse
+closures. Needed by 05 (G.2.6 select synthesis) and 07 (checks).
+TODO 05 scope.
+
+## 4. Semantic checks (kernel/check.lisp, 835 lines)
+
+check-p11 methods per node type: schema-level (interface resource
+existence, per-kind resolution rules), entity (attribute redeclare
+compatibility, inverse target/entity/inverse-attr existence, derive
+aggregation rules), function/procedure/rule parameters, subtype
+constraints, types. Check severity/notes via check-notes.lisp.
+
+**expressir today:** `RepositoryValidator` (interface-level only) +
+`validate ascii` CLI. Gap tracked in TODO 07.
+
+## 5. Writers (plugins/p11/wo-*.lisp)
+
+| eeng | semantics | expressir |
+|---|---|---|
+| wo-pretty | normal encoding, 2-space indent | Formatter clean mode — parity specs landed (bug39/40) |
+| wo-flat | `:flat` encoding, downcased symbols, typeof-option banner | 05 (Annex G — note eeng's flat KEEPS 2003 constructs: partial) |
+| wo-list | declaration listing | 09 |
+| wo-smrl-xml | `<concatenated_express_file_content_list>` index for longform/concatenated modes | 09 |
+| wo-lexp | lisp S-expression dump | not planned (internal debugging) |
+
+## 6. Bugs found by this audit
+
+- **expressir#373** — Ruby parse path corrupts small USE FROM schemas
+  (<~70B, non-monotonic; core path correct). Found by the concatenator
+  fixtures; `dependency_resolver_spec` failures on main are the same
+  root cause. Fixed by the #371/#370 stack.
+- eeng partial Annex G coverage — documented in
+  `spec/fixtures/shtolo/annex-g-rules.md` (eeng is a partial oracle;
+  the annex is the contract).

data/exe/expressir

@@ -1,5 +1,11 @@
 #!/usr/bin/env ruby
 # encoding: UTF-8
 
+# Ruby block-buffers stdout when it is not a terminal, so progress from a long
+# run reaches a CI log only when the buffer fills or the process exits. A job
+# that is working then looks hung. Set here rather than in the library, which
+# runs inside other people's processes and does not own their streams.
+$stdout.sync = true
+
 require_relative "../lib/expressir"
 Expressir::Cli.start(ARGV)

data/ext/expressir_core/Cargo.toml

@@ -0,0 +1,24 @@
+[package]
+name = "expressir_core"
+version = "0.1.0"
+edition = "2021"
+publish = false
+
+[lib]
+name = "expressir_core"
+crate-type = ["cdylib"]
+
+[dependencies]
+magnus = { version = "0.9" }
+expressir-rs = { git = "https://github.com/lutaml/expressir-rs", branch = "main" }
+serde_json = "1.0"
+
+# Standalone workspace: the repo root workspace owns crates/, the ext
+# is built by rb_sys with its own cargo invocation.
+[workspace]
+
+# Same magnus/rb-sys pins as parsanol-ruby (Ruby 4.0 support).
+[patch.crates-io]
+magnus = { git = "https://github.com/matsadler/magnus", rev = "9407bd9" }
+magnus-macros = { git = "https://github.com/matsadler/magnus", rev = "9407bd9" }
+rb-sys = { git = "https://github.com/oxidize-rb/rb-sys.git", rev = "daa12b6521a0c442a705d7f9a929029119aab1d6" }

data/ext/expressir_core/extconf.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require "mkmf"
+
+# The Rust extension only builds against MRI's C API. Other engines
+# fall back to the pure-Ruby parser path and must install cleanly.
+if RUBY_ENGINE != "ruby" || ENV["EXPRESSIR_CORE"] == "0"
+  File.write("Makefile", dummy_makefile("").to_s)
+  warn "expressir: skipping the Rust core extension on #{RUBY_ENGINE}"
+  exit 0
+end
+
+# Pure-Ruby fallback when no Rust toolchain is available: the gem must
+# install and keep working without the native core
+# (Expressir::Core::NATIVE_AVAILABLE stays false). Prebuilt platform
+# gems are the follow-up so cargo-less MRI installs get binaries too.
+unless find_executable("cargo")
+  File.write("Makefile", dummy_makefile("").to_s)
+  warn "expressir: cargo not found — skipping the Rust core extension " \
+       "(Expressir::Core falls back to the Ruby parser)"
+  exit 0
+end
+
+require "rb_sys/mkmf"
+
+create_rust_makefile("expressir/expressir_core") do |r|
+  r.profile = ENV.fetch("RB_SYS_CARGO_PROFILE", :dev).to_sym
+  r.use_stable_api_compiled_fallback = true
+  r.force_install_rust_toolchain = false
+end