lutaml-xsd 1.1.13 → 1.2.0

data/TODO.conformance.md

@@ -0,0 +1,99 @@
+# TODO.conformance — measured validation conformance (W3C XSD Test Suite)
+
+Goal: conformance is a number, not a feeling. Vendor the W3C XML Schema
+test-suite categories, build the harness, publish the pass-rate, grow it.
+
+## Baseline (2026-10-22, 20 cases / 42 checks)
+
+Corpus: 20 case classes under `spec/conformance/cases/`, modelled on the
+W3C XML Schema Test Suite category taxonomy — self-authored because the
+canonical 2007 suite distribution is no longer served from w3.org (owner
+can vendor a tarball later; provenance README goes with it).
+
+| mode | pass-rate |
+|---|---|
+| engine (Moxml::XSD tier-1, moxml 0.5.150) | 35/42 (83.3%) |
+| pipeline (ValidationJob + wired rule engine) | 40/42 (95.2%) |
+
+The pipeline outperforms the engine by five checks: the root-declaration
+check catches a wrong-namespace root that engine root-matching silently
+skips; the identity-constraints rule catches duplicate keys the engine
+lacks; xsi:nil handling rejects non-nillable nils; strict wildcards
+(element and attribute) reject undeclared content. Zero false positives
+retained.
+
+Gap register (each failure is a tracked gap, not a test bug):
+
+- `attributes-any` valid-known — tier-1 engine reports a qualified
+  global attribute as "not declared" under strict anyAttribute (engine
+  limitation; pipeline accepts it correctly)
+- `identity-keykeyref` — dangling keyref not detected: implemented in
+  `IdentityConstraintsRule`, but activation needs lutaml-model to map the
+  `refer` attribute on `xs:keyref` (upstream fix in flight)
+- `xsi-type` — a VALID instance derived via `xsi:type` is rejected by the
+  tier-1 engine (false positive, structural: engine scope is stricter than
+  derivation); the unknown-type half passes in pipeline mode
+- engine-only: `namespaces-qualified` wrong-namespace root accepted
+  (EngineContext root-matching skips it; pipeline's root check catches it)
+
+History: 2026-10-20 baseline 20/25 (80.0%) with pipeline == engine
+(empty `RuleRegistry`, mismatched rule signatures); 2026-10-21 wiring
+baseline 34/39 (87.2%); gap rules 37/39 (94.9%); anyAttribute lifted the
+pipeline to 40/42 (95.2%).
+
+## Harness
+
+- [x] Conformance corpus under `spec/conformance/cases/` (directory
+      convention: schema.xsd + valid-*.xml + invalid-*.xml)
+- [x] `Lutaml::Xsd::Conformance::Runner`: discovery, per-mode
+      classification, pass-rate report (engine + production pipeline)
+- [x] `rake conformance` → prints report and writes
+      `tmp/conformance_report.md`
+- [x] Deliverable doc: `docs/CONFORMANCE.adoc` (methodology, baseline,
+      gap register pointer, how to add cases)
+- [ ] Vendor a real W3C XSD TS subset — DEFERRED: upstream distribution
+      URLs are dead as of 2026-10-20; the owner supplies the corpus
+      tarball and a provenance/license README is added with it
+- [ ] Harness run against the rule engine with registries explicitly
+      populated — DEFERRED: subsumed by the rule-registry wiring design
+      below
+
+## Rule-engine wiring (done 2026-10-21)
+
+- [x] Design note: `docs/superpowers/specs/2026-10-21-rule-engine-wiring-design.md`
+- [x] Element-scoped dispatch contract: `ValidationRule#schema_argument`
+      (defaults to the resolved declaration; occurrence rule returns the
+      content-model particle) and `runs_without_schema?`
+- [x] `SchemaDefinitionResolver`: top-level + inline element index with
+      namespace matching and named-type resolution
+- [x] `RuleRegistry.default(config)`: production rule set, feature-gated;
+      shared by `Validator` and the conformance harness
+- [x] Rules resolve `type="prefix:Name"` references via the repository
+- [x] Substitution groups: content-model and occurrence matching are
+      substitution-aware (substitution case now passes in pipeline mode)
+- [x] Facet-collection handling fixed (empty arrays were treated as
+      present facets; `XmlAttribute` namespace ported to `.uri`)
+- [x] Integration specs prove rule findings fire with the engine phase
+      disabled (ordering, missing content) and valid instances stay clean
+
+## Rule-engine depth (remaining)
+
+- [x] Identity constraints: `key`, `keyref`, `unique` —
+      `IdentityConstraintsRule` (selector/field XPath evaluation,
+      duplicate detection, keyref resolution); dangling-keyref activation
+      waits on lutaml-model `refer` mapping
+- [x] Wildcards: `any` namespace constraints (##any/##other/##local/
+      ##targetNamespace/lists) + `processContents` strict declaration
+      checking in the content-model walk; `anyAttribute` honors namespace
+      constraints and strict checking where the parse layer exposes the
+      declaration (attributeGroup, content-model restriction) —
+      complexType-level and extension-shape anyAttribute mappings are
+      missing upstream (lutaml-model), keeping those on pre-wildcard
+      behavior until mapped
+- [x] `xsi:nil` (nillable + empty-content checks) and `xsi:type`
+      resolution (unknown-type rejection; content walk follows the
+      override); valid-derived acceptance needs engine-scope rework
+- [ ] Particle checking: UPA detection — DEFERRED (engine covers the
+      tier-1 subset; UPA is schema-side checking)
+- [ ] XSD 1.1 constructs behind a version flag — DEFERRED: 1.1 strategy
+      decision needed (see TODO.som.md)

data/TODO.docs-spa.md

@@ -0,0 +1,46 @@
+# TODO.docs-spa — comprehension SOTA: interactive, live-validated documentation
+
+Goal: leapfrog static commercial doc-generators (Oxygen/Altova) with an
+interactive corpus browser where the documentation *executes*.
+
+Status 2026-10-20: DEFERRED as a pillar. Reason: the SPA features below
+need the frontend development loop (`rake build_frontend`, Vue dev server)
+and a backend seam decision (CLI vs HTTP). The validation core these
+features depend on is ready (EngineContext + `engine_validation` +
+Conformance runner), so the UI work can start against a stable base. The
+one prerequisite wiring — rule-registry gaps surfacing as findings — is
+tracked in TODO.conformance.md.
+
+## Live validation playground (the killer feature)
+
+- [ ] SPA page: paste/upload an XML instance → validate against the loaded
+      package (engine + rule stack via a CLI/HTTP seam) → findings with
+      line/column/path — DEFERRED: backend seam decision + frontend build
+      loop
+- [ ] Click-through: finding → schema component page — DEFERRED (needs
+      the playground first)
+- [ ] Instance tree view with per-node validity — DEFERRED (needs
+      element-path findings in the UI; engine already reports paths)
+
+## Navigation & comprehension
+
+- [ ] Global search across types/elements/attributes/groups — DEFERRED:
+      the search index exists (`build_index`); ranking design + UI work
+- [ ] Usage navigation matrix for attributes and groups — DEFERRED:
+      liquid methods pattern exists; extend `used_by` coverage first
+- [ ] Type hierarchy graph in the SPA — DEFERRED (pairs with TODO.som.md)
+- [ ] Dependency graph page — DEFERRED
+
+## Evolution
+
+- [ ] Schema diff between package versions — DEFERRED: structural diff
+      design needed (component identity across versions)
+- [ ] Coverage view wired to `coverage_analyzer` — DEFERRED
+
+## Polish
+
+- [ ] SPA generation performance on large corpora — DEFERRED until the
+      post-recalibration corpus exists (TODO.performance.md)
+- [ ] Accessibility pass on the SPA — DEFERRED (rides the next UI change)
+- [ ] Nightly demo package regeneration in CI — DEFERRED: cimas-owned
+      workflow; needs an owner decision on schedule

data/TODO.interop.md

@@ -0,0 +1,52 @@
+# TODO.interop — bridges: RNG, JSON Schema, LutaML cross-model
+
+Goal: no schema format is an island; every bridge lands in the LutaML
+model-driven documentation pipeline.
+
+Status 2026-10-20: RNG bridge migrated to Moxml; JSON Schema exporter
+delivered; remaining items deferred with reasons below.
+
+## RelaxNG bridge (rng gem, ~/src/lutaml/rng — its own repo + PR flow)
+
+- [x] Migrate rng gem to Moxml (rng PR #35): five Nokogiri files ported
+      (rnc_to_rng_converter, rnc_parser, test_suite_parser,
+      schema_validator, external_ref_resolver); `nokogiri` removed from
+      the runtime dependencies, `moxml` added; dev Gemfile runs leptris
+- [x] Round-trip coverage: the rng suite's RNC ↔ RNG round-trip and
+      spec-test suites pass on the Moxml converter (1056 examples; two
+      environmental timing guards were widened to pathological-regression
+      budgets — they also failed on pre-migration main on loaded machines)
+- [x] rng 0.3.9 baseline recalibration (2026-10-21): converter expectations
+      re-baselined against the fixed include processor — complex types
+      170, simple types 13, elements 203, groups 237, attribute_groups
+      44; sections sequence: 7 choices / 1 group / 1 element (`sections`
+      present via bsi's override, trace in rng PR #34)
+- [x] `rng ~> 0.3.9` pinned in lutaml-xsd (2026-10-21)
+
+## JSON Schema bridge
+
+- [x] XSD → JSON Schema (draft 2020-12) exporter:
+      `Lutaml::Xsd::JsonSchemaExporter` — objects from complexTypes,
+      properties/required from sequences and attributes, arrays from
+      repeated particles, enum/pattern/length/range facets, builtin type
+      mapping, `const` for fixed values, nillable → null union
+- [x] Loss table documented in docs/JSON_SCHEMA.adoc (XSD regex dialect,
+      identity constraints, substitution groups, derivation flattening,
+      wildcards, defaults, mixed content, XSD 1.1)
+- [ ] SPA "export JSON Schema" action per package — DEFERRED: needs the
+      SPA work in TODO.docs-spa.md
+
+## LutaML cross-model (the differentiator)
+
+- [ ] XSD → LutaML model export — DEFERRED: there is no cross-model
+      contract in this repo yet (LutaML class/association export needs a
+      design note agreed with the lutaml-model owners); the exporter
+      here is the template for that work
+- [ ] Metanorma integration for schema-package documentation — DEFERRED:
+      depends on the SPA data contract being frozen
+
+## Adjacent (evaluate, don't commit)
+
+- [ ] ISO Schematron rule-engine plugin — DEFERRED: evaluate demand;
+      the rule_registry pattern fits, and the conformance harness gives
+      the measurement hook when it is taken up

data/TODO.performance.md

@@ -0,0 +1,80 @@
+# TODO.performance — measured speed on the fastest Ruby XML engine
+
+Goal: stay the fastest, prove it continuously, and adopt every new engine
+surface as moxml/leptris ship it.
+
+## Engine integration maturation
+
+- [x] Track https://github.com/lutaml/moxml/issues/410 — FIXED and
+      RELEASED (moxml 0.5.149, moxml PR #411). lutaml-xsd now locks
+      0.5.149: `Schema#content_valid?` answers correctly on leptris and
+      the engine content-seam specs run live.
+- [x] Content delegation verified live: engine answers true/false for
+      matching/mismatching child sequences (engine_context_spec + the
+      formerly gated content-model rule specs).
+- [ ] Adopt moxml `Plan`/`StructPlan` materialization for the hot
+      parse path if lutaml-model exposes seams for it (profile first)
+- [ ] Engine error payload richness: once line/column/path are stable
+      across adapters, surface them in CLI output and the SPA findings UI
+      — PARTIAL: CLI JSON already carries code/message/severity/
+      location/line_number/context (verified 2026-10-21); SPA wiring
+      rides TODO.docs-spa.md
+
+## Benchmarks
+
+- [x] In-repo benchmark suite (`benchmark/engines.rb` + `rake
+      benchmark:engines`): engine parse/xpath/serialize, pipeline parse,
+      schema to_xml, compiled-schema validation
+- [x] Baseline table: leptris vs nokogiri adapter (2026-10-20, Ruby 3.4.8,
+      arm64-darwin, 10.8 KB corpus converted from
+      metanorma-model-iso-grammars/bsi.rnc under rng 0.3.7 — corpus is
+      SMALL because rng 0.3.7 partially resolves the include tree;
+      re-run after the rng 0.3.9 recalibration for the full corpus):
+
+| measurement | leptris | nokogiri |
+|---|---|---|
+| engine_parse | 0.3 ms | 0.4 ms |
+| engine_xpath_elements | 0.9 ms | 0.4 ms |
+| engine_serialize | 0.7 ms | 0.1 ms |
+| pipeline_schema_parse | 274 ms | 377 ms |
+| pipeline_schema_to_xml | 224 ms | 223 ms |
+| engine_xsd_compile | 0.1 ms | n/a (SyntaxError on rng-0.3.7 corpus) |
+| engine_xsd_validate | 0.0 ms | n/a |
+
+  Caveat: per-op values are noisy run-to-run (0.1–12 ms spread observed)
+  on a small corpus; the harness is the deliverable, numbers firm up on
+  the post-recalibration corpus.
+- [ ] CI-optional benchmark run (manual dispatch) with trend notes —
+      no PR-blocking perf gates yet
+
+## Profiling targets
+
+- [ ] SPA generation memory profile on 100+ schema corpora — DEFERRED:
+      needs the post-recalibration corpus (TODO.cleanup.md) and the SPA
+      pillar (TODO.docs-spa.md); the marshal-cleanup code is already in
+      place
+- [ ] `EngineContext.compile` cost on multi-schema repositories —
+      DEFERRED: measured at 0.1 ms/compile on the benchmark corpus (well
+      below noise); revisit only when the full-include corpus changes the
+      picture
+- [ ] XmlNavigator walk cost on deep instance trees — DEFERRED: no
+      evidence of a hotspot yet; take it with the rule-engine wiring,
+      which is the walker's main consumer
+
+## moxml Builder DSL profile (2026-10-21, reported to moxml sota/04)
+
+rng's converter drives `Moxml::Builder`; the same 500-define synthetic
+tree through the same DSL:
+
+| path | time |
+|---|---|
+| Moxml Builder + nokogiri adapter | 0.059 s |
+| Moxml Builder + leptris adapter | 0.463 s |
+| direct `Leptris::XML::Builder` | 0.007 s |
+
+The ~8x gap is in moxml's leptris builder-face interaction (create_tag /
+method_missing / face flush), not the binding — the DSL through the
+nokogiri adapter is only ~2x native Nokogiri's builder. moxml's scoreboard
+covers the mutation API, not the Builder DSL; this is offered as a row +
+profile lead for the moxml perf branch. Work belongs in moxml, not here.
+

data/TODO.som.md

@@ -0,0 +1,47 @@
+# TODO.som — schema object model completeness (XSOM parity)
+
+Goal: the Ruby SOM must answer every structural question XSOM can, with
+XSD 1.1 constructs layered on.
+
+Status 2026-10-20: DEFERRED as a pillar. Reason: every item here is a
+specification-parity design cycle (component resolution semantics,
+derivation graphs, 1.1 constructs), and the conformance gap register
+(rule-engine wiring first) and the release-gated rng recalibration are
+higher-leverage prerequisites. Track this file as the design backlog; do
+not start mid-cycle.
+
+## Parse-side completeness
+
+- [ ] `xs:redefine` semantics (redefinition chains, component shadowing)
+      — DEFERRED: needs a component-resolution design note first
+- [ ] Chameleon includes (namespace adoption rules) — DEFERRED: verify
+      current behavior against the rng include fixes, then design
+- [ ] Substitution group closure and membership enumeration on the model
+      — DEFERRED: pairs with the conformance substitution-group gap
+- [ ] `xs:group` / `xs:attributeGroup` reference resolution including
+      redefined groups — DEFERRED (same design cycle)
+- [ ] Complex type derivation graph: extension/restriction chains,
+      `{base type definition}` reachability — DEFERRED: the type-hierarchy
+      analyzer exists; extending it into a full derivation graph is the
+      design entry point
+- [ ] Simple type facets carried losslessly — DEFERRED: covered
+      opportunistically by the JSON Schema exporter tests; a dedicated
+      round-trip matrix should come with the recalibration
+
+## Query surfaces
+
+- [ ] Structural queries via moxml XPath where trees are involved —
+      DEFERRED: profile first (EngineContext already avoids Ruby walks)
+- [ ] `SchemaRepository#types_in_namespace` deterministic ordering and
+      stable public docs — DEFERRED: small, take it with the next
+      repository-touching PR
+- [ ] Type hierarchy export (DOT/mermaid) into the SPA — DEFERRED: needs
+      the SPA work in TODO.docs-spa.md
+
+## 1.1 constructs (behind `version: "1.1"`)
+
+- [ ] `xs:assert` / `xs:assertion` model + evaluation hooks — DEFERRED:
+      XSD 1.1 strategy decision needed (engine support vs pure Ruby)
+- [ ] `xs:alternative` conditional types; `openContent` modes — DEFERRED
+      (same 1.1 strategy)
+- [ ] `xs:override` and `xs:defaultOpenContent` — DEFERRED (same)

data/benchmark/engines.rb

@@ -0,0 +1,77 @@
+# frozen_string_literal: true
+
+# Engine benchmark: leptris vs nokogiri through Moxml, on a real corpus.
+#
+#   ADAPTER=leptris bundle exec ruby benchmark/engines.rb
+#
+# Per-adapter numbers land on stdout as `key: value` lines; the rake task
+# (`rake benchmark:engines`) runs both adapters for side-by-side recording
+# in TODO.performance.md.
+
+require "benchmark"
+require "tmpdir"
+require "fileutils"
+
+ADAPTER = ENV.fetch("ADAPTER", "leptris")
+
+require "rng" if ADAPTER == "leptris" # load first so Moxml picks it
+require "lutaml/xsd"
+
+def corpus_path
+  corpus = File.join(__dir__, "..", "tmp", "bench_corpus.xsd")
+  unless File.exist?(corpus)
+    rnc = File.expand_path(
+      "../spec/fixtures/metanorma-model-iso-grammars/bsi.rnc", __dir__
+    )
+    grammar = Rng.parse_file(rnc)
+    schema = Lutaml::Xsd::RngToXsdConverter.new(grammar, file_path: rnc).convert
+    FileUtils.mkdir_p(File.dirname(corpus))
+    File.write(corpus, schema.to_xml)
+  end
+  corpus
+end
+
+def measure(label, iterations: 5, &)
+  gc_was = GC.enable
+  GC.start
+  elapsed = Benchmark.realtime { iterations.times(&) }
+  GC.start if gc_was
+  format("%-34s %8.1f ms/op", label, (elapsed / iterations) * 1000)
+end
+
+corpus = File.read(corpus_path)
+ctx = Moxml::Context.new(ADAPTER.to_sym)
+
+puts "adapter: #{ADAPTER}"
+puts "corpus_bytes: #{corpus.bytesize}"
+
+# Engine-level parse
+doc = nil
+puts measure("engine_parse") { doc = ctx.parse(corpus) }
+
+# Engine-level XPath
+puts measure("engine_xpath_elements") { doc.xpath("//*[local-name() = 'element']") }
+
+# Engine-level serialize
+puts measure("engine_serialize") { doc.to_xml }
+
+# Schema pipeline parse (lutaml-model over Moxml)
+Lutaml::Xml::Schema::Xsd.parse(corpus) # warm autoloads/registration
+schema = nil
+puts measure("pipeline_schema_parse") { schema = Lutaml::Xml::Schema::Xsd.parse(corpus) }
+
+# Schema model serialization
+puts measure("pipeline_schema_to_xml") { schema.to_xml }
+
+# Compiled-schema instance validation (tier-1 engines)
+instance = <<~XML
+  <bible xmlns="http://example.com/bsi"><content><section-title>t</section-title></content></bible>
+XML
+begin
+  compiled = ctx.xsd(corpus)
+  puts measure("engine_xsd_compile", iterations: 3) { ctx.xsd(corpus) }
+  puts measure("engine_xsd_validate") { compiled.validate(instance) }
+rescue StandardError => e
+  puts "engine_xsd_compile                   n/a (#{e.class})"
+  puts "engine_xsd_validate                  n/a (#{e.class})"
+end

data/config/validation.yml

@@ -48,6 +48,10 @@ validation:
     # Validate namespace declarations
     validate_namespaces: true
 
+    # Engine-native XSD validation (Moxml::XSD) when the adapter
+    # provides an engine (leptris/nokogiri); findings are additive
+    engine_validation: true
+
   # Error reporting configuration
   error_reporting:
     # Include XPath location in error messages

data/docs/CONFORMANCE.adoc

@@ -0,0 +1,43 @@
+# Conformance suite
+
+`spec/conformance/cases/` holds one directory per case class, modelled on
+the W3C XML Schema Test Suite category taxonomy. The canonical 2007 suite
+distribution is no longer served from w3.org (checked 2026-10-20); cases
+here are original, and a vendored upstream subset can be added later with
+a provenance + license README alongside it.
+
+## Case layout
+
+```
+spec/conformance/cases/<case-name>/
+  schema.xsd        # the schema under test
+  valid-*.xml       # instances that MUST validate
+  invalid-*.xml     # instances that MUST NOT validate
+```
+
+## Running
+
+```bash
+bundle exec rake conformance
+```
+
+Prints a per-mode report and writes `tmp/conformance_report.md`.
+
+Two modes are measured independently:
+
+- **engine** — `Moxml::XSD` tier-1 validation through `EngineContext`
+  (leptris when the adapter ships an engine)
+- **pipeline** — the production `Validator` stack (`ValidationJob` with
+  the default rule registry)
+
+## Baseline and gap register
+
+Recorded in `TODO.conformance.md` (2026-10-20: 20/25 = 80.0% in both
+modes). Each failing check maps to a tracked gap; the gap register is the
+work queue for the conformance pillar.
+
+## Adding a case
+
+Create the directory, add `schema.xsd`, at least one `valid-*.xml` and one
+`invalid-*.xml`, and re-run the rake task. No registration needed — the
+runner discovers directories containing `schema.xsd`.

data/docs/JSON_SCHEMA.adoc

@@ -0,0 +1,42 @@
+# JSON Schema export
+
+`Lutaml::Xsd::JsonSchemaExporter` writes an XSD schema as a JSON Schema
+2020-12 document.
+
+```ruby
+schema = Lutaml::Xml::Schema::Xsd.parse(File.read("schema.xsd"))
+json = Lutaml::Xsd::JsonSchemaExporter.new(schema).to_json
+File.write("schema.json", json)
+```
+
+## What maps
+
+- top-level `complexType`/`simpleType`/`element` declarations → `$defs`
+- `sequence` / `all` elements → `properties`; `minOccurs > 0` → `required`
+- `maxOccurs > 1` / `unbounded` → `{"type": "array", "items": ...}`
+- `choice` and top-level elements → `oneOf`
+- attributes → `properties`; `use="required"` → `required`; `fixed` → `const`
+- `simpleType` restrictions: `enumeration` → `enum`; `pattern` → `pattern`;
+  length/range facets → `minLength`/`maxLength`/`minimum`/`maximum`/
+  `exclusiveMinimum`/`exclusiveMaximum`
+- built-in types to JSON types/formats (date → `format: date`, dateTime →
+  `date-time`, time → `time`, anyURI → `uri`, integer family → `integer`,
+  decimal/float/double → `number`)
+- `nillable="true"` → the JSON type accepts `null`
+
+## Loss table (deliberate)
+
+| XSD | JSON Schema outcome |
+|---|---|
+| XSD regular expressions | emitted verbatim; XSD regex dialect differs from ECMA-262 — patterns may over/under-match |
+| `key` / `keyref` / `unique` identity constraints | not representable; dropped |
+| substitution groups | members are separate `$defs`; head/member relation is not encoded |
+| complex-type derivation (extension/restriction) | flattened into the derived type; base linkage lost |
+| `anyAttribute` / open content | not represented (`additionalProperties: false` is emitted) |
+| wildcards (`xs:any`) | not represented |
+| default values | not emitted (defaults are instance-processing semantics) |
+| mixed content | modelled as element properties only; interleaved text is not captured |
+| XSD 1.1 assertions / alternatives / openContent | not represented |
+
+The exporter targets documentation, tooling interop, and contract
+publication; it is not a validation-equivalence layer.