expressir 2.4.9 → 2.4.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1dac0cba9583d9371198d313d26da56bb31fa9b62392acd9548981eda41a115a
-  data.tar.gz: e0b069f89d8358e1244d930ddac24f56be78bfe2aa6e3bf9ab91b895049326fc
+  metadata.gz: 4a8a349971d0d91880df982145a16571130609ca66be9082aa512dad577e2156
+  data.tar.gz: 19c5b335acdf93eb742096e0405424f7a45fa271aeaa9fd6e7f000787e0574e4
 SHA512:
-  metadata.gz: c2f3cf0c7f2d375454ebcd182eb0bfcbffbf3b1725b484b9c8e81a92fbef9de65384509eff26aea2462adf4888c375d0d7e7bd5228ac59cff974cdc7c7c78d1a
-  data.tar.gz: d61dcd5186a1c52bf1a259fa3146bc6e3ca6701459785a3854d5a2e3c430bb1ded81267ad90fba4ee345531f6822baee5ebe2535d848bc3d5f19d7d4ddaa49f6
+  metadata.gz: 431c803fc5c99f82c513734225fb7cf37f039c7dae5da6f85b130e778cf1b2f2b34bd37a5288b86ad2fd8a2c6861441263e6313501dc484d88ca15b80ece156f
+  data.tar.gz: 873c8f40bc8c08970258106903cccc04e18b758a5af3a3cd48bf685c909f9cefe3231c4570c64fe87f96bb4e74aced28883e5be2174d3f50683905a731459549

data/TODO.parity-ee/19-architecture.md

@@ -0,0 +1,97 @@
+# 19 — End-state architecture: Expressir in Rust, bound to metanorma via lutaml-model
+
+**Question (2026-09-23): should Expressir ultimately be in Rust, bound to
+metanorma via lutaml-model?**
+
+**Answer: yes as a direction, no as a rewrite.** The end state is
+*Rust engine, Ruby presentation* — reached in three milestones, not by
+porting everything at once. The binding to metanorma goes through
+lutaml-model's Liquid Drop layer, which is exactly the seam that
+already works (expressir#255, #319, #421, #426).
+
+## Current state (v2.4.9, measured)
+
+```
+.exp ──► parsanol-rs 0.8.8 (parse, wire tree)
+          └─► expressir_core ext (batch, EXSCS1 artifact, warm start)
+                └─► instantiate → Expressir::Model (lutaml-model, Ruby)
+                      └─► Checker / Shtolo / formatters (Ruby semantics)
+                            └─► Liquid Drops ──► metanorma / suma
+```
+
+- Parse is Rust and fast (SRL cold 36 s CPU, SMRL 558 s CPU / 1448
+  schemas); parsanol 0.8.8's selective memoization is adopted.
+- The measured floor is now the **wire→Ruby-object hydration**
+  (71 % of warm CPU) and the overlays (29 %). Rendering (metanorma)
+  touches a small fraction of the model.
+- Semantics (Checker subset, SHTOLO with the 50-module eengine
+  differential, PrettyFormatter, SMRL XML) live in Ruby, eeng-parity
+  validated, with Ruby CI harnesses.
+
+## Options considered
+
+**A — Status quo+.** Rust keeps parse+artifact; all new semantics
+(Checker catalog, mapping validation) land in Ruby. No rewrite risk,
+parity harnesses stay, but the hydration floor remains and the Rust
+surface stays private.
+
+**B — Lazy binding (the recommended milestone).** Keep the Ruby model
+as the canonical object model for feature work, and add a READ path
+that serves metanorma directly from Rust-held compiled-set data:
+expressir_core exposes accessors over the loaded set (schema by id,
+kind listings, field reads), and a lutaml-model Drop proxies them on
+demand — rendering never instantiates the full object graph.
+lutaml-model keeps ownership of mappings/serialization/Drop
+integration; the Drop becomes the literal "Expressir in Rust bound to
+metanorma via lutaml-model".
+
+**C — Full Rust Expressir.** Model + semantics in Rust; Ruby keeps
+only Drops and CLI glue; publish expressir-rs on crates.io; unlock
+non-Ruby consumers (standalone binary, WASM). Rejected for now: it
+re-implements eeng-validated semantics with parity risk, doubles CI,
+and metanorma still needs a Drop/FFI layer — the boundary cost moves,
+it does not disappear.
+
+**Decision: B now; C deferred behind explicit triggers**
+(a non-Ruby consumer materializes; the Ruby semantic layers become a
+measured bottleneck — they are not today; or crate publishing is
+wanted for supply-chain reasons independent of consumers).
+
+## Milestones
+
+### M1 — rev-pinned core dependency (supply chain, one line)
+The ext pins `expressir-rs = { git …, branch = "main" }`; a released
+gem can pick up a shifted core. Pin `rev = "<sha>"` per expressir
+release (2.4.8 ↔ today's expressir-rs main), bump per release.
+
+### M2 — lazy Drops for the metanorma hot path
+1. expressir-rs: read accessors over the loaded compiled set
+   (`schema(id)`, kind listings, per-declaration field reads,
+   remark/refs overlay application in place).
+2. expressir_core: magnus wrappers returning primitives.
+3. Ruby: `LazySchemaDrop` (lutaml-model #319 custom Drop, the same
+   seam as #255/#426) implementing id / declaration listings / source
+   faces via the accessors.
+4. Gates: suma-style schema-doc pages render output-comparable with
+   the eager path on a fixture corpus; warm page-render benchmark;
+   SMRL warm-start CPU target ≤ 30 % of today's 128 s for the
+   read-only path.
+
+### M3 — semantics placement rule (standing)
+New semantic features land in Ruby **unless** they are
+parse-adjacent or graph-scale (then Rust in expressir-rs). The
+Checker catalog (12) and mapping validation (18) stay Ruby; SHTOLO
+stays Ruby with its eengine differential as the gate.
+
+### M4 — trigger review for option C
+Reassess on any of: a non-Ruby consumer request; hydrate/overlay CPU
+re-emerging as the top cost after M2; a crates.io publication ask.
+M2's accessors are exactly the seed of C's public API — nothing in B
+is throwaway.
+
+## What lutaml-model owns in every option
+
+The mapping DSL (yaml/json faces — incl. the #88 `mapping.yaml`
+reader), Liquid integration (#319 Drops), artifact serialization —
+the binding contract between the engine and metanorma. Rust never
+duplicates it.

data/lib/expressir/express/checker.rb

@@ -93,6 +93,7 @@ module Expressir
         schema.entities.each { |e| check_entity(schema, e) }
         schema.types.each do |t|
           check_type(schema, t)
+          check_select_items(schema, t)
           check_where_rules(schema, t, t.where_rules)
         end
         schema.rules.each { |r| check_where_rules(schema, r, r.where_rules) }
@@ -208,7 +209,15 @@ module Expressir
 
           note!(:check_subtype_ref, :error, schema,
                 "ENTITY #{entity.id}: subtype '#{id}' not found", entity)
+          next unless declared_kind(schema, id) == :type
+
+          # eeng check-subtypeof-invalid: the name exists but is not
+          # an entity, so it cannot be a supertype.
+          note!(:check_subtypeof_invalid, :error, schema,
+                "ENTITY #{entity.id}: subtype '#{id}' is a TYPE, " \
+                "not an entity", entity)
         end
+        check_supertype_refs(schema, entity)
         check_subtype_cycles(schema, entity)
 
         check_where_rules(schema, entity, entity.where_rules)
@@ -261,6 +270,66 @@ module Expressir
         cyclic
       end
 
+      # eeng check-supertype-ref: every entity named in the SUPERTYPE
+      # expression must resolve to an entity of this schema.
+      def check_supertype_refs(schema, entity)
+        expr = entity.supertype_expression
+        return unless expr
+
+        each_reference(expr) do |ref|
+          id = ref.id
+          next if id.nil? || find_entity(schema, id)
+
+          note!(:check_supertype_ref, :error, schema,
+                "ENTITY #{entity.id}: supertype '#{id}' not found", entity)
+        end
+      end
+
+      def each_reference(node, &)
+        case node
+        when Model::References::SimpleReference then yield node
+        when Model::ModelElement
+          node.class.attributes.each_key do |attr|
+            next if Model::ModelElement::SKIP_ATTRIBUTES.include?(attr) || attr == :parent
+
+            value = node.public_send(attr)
+            case value
+            when Array then value.each { |item| each_reference(item, &) }
+            when Model::ModelElement then each_reference(value, &)
+            end
+          end
+        end
+      end
+
+      # eeng check-select-named-type: SELECT items must name types.
+      def check_select_items(schema, type)
+        return unless type.underlying_type.is_a?(Model::DataTypes::Select)
+
+        Array(type.underlying_type.items).each do |item|
+          next if item.id.nil?
+          next if find_type(schema, item.id)
+          next if find_entity(schema, item.id).nil? # unresolved handled elsewhere
+
+          note!(:check_select_named_type, :error, schema,
+                "TYPE #{type.id}: select item '#{item.id}' is an entity, " \
+                "not a named type", type)
+        end
+      end
+
+      def find_type(schema, id)
+        key = id.safe_downcase
+        return schema.types.find { |t| t.id.safe_downcase == key } if schema
+
+        nil
+      end
+
+      # eeng declared-kind probe: :type when a TYPE of that name is
+      # visible in the schema (used to separate not-found from
+      # not-an-entity in subtype diagnostics).
+      def declared_kind(schema, id)
+        :type if find_type(schema, id)
+      end
+
       def check_where_rules(schema, owner, rules)
         Array(rules).each do |wr|
           next if wr.id.nil? || wr.id.match?(WHERE_LABEL)

data/lib/expressir/express/xsd.rb

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+module Expressir
+  module Express
+    # EXPRESS → XML Schema writer (expressir #276 step 2,
+    # TODO.parity-ee 15) — the express2xsd counterpart, built on
+    # moxml.
+    #
+    # Mapping (v1):
+    #   ENTITY      → xs:element + xs:complexType (explicit attributes
+    #                 in declaration order); SUBTYPE OF a schema-local
+    #                 entity becomes a substitutionGroup reference.
+    #   ENUMERATION → xs:simpleType with enumeration facets.
+    #   SELECT      → xs:element of xs:anyType (full Part 28 select
+    #                 mapping is a documented v1 limitation).
+    #   defined TYPE over a simple type → xs:simpleType restriction.
+    #   attributes  → xs:element inside the complexType sequence;
+    #                 LIST/SET/BAG/ARRAY → maxOccurs="unbounded".
+    #
+    # Built-in EXPRESS types map to XML Schema built-ins (STRING→
+    # xs:string, INTEGER→xs:integer, REAL/NUMBER→xs:double,
+    # BOOLEAN→xs:boolean, LOGICAL→string restriction, BINARY→
+    # xs:hexBinary).
+    module Xsd
+      BUILTIN_TYPES = {
+        "string" => "xs:string",
+        "integer" => "xs:integer",
+        "real" => "xs:double",
+        "number" => "xs:double",
+        "boolean" => "xs:boolean",
+        "binary" => "xs:hexBinary",
+      }.freeze
+
+      AGGREGATES = %w[array list set bag].freeze
+
+      module_function
+
+      # @param schema [Model::Declarations::Schema]
+      # @return [String] the serialized xs:schema document
+      def format(schema)
+        ctx = Moxml::Context.new
+        doc = ctx.create_document
+        root = doc.create_element("xs:schema")
+        root.set_attributes("xmlns:xs" => "http://www.w3.org/2001/XMLSchema",
+                            "elementFormDefault" => "qualified")
+        doc.add_child(root)
+        schema.types.to_a.each { |type| root.add_child(type_node(doc, schema, type)) }
+        schema.entities.to_a.each do |entity|
+          root.add_child(entity_element(doc, schema, entity))
+          root.add_child(entity_type(doc, schema, entity))
+        end
+        doc.to_xml
+      end
+
+      def type_node(doc, schema, type)
+        underlying = type.underlying_type
+        node = doc.create_element("xs:simpleType")
+        node.set_attributes("name" => type.id)
+        restriction = doc.create_element("xs:restriction")
+        node.add_child(restriction)
+        if underlying.is_a?(Expressir::Model::DataTypes::Enumeration)
+          restriction.set_attributes("base" => "xs:string")
+          underlying.items.to_a.each do |item|
+            facet = doc.create_element("xs:enumeration")
+            facet.set_attributes("value" => item.id)
+            restriction.add_child(facet)
+          end
+        else
+          restriction.set_attributes("base" => xs_type(underlying, schema))
+        end
+        node
+      end
+
+      def entity_element(doc, schema, entity)
+        node = doc.create_element("xs:element")
+        node.set_attributes("name" => entity.id)
+        node.set_attributes("type" => entity.id)
+        parent = supertype_name(schema, entity)
+        node.set_attributes("substitutionGroup" => parent) if parent
+        node
+      end
+
+      def entity_type(doc, schema, entity)
+        node = doc.create_element("xs:complexType")
+        node.set_attributes("name" => entity.id)
+        sequence = doc.create_element("xs:sequence")
+        node.add_child(sequence)
+        entity.attributes.to_a.each do |attr|
+          skip = attr.kind == Expressir::Model::Declarations::Attribute::INVERSE ||
+            (attr.respond_to?(:derive?) && attr.derive?)
+          next if skip
+
+          sequence.add_child(attribute_element(doc, schema, attr))
+        end
+        node
+      end
+
+      def attribute_element(doc, schema, attr)
+        node = doc.create_element("xs:element")
+        node.set_attributes("name" => attr.id)
+        node.set_attributes("type" => xs_type(attr.type, schema))
+        kind = attr.type.class.name.split("::").last.downcase
+        node.set_attributes("maxOccurs" => "unbounded") if AGGREGATES.include?(kind)
+        node
+      end
+
+      def xs_type(type, schema)
+        if type.is_a?(Expressir::Model::References::SimpleReference)
+          id = type.respond_to?(:base_path) && type.base_path ? base_id(type.base_path) : type.id
+          return schema.types.to_a.any? { |t| t.id.safe_downcase == id.safe_downcase } ? id : "xs:anyType"
+        end
+
+        if type.is_a?(Expressir::Model::DataTypes::Enumeration)
+          return type.id || "xs:string"
+        end
+
+        if type.respond_to?(:base_type)
+          inner = type.base_type
+          return xs_type(inner, schema) if inner
+
+          return "xs:anyType"
+        end
+
+        name = type.class.name.split("::").last.downcase
+        BUILTIN_TYPES.fetch(name, "xs:anyType")
+      end
+
+      def supertype_name(schema, entity)
+        ref = Array(entity.subtype_of).first
+        return nil unless ref
+
+        id = ref.is_a?(String) ? ref : ref.id
+        return nil unless id
+        return nil unless schema.entities.to_a.any? { |e| e.id.safe_downcase == id.safe_downcase }
+
+        schema.entities.find { |e| e.id.safe_downcase == id.safe_downcase }.id
+      end
+
+      def base_id(base_path)
+        parts = base_path.split(".")
+        parts[-1]
+      end
+    end
+  end
+end

data/lib/expressir/express.rb

@@ -45,6 +45,7 @@ module Expressir
     autoload :SchemaSourceFormatter, "#{__dir__}/express/schema_source_formatter"
     autoload :ScopeResolver, "#{__dir__}/express/scope_resolver"
     autoload :SmrlXml, "#{__dir__}/express/smrl_xml"
+    autoload :Xsd, "#{__dir__}/express/xsd"
     autoload :SourceFormatter, "#{__dir__}/express/source_formatter"
     autoload :StreamingBuilder, "#{__dir__}/express/streaming_builder"
   end

data/lib/expressir/mapping.rb

@@ -0,0 +1,141 @@
+# frozen_string_literal: true
+
+module Expressir
+  # MIM application-element mapping model (expressir#88,
+  # TODO.parity-ee 18): the typed reader for the `mapping.yaml` files
+  # each STEP module carries, and validation of the
+  # `<<express:SCHEMA.ITEM,ITEM>>` links they contain against a
+  # resolved ARM/MIM repository.
+  #
+  # The model is lutaml-model Serializable — the yaml face is the
+  # module corpus format (top-level `ae` + `sc` keys); fields the
+  # corpus carries that v1 does not model (alt_map, descriptions)
+  # are ignored by the mapping and survive round-trips untouched in
+  # the source files.
+  module Mapping
+    EXPRESS_LINK = /<<express:([^,>]+)(?:,([^>]+))?>>/
+
+    class ReferencePath < Lutaml::Model::Serializable
+      attribute :content, :string
+
+      key_value do
+        map "content", to: :content
+      end
+    end
+
+    class AttributeMapping < Lutaml::Model::Serializable
+      attribute :attribute, :string
+      attribute :aimelt, :string
+      attribute :inherited_from_entity, :string
+      attribute :assertion_to, :string
+      attribute :refpath, ReferencePath
+
+      key_value do
+        map "attribute", to: :attribute
+        map "aimelt", to: :aimelt
+        map "inherited_from_entity", to: :inherited_from_entity
+        map "assertion_to", to: :assertion_to
+        map "refpath", to: :refpath
+      end
+    end
+
+    class ApplicationElement < Lutaml::Model::Serializable
+      attribute :entity, :string
+      attribute :aimelt, :string
+      attribute :extensible, :string
+      attribute :original_module, :string
+      attribute :refpath, ReferencePath
+      attribute :aa, AttributeMapping, collection: true
+
+      key_value do
+        map "entity", to: :entity
+        map "aimelt", to: :aimelt
+        map "extensible", to: :extensible
+        map "original_module", to: :original_module
+        map "refpath", to: :refpath
+        map "aa", to: :aa
+      end
+    end
+
+    class SubtypeConstraintMapping < Lutaml::Model::Serializable
+      attribute :constraint, :string
+      attribute :entity, :string
+
+      key_value do
+        map "constraint", to: :constraint
+        map "entity", to: :entity
+      end
+    end
+
+    class Document < Lutaml::Model::Serializable
+      attribute :ae, ApplicationElement, collection: true
+      attribute :sc, SubtypeConstraintMapping, collection: true
+
+      key_value do
+        map "ae", to: :ae
+        map "sc", to: :sc
+      end
+    end
+
+    UnknownLink = Struct.new(:text, :schema, :item, keyword_init: true)
+
+    module_function
+
+    # @param path [String] path to a module mapping.yaml
+    # @return [Document]
+    def load_file(path)
+      Document.from_yaml(File.read(path))
+    end
+
+    # Every `<<express:...>>` link in the document, as
+    # [schema_id, item_id] pairs (duplicates preserved).
+    def links(document)
+      values(document).flat_map do |value|
+        value.to_s.scan(EXPRESS_LINK).map do |path, _id|
+          parts = path.split(".")
+          [parts[-2], parts[-1]]
+        end
+      end
+    end
+
+    # Links whose schema or item cannot be found in +repository+.
+    # A link with no item part resolves against schema presence only.
+    def unknown_links(document, repository)
+      known = repository.schemas.to_h do |schema|
+        [schema.id.safe_downcase, schema]
+      end
+      links(document).filter_map do |schema_id, item_id|
+        schema = known[schema_id&.safe_downcase]
+        if schema.nil?
+          next UnknownLink.new(text: "#{schema_id}.#{item_id}",
+                               schema: schema_id, item: item_id)
+        end
+        next if item_id.nil?
+
+        unless item_declared?(schema, item_id)
+          next UnknownLink.new(text: "#{schema_id}.#{item_id}",
+                               schema: schema_id, item: item_id)
+        end
+
+        nil
+      end
+    end
+
+    def item_declared?(schema, item_id)
+      key = item_id.safe_downcase
+      %i[types entities functions procedures rules constants
+         subtype_constraints].any? do |coll|
+        Array(schema.public_send(coll)).any? do |decl|
+          decl.respond_to?(:id) && decl.id&.safe_downcase == key
+        end
+      end
+    end
+
+    def values(document)
+      document.ae.to_a.flat_map do |element|
+        [element.entity, element.aimelt] +
+          element.aa.to_a.flat_map { |aa| [aa.attribute, aa.aimelt] }
+      end.compact
+    end
+  end
+end

data/lib/expressir/version.rb

@@ -3,6 +3,6 @@ module Expressir
   # autoload it (Ruby autoload only works for module/class constants, not
   # for string constants like `Expressir::VERSION`).
   module Version
-    VERSION = "2.4.9".freeze
+    VERSION = "2.4.10".freeze
   end
 end

data/lib/expressir.rb

@@ -29,6 +29,7 @@ module Expressir
   extend Config
 
   autoload :Version, "#{__dir__}/expressir/version"
+  autoload :Mapping, "#{__dir__}/expressir/mapping"
 
   # Error classes — autoload the same errors.rb file from each name.
   # The first reference triggers the load; the file defines all 21 classes.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: expressir
 version: !ruby/object:Gem::Version
-  version: 2.4.9
+  version: 2.4.10
 platform: ruby
 authors:
 - Ribose Inc.
@@ -318,6 +318,7 @@ files:
 - TODO.parity-ee/15-part28-xml.md
 - TODO.parity-ee/16-compare-patch.md
 - TODO.parity-ee/17-official-tests.md
+- TODO.parity-ee/19-architecture.md
 - TODO.parity-ee/parity-matrix.md
 - benchmark/srl_benchmark.rb
 - benchmark/srl_native_benchmark.rb
@@ -526,10 +527,12 @@ files:
 - lib/expressir/express/smrl_xml.rb
 - lib/expressir/express/source_formatter.rb
 - lib/expressir/express/streaming_builder.rb
+- lib/expressir/express/xsd.rb
 - lib/expressir/liquid.rb
 - lib/expressir/manifest.rb
 - lib/expressir/manifest/resolver.rb
 - lib/expressir/manifest/validator.rb
+- lib/expressir/mapping.rb
 - lib/expressir/model.rb
 - lib/expressir/model/cache.rb
 - lib/expressir/model/concerns.rb