lutaml-model 0.8.95 → 0.8.97

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c5933a3971f4f39e30541f504636641cd059e7a97751ba67a710203dbfdcc45a
-  data.tar.gz: c23b70227e80d99427b9d4125a09e33d9751112bc72a5e6abaca73d6a3897c60
+  metadata.gz: 2dcefe56e003b241812971d37fc9d76d6b6945cb5a138a9fa15f9ce2aa4fbd7d
+  data.tar.gz: 835dfecf768169c9fbc6ab772c2e11dbe593f3f5e9deb7016b09c8088706117c
 SHA512:
-  metadata.gz: 473b9970246c24172d6040a9c5062fad06d0fea413a5e04956ac98c0990324f2543668436c9259ad2a30b8647555d1e00102f6346afd3018814d68c2841ca8e5
-  data.tar.gz: f48b568bdc715a7649a262ca5757474f56ecc52d775121be99f18f067570d48d5401ff29cd7141ff768e9ea0a3e0aef5996d32129b8907c386cfd3c72e9f5370
+  metadata.gz: 7efe4b0bc24f20b9d48e37c33b5e23cf23af7397900a7d788912275a4b396318d916c721b21bc6373de53e99f3f4f0dc29e459c94bba2e2baaf8c4da67232c14
+  data.tar.gz: 59d5412393469b1904c4c6e29bc0ff58b0c6333b5e3710e531d9e7a8ae465e7762993288f04e6147a12a204a8fb581059a24491d9dc90d7f309e78b1bb39b97a

data/docs/_guides/grammar-backed-formats.adoc

@@ -0,0 +1,69 @@
+= Grammar-backed string formats
+
+A serialization format whose input is a *grammar*, not a data structure:
+identifier strings, query languages, DSLs — text that is parsed, not
+decoded. lutaml-model supports registering these as first-class formats
+through the same `FormatRegistry` extension point as xml/json/yaml; the
+parse side is implemented by a grammar engine (the reference
+implementation is https://github.com/parsanol/parsanol-ruby[Parsanol]'s
+`Parsanol::PARG::Lutaml.register`, which consumes a compiled PARG
+artifact plus its bindings).
+
+== The contract
+
+A grammar-backed format needs three things:
+
+* a *parse* — text → attribute hash (the grammar engine's job)
+* *bindings* — capture names → model attributes (the artifact's
+  bindings section)
+* a *model class* — a regular `Serializable` with attributes; no
+  hand-written `from_h`/`to_h` anywhere
+
+== Reference implementation (Parsanol)
+
+[source,ruby]
+----
+require "parsanol/parg/lutaml"
+
+Parsanol::PARG::Lutaml.register(
+  IsoIdentifier,
+  format_name: :pubid_iso,
+  artifact: "artifacts/iso.json", # compiled grammar + bindings + tests
+  entry: "identifier",
+)
+
+IsoIdentifier.from_pubid_iso("ISO/IEC 12345-1:2020")
+# => #<IsoIdentifier number: "12345-1", year: 2020, ...>
+----
+
+What `register` does:
+
+. loads the checksummed artifact (parse failures raise the engine's
+  error type — register it via `error_types:` so users can rescue it);
+. registers the format through `FormatRegistry.register` with an
+  adapter whose `parse` is artifact-parse-then-bind;
+. defines `from_<format>` on the model class.
+
+The artifact is the parser of record: it carries the grammar, the
+bindings, and the conformance tests (the grammar's own test block is
+baked in and can be re-run by consumers), and its checksum pins the
+pinned version of the language.
+
+== Writing the grammar side
+
+Grammars are authored in PARG (`.parg` — ABNF-flavoured surface, PEG
+semantics, one rule per line) and compiled with the `parsanol` CLI:
+
+----
+parsanol compile grammars/iso.parg --tables tables
+----
+
+The compile-time lint rejects left recursion and shadowed
+alternatives; the embedded test block runs at compile time, so a
+grammar that stops accepting its documented inputs fails the build
+rather than the consumer.
+
+== See also
+
+* parsanol-ruby `docs/pg-language-guide.md` — the PARG language
+* `Parsanol::PARG::Lutaml.register` — the bridge (reference implementation)

data/docs/_guides/index.adoc

@@ -26,6 +26,7 @@ Task-oriented guides for accomplishing specific goals with Lutaml::Model.
 == Advanced mapping
 
 * link:../advanced-mapping[Advanced Attribute Mapping] - Same-name dispatch on attribute values (`when_attribute`), grouped declarations, custom methods, delegation
+* link:../grammar-backed-formats[Grammar-Backed String Formats] - Register a grammar engine as a serialization format (identifier strings, DSLs)
 
 == Performance
 
@@ -47,6 +48,7 @@ See link:../references/rdf-namespaces[RDF Namespaces] for namespace classes,
 * link:../value-transformations[Value Transformations] - Transform values during serialization
 * link:../missing-values-handling[Missing Values Handling] - Handle nil/empty/omitted
 * link:../advanced-mapping[Advanced Mapping] - Custom methods and delegation
+* link:../grammar-backed-formats[Grammar-Backed String Formats] - Grammar engines as serialization formats
 * link:../schema-generation[Schema Generation] - Generate schemas from models
 * link:../creating-xsd[Creating XSD Schemas] - Complete reference for XSD generation from LutaML models
 * link:../schema-import[Schema Import] - Import schemas to create models

data/lib/lutaml/model/serialize/attribute_definition.rb

@@ -8,6 +8,24 @@ module Lutaml
       # Extracted from serialize.rb to improve code organization.
       # Provides methods for defining and validating model attributes.
       module AttributeDefinition
+        # Whether the per-class accessors and the state-defaults seed are
+        # compiled from Ruby source with a String class_eval (the fast
+        # path). Opal has no runtime compiler unless opal-parser is
+        # bundled, so there every such method is defined from a block
+        # with define_method instead.
+        # Core dispatch for the block-form methods, bound per call so an
+        # attribute or method a model defines with one of these names
+        # (send, __send__, instance_variable_get, const_get, ...) is never invoked;
+        # the compiled form reads ivars and calls methods directly.
+        IVAR_GET = ::Kernel.instance_method(:instance_variable_get)
+        IVAR_SET = ::Kernel.instance_method(:instance_variable_set)
+        SEND = ::BasicObject.instance_method(:__send__)
+        CONST_GET = ::Module.instance_method(:const_get)
+
+        def self.source_compilation?
+          !Lutaml::Model.opal?
+        end
+
         # Define attribute methods on the model class
         #
         # @param attr [Attribute] The attribute to define methods for
@@ -44,6 +62,26 @@ module Lutaml
             return
           end
 
+          unless AttributeDefinition.source_compilation?
+            seeds = attrs.map do |name, attr|
+              sentinel = if attr.collection?
+                           Lutaml::Model::Serialize::LAZY_EMPTY_COLLECTION
+                         else
+                           Lutaml::Model::UninitializedClass.instance
+                         end
+              [:"@#{name}", sentinel]
+            end
+            # Raw readers for every attribute the register sees, as the
+            # compiled form below does (#933).
+            attrs.each_key { |name| define_raw_state_reader(name) }
+            define_method(method_name) do
+              # Returns the last seeded value, as the compiled form's final
+              # assignment does.
+              seeds.reduce(nil) { |_, (ivar, sentinel)| IVAR_SET.bind_call(self, ivar, sentinel) }
+            end
+            return
+          end
+
           lines = attrs.map do |name, attr|
             if attr.collection?
               "@#{name} = Lutaml::Model::Serialize::LAZY_EMPTY_COLLECTION"
@@ -82,6 +120,14 @@ module Lutaml
           RUBY
         end
 
+        # Block-form raw-state reader (no String eval), defined on self.
+        def define_raw_state_reader(name)
+          ivar = :"@#{name}"
+          define_method(:"__lutaml_raw_#{name}") do
+            IVAR_GET.bind_call(self, ivar)
+          end
+        end
+
         # Historical getter shape for punctuation-named attributes and
         # any name the compiled form cannot express.
         def define_reflective_attribute_methods(name, attr)
@@ -145,11 +191,16 @@ module Lutaml
           # the state-defaults compilation) so late declarations get it
           # too -- XMI's extension loader declares attributes after
           # first parse.
-          if name.to_s.match?(PLAIN_NAME)
-            # class_eval on self: a bare `model` receiver resolves
-            # through the host class's own namespace (omml's Base
-            # answers a Store model) and defined the reader on the
-            # wrong object (#933).
+          # Without a source compiler, compile_state_defaults! defines
+          # its raw readers from blocks; define one for every name here
+          # too (define_method takes any name, not only plain ones).
+          # Both forms define on self: a bare `model` receiver resolves
+          # through the host class's own namespace (omml's Base answers
+          # a Store model) and defined the reader on the wrong object
+          # (#933).
+          if !AttributeDefinition.source_compilation?
+            define_raw_state_reader(name)
+          elsif name.to_s.match?(PLAIN_NAME)
             class_eval(<<~RUBY, __FILE__, __LINE__ + 1) # rubocop:disable Style/DocumentDynamicEvalDefinition
               def __lutaml_raw_#{name}
                 @#{name}
@@ -263,7 +314,9 @@ module Lutaml
           # Array on EVERY read — the largest per-call allocation source
           # on instance-heavy parses (TODO.max-perf/06). The optional-arg
           # form allocates nothing, and the read is a direct @ivar.
-          if attr.collection?
+          if !AttributeDefinition.source_compilation?
+            define_block_attribute_getter(name, attr)
+          elsif attr.collection?
             # class_eval interpolates, e.g.:
             #   def items(arg = Lutaml::Model::Serialize::NO_ARG)
             #     if arg.equal?(Lutaml::Model::Serialize::NO_ARG)
@@ -319,7 +372,9 @@ module Lutaml
                            end
           define_method(attr_reader_method) { attr } unless reader_defined
 
-          if attr.collection?
+          if !AttributeDefinition.source_compilation?
+            define_block_attribute_setters(name, attr)
+          elsif attr.collection?
             # class_eval interpolates, e.g.:
             #   def items=(value)
             #     value_set_for(:items)
@@ -379,6 +434,77 @@ module Lutaml
           end
         end
 
+        # Block form of the compiled getter in define_regular_attribute_methods,
+        # for runtimes without a source compiler (see source_compilation?).
+        def define_block_attribute_getter(name, attr)
+          name = name.to_sym # the compiled form passes :name literals
+          ivar = :"@#{name}"
+          if attr.collection?
+            # The compiled getter resolves Array in the model class's scope.
+            scope = self
+            define_method(name) do |arg = Lutaml::Model::Serialize::NO_ARG|
+              if arg.equal?(Lutaml::Model::Serialize::NO_ARG)
+                materialize_lazy_collection(name)
+              else
+                current = IVAR_GET.bind_call(self, ivar) || []
+                new_value = current.is_a?(CONST_GET.bind_call(scope, :Array)) ? current + [arg] : arg
+                IVAR_SET.bind_call(self, ivar, new_value)
+                record_mutation(name, arg)
+                arg
+              end
+            end
+          else
+            define_method(name) do |arg = Lutaml::Model::Serialize::NO_ARG|
+              if arg.equal?(Lutaml::Model::Serialize::NO_ARG)
+                IVAR_GET.bind_call(self, ivar)
+              else
+                public_send(:"#{name}=", arg)
+                arg
+              end
+            end
+          end
+        end
+
+        # Block form of the compiled setters in define_regular_attribute_methods.
+        # The Attribute is read through the same hidden accessor the
+        # compiled form calls, not captured: a setter redefined over an
+        # enum shorthand keeps casting with the accessor's Attribute.
+        def define_block_attribute_setters(name, attr)
+          name = name.to_sym # the compiled form passes :name literals
+          ivar = :"@#{name}"
+          handle = :"__attribute_definition_#{name}"
+          if attr.collection?
+            assign = lambda do |model, value|
+              current = IVAR_GET.bind_call(model, ivar)
+              unless current.equal?(Lutaml::Model::Serialize::LAZY_EMPTY_COLLECTION) &&
+                  (value.nil? || Lutaml::Model::Utils.uninitialized?(value))
+                IVAR_SET.bind_call(model, ivar, value)
+              end
+              SEND.bind_call(model, :record_mutation_collection, name, value)
+            end
+            define_method(:"#{name}=") do |value|
+              value_set_for(name)
+              assign.call(self, SEND.bind_call(self, handle).cast_value(value, lutaml_register))
+            end
+            define_method(:"__assign_parsed_#{name}=") do |value|
+              value_set_for(name)
+              assign.call(self, value)
+            end
+          else
+            define_method(:"#{name}=") do |value|
+              value_set_for(name)
+              value = SEND.bind_call(self, handle).cast_value(value, lutaml_register)
+              IVAR_SET.bind_call(self, ivar, value)
+              record_mutation(name, value)
+            end
+            define_method(:"__assign_parsed_#{name}=") do |value|
+              value_set_for(name)
+              IVAR_SET.bind_call(self, ivar, value)
+              record_mutation(name, value)
+            end
+          end
+        end
+
         # Define an attribute for the model
         #
         # @param name [Symbol] The attribute name

data/lib/lutaml/model/serialize/format_conversion.rb

@@ -95,7 +95,12 @@ module Lutaml
             options.key?(:mappings) || options.key?(:adapter) ||
             options.key?(:_adapter_override) ||
             options.key?(:indent) || options.key?(:xml_declaration) ||
-            options.key?(:declaration) || options.key?(:doctype)
+            options.key?(:declaration) || options.key?(:doctype) ||
+            # prefix requests ride the interpretive writer's use_prefix
+            # option (planner Tier-1) — the plan emission has no
+            # option-aware spelling. The instance-level option prep
+            # rewrites :prefix to :use_prefix before this gate.
+            options.key?(:prefix) || options.key?(:use_prefix)
 
           adapter_name = Lutaml::Model::Config.adapter_for(:xml)
           adapter_name &&= adapter_name.name
@@ -103,9 +108,14 @@ module Lutaml
 
           register = Lutaml::Model::Config.default_register
           plan = Lutaml::Xml::Leptris::PlanCompiler.compile(self, register)
-          return nil unless plan && !plan[:namespaced] &&
+          return nil unless plan &&
             Lutaml::Xml::Leptris::PlanSerializer.serializable?(plan)
 
+          # Namespaced plans ride the plan serializer too: the DOM
+          # emission spells prefixed names and hoists declarations
+          # (the #847 xmlns drop is fixed in PlanSerializer); any
+          # unspellable shape answers nil there and falls back
+          # interpretively.
           Lutaml::Xml::Leptris::PlanSerializer.call(instance, plan)
         end
 
@@ -128,6 +138,7 @@ module Lutaml
           register = Lutaml::Model::Config.default_register
           plan = Lutaml::Xml::Leptris::PlanCompiler.compile(self, register)
           return nil unless plan
+          return nil unless Lutaml::Xml::Leptris::PlanCompiler.engine_walk_safe?(plan)
 
           root = ::Leptris::XML.parse(data.to_s).root
           return nil if root.nil?

data/lib/lutaml/model/serialize/initialization.rb

@@ -172,12 +172,30 @@ module Lutaml
         #
         # @param register [Symbol, nil] The register context
         def ensure_imports!(register = nil)
-          ensure_model_imports!(register)
-          ensure_choice_imports!(register)
-          ensure_restrict_attributes!(register)
-          # Hook for format-specific mapping import resolution.
-          # XML overrides this to call mappings[:xml]&.ensure_mappings_imported!(register)
-          ensure_format_mapping_imports!(register)
+          # Bounded re-entry for the resolution chain. Each leg's
+          # record_import_attempt tail calls attributes(), which calls
+          # back into ensure_imports!; nested passes converge imports
+          # whose targets only become resolvable after an earlier leg
+          # (so they must not be dropped), but with a permanently
+          # pending import the tail re-entered unboundedly and crashed
+          # with SystemStackError (#938). Two levels allow one nested
+          # convergence pass; deeper re-entry is suppressed per
+          # register and the counter unwinds on error.
+          register_id = extract_register_id(register)
+          (@ensure_imports_depth ||= {})[register_id] ||= 0
+          return if @ensure_imports_depth[register_id] >= 2
+
+          @ensure_imports_depth[register_id] += 1
+          begin
+            ensure_model_imports!(register)
+            ensure_choice_imports!(register)
+            ensure_restrict_attributes!(register)
+            # Hook for format-specific mapping import resolution.
+            # XML overrides this to call mappings[:xml]&.ensure_mappings_imported!(register)
+            ensure_format_mapping_imports!(register)
+          ensure
+            @ensure_imports_depth[register_id] -= 1
+          end
         end
 
         # Hook for format-specific mapping import resolution.
@@ -357,7 +375,7 @@ module Lutaml
           # chain's frames per level pushed that recursion past the
           # stack limit — models without pending deferred imports skip
           # it entirely.
-          ensure_imports!(register_id) if finalized? && deferred_imports_pending?
+          ensure_imports!(register_id) if finalized? && deferred_imports_pending?(register_id)
           instance = allocate
           instance.finalize_deserialization(register_id)
           instance

data/lib/lutaml/model/serialize/model_import.rb

@@ -162,11 +162,39 @@ module Lutaml
         # @param register_id [Symbol, nil] The register context
         # Whether deferred (symbol-form) imports are still pending —
         # gates the allocation-time ensure chain (#933).
-        def deferred_imports_pending?
-          importable_models.any? { |_method, models| models.any? } ||
-            importable_choices.any? do |_choice, imports|
-              imports.any? { |_method, models| models.any? }
+        # The attribute-set fingerprint at the moment an import
+        # resolution attempt ran — deferred_imports_pending? compares
+        # against it to skip re-resolution on stable sets.
+        def record_import_attempt(register_id)
+          # Raw stores only — the lazy attributes() re-enters
+          # ensure_imports! and, when @models_imported was just set to
+          # false (unresolved imports), cycles through this very
+          # method forever (the uniword register_in SystemStackError).
+          register_attrs = (@register_records ||= {})[register_id]
+          (@import_attempt_fingerprints ||= {})[register_id] =
+            @attributes.size + @choice_attributes.to_a.size +
+            (register_attrs ? register_attrs[:attributes].to_a.size : 0)
+        end
+
+        def deferred_imports_pending?(register_id = nil)
+          register_id ||= Lutaml::Model::Config.default_register
+          has_importables = importable_models.any? { |_m, models| models.any? } ||
+            importable_choices.any? do |_c, imports|
+              imports.any? { |_m, models| models.any? }
             end || restrict_attributes.any?
+          return false unless has_importables
+
+          # Unresolvable imports stay pending forever; re-running the
+          # ensure chain (and its state-defaults invalidation) on every
+          # allocation thrashed import-heavy models quadratically.
+          # Resolution re-arms only when the attribute set CHANGED
+          # since the last attempt — a late declaration after a failed
+          # resolution re-attempts; a stable set skips.
+          attempted = (@import_attempt_fingerprints ||= {})[register_id]
+          return true if attempted.nil?
+
+          current = attributes(register_id).size + @choice_attributes.to_a.size
+          current != attempted
         end
 
         def ensure_model_imports!(register_id = nil)
@@ -200,6 +228,7 @@ module Lutaml
           end
 
           @models_imported[register_id] = all_resolved
+          record_import_attempt(register_id)
         end
 
         # Ensure all choice imports are resolved for a specific register
@@ -239,6 +268,7 @@ module Lutaml
           end
 
           @choices_imported[register_id] = true if all_resolved
+          record_import_attempt(register_id)
           # Choice imports can add attributes without passing through
           # define_attribute_methods — recompile state defaults so the
           # seeding and raw readers cover them (#933). Only when an
@@ -268,6 +298,11 @@ module Lutaml
         # Uses TracePoint to detect when a class definition is complete,
         # allowing deferred import resolution.
         def setup_trace_point
+          # Opal has no TracePoint. Symbol imports still resolve lazily
+          # on first use (ensure_imports!), so there the class is marked
+          # finalized at once instead of at the end of its body.
+          return @finalized = true if Lutaml::Model.opal?
+
           @trace ||= TracePoint.new(:end) do |_tp|
             if include?(Lutaml::Model::Serialize)
               @finalized = true

data/lib/lutaml/model/version.rb

@@ -2,6 +2,6 @@
 
 module Lutaml
   module Model
-    VERSION = "0.8.95"
+    VERSION = "0.8.97"
   end
 end

data/lib/lutaml/model.rb

@@ -294,3 +294,10 @@ require "#{__dir__}/xml/format"
 # Builder must be prepended AFTER XML so its initialize runs first
 # (Builder -> XML InstanceMethods -> Serialize)
 Lutaml::Model::Serialize.prepend(Lutaml::Model::Serialize::Builder)
+
+# Opal does not propagate a module prepended into an already-included
+# module to classes that included it earlier (see lib/lutaml/xml/format.rb).
+# Serializable included Serialize at boot, so Builder goes onto it directly.
+if Lutaml::Model.opal?
+  Lutaml::Model::Serializable.prepend(Lutaml::Model::Serialize::Builder)
+end

data/lib/lutaml/xml/format.rb

@@ -243,9 +243,11 @@ Lutaml::Model::Serialize.prepend(
 #
 # Anonymous model classes (Class.new { include Serialize }) extend
 # Serialize::ClassMethods rather than inheriting from Serializable, so the
-# ModelImportExt override (root?) must land on ClassMethods too. The other
-# two (FormatConversion, InstanceMethods) are already prepended unconditionally
-# above, so Opal's no-double-prepend rule means we don't repeat them here.
+# ModelImportExt override (root?) must land on ClassMethods too.
+#
+# On Opal (and TruffleRuby, below) the ClassMethods.prepend(FormatConversion)
+# above does not reach Serializable's singleton chain, which extended
+# ClassMethods before it ran, so it is prepended onto that singleton directly.
 #
 # TruffleRuby (34.0.1) has the same propagation gap, on both the include
 # side and the extend/singleton side — a module prepended into an
@@ -273,13 +275,9 @@ if Lutaml::Model.opal? || RUBY_ENGINE == "truffleruby"
     Lutaml::Xml::Serialization::ModelImportExt,
   )
 
-  # TruffleRuby only: Opal's no-double-prepend rule makes the unconditional
-  # ClassMethods.prepend(FormatConversion) above sufficient there.
-  if RUBY_ENGINE == "truffleruby"
-    Lutaml::Model::Serializable.singleton_class.prepend(
-      Lutaml::Xml::Serialization::FormatConversion,
-    )
-  end
+  Lutaml::Model::Serializable.singleton_class.prepend(
+    Lutaml::Xml::Serialization::FormatConversion,
+  )
 end
 
 # Register XML-specific attribute override warning names