lutaml-model 0.6.7 → 0.7.2

data/RELEASE_NOTES.adoc

@@ -0,0 +1,346 @@
+= Release Notes: v0.6.7 to v0.7.1
+
+== Overview
+
+This release introduces significant enhancements to the lutaml-model library, focusing on improved handling of missing values, comprehensive polymorphic model support, and important bug fixes. This document outlines the key changes and provides guidance for upgrading your codebase.
+
+== Key changes
+
+=== Missing values handling family
+
+A comprehensive set of features for handling the "missing values" family (empty,
+non-existent, and undefined values) across different serialization formats has
+been introduced.
+
+WARNING: The default behavior for handling uninitialized collection attributes
+is changed from before.
+
+In version 0.6.7, an uninitialized collection attribute in Lutaml::Model is
+represented as an empty array (`[]`).
+
+In version 0.7.1, an uninitialized collection attribute in Lutaml::Model is
+represented as `nil`.
+
+This change is made to ensure consistency across different serialization formats
+and to provide a predictable and consistent behavior based on explicit
+declarations when dealing with missing values.
+
+Previously, Lutaml::Model had a limited ability to handle the missing value types,
+for example, the appearance and round-trip behavior of the blank XML element
+is uncertain. Lutaml::Model also did not support XML `nil` and YAML `null` values
+in a consistent manner, nor the inconsistency where TOML does not support
+the notion of `nil` or `null` values.
+
+This change may break existing code that relies on the previous behavior. If you
+were using uninitialized collection attributes in your code, you may need to
+update your code to handle the new behavior.
+
+There is now a series of features to support the missing values family, including:
+
+* Attribute-level `initialize_empty` option, which controls the default
+initialization behavior of collection attributes. Previously, collection
+attributes defaulted to an empty array (`[]`) when uninitialized. The default
+is now the value `nil`.
++
+Revert to legacy behavior: You can return to the previous behavior by setting
+`initialize_empty: true` in the attribute definition.
+
+* Mapping-rule-level `render_nil` and `render_empty` options have been revamped
+into accepting additional options that are used to control how nil and empty
+values are parsed from and rendered in serialization formats.
++
+The behavior of `render_nil: true` and `render_empty: true` is now equivalent to
+`render_nil: :as_empty` and `render_empty: :as_empty`, respectively.
+It is strongly recommended that the legacy behavior be replaced with the new
+options.
++
+The reason being that legacy behavior breaks round-trips between conversions of
+the same format. In the case of `render_nil: true`, a model value of `nil` is
+serialized into an XML blank element, but when the XML blank element is
+deserialized, it is deserialized into an empty string (`""`, as per the W3C XML
+standard), not `nil`.
+
+* Mapping-rule-level `value_map` option, which provides fine-grained control
+over how missing values are handled during serialization and deserialization.
+
+* Support for a new `Uninitialized` value at the `Lutaml::Model` level, which is
+used to represent uninitialized attributes.
++
+This means that if you have a model with an attribute that is not set, it will
+be represented as `Uninitialized` instead of `nil`. This is useful for
+distinguishing between an attribute that is not set and an attribute that is
+set to `nil`, and as a result will preserve the round-trip behavior of
+serialization formats.
+
+* Support for `nil` values in JSON and YAML, and XML serialization formats.
++
+The `nil` value is now supported in JSON, YAML and XML serialization formats.
+This means that if you have a model with an attribute that is set to `nil`, it
+will be serialized as `null` in JSON/YAML and as an empty element in XML. This
+is useful for representing missing values in a consistent manner across
+different serialization formats.
+
+
+
+==== `initialize_empty` option
+
+The `initialize_empty` option controls the default initialization behavior of
+collection attributes:
+
+[source,ruby]
+----
+# Default to `nil`
+class SomeModel < Lutaml::Model::Serializable
+  attribute :coll, :string, collection: true
+
+  xml do
+    root "some-model"
+    map_element 'collection', to: :coll
+  end
+end
+puts SomeModel.new.coll  # => nil
+
+# Default to empty array
+class SomeModel < Lutaml::Model::Serializable
+  attribute :coll, :string, collection: true, initialize_empty: true
+
+  xml do
+    map_element 'collection', to: :coll
+  end
+end
+puts SomeModel.new.coll  # => []
+----
+
+==== `value_map` option
+
+The `value_map` option provides fine-grained control over how missing values are handled during serialization and deserialization. This is especially useful when different serialization formats represent missing values differently.
+
+[source,ruby]
+----
+class ExampleClass < Lutaml::Model::Serializable
+  attribute :status, :string
+
+  xml do
+    map_element 'status', to: :status, value_map: {
+      from: { empty: :nil, omitted: :omitted, nil: :nil },
+      to: { empty: :nil, omitted: :omitted, nil: :nil }
+    }
+  end
+
+  json do
+    map 'status', to: :status, value_map: {
+      from: { empty: :nil, omitted: :omitted, nil: :nil },
+      to: { empty: :nil, omitted: :omitted, nil: :nil }
+    }
+  end
+
+  toml do
+    map 'status', to: :status, value_map: {
+      from: { empty: :nil, omitted: :omitted },
+      to: { empty: :nil, omitted: :omitted, nil: :omitted }
+    }
+  end
+end
+----
+
+For collection attributes, the value_map behaves differently depending on the `initialize_empty` setting.
+
+==== `render_nil` and `render_empty` modes
+
+These options provide shorthand methods to control how nil and empty values are rendered in serialization formats:
+
+* `render_nil: true | :as_empty | :as_blank | :nil | :omit` - Controls how nil values are rendered
+* `render_empty: :as_empty | :as_blank | :nil | :omit` - Controls how empty collections are rendered
+
+[source,ruby]
+----
+class SomeModel < Lutaml::Model::Serializable
+  attribute :coll, :string, collection: true
+
+  xml do
+    root "some-model"
+    map_element 'collection', to: :coll, render_nil: :omit
+  end
+
+  json do
+    map 'collection', to: :coll, render_empty: :as_nil
+  end
+end
+----
+
+=== Polymorphic model support
+
+From version 0.7.1, Lutaml::Model now supports polymorphic models for attribute
+types.
+
+Comprehensive support for polymorphic models has been introduced, allowing for
+flexible modeling of inheritance relationships and proper
+serialization/deserialization.
+
+This means that you can define attributes that can accept multiple types of
+objects, and the library will handle serialization and deserialization for these
+types seamlessly.
+
+Specifically, the following features have been added:
+
+* Polymorphic attribute definition
+* Polymorphic class differentiation in model and serializations
+
+==== Polymorphic attribute definition
+
+Polymorphic attributes can be defined using the `polymorphic` option.
+
+It is possible to define polymorphic attribute classes in the superclass
+and subclasses.
+
+The `polymorphic` option can be set to a collection of classes, and the
+`polymorphic_class` option can be set to `true` to indicate acceptance of any
+subclass of the polymorphic class. Alternatively, you can specify a collection
+of classes to restrict the accepted types.
+
+[source,ruby]
+----
+class ReferenceSet < Lutaml::Model::Serializable
+  attribute :references, Reference, collection: true, polymorphic: [
+    DocumentReference,
+    AnchorReference,
+  ]
+end
+----
+
+When you are not requiring a specific set of subclasses, you can use the
+`polymorphic: true` option to indicate that any subclass of the specified class is
+acceptable.
+
+[source,ruby]
+----
+class ReferenceSet < Lutaml::Model::Serializable
+  attribute :references, Reference, collection: true, polymorphic: true
+end
+----
+
+
+==== Polymorphic class differentiator
+
+When serializing polymorphic models, a differentiator attribute is required to
+also be serialized to identify the specific subclass of the polymorphic class.
+
+This differentiator attribute is typically a string that indicates the type of
+the object being serialized. The differentiator attribute can be defined in the
+superclass or subclasses of the polymorphic class.
+
+Typically, the differentiator attribute is an XML element or attribute (e.g. `type="document-ref"`), or in JSON a `@`-prefixed key (e.g. `@type`).
+
+A polymorphic differentiator attribute can be set in either the superclass or subclasses:
+
+[source,ruby]
+----
+# In superclass
+class Reference < Lutaml::Model::Serializable
+  attribute :_class, :string, polymorphic_class: true
+  # ...
+end
+
+# Or in subclasses
+class DocumentReference < Reference
+  attribute :_class, :string, polymorphic_class: true
+  # ...
+end
+
+# Or in subclasses
+class AnchorReference < Reference
+  attribute :_class, :string, polymorphic_class: true
+  # ...
+end
+----
+
+Given the differentiator attribute being `_class`, we still need to define the
+mapping for the differentiator attribute in the superclass or subclasses.
+
+Polymorphic mapping in serialization is supported through the `polymorphic_map` option:
+
+[source,ruby]
+----
+class Reference < Lutaml::Model::Serializable
+  attribute :_class, :string, polymorphic_class: true
+
+  xml do
+    map_attribute "reference-type", to: :_class, polymorphic_map: {
+      "document-ref" => "DocumentReference",
+      "anchor-ref" => "AnchorReference"
+    }
+  end
+
+  key_value do
+    map "_class", to: :_class, polymorphic_map: {
+      "Document" => "DocumentReference",
+      "Anchor" => "AnchorReference"
+    }
+  end
+end
+----
+
+The `polymorphic_map` option is used to indicate that when serializing a
+`DocumentReference` object, the `_class` attribute will be serialized as
+`document-ref`, and when serializing an `AnchorReference` object, the `_class`
+attribute will be serialized as `anchor-ref`.
+
+This is a mapping-level option so that it can be used in serialization formats
+independently.
+
+This will produce a differentiator attribute in the serialized output as such.
+
+[source,yaml]
+----
+---
+references:
+- _class: Document
+  # other attributes...
+- _class: Anchor
+  # other attributes...
+----
+
+
+=== Importable model improvements
+
+Importable model functionality has been improved, with better support for reusable models:
+
+* `import_model` - Imports both attributes and mappings
+* `import_model_attributes` - Imports only attributes
+* `import_model_mappings` - Imports only mappings
+
+Bug fixes for the import_model functionality ensure more reliable model reuse.
+
+=== Circular reference handling
+
+Improved handling of circular references in the `ComparableModel` module prevents stack overflow errors when comparing models with self-referential structures.
+
+== Upgrade guide
+
+=== Missing values handling
+
+If you were previously using `render_nil: true`, you can continue using it, but
+you may want to explore the more flexible `value_map` option for fine-grained
+control over different serialization formats.
+
+For collection attributes, consider whether you want collections to initialize
+as `nil` or as an empty array by setting the `initialize_empty` option
+accordingly.
+
+=== Polymorphic models
+
+If you were previously using class detection for polymorphic models without
+explicit differentiators, you should now define a polymorphic differentiator
+attribute and use the `polymorphic_class: true` option.
+
+== Contributors
+
+Thank you to all contributors who made this release possible, especially:
+
+* HassanAkbar
+* Ronald Tse
+* suleman-uzair
+
+== Compatibility
+
+This release maintains compatibility with Ruby 2.7 and above.

data/docs/custom_adapters.adoc

@@ -0,0 +1,144 @@
+= Custom Adapters in Lutaml::Model
+
+Lutaml::Model provides a flexible system for creating custom adapters to handle different data formats. This guide explains how to create and use custom adapters in your application.
+
+== Overview
+
+Custom adapters allow you to extend Lutaml::Model to support additional data formats beyond the built-in ones (TOML, YAML, JSON, XML). Each adapter consists of three main components:
+
+. An adapter class that handles parsing and serialization
+. A mapping class that defines how data maps to your model
+. A transform class that handles data transformation
+
+== Creating a Custom Adapter
+
+=== 1. Adapter Class
+
+The adapter class is responsible for parsing input data and converting it back to the target format. It must implement:
+
+* `self.parse(data, options = {})` - Class method to parse input data
+* `to_<format_name>` - Instance method to convert data back to the target format
+
+.Example of a custom adapter for parsing string pairs
+[source,ruby]
+----
+class PairsAdapter < Lutaml::Model::KeyValueDocument
+  attr_reader :parsed_data
+
+  def initialize(parsed_data)
+    @parsed_data = parsed_data
+  end
+
+  def self.parse(data, options = {})
+    # Example input: "name:John|age:30"
+    parsed = data.split("|").map { |pair| pair.split(":") }.to_h
+    new(parsed)
+  end
+
+  def to_pairs_format
+    # Example output: "name:John|age:30"
+    parsed_data.map { |k, v| "#{k}:#{v}" }.join("|")
+  end
+end
+----
+
+=== 2. Mapping Class
+
+The mapping class defines how data fields map to your model attributes. It should inherit from `Lutaml::Model::Mapping`.
+
+[source,ruby]
+----
+class PairsMappingRule < Lutaml::Model::MappingRule
+end
+
+class PairsMapping < Lutaml::Model::Mapping
+  def map_field(name, to:)
+    @mappings << PairsMappingRule.new(name, to: to)
+  end
+
+  def mappings
+    @mappings
+  end
+end
+----
+
+=== 3. Transform Class
+
+The transform class handles the conversion between your data format and the model. It should inherit from `Lutaml::Model::Transform`.
+
+[source,ruby]
+----
+class PairsTransform < Lutaml::Model::Transform
+  def self.data_to_model(context, data, format, options = {})
+    model = context.model.new
+    model.name = data["name"]
+    model.age = data["age"]
+    model
+  end
+
+  def self.model_to_data(context, model, format, options = {})
+    { "name" => model.name, "age" => model.age }
+  end
+end
+----
+
+== Registering Your Adapter
+
+Register your custom adapter with the FormatRegistry:
+
+[source,ruby]
+----
+Lutaml::Model::FormatRegistry.register(
+  :pairs,
+  mapping_class: PairsMapping,
+  adapter_class: PairsAdapter,
+  transformer: PairsTransform
+)
+----
+
+== Using Custom Adapters
+
+Once registered, you can use your custom adapter with your model classes:
+
+[source,ruby]
+----
+class Person < Lutaml::Model::Serializable
+  attribute :name, :string
+  attribute :age, :string
+
+  pairs do
+    map_field "name", to: :name
+    map_field "age", to: :age
+  end
+end
+----
+
+=== Step-by-step Example
+
+[source,ruby]
+----
+input = "name:John|age:30"
+
+person = Person.from_pairs(input)
+# => #<Person @name="John", @age="30">
+
+output = person.to_pairs_format
+# => "name:John|age:30"
+----
+
+== Best Practices
+
+. *Separation of Concerns*: Keep parsing, mapping, and transformation logic in their respective classes.
+. *Error Handling*: Add meaningful error handling in your adapter's `.parse` method.
+. *Documentation*: Clearly document the format and adapter usage.
+. *Testing*: Write tests for your adapter's logic and behavior.
+. *Leverage Format Features*: Support format-specific options while staying consistent with Lutaml::Model’s interfaces.
+
+== Example Implementations
+
+For complete examples of custom adapter implementations, see:
+
+* `spec/lutaml/model/custom_bibtex_adapter_spec.rb`
+* `spec/lutaml/model/custom_vobject_adapter_spec.rb`
+
+These demonstrate how to build complete, custom adapters that integrate cleanly with `Lutaml::Model`.

data/lib/lutaml/model/attribute.rb

@@ -14,6 +14,9 @@ module Lutaml
         choice
         sequence
         method_name
+        polymorphic
+        polymorphic_class
+        initialize_empty
       ].freeze
 
       def initialize(name, type, options = {})
@@ -25,6 +28,10 @@ module Lutaml
         process_options!
       end
 
+      def polymorphic?
+        @options[:polymorphic_class]
+      end
+
       def derived?
         type.nil?
       end
@@ -41,6 +48,10 @@ module Lutaml
         @options[:method_name]
       end
 
+      def initialize_empty?
+        @options[:initialize_empty]
+      end
+
       def cast_type!(type)
         case type
         when Symbol
@@ -97,11 +108,17 @@ module Lutaml
           type.attributes[to].default
         elsif options[:default].is_a?(Proc)
           options[:default].call
-        else
+        elsif options.key?(:default)
           options[:default]
+        else
+          Lutaml::Model::UninitializedClass.instance
         end
       end
 
+      def default_set?
+        !Utils.uninitialized?(default_value)
+      end
+
       def pattern
         options[:pattern]
       end
@@ -121,6 +138,7 @@ module Lutaml
       def valid_value!(value)
         return true if value.nil? && singular?
         return true unless enum?
+        return true if Utils.uninitialized?(value)
 
         unless valid_value?(value)
           raise Lutaml::Model::InvalidValueError.new(name, value, enum_values)
@@ -162,7 +180,21 @@ module Lutaml
 
         valid_value!(value) &&
           valid_collection!(value, self) &&
-          valid_pattern!(value)
+          valid_pattern!(value) &&
+          validate_polymorphic!(value)
+      end
+
+      def validate_polymorphic(value)
+        return value.all? { |v| validate_polymorphic!(v) } if value.is_a?(Array)
+        return true unless options[:polymorphic]
+
+        valid_polymorphic_type?(value)
+      end
+
+      def validate_polymorphic!(value)
+        return true if validate_polymorphic(value)
+
+        raise Lutaml::Model::PolymorphicError.new(value, options, type)
       end
 
       def validate_collection_range
@@ -200,9 +232,6 @@ module Lutaml
 
         return true unless collection?
 
-        # Allow nil values for collections during initialization
-        return true if value.nil?
-
         # Allow any value for unbounded collections
         return true if options[:collection] == true
 
@@ -235,7 +264,8 @@ module Lutaml
       end
 
       def serialize(value, format, options = {})
-        return if value.nil?
+        value ||= [] if collection? && initialize_empty?
+        return value if value.nil? || Utils.uninitialized?(value)
         return value if derived?
         return serialize_array(value, format, options) if value.is_a?(Array)
         return serialize_model(value, format, options) if type <= Serialize
@@ -244,25 +274,66 @@ module Lutaml
       end
 
       def cast(value, format, options = {})
-        return value if type <= Serialize && value.is_a?(type.model)
-
-        value ||= [] if collection?
+        value ||= [] if collection? && !value.nil?
         return value.map { |v| cast(v, format, options) } if value.is_a?(Array)
 
-        if type <= Serialize && castable?(value, format)
-          type.apply_mappings(value, format, options)
-        elsif !value.nil? && !value.is_a?(type)
-          type.send(:"from_#{format}", value)
+        return value if already_serialized?(type, value)
+
+        klass = resolve_polymorphic_class(type, value, options)
+
+        if can_serialize?(klass, value, format)
+          klass.apply_mappings(value, format, options)
+        elsif needs_conversion?(klass, value)
+          klass.send(:"from_#{format}", value)
         else
-          type.cast(value)
+          klass.cast(value)
         end
       end
 
+      def serializable?
+        type <= Serialize
+      end
+
+      def deep_dup
+        self.class.new(name, type, Utils.deep_dup(options))
+      end
+
       private
 
+      def resolve_polymorphic_class(type, value, options)
+        return type unless polymorphic_map_defined?(options, value)
+
+        val = value[options[:polymorphic][:attribute]]
+        klass_name = options[:polymorphic][:class_map][val]
+        Object.const_get(klass_name)
+      end
+
+      def polymorphic_map_defined?(polymorphic_options, value)
+        !value.nil? &&
+          polymorphic_options[:polymorphic] &&
+          !polymorphic_options[:polymorphic].empty? &&
+          value[polymorphic_options[:polymorphic][:attribute]]
+      end
+
       def castable?(value, format)
         value.is_a?(Hash) ||
-          (format == :xml && value.is_a?(Lutaml::Model::XmlAdapter::XmlElement))
+          (format == :xml && value.is_a?(Lutaml::Model::Xml::XmlElement))
+      end
+
+      def castable_serialized_type?(value)
+        type <= Serialize && value.is_a?(type.model)
+      end
+
+      def can_serialize?(klass, value, format)
+        klass <= Serialize && castable?(value, format)
+      end
+
+      def needs_conversion?(klass, value)
+        !value.nil? && !value.is_a?(klass)
+      end
+
+      def already_serialized?(klass, value)
+        klass <= Serialize && value.is_a?(klass.model)
       end
 
       def serialize_array(value, format, options)
@@ -300,7 +371,7 @@ module Lutaml
 
       def set_default_for_collection
         validate_collection_range
-        @options[:default] ||= -> { [] }
+        @options[:default] ||= -> { [] } if initialize_empty?
       end
 
       def validate_options!(options)
@@ -315,6 +386,10 @@ module Lutaml
                 "`pattern` is only allowed for :string type"
         end
 
+        if initialize_empty? && !collection?
+          raise StandardError,
+                "Invalid option `initialize_empty` given without `collection: true` option"
+        end
         true
       end
 
@@ -325,6 +400,16 @@ module Lutaml
         raise ArgumentError,
               "Invalid type: #{type}, must be a Symbol, String or a Class"
       end
+
+      def valid_polymorphic_type?(value)
+        return value.is_a?(type) unless has_polymorphic_list?
+
+        options[:polymorphic].include?(value.class) && value.is_a?(type)
+      end
+
+      def has_polymorphic_list?
+        options[:polymorphic]&.is_a?(Array)
+      end
     end
   end
 end

data/lib/lutaml/model/choice.rb

@@ -15,6 +15,13 @@ module Lutaml
         raise Lutaml::Model::InvalidChoiceRangeError.new(@min, @max) if @min.negative? || @max.negative?
       end
 
+      def ==(other)
+        @attributes == other.attributes &&
+          @min == other.min &&
+          @max == other.max &&
+          @model == other.model
+      end
+
       def attribute(name, type, options = {})
         options[:choice] = self
         @attributes << @model.attribute(name, type, options)