moxml 0.1.10 → 0.1.12

data/docs/ENTITY_SUPPORT_FOR_LUTAML_MODEL.md

@@ -0,0 +1,102 @@
+# Entity Support for lutaml-model Team
+
+## Overview
+
+Moxml now supports entity restoration during parsing. This feature ensures that XML entities (like `&`, `<`, `>`, `"`, `'`) are preserved as `EntityReference` nodes rather than being resolved to their character values during parsing.
+
+## Key Concept: Entity Restoration
+
+By default, XML parsers resolve entities during parsing:
+- Input: `<root>foo&amp;bar</root>`
+- Default behavior: Text node contains `foo&bar` (resolved `&`)
+- With entity restoration: Text node contains `foo` + EntityReference `&amp;` + Text node `bar`
+
+## Enabling Entity Restoration
+
+### Option 1: Per-Context Configuration
+
+```ruby
+context = Moxml.new(:nokogiri, restore_entities: true)
+doc = context.parse('<root>foo&amp;bar</root>')
+# doc.to_xml will preserve &amp; as EntityReference
+```
+
+### Option 2: Global Configuration
+
+```ruby
+Moxml.configure do |config|
+  config.restore_entities = true
+end
+```
+
+## Preloading Entity Sets
+
+You can preload standard entity sets (HTML5, MathML, ISO) for faster entity resolution:
+
+```ruby
+context = Moxml.new(:nokogiri,
+  restore_entities: true,
+  preload_entity_sets: [:html5, :mathml]
+)
+```
+
+## W3C XML Core WG Compliance
+
+Per W3C XML Core WG guidance:
+- Standard XML entities (`amp`, `lt`, `gt`, `quot`, `apos`) are implicitly declared per XML spec
+- The `EntityRegistry` class tracks all known entities and their Unicode codepoints
+- Entity names are preserved through round-trip serialization
+
+## What lutaml-model Needs to Know
+
+### 1. Document Structure with Entities
+
+When entity restoration is enabled, documents containing entities will have mixed node types:
+
+```
+Document
+└── Element: root
+    ├── Text: "foo"
+    ├── EntityReference: "amp"  # Represents &
+    └── Text: "bar"
+```
+
+### 2. Serialization
+
+`doc.to_xml` will serialize EntityReference nodes as proper XML entity syntax:
+- `EntityReference("amp")` → `&amp;`
+- `EntityReference("lt")` → `&lt;`
+- etc.
+
+### 3. XPath Queries
+
+EntityReference nodes participate in XPath queries like any other node. You can query for them specifically if needed.
+
+### 4. Configuration Inheritance
+
+When using `Moxml::Context`, the entity restoration setting is preserved through document operations. However, when creating new contexts, you need to set the option explicitly.
+
+## Example Usage in lutaml-model
+
+```ruby
+# Parse XML with entities preserved
+context = Moxml.new(:nokogiri, restore_entities: true)
+doc = context.parse(your_xml_string)
+
+# Serialize back - entities are preserved
+output = doc.to_xml
+```
+
+## Testing Considerations
+
+When writing tests for models that handle XML with entities:
+1. Enable `restore_entities: true` in your test context
+2. Verify that EntityReference nodes are created for entities in text
+3. Test round-trip: parse → serialize → parse should preserve entities
+
+## Files of Interest
+
+- `lib/moxml/entity_registry.rb` - Entity definitions and lookup
+- `lib/moxml/config.rb` - Configuration options
+- `lib/moxml/document_builder.rb` - Entity restoration logic
+- `lib/moxml/entity_reference.rb` - EntityReference node class

data/docs/_pages/adapters/ox.adoc

@@ -48,7 +48,37 @@ doc.xpath("//book").find { |book| book["id"] == "123" }
 IMPORTANT: For complete XPath 1.0 specification with zero limitations today, use
 Nokogiri or Oga adapters.
 
+==== Element Ordering in Round-Trip Tests
 
+[IMPORTANT]
+.Round-trip tests for Ox are experimental
+--
+Ox round-trip tests (`nokogiri-ox` CI job) are **experimental** because Ox lacks full XML
+conformance:
+
+* **Namespace support**: Ox does not properly handle namespaced elements in XPath queries
+* **XPath limitations**: Uses `locate()` instead of standard XPath; no attribute value
+  predicates, no logical operators, no position predicates
+* **Element ordering**: Ox produces elements in a different order than Nokogiri/Oga for
+  certain complex fixtures
+
+For production use with complex XML, prefer Nokogiri or Oga adapters.
+--
+
+The Ox adapter produces elements in a different order than other adapters for certain
+fixtures with complex nested structures. In round-trip tests, this causes the
+`elements_with_attributes` array length comparison to fail, even though the semantic
+equivalence check (double round-trip) passes.
+
+Known affected fixtures:
+
+* `niso-jats/element_citation.xml`
+* `niso-jats/pnas_sample.xml`
+* `metanorma/collection1nested.xml`
+
+This is a known limitation tracked in the round-trip test suite via
+`KNOWN_ELEMENT_ORDERING_ISSUES`. The `elements_with_attributes` comparison is
+automatically skipped for these Ox adapter pairs.
 
 See also:
 

data/docs/_pages/configuration.adoc

@@ -97,6 +97,49 @@ context.config.default_encoding = 'UTF-16'
 
 **Default:** `"UTF-8"`
 
+==== Namespace URI validation mode
+
+Control how strictly namespace URIs are validated:
+
+[source,ruby]
+----
+# Strict mode (default) — validates namespace URIs against RFC 3986
+context.config.namespace_uri_mode = :strict
+doc = context.parse(xml)  # Raises ValidationError for invalid URIs
+
+# Lenient mode — accepts any string as a namespace URI
+context.config.namespace_uri_mode = :lenient
+doc = context.parse(xml)  # Accepts non-standard namespace URIs
+----
+
+**Default:** `:strict`
+
+**Modes:**
+
+`:strict`:: Validates namespace URIs against the
+https://www.rfc-editor.org/rfc/rfc3986[RFC 3986] URI-reference specification, as
+required by https://www.w3.org/TR/xml-names/[Namespaces in XML]. Invalid URIs
+raise a `Moxml::ValidationError`. This is the recommended mode for
+standards-compliant XML processing.
+
+`:lenient`:: Accepts any string as a namespace URI, only rejecting strings
+containing XML-invalid control characters (`0x00`-`0x08`, `0x0B`, `0x0C`,
+`0x0E`-`0x1F`). Use this mode when processing XML documents that use
+non-standard namespace identifiers such as URNs or other non-URI strings.
+
+**Example:**
+
+[source,ruby]
+----
+# Process documents with non-standard namespace URIs
+context = Moxml.new do |config|
+  config.namespace_uri_mode = :lenient
+end
+
+xml = '<root xmlns:ex="not a valid URI but accepted in lenient mode"/>'
+doc = context.parse(xml)  # Parses successfully
+----
+
 === Context switching
 
 Use different configurations for different tasks:

data/docs/_pages/node-api-reference.adoc

@@ -32,6 +32,10 @@ All node types in Moxml support the `#identifier` method, which returns the prim
 | `nil` (no identifier)
 | `nil`
 
+| EntityReference
+| The entity name
+| `"nbsp"`, `"copy"`
+
 | Declaration
 | `nil` (no identifier)
 | `nil`
@@ -137,3 +141,34 @@ doctype.identifier   # => "html"
 * `identifier` - Returns the primary identifier (same as `name`)
 
 All Doctype accessor methods are fully implemented across all 6 adapters.
+
+=== EntityReference nodes
+
+EntityReference nodes represent XML entity references like ` `, `©`, or custom entities declared in the DOCTYPE.
+
+[source,ruby]
+----
+# Create programmatically
+ref = doc.create_entity_reference('nbsp')
+element.add_child(ref)
+
+# Or via builder
+doc = Moxml::Builder.new(Moxml.new).build do
+  element 'text' do
+    entity_reference 'ndash'
+  end
+end
+----
+
+*Available methods:*
+
+* `name` - Returns the entity name (e.g., `"nbsp"`, `"copy"`)
+* `identifier` - Returns the primary identifier (same as `name`)
+* `text` - Returns empty string (`""`) since entity has no text content
+* `content` - Returns empty string (entity content is in the name)
+* `to_xml` - Returns the entity syntax (e.g., `" "`)
+
+*Adapter notes:*
+
+* *Nokogiri*: Preserves custom declared entities as `EntityReference` nodes
+* *Ox, Oga*: These adapters resolve entities during parsing and do not expose entity reference nodes. Use Nokogiri or LibXML for entity preservation.

data/docs/_tutorials/namespace-handling.adoc

@@ -276,6 +276,27 @@ puts all_children.length  # => 2
 * link:../pages/adapters/rexml[REXML] - ⚠️ No namespace XPath
 * link:../pages/adapters/ox[Ox] - ⚠️ Basic only, no XPath
 
+=== Namespace URI validation
+
+By default, Moxml validates namespace URIs against
+https://www.rfc-editor.org/rfc/rfc3986[RFC 3986] (strict mode). To accept
+non-standard namespace identifiers, use lenient mode:
+
+[source,ruby]
+----
+# Strict mode (default) — validates URIs per RFC 3986
+context = Moxml.new do |config|
+  config.namespace_uri_mode = :strict
+end
+
+# Lenient mode — accepts any string as namespace URI
+context = Moxml.new do |config|
+  config.namespace_uri_mode = :lenient
+end
+----
+
+See link:../pages/configuration[Configuration] for details.
+
 === Troubleshooting
 
 **Namespace XPath not working:**

data/examples/rss_parser/rss_parser.rb

@@ -179,9 +179,7 @@ if __FILE__ == $0
 
   # Count categories
   all_categories = articles.flat_map(&:categories)
-  category_counts = all_categories.each_with_object(Hash.new(0)) do |cat, counts|
-    counts[cat] += 1
-  end
+  category_counts = all_categories.tally
   puts "Categories: #{category_counts.map do |cat, count|
     "#{cat} (#{count})"
   end.join(', ')}"

data/lib/moxml/adapter/base.rb

@@ -98,12 +98,24 @@ module Moxml
           create_native_declaration(version, encoding, standalone)
         end
 
-        def create_namespace(element, prefix, uri)
+        def create_namespace(element, prefix, uri, namespace_uri_mode: :strict)
+          if prefix && uri.to_s.empty?
+            raise NamespaceError.new(
+              "Prefixed namespace declaration cannot have an empty URI",
+              prefix: prefix,
+              uri: uri,
+            )
+          end
           validate_prefix(prefix) if prefix
-          validate_uri(uri)
+          validate_uri(uri, mode: namespace_uri_mode)
           create_native_namespace(element, prefix, uri)
         end
 
+        def create_entity_reference(name)
+          validate_entity_reference_name(name)
+          create_native_entity_reference(name)
+        end
+
         def set_attribute_name(attribute, name)
           attribute.name = name
         end
@@ -112,6 +124,10 @@ module Moxml
           attribute.value = value
         end
 
+        def entity_reference_name(node)
+          node.name
+        end
+
         def duplicate_node(node)
           node.dup
         end
@@ -193,6 +209,14 @@ module Moxml
             adapter: name,
           )
         end
+
+        def create_native_entity_reference(_name)
+          raise Moxml::NotImplementedError.new(
+            "create_native_entity_reference not implemented",
+            feature: "create_native_entity_reference",
+            adapter: name,
+          )
+        end
       end
     end
   end

data/lib/moxml/adapter/headed_ox.rb

@@ -26,7 +26,7 @@ module Moxml
     class HeadedOx < Ox
       class << self
         # Override parse to use HeadedOx context instead of Ox context
-        def parse(xml, _options = {})
+        def parse(xml, _options = {}, _context = nil)
           native_doc = begin
             result = ::Ox.parse(xml)
 
@@ -45,8 +45,9 @@ module Moxml
             )
           end
 
-          # Use :headed_ox context instead of :ox
-          DocumentBuilder.new(Context.new(:headed_ox)).build(native_doc)
+          # Use provided context if available, otherwise create new one
+          ctx = _context || Context.new(:headed_ox)
+          DocumentBuilder.new(ctx).build(native_doc)
         end
 
         # Execute XPath query using Moxml's XPath engine
@@ -66,7 +67,7 @@ module Moxml
             ctx = Context.new(:headed_ox)
 
             # Wrap the native node - don't rebuild the whole document
-            node = Node.wrap(node, ctx)
+            node = Moxml::Node.wrap(node, ctx)
           end
 
           # Parse XPath expression to AST

data/lib/moxml/adapter/libxml.rb

@@ -48,7 +48,7 @@ module Moxml
           doc.root = element
         end
 
-        def parse(xml, options = {})
+        def parse(xml, options = {}, _context = nil)
           # LibXML doesn't preserve DOCTYPE during parsing, so we need to extract it manually
           xml_string = if xml.is_a?(String)
                          xml
@@ -94,7 +94,8 @@ module Moxml
             native_doc.instance_variable_set(:@moxml_doctype, doctype_wrapper)
           end
 
-          DocumentBuilder.new(Context.new(:libxml)).build(native_doc)
+          ctx = _context || Context.new(:libxml)
+          DocumentBuilder.new(ctx).build(native_doc)
         end
 
         # SAX parsing implementation for LibXML

data/lib/moxml/adapter/nokogiri.rb

@@ -11,7 +11,7 @@ module Moxml
           doc.root = element
         end
 
-        def parse(xml, options = {})
+        def parse(xml, options = {}, _context = nil)
           native_doc = begin
             if options[:fragment]
               ::Nokogiri::XML::DocumentFragment.parse(xml) do |config|
@@ -29,7 +29,9 @@ module Moxml
                                                    column: e.column)
           end
 
-          DocumentBuilder.new(Context.new(:nokogiri)).build(native_doc)
+          # Use provided context if available, otherwise create new one
+          ctx = _context || Context.new(:nokogiri)
+          DocumentBuilder.new(ctx).build(native_doc)
         end
 
         # SAX parsing implementation for Nokogiri
@@ -104,6 +106,14 @@ module Moxml
           )
         end
 
+        def create_native_entity_reference(name)
+          ::Nokogiri::XML::EntityReference.new(create_document, name)
+        end
+
+        def entity_reference_name(node)
+          node.name
+        end
+
         def declaration_attribute(declaration, attr_name)
           return nil unless declaration.content
 
@@ -150,6 +160,7 @@ module Moxml
           when ::Nokogiri::XML::ProcessingInstruction then :processing_instruction
           when ::Nokogiri::XML::Document, ::Nokogiri::XML::DocumentFragment then :document
           when ::Nokogiri::XML::DTD then :doctype
+          when ::Nokogiri::XML::EntityReference then :entity_reference
           else :unknown
           end
         end
@@ -277,7 +288,9 @@ module Moxml
         end
 
         def inner_text(node)
-          text_children = node.children - node.element_children
+          text_children = node.children.reject do |c|
+            c.element? || c.comment?
+          end
           text_children.map(&:content).join
         end
 

data/lib/moxml/adapter/oga.rb

@@ -9,17 +9,24 @@ module Moxml
   module Adapter
     class Oga < Base
       class << self
+        # Standard XML entities handled natively by parsers
+        STANDARD_XML_ENTITIES = %w[amp lt gt quot apos].freeze
+
         def set_root(doc, element)
           # Clear existing root element if any - Oga's NodeSet needs special handling
           # We need to manually remove elements since NodeSet doesn't support clear or delete_if
-          elements_to_remove = doc.children.select { |child| child.is_a?(::Oga::XML::Element) }
+          elements_to_remove = doc.children.grep(::Oga::XML::Element)
           elements_to_remove.each { |elem| doc.children.delete(elem) }
           doc.children << element
         end
 
-        def parse(xml, options = {})
+        def parse(xml, options = {}, _context = nil)
+          # Pre-process XML to convert named entities to marker form (\x01name;).
+          # Oga drops named entity references like &nbsp; during parsing.
+          processed_xml = preprocess_named_entities(xml)
+
           native_doc = begin
-            ::Oga.parse_xml(xml, strict: options[:strict])
+            ::Oga.parse_xml(processed_xml, strict: options[:strict])
           rescue LL::ParserError => e
             raise Moxml::ParseError.new(
               e.message,
@@ -27,7 +34,8 @@ module Moxml
             )
           end
 
-          DocumentBuilder.new(Context.new(:oga)).build(native_doc)
+          ctx = _context || Context.new(:oga)
+          DocumentBuilder.new(ctx).build(native_doc)
         end
 
         # SAX parsing implementation for Oga
@@ -61,7 +69,7 @@ module Moxml
         end
 
         def create_native_text(content)
-          ::Oga::XML::Text.new(text: content)
+          ::Oga::XML::Text.new(text: encode_entity_markers(content))
         end
 
         def create_native_cdata(content)
@@ -74,7 +82,8 @@ module Moxml
 
         def create_native_doctype(name, external_id, system_id)
           ::Oga::XML::Doctype.new(
-            name: name, public_id: external_id, system_id: system_id, type: "PUBLIC",
+            name: name, public_id: external_id, system_id: system_id,
+            type: external_id ? "PUBLIC" : "SYSTEM"
           )
         end
 
@@ -224,7 +233,7 @@ module Moxml
           attr = ::Oga::XML::Attribute.new(
             name: name.to_s,
             namespace_name: namespace_name,
-            value: value.to_s,
+            value: encode_entity_markers(value.to_s),
           )
           element.add_attribute(attr)
         end
@@ -234,7 +243,7 @@ module Moxml
         end
 
         def get_attribute_value(element, name)
-          element[name.to_s]
+          restore_entity_markers(element[name.to_s])
         end
 
         def remove_attribute(element, name)
@@ -303,24 +312,25 @@ module Moxml
         end
 
         def text_content(node)
-          node.text
+          restore_entity_markers(node.text)
         end
 
         def inner_text(node)
-          if node.respond_to?(:inner_text)
-            node.inner_text
-          else
-            # Oga::XML::Text node for example
-            node.text
-          end
+          text = if node.respond_to?(:inner_text)
+                   node.inner_text
+                 else
+                   # Oga::XML::Text node for example
+                   node.text
+                 end
+          restore_entity_markers(text)
         end
 
         def set_text_content(node, content)
+          encoded = encode_entity_markers(content)
           if node.respond_to?(:inner_text=)
-            node.inner_text = content
+            node.inner_text = encoded
           else
-            # Oga::XML::Text node for example
-            node.text = content
+            node.text = encoded
           end
         end
 
@@ -402,6 +412,53 @@ module Moxml
         end
 
         def serialize(node, options = {})
+          output = serialize_without_entity_processing(node, options)
+          # Post-process: convert entity markers back to entity references
+          output.gsub(ENTITY_MARKER_REGEX, '&\1;')
+        end
+
+        # Shared entity name pattern (W3C: 2-31 chars, starts with alpha)
+        ENTITY_PATTERN = "([a-zA-Z][a-zA-Z0-9]{1,30})"
+
+        # Marker character for entity preservation through Oga's parser.
+        # U+0001 is preserved literally by Oga through parse/serialize cycle.
+        ENTITY_MARKER = "\x01"
+
+        # Regular expression for entity marker post-processing
+        ENTITY_MARKER_REGEX = /#{ENTITY_MARKER}#{ENTITY_PATTERN};/
+
+        # Simple entity-only regex with no nested quantifiers
+        ENTITY_REF_REGEX = /&#{ENTITY_PATTERN};/
+
+        private
+
+        # Convert &entity; back to \x01entity; for Oga text storage.
+        # Used when setting text content programmatically (not from parsing).
+        def encode_entity_markers(text)
+          return text unless text&.include?("&")
+
+          text.gsub(ENTITY_REF_REGEX) do
+            name = ::Regexp.last_match(1)
+
+            next ::Regexp.last_match(0) if STANDARD_XML_ENTITIES.include?(name)
+
+            codepoint = Moxml::EntityRegistry.default.codepoint_for_name(name)
+            if codepoint
+              "#{ENTITY_MARKER}#{name};"
+            else
+              ::Regexp.last_match(0)
+            end
+          end
+        end
+
+        # Convert \x01entity; back to &entity; for text accessors.
+        def restore_entity_markers(text)
+          return text unless text
+
+          text.gsub(ENTITY_MARKER_REGEX, '&\1;')
+        end
+
+        def serialize_without_entity_processing(node, options = {})
           # Oga's XmlGenerator doesn't support options directly
           # We need to handle declaration options ourselves for Document nodes
           if node.is_a?(::Oga::XML::Document)
@@ -416,7 +473,11 @@ module Moxml
                                     node.xml_declaration ? true : false
                                   end
 
-            if should_include_decl && !node.xml_declaration
+            # Fix: Check if declaration already exists in children
+            # This prevents duplicate declarations when document already has one
+            has_existing_declaration = node.children.any?(::Oga::XML::XmlDeclaration)
+
+            if should_include_decl && !node.xml_declaration && !has_existing_declaration
               # Need to add declaration - create default one
               output = +""
               output << '<?xml version="1.0" encoding="UTF-8"?>'
@@ -450,7 +511,50 @@ module Moxml
           end
 
           # Default: use XmlGenerator
-          ::Moxml::Adapter::CustomizedOga::XmlGenerator.new(node).to_xml
+          # But first check if we need to handle declaration specially
+          if node.is_a?(::Oga::XML::Document) && node.xml_declaration
+            # Document has declaration - use custom handling to avoid duplicates
+            output = +""
+
+            # Serialize children, but skip XmlDeclaration if it would cause duplication
+            node.children.each do |child|
+              # Check if this would cause duplication by seeing if we already have one in output
+              if child.is_a?(::Oga::XML::XmlDeclaration) && output.include?("<?xml")
+                next # Skip duplicate declaration
+              end
+
+              output << ::Moxml::Adapter::CustomizedOga::XmlGenerator.new(child).to_xml
+            end
+
+            output
+          else
+            # Normal case - use XmlGenerator directly
+            ::Moxml::Adapter::CustomizedOga::XmlGenerator.new(node).to_xml
+          end
+        end
+
+        # Pre-process XML to convert named entities to marker format.
+        # Oga drops named entity references like &nbsp; but preserves control chars.
+        # By converting known named entities to marker form (\x01name;), we can
+        # reconstruct them during serialization.
+        #
+        # @param xml [String, #to_s] The XML string to process
+        # @return [String] The XML with known named entities converted to marker form
+        def preprocess_named_entities(xml)
+          return xml unless xml.is_a?(String)
+
+          xml.gsub(ENTITY_REF_REGEX) do
+            name = Regexp.last_match(1)
+
+            next Regexp.last_match(0) if STANDARD_XML_ENTITIES.include?(name)
+
+            codepoint = Moxml::EntityRegistry.default.codepoint_for_name(name)
+            if codepoint
+              "#{ENTITY_MARKER}#{name};"
+            else
+              Regexp.last_match(0)
+            end
+          end
         end
       end
     end

data/lib/moxml/adapter/ox.rb

@@ -17,7 +17,7 @@ module Moxml
           replace_children(doc, [element])
         end
 
-        def parse(xml, _options = {})
+        def parse(xml, _options = {}, _context = nil)
           native_doc = begin
             result = ::Ox.parse(xml)
 
@@ -36,7 +36,8 @@ module Moxml
             )
           end
 
-          DocumentBuilder.new(Context.new(:ox)).build(native_doc)
+          ctx = _context || Context.new(:ox)
+          DocumentBuilder.new(ctx).build(native_doc)
         end
 
         # SAX parsing implementation for Ox
@@ -452,7 +453,7 @@ module Moxml
         def inner_text(node)
           return "" unless node.respond_to?(:nodes)
 
-          node.nodes.select { _1.is_a?(String) }.join
+          node.nodes.grep(String).join
         end
 
         def set_text_content(node, content)