moxml 0.1.14 → 0.1.16

data/TODO.remaining/8-xpath-predicate-gaps.md

@@ -0,0 +1,68 @@
+# TODO 8: XPath Engine Predicate Gaps (5 xit Tests)
+
+## Problem
+
+The pure-Ruby XPath engine (used by HeadedOx) does not fully support
+`position()`, `last()`, and `id()` inside predicates. Five tests are marked
+`xit` pending predicate support.
+
+These gaps affect the XPath engine in `lib/moxml/xpath/` — they are not
+adapter-specific.
+
+## Failing Tests
+
+### `position()` in Predicates (2 tests)
+
+`spec/moxml/xpath/functions/position_functions_spec.rb`
+
+```ruby
+xit "returns current position in predicate" do
+  # /root/item[position() = 2]
+end
+
+xit "works with position comparison" do
+  # /root/item[position() > 1]
+end
+```
+
+### `last()` in Predicates (2 tests)
+
+`spec/moxml/xpath/functions/position_functions_spec.rb`
+
+```ruby
+xit "returns size of context in predicate" do
+  # /root/item[position() = last()]
+end
+
+xit "works with last() - 1" do
+  # /root/item[position() = last() - 1]
+end
+```
+
+### `id()` with Nodeset Argument (1 test)
+
+`spec/moxml/xpath/functions/special_functions_spec.rb:69`
+
+```ruby
+xit "accepts nodeset argument containing IDs" do
+  # id(nodeset) where nodeset is path-evaluated
+end
+```
+
+## Investigation Needed
+
+- The XPath compiler likely needs to pass predicate context (position, size)
+  into the evaluation environment when compiling predicate expressions.
+- `position()` and `last()` are defined but raise `InvalidContextError` when
+  used inside predicates — the predicate evaluation path doesn't set up the
+  context they need.
+- `id()` with a nodeset argument requires evaluating the argument as an XPath
+  path first, then extracting ID values from the resulting nodes.
+
+## Files
+
+- `lib/moxml/xpath/compiler.rb` — predicate compilation
+- `lib/moxml/xpath/engine.rb` — runtime evaluation context
+- `lib/moxml/xpath/context.rb` — context setup for position/last
+- `spec/moxml/xpath/functions/position_functions_spec.rb`
+- `spec/moxml/xpath/functions/special_functions_spec.rb`

data/TODO.remaining/9-cleanup-hygiene.md

@@ -0,0 +1,42 @@
+# TODO 9: Cleanup and Hygiene
+
+Small items that don't affect functionality but should be addressed.
+
+## 9a. Stale Doc Links in Skip Messages
+
+15+ test skip messages reference `docs/HEADED_OX_LIMITATIONS.md` but the
+actual file is at `docs/_pages/headed-ox-limitations.adoc`. The referenced
+path does not exist.
+
+**Fix:** Update all skip messages to reference
+`docs/_pages/headed-ox-limitations.adoc` instead.
+
+**Affected files:**
+- `spec/integration/headed_ox_integration_spec.rb`
+- `spec/integration/shared_examples/integration_workflows.rb`
+- `spec/integration/shared_examples/node_wrappers/node_behavior.rb`
+- `spec/integration/shared_examples/node_wrappers/cdata_behavior.rb`
+- `spec/integration/shared_examples/edge_cases.rb`
+- `spec/moxml/xpath/axes_spec.rb`
+- `spec/moxml/xpath/compiler_spec.rb`
+- `spec/moxml/adapter/headed_ox_spec.rb`
+
+## 9b. Untracked `scripts/` Directory
+
+`scripts/format_xml.rb` and `scripts/pretty_format_xml.rb` exist as untracked
+files. Decide whether to commit (and add to `.gitignore` pattern or document)
+or remove.
+
+## 9c. Superseded Root TODO Files
+
+The following root-level files are marked as superseded in
+`TODO.remaining/README.md` but still exist:
+
+- `TODO.entities-work.md`
+- `TODO.entity-handling.md`
+- `TODO.entity-support.md`
+- `TODO.full-entity-support.md`
+- `TODO.full-entity.md`
+- `TODO.mn-bilingual-round-trip.md`
+
+Once all work is confirmed tracked in `TODO.remaining/`, these can be deleted.

data/TODO.remaining/README.md

@@ -0,0 +1,54 @@
+# TODO.remaining — Consolidated Action Items
+
+Consolidated from: TODO.entities-work.md, TODO.entity-support.md,
+TODO.full-entity.md, TODO.full-entity-support.md, TODO.entity-handling.md,
+TODO.mn-bilingual-round-trip.md, plus code audit (2026-04-22).
+
+Those root files are superseded and can be removed.
+
+## Dependency Order
+
+```
+TODO 1 (Adapter Support)
+   |
+   v
+TODO 2 (Model-Driven Restoration) ---> TODO 4 (Lenient Entities Mode)
+   |
+   v
+TODO 3 (Test Coverage)
+
+TODO 5 (Fixture Integrity) — independent
+TODO 6 (Ox Element Ordering) — independent
+TODO 7 (HeadedOx Limitations) — independent
+TODO 8 (XPath Predicate Gaps) — independent
+TODO 9 (Cleanup Hygiene) — independent
+```
+
+## Summary
+
+| # | File | Description | Status |
+|---|------|-------------|--------|
+| 1 | `1-entity-reference-adapter-support.md` | EntityReference in Ox, Oga, REXML, LibXML, HeadedOx | Not started |
+| 2 | `2-entity-restoration-model-driven.md` | Use EntityRegistry as source of truth for restoration | Not started |
+| 3 | `3-entity-reference-test-coverage.md` | Tests for EntityReference nodes and round-trips | Not started |
+| 4 | `4-lenient-entities-mode.md` | Strict vs lenient entity restoration mode | Not started |
+| 5 | `5-fixture-integrity.md` | Bilingual fixture verification + CI validation | Not started |
+| 6 | `6-ox-element-ordering-bug.md` | Ox adapter reorders elements in certain fixtures | Not started |
+| 7 | `7-headed-ox-limitations.md` | 15 skipped tests across 7 HeadedOx limitation areas | Not started |
+| 8 | `8-xpath-predicate-gaps.md` | position()/last()/id() not working in XPath predicates | Not started |
+| 9 | `9-cleanup-hygiene.md` | Stale doc links, untracked scripts, superseded files | Not started |
+
+## What's Already Done
+
+- EntityReference node class (`lib/moxml/entity_reference.rb`)
+- EntityRegistry with 2125 W3C entities (`lib/moxml/entity_registry.rb`)
+- Node type registry includes `:entity_reference`
+- Base adapter template: `create_entity_reference`, `validate_entity_reference_name`
+- Nokogiri adapter: full native EntityReference support
+- Document factory: `create_entity_reference(name)`
+- DocumentBuilder: `visit_entity_reference` + partial `restore_entities_in_text`
+- Builder DSL: `entity_reference(name)`
+- Config: `restore_entities`, `entity_load_mode`, `entity_provider`, `preload_entity_sets`
+- Context: entity registry integration
+- EntityRegistry tests (24 examples passing)
+- HeadedOx limitations documented in `docs/_pages/headed-ox-limitations.adoc`

data/benchmarks/generate_report.rb

@@ -488,7 +488,7 @@ class MoxmlBenchmarkReport
     f.puts "- Can accept 99.20% pass rate (16 documented Ox limitations)"
     f.puts ""
     f.puts "**Note:** HeadedOx = Ox parsing speed + full XPath features."
-    f.puts "See docs/HEADED_OX_LIMITATIONS.md for complete details."
+    f.puts "See docs/_pages/headed-ox-limitations.adoc for complete details."
     f.puts ""
   end
 

data/docs/_pages/configuration.adoc

@@ -97,19 +97,21 @@ context.config.default_encoding = 'UTF-16'
 
 **Default:** `"UTF-8"`
 
-==== Namespace URI validation mode
+==== Namespace validation mode
 
-Control how strictly namespace URIs are validated:
+Control how strictly namespace URIs and prefixes are validated:
 
 [source,ruby]
 ----
-# Strict mode (default) — validates namespace URIs against RFC 3986
-context.config.namespace_uri_mode = :strict
-doc = context.parse(xml)  # Raises ValidationError for invalid URIs
-
-# Lenient mode — accepts any string as a namespace URI
-context.config.namespace_uri_mode = :lenient
-doc = context.parse(xml)  # Accepts non-standard namespace URIs
+# Strict mode (default) — validates namespace URIs against RFC 3986 and
+# prefixes against NCName rules
+context.config.namespace_validation_mode = :strict
+doc = context.parse(xml)  # Raises ValidationError for invalid URIs/prefixes
+
+# Lenient mode — accepts any URI string and defers prefix validation to the
+# underlying XML parser
+context.config.namespace_validation_mode = :lenient
+doc = context.parse(xml)  # Accepts non-standard namespace URIs/prefixes
 ----
 
 **Default:** `:strict`
@@ -117,23 +119,24 @@ doc = context.parse(xml)  # Accepts non-standard namespace URIs
 **Modes:**
 
 `:strict`:: Validates namespace URIs against the
-https://www.rfc-editor.org/rfc/rfc3986[RFC 3986] URI-reference specification, as
-required by https://www.w3.org/TR/xml-names/[Namespaces in XML]. Invalid URIs
-raise a `Moxml::ValidationError`. This is the recommended mode for
-standards-compliant XML processing.
+https://www.rfc-editor.org/rfc/rfc3986[RFC 3986] URI-reference specification and
+namespace prefixes against NCName rules, as required by
+https://www.w3.org/TR/xml-names/[Namespaces in XML]. Invalid values raise a
+`Moxml::ValidationError`. This is the recommended mode for standards-compliant
+XML processing.
 
-`:lenient`:: Accepts any string as a namespace URI, only rejecting strings
-containing XML-invalid control characters (`0x00`-`0x08`, `0x0B`, `0x0C`,
-`0x0E`-`0x1F`). Use this mode when processing XML documents that use
-non-standard namespace identifiers such as URNs or other non-URI strings.
+`:lenient`:: Accepts any string as a namespace URI (only rejecting control
+characters) and defers prefix validation to the underlying XML parser. Use this
+mode when processing XML documents that use non-standard namespace identifiers
+or prefixes (e.g., `xmlns_1.0`).
 
 **Example:**
 
 [source,ruby]
 ----
-# Process documents with non-standard namespace URIs
+# Process documents with non-standard namespace URIs and prefixes
 context = Moxml.new do |config|
-  config.namespace_uri_mode = :lenient
+  config.namespace_validation_mode = :lenient
 end
 
 xml = '<root xmlns:ex="not a valid URI but accepted in lenient mode"/>'

data/docs/_tutorials/namespace-handling.adoc

@@ -280,18 +280,18 @@ puts all_children.length  # => 2
 
 By default, Moxml validates namespace URIs against
 https://www.rfc-editor.org/rfc/rfc3986[RFC 3986] (strict mode). To accept
-non-standard namespace identifiers, use lenient mode:
+non-standard namespace identifiers or prefixes, use lenient mode:
 
 [source,ruby]
 ----
-# Strict mode (default) — validates URIs per RFC 3986
+# Strict mode (default) — validates URIs per RFC 3986 and prefixes per NCName
 context = Moxml.new do |config|
-  config.namespace_uri_mode = :strict
+  config.namespace_validation_mode = :strict
 end
 
-# Lenient mode — accepts any string as namespace URI
+# Lenient mode — accepts any URI string and defers prefix validation to parser
 context = Moxml.new do |config|
-  config.namespace_uri_mode = :lenient
+  config.namespace_validation_mode = :lenient
 end
 ----
 

data/lib/moxml/adapter/base.rb

@@ -98,7 +98,8 @@ module Moxml
           create_native_declaration(version, encoding, standalone)
         end
 
-        def create_namespace(element, prefix, uri, namespace_uri_mode: :strict)
+        def create_namespace(element, prefix, uri,
+namespace_validation_mode: :strict)
           if prefix && uri.to_s.empty?
             raise NamespaceError.new(
               "Prefixed namespace declaration cannot have an empty URI",
@@ -106,8 +107,12 @@ module Moxml
               uri: uri,
             )
           end
-          validate_prefix(prefix) if prefix
-          validate_uri(uri, mode: namespace_uri_mode)
+          if namespace_validation_mode == :strict
+            validate_prefix(prefix) if prefix
+            validate_uri(uri, mode: :strict)
+          else
+            validate_uri(uri, mode: :lenient)
+          end
           create_native_namespace(element, prefix, uri)
         end
 
@@ -144,6 +149,20 @@ module Moxml
           node
         end
 
+        # Check if the native document has an XML declaration
+        # @param native_doc the native document object
+        # @param wrapper [Moxml::Document] the wrapper with has_xml_declaration flag
+        # @return [Boolean]
+        def has_declaration?(_native_doc, wrapper)
+          wrapper.has_xml_declaration
+        end
+
+        # Return the actual native node after an add_child operation.
+        # Override for adapters where node identity may change (e.g., LibXML doc.root=).
+        def actual_native(child_native, _parent_native)
+          child_native
+        end
+
         protected
 
         def create_native_element(_name, _owner_doc = nil)

data/lib/moxml/adapter/customized_libxml/declaration.rb

@@ -9,7 +9,7 @@ module Moxml
       # are read-only after creation. This wrapper allows mutation by
       # storing values internally and regenerating XML when needed.
       class Declaration
-        attr_accessor :version, :encoding
+        attr_accessor :version, :encoding, :removed, :parent_doc
         attr_reader :native
 
         def initialize(native_doc, version = nil, encoding = nil,

data/lib/moxml/adapter/customized_libxml/entity_reference.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    module CustomizedLibxml
+      class EntityReference
+        attr_reader :name
+
+        def initialize(name)
+          @name = name
+        end
+
+        def to_xml
+          "&#{@name};"
+        end
+
+        def ==(other)
+          other.is_a?(self.class) && @name == other.name
+        end
+      end
+    end
+  end
+end

data/lib/moxml/adapter/customized_libxml.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    module CustomizedLibxml
+      autoload :Cdata, "moxml/adapter/customized_libxml/cdata"
+      autoload :Comment, "moxml/adapter/customized_libxml/comment"
+      autoload :Declaration, "moxml/adapter/customized_libxml/declaration"
+      autoload :Element, "moxml/adapter/customized_libxml/element"
+      autoload :EntityReference,
+               "moxml/adapter/customized_libxml/entity_reference"
+      autoload :Node, "moxml/adapter/customized_libxml/node"
+      autoload :ProcessingInstruction,
+               "moxml/adapter/customized_libxml/processing_instruction"
+      autoload :Text, "moxml/adapter/customized_libxml/text"
+    end
+  end
+end

data/lib/moxml/adapter/customized_oga.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    module CustomizedOga
+      autoload :XmlDeclaration, "moxml/adapter/customized_oga/xml_declaration"
+      autoload :XmlGenerator, "moxml/adapter/customized_oga/xml_generator"
+    end
+  end
+end

data/lib/moxml/adapter/customized_ox/entity_reference.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    module CustomizedOx
+      class EntityReference
+        attr_reader :name
+        attr_accessor :parent
+
+        def initialize(name)
+          @name = name
+          @parent = nil
+        end
+
+        def to_xml
+          "&#{@name};"
+        end
+
+        def ==(other)
+          other.is_a?(self.class) && @name == other.name
+        end
+      end
+    end
+  end
+end

data/lib/moxml/adapter/customized_ox.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    module CustomizedOx
+      autoload :Attribute, "moxml/adapter/customized_ox/attribute"
+      autoload :EntityReference, "moxml/adapter/customized_ox/entity_reference"
+      autoload :Namespace, "moxml/adapter/customized_ox/namespace"
+      autoload :Text, "moxml/adapter/customized_ox/text"
+    end
+  end
+end

data/lib/moxml/adapter/customized_rexml/entity_reference.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    module CustomizedRexml
+      class EntityReference
+        attr_reader :name
+
+        def initialize(name)
+          @name = name
+        end
+
+        def ==(other)
+          other.is_a?(self.class) && @name == other.name
+        end
+      end
+    end
+  end
+end

data/lib/moxml/adapter/customized_rexml/formatter.rb

@@ -7,18 +7,21 @@ module Moxml
     module CustomizedRexml
       # Custom REXML formatter that fixes indentation and wrapping issues
       class Formatter < ::REXML::Formatters::Pretty
-        def initialize(indentation: 2, self_close_empty: false)
+        def initialize(indentation: 2, self_close_empty: false, adapter: nil)
           @indentation = " " * indentation
           @level = 0
           @compact = true
           @width = -1 # Disable line wrapping
           @self_close_empty = self_close_empty
+          @adapter = adapter
         end
 
         def write(node, output)
           case node
           when ::REXML::XMLDecl
             write_declaration(node, output)
+          when ::Moxml::Adapter::CustomizedRexml::EntityReference
+            output << "&#{node.name};"
           else
             super
           end
@@ -29,7 +32,13 @@ module Moxml
           output << "<#{node.expanded_name}"
           write_attributes(node, output)
 
-          if node.children.empty? && @self_close_empty
+          # Check for entity refs stored in adapter attachments
+          entity_refs = @adapter&.attachments&.get(node, :entity_refs)
+          child_sequence = @adapter&.attachments&.get(node, :child_sequence)
+
+          has_no_children = node.children.empty? && !(entity_refs && !entity_refs.empty?)
+
+          if has_no_children && @self_close_empty
             output << "/>"
             return
           end
@@ -42,26 +51,41 @@ module Moxml
           mixed = has_text && has_elements
 
           # Handle children based on content type
-          unless node.children.empty?
+          all_children_empty = node.children.empty? && !(entity_refs && !entity_refs.empty?)
+          unless all_children_empty
             @level += @indentation.length unless mixed
 
-            node.children.each_with_index do |child, _index|
-              # Skip insignificant whitespace
-              next if child.is_a?(::REXML::Text) &&
-                child.to_s.strip.empty? &&
-                !(child.next_sibling.nil? && child.previous_sibling.nil?)
-
-              # Indent non-text nodes in non-mixed content
-              # if !mixed && !child.is_a?(::REXML::Text)
-              #   output << ' ' * @level
-              # end
-
-              write(child, output)
-
-              # Add newlines between elements in non-mixed content
-              # if !mixed && !child.is_a?(::REXML::Text) && index < node.children.size - 1
-              #   output << "\n"
-              # end
+            if entity_refs && !entity_refs.empty? && child_sequence
+              # Interleave native children with entity refs using tracked sequence
+              eref_idx = 0
+              native_idx = 0
+              child_sequence.each do |type|
+                case type
+                when :native
+                  if native_idx < node.children.size
+                    child = node.children[native_idx]
+                    native_idx += 1
+                    next if child.is_a?(::REXML::Text) &&
+                      child.to_s.strip.empty? &&
+                      !(child.next_sibling.nil? && child.previous_sibling.nil?)
+                    write(child, output)
+                  end
+                when :eref
+                  if eref_idx < entity_refs.size
+                    write(entity_refs[eref_idx], output)
+                    eref_idx += 1
+                  end
+                end
+              end
+            else
+              node.children.each_with_index do |child, _index|
+                # Skip insignificant whitespace
+                next if child.is_a?(::REXML::Text) &&
+                  child.to_s.strip.empty? &&
+                  !(child.next_sibling.nil? && child.previous_sibling.nil?)
+
+                write(child, output)
+              end
             end
 
             # Reset indentation for closing tag in non-mixed content

data/lib/moxml/adapter/customized_rexml.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    module CustomizedRexml
+      autoload :EntityReference,
+               "moxml/adapter/customized_rexml/entity_reference"
+      autoload :Formatter, "moxml/adapter/customized_rexml/formatter"
+    end
+  end
+end

data/lib/moxml/adapter/headed_ox.rb

@@ -25,17 +25,23 @@ module Moxml
     #
     class HeadedOx < Ox
       class << self
-        # Override parse to use HeadedOx context instead of Ox context
-        def parse(xml, _options = {}, _context = nil)
+        # Override parse to use lazy wrapping like the Ox adapter.
+        # Previously used DocumentBuilder (eager tree construction causing
+        # ~176K allocations per 100-element parse). Lazy parse defers wrapper
+        # creation until nodes are accessed, matching Ox adapter behavior.
+        def parse(xml, options = {}, _context = nil)
           native_doc = begin
             result = ::Ox.parse(xml)
 
             # result can be either Document or Element
             if result.is_a?(::Ox::Document)
+              assign_parents(result)
+              validate_single_root(result) if options[:strict]
               result
             else
               doc = ::Ox::Document.new
               doc << result
+              assign_parents(doc)
               doc
             end
           rescue ::Ox::ParseError => e
@@ -47,17 +53,17 @@ module Moxml
 
           # Use provided context if available, otherwise create new one
           ctx = _context || Context.new(:headed_ox)
-          DocumentBuilder.new(ctx).build(native_doc)
+          Document.new(native_doc, ctx)
         end
 
         # Execute XPath query using Moxml's XPath engine
         #
         # This overrides the Ox adapter's xpath method which uses locate().
         #
-        # @param [Moxml::Node] node Starting node (wrapped Moxml node)
+        # @param node Starting node (native or wrapped)
         # @param [String] expression XPath expression
         # @param [Hash] namespaces Namespace prefix mappings
-        # @return [Moxml::NodeSet, Object] Query results
+        # @return [Array, Object] Native node array or scalar value
         def xpath(node, expression, namespaces = {})
           # If we receive a native node, wrap it first
           # Document#xpath passes @native, but our compiled XPath needs Moxml nodes
@@ -79,16 +85,33 @@ module Moxml
           # Execute on the node (now guaranteed to be wrapped Moxml node)
           result = proc.call(node)
 
-          # Wrap Array results in NodeSet, return other types directly
+          # Return native arrays for Node#xpath to wrap, scalars directly.
+          # The adapter contract: xpath() returns Array<native> | scalar.
           case result
           when Array
-            # Deduplicate by native object identity to handle descendant-or-self
-            # which may yield the same native node multiple times
-            nodeset = NodeSet.new(result, node.context)
-            nodeset.uniq_by_native
+            # XPath engine returns wrapped Moxml::Node objects.
+            # Extract native nodes and deduplicate by object identity.
+            native_nodes = result.map { |n| n.is_a?(Moxml::Node) ? n.native : n }
+            seen = {}
+            native_nodes.select do |native|
+              id = native.object_id
+              if seen[id]
+                false
+              else
+                seen[id] = true
+              end
+            end
           when NodeSet
-            # Deduplicate NodeSet results as well
-            result.uniq_by_native
+            # NodeSet from intermediate evaluation - extract natives and deduplicate
+            seen = {}
+            result.to_a.map(&:native).select do |native|
+              id = native.object_id
+              if seen[id]
+                false
+              else
+                seen[id] = true
+              end
+            end
           else
             # Scalar values (string, number, boolean) - return as-is
             result
@@ -107,10 +130,10 @@ module Moxml
         # @param [Moxml::Node] node Starting node
         # @param [String] expression XPath expression
         # @param [Hash] namespaces Namespace prefix mappings
-        # @return [Moxml::Node, Object, nil] First result or nil
+        # @return [Object, nil] First native node or scalar value
         def at_xpath(node, expression, namespaces = {})
           result = xpath(node, expression, namespaces)
-          result.is_a?(NodeSet) ? result.first : result
+          result.is_a?(Array) ? result.first : result
         end
 
         # Check if XPath is supported