moxml 0.1.15 → 0.1.16

data/TODO.remaining/5-fixture-integrity.md

@@ -0,0 +1,65 @@
+# TODO 5: Fixture Integrity and CI Validation
+
+## Problem
+
+The Metanorma bilingual presentation fixture was previously corrupted (error
+messages appended after valid XML). It was truncated to fix the corruption,
+but the truncated version has not been verified against the upstream source.
+
+Additionally, there is no automated validation of XML fixtures in CI — a
+corrupted fixture could be introduced and not caught until round-trip tests
+fail with confusing errors.
+
+## Remaining Tasks
+
+### 1. Verify Bilingual Fixture Against Upstream
+
+The file `spec/fixtures/round-trips/metanorma/bilingual.presentation.xml`
+was truncated from 111,606 lines to fix corruption. Need to:
+
+- Obtain a clean copy from the Metanorma project
+- Compare with the current truncated version (21,211 lines — different from
+  the 55,802 lines mentioned in the original TODO, suggesting further changes)
+- Confirm no data loss occurred in truncation
+
+### 2. Add CI Fixture Validation
+
+Add a Rake task or RSpec test that validates all XML fixtures are well-formed
+before running round-trip tests. This prevents silent corruption.
+
+**Option A**: Rake task using `xmllint`:
+```ruby
+# In Rakefile
+namespace :spec do
+  task :validate_fixtures do
+    errors = []
+    Dir.glob("spec/fixtures/**/*.xml").each do |path|
+      output = `xmllint --noout "#{path}" 2>&1`
+      errors << "#{path}: #{output}" unless $?.success?
+    end
+    raise "Invalid fixtures:\n#{errors.join("\n")}" unless errors.empty?
+  end
+end
+task spec: ["spec:validate_fixtures"]
+```
+
+**Option B**: RSpec test:
+```ruby
+# spec/integration/fixture_validation_spec.rb
+RSpec.describe "XML fixtures" do
+  Dir.glob("spec/fixtures/**/*.xml").each do |path|
+    it "#{path} is valid XML" do
+      ctx = Moxml.new(:nokogiri)
+      expect { ctx.parse(File.read(path)) }.not_to raise_error
+    end
+  end
+end
+```
+
+Option A is preferred — `xmllint` is stricter and catches issues that
+lenient parsers might silently accept.
+
+## Files to Create/Modify
+
+- `Rakefile` — add `spec:validate_fixtures` task
+- Verify/replace `spec/fixtures/round-trips/metanorma/bilingual.presentation.xml`

data/TODO.remaining/6-ox-element-ordering-bug.md

@@ -0,0 +1,36 @@
+# TODO 6: Ox Adapter Element Ordering Bug
+
+## Problem
+
+When round-tripping certain XML fixtures through the Ox adapter, child elements
+are produced in a different order compared to Nokogiri, Oga, and REXML. This
+causes cross-adapter consistency failures for `elements_with_attributes`
+comparisons.
+
+The semantic equivalence check (double round-trip) still passes, so the
+document content is correct — only the ordering is wrong.
+
+## Current State
+
+Suppressed in `spec/consistency/round_trip_spec.rb:332` via
+`KNOWN_ELEMENT_ORDERING_ISSUES` set. Affected fixture/adapter combinations:
+
+```
+niso-jats/element_citation.xml  nokogiri <-> ox
+niso-jats/element_citation.xml  ox <-> oga
+niso-jats/element_citation.xml  rexml <-> ox
+```
+
+## Investigation Needed
+
+- Determine whether Ox's DOM building reorders nodes or if the issue is in
+  Moxml's tree traversal during serialization.
+- Check if Ox's `Ox::Element#nodes` preserves insertion order.
+- Compare Ox's native serialization (`Ox.dump`) with Moxml's custom serializer
+  to narrow down where the reorder happens.
+
+## Files
+
+- `spec/consistency/round_trip_spec.rb` — suppression set
+- `lib/moxml/adapter/ox.rb` — serialization path
+- `lib/moxml/adapter/customized_ox/` — wrapper classes involved in tree walk

data/TODO.remaining/7-headed-ox-limitations.md

@@ -0,0 +1,95 @@
+# TODO 7: HeadedOx Adapter Limitations (15 Skipped Tests)
+
+## Problem
+
+HeadedOx (Ox + pure-Ruby XPath engine) has 15 skipped tests representing 7
+distinct limitation areas. Some require upstream Ox gem enhancements; others
+need investigation or Moxml-side fixes.
+
+Full details in `docs/_pages/headed-ox-limitations.adoc`.
+
+## Limitation Areas
+
+### 7a. XPath `@*` Attribute Wildcard (3 tests)
+
+The XPath parser does not support wildcard in the attribute axis.
+
+**Tests:**
+- `spec/moxml/xpath/compiler_spec.rb:156` — descendant-or-self wildcards
+- `spec/moxml/xpath/compiler_spec.rb:192` — attribute axis wildcards
+- `spec/moxml/xpath/axes_spec.rb:225` — attribute + predicate combinations
+
+**Workaround:** Use `element.attributes.values` via Ruby enumeration.
+
+### 7b. Namespace Methods (4 tests)
+
+Ox does not expose namespace information through its public API. The adapter
+cannot implement `node.namespace`, `node.namespaces`, or namespace inheritance.
+
+**Tests:**
+- `spec/integration/shared_examples/edge_cases.rb:93` — default namespace changes
+- `spec/integration/shared_examples/edge_cases.rb:119` — recursive namespace defs
+- `spec/integration/shared_examples/edge_cases.rb:139` — namespace-prefixed attr access
+- `spec/integration/shared_examples/integration_workflows.rb:83` — complex namespaces
+
+**Requires:** Ox gem API enhancement (namespace accessors on `Ox::Element`).
+
+### 7c. Text Content from Nested XPath Results (4 tests)
+
+Accessing text content from child elements of XPath result nodes returns empty
+strings. Likely a node wrapping or text node handling issue in HeadedOx.
+
+**Tests:**
+- `spec/moxml/adapter/headed_ox_spec.rb:74` — string functions in predicates
+- `spec/moxml/adapter/headed_ox_spec.rb:82` — position functions
+- `spec/moxml/adapter/headed_ox_spec.rb:304` — last() function
+- `spec/integration/shared_examples/node_wrappers/node_behavior.rb:113` — XPath text access
+
+**Needs:** Investigation — check node wrapping and text node registration.
+
+### 7d. CDATA `]]>` Escaping (2 tests)
+
+Ox serializes CDATA sections as-is without splitting on `]]>` sequences, which
+violates the XML spec.
+
+**Tests:**
+- `spec/integration/shared_examples/edge_cases.rb:39`
+- `spec/integration/shared_examples/node_wrappers/cdata_behavior.rb:44`
+
+**Requires:** Ox gem enhancement or Moxml-side serialization override.
+
+### 7e. Parent Node Setter (1 test)
+
+Ox has no native method to change a node's parent after creation.
+
+**Test:**
+- `spec/integration/shared_examples/integration_workflows.rb:126`
+
+**Requires:** Ox gem reparenting API or workaround via remove + re-add.
+
+### 7f. Namespace-Aware XPath with Predicates (1 test)
+
+Queries like `//xmlns:item[@id="123"]` return empty results under HeadedOx.
+
+**Test:**
+- `spec/integration/shared_examples/integration_workflows.rb:63`
+
+**Needs:** Investigation — check namespace resolution in predicate context.
+
+### 7g. Wildcard Element Counting (1 test)
+
+`//*` returns a different count (6) vs Nokogiri (7+), likely due to Ox's DOM
+structure.
+
+**Test:**
+- `spec/moxml/xpath/compiler_spec.rb:156`
+
+**Impact:** Low — real-world queries typically use specific element names.
+
+## Files
+
+- `docs/_pages/headed-ox-limitations.adoc` — full documentation
+- `lib/moxml/adapter/headed_ox.rb`
+- `lib/moxml/adapter/ox.rb`
+- `lib/moxml/xpath/` — pure-Ruby XPath engine
+- All spec files listed above

data/TODO.remaining/8-xpath-predicate-gaps.md

@@ -0,0 +1,68 @@
+# TODO 8: XPath Engine Predicate Gaps (5 xit Tests)
+
+## Problem
+
+The pure-Ruby XPath engine (used by HeadedOx) does not fully support
+`position()`, `last()`, and `id()` inside predicates. Five tests are marked
+`xit` pending predicate support.
+
+These gaps affect the XPath engine in `lib/moxml/xpath/` — they are not
+adapter-specific.
+
+## Failing Tests
+
+### `position()` in Predicates (2 tests)
+
+`spec/moxml/xpath/functions/position_functions_spec.rb`
+
+```ruby
+xit "returns current position in predicate" do
+  # /root/item[position() = 2]
+end
+
+xit "works with position comparison" do
+  # /root/item[position() > 1]
+end
+```
+
+### `last()` in Predicates (2 tests)
+
+`spec/moxml/xpath/functions/position_functions_spec.rb`
+
+```ruby
+xit "returns size of context in predicate" do
+  # /root/item[position() = last()]
+end
+
+xit "works with last() - 1" do
+  # /root/item[position() = last() - 1]
+end
+```
+
+### `id()` with Nodeset Argument (1 test)
+
+`spec/moxml/xpath/functions/special_functions_spec.rb:69`
+
+```ruby
+xit "accepts nodeset argument containing IDs" do
+  # id(nodeset) where nodeset is path-evaluated
+end
+```
+
+## Investigation Needed
+
+- The XPath compiler likely needs to pass predicate context (position, size)
+  into the evaluation environment when compiling predicate expressions.
+- `position()` and `last()` are defined but raise `InvalidContextError` when
+  used inside predicates — the predicate evaluation path doesn't set up the
+  context they need.
+- `id()` with a nodeset argument requires evaluating the argument as an XPath
+  path first, then extracting ID values from the resulting nodes.
+
+## Files
+
+- `lib/moxml/xpath/compiler.rb` — predicate compilation
+- `lib/moxml/xpath/engine.rb` — runtime evaluation context
+- `lib/moxml/xpath/context.rb` — context setup for position/last
+- `spec/moxml/xpath/functions/position_functions_spec.rb`
+- `spec/moxml/xpath/functions/special_functions_spec.rb`

data/TODO.remaining/9-cleanup-hygiene.md

@@ -0,0 +1,42 @@
+# TODO 9: Cleanup and Hygiene
+
+Small items that don't affect functionality but should be addressed.
+
+## 9a. Stale Doc Links in Skip Messages
+
+15+ test skip messages reference `docs/HEADED_OX_LIMITATIONS.md` but the
+actual file is at `docs/_pages/headed-ox-limitations.adoc`. The referenced
+path does not exist.
+
+**Fix:** Update all skip messages to reference
+`docs/_pages/headed-ox-limitations.adoc` instead.
+
+**Affected files:**
+- `spec/integration/headed_ox_integration_spec.rb`
+- `spec/integration/shared_examples/integration_workflows.rb`
+- `spec/integration/shared_examples/node_wrappers/node_behavior.rb`
+- `spec/integration/shared_examples/node_wrappers/cdata_behavior.rb`
+- `spec/integration/shared_examples/edge_cases.rb`
+- `spec/moxml/xpath/axes_spec.rb`
+- `spec/moxml/xpath/compiler_spec.rb`
+- `spec/moxml/adapter/headed_ox_spec.rb`
+
+## 9b. Untracked `scripts/` Directory
+
+`scripts/format_xml.rb` and `scripts/pretty_format_xml.rb` exist as untracked
+files. Decide whether to commit (and add to `.gitignore` pattern or document)
+or remove.
+
+## 9c. Superseded Root TODO Files
+
+The following root-level files are marked as superseded in
+`TODO.remaining/README.md` but still exist:
+
+- `TODO.entities-work.md`
+- `TODO.entity-handling.md`
+- `TODO.entity-support.md`
+- `TODO.full-entity-support.md`
+- `TODO.full-entity.md`
+- `TODO.mn-bilingual-round-trip.md`
+
+Once all work is confirmed tracked in `TODO.remaining/`, these can be deleted.

data/TODO.remaining/README.md

@@ -0,0 +1,54 @@
+# TODO.remaining — Consolidated Action Items
+
+Consolidated from: TODO.entities-work.md, TODO.entity-support.md,
+TODO.full-entity.md, TODO.full-entity-support.md, TODO.entity-handling.md,
+TODO.mn-bilingual-round-trip.md, plus code audit (2026-04-22).
+
+Those root files are superseded and can be removed.
+
+## Dependency Order
+
+```
+TODO 1 (Adapter Support)
+   |
+   v
+TODO 2 (Model-Driven Restoration) ---> TODO 4 (Lenient Entities Mode)
+   |
+   v
+TODO 3 (Test Coverage)
+
+TODO 5 (Fixture Integrity) — independent
+TODO 6 (Ox Element Ordering) — independent
+TODO 7 (HeadedOx Limitations) — independent
+TODO 8 (XPath Predicate Gaps) — independent
+TODO 9 (Cleanup Hygiene) — independent
+```
+
+## Summary
+
+| # | File | Description | Status |
+|---|------|-------------|--------|
+| 1 | `1-entity-reference-adapter-support.md` | EntityReference in Ox, Oga, REXML, LibXML, HeadedOx | Not started |
+| 2 | `2-entity-restoration-model-driven.md` | Use EntityRegistry as source of truth for restoration | Not started |
+| 3 | `3-entity-reference-test-coverage.md` | Tests for EntityReference nodes and round-trips | Not started |
+| 4 | `4-lenient-entities-mode.md` | Strict vs lenient entity restoration mode | Not started |
+| 5 | `5-fixture-integrity.md` | Bilingual fixture verification + CI validation | Not started |
+| 6 | `6-ox-element-ordering-bug.md` | Ox adapter reorders elements in certain fixtures | Not started |
+| 7 | `7-headed-ox-limitations.md` | 15 skipped tests across 7 HeadedOx limitation areas | Not started |
+| 8 | `8-xpath-predicate-gaps.md` | position()/last()/id() not working in XPath predicates | Not started |
+| 9 | `9-cleanup-hygiene.md` | Stale doc links, untracked scripts, superseded files | Not started |
+
+## What's Already Done
+
+- EntityReference node class (`lib/moxml/entity_reference.rb`)
+- EntityRegistry with 2125 W3C entities (`lib/moxml/entity_registry.rb`)
+- Node type registry includes `:entity_reference`
+- Base adapter template: `create_entity_reference`, `validate_entity_reference_name`
+- Nokogiri adapter: full native EntityReference support
+- Document factory: `create_entity_reference(name)`
+- DocumentBuilder: `visit_entity_reference` + partial `restore_entities_in_text`
+- Builder DSL: `entity_reference(name)`
+- Config: `restore_entities`, `entity_load_mode`, `entity_provider`, `preload_entity_sets`
+- Context: entity registry integration
+- EntityRegistry tests (24 examples passing)
+- HeadedOx limitations documented in `docs/_pages/headed-ox-limitations.adoc`

data/benchmarks/generate_report.rb

@@ -488,7 +488,7 @@ class MoxmlBenchmarkReport
     f.puts "- Can accept 99.20% pass rate (16 documented Ox limitations)"
     f.puts ""
     f.puts "**Note:** HeadedOx = Ox parsing speed + full XPath features."
-    f.puts "See docs/HEADED_OX_LIMITATIONS.md for complete details."
+    f.puts "See docs/_pages/headed-ox-limitations.adoc for complete details."
     f.puts ""
   end
 

data/lib/moxml/adapter/base.rb

@@ -149,6 +149,20 @@ namespace_validation_mode: :strict)
           node
         end
 
+        # Check if the native document has an XML declaration
+        # @param native_doc the native document object
+        # @param wrapper [Moxml::Document] the wrapper with has_xml_declaration flag
+        # @return [Boolean]
+        def has_declaration?(_native_doc, wrapper)
+          wrapper.has_xml_declaration
+        end
+
+        # Return the actual native node after an add_child operation.
+        # Override for adapters where node identity may change (e.g., LibXML doc.root=).
+        def actual_native(child_native, _parent_native)
+          child_native
+        end
+
         protected
 
         def create_native_element(_name, _owner_doc = nil)

data/lib/moxml/adapter/customized_libxml/declaration.rb

@@ -9,7 +9,7 @@ module Moxml
       # are read-only after creation. This wrapper allows mutation by
       # storing values internally and regenerating XML when needed.
       class Declaration
-        attr_accessor :version, :encoding
+        attr_accessor :version, :encoding, :removed, :parent_doc
         attr_reader :native
 
         def initialize(native_doc, version = nil, encoding = nil,

data/lib/moxml/adapter/customized_rexml/formatter.rb

@@ -7,12 +7,13 @@ module Moxml
     module CustomizedRexml
       # Custom REXML formatter that fixes indentation and wrapping issues
       class Formatter < ::REXML::Formatters::Pretty
-        def initialize(indentation: 2, self_close_empty: false)
+        def initialize(indentation: 2, self_close_empty: false, adapter: nil)
           @indentation = " " * indentation
           @level = 0
           @compact = true
           @width = -1 # Disable line wrapping
           @self_close_empty = self_close_empty
+          @adapter = adapter
         end
 
         def write(node, output)
@@ -31,7 +32,13 @@ module Moxml
           output << "<#{node.expanded_name}"
           write_attributes(node, output)
 
-          if node.children.empty? && @self_close_empty
+          # Check for entity refs stored in adapter attachments
+          entity_refs = @adapter&.attachments&.get(node, :entity_refs)
+          child_sequence = @adapter&.attachments&.get(node, :child_sequence)
+
+          has_no_children = node.children.empty? && !(entity_refs && !entity_refs.empty?)
+
+          if has_no_children && @self_close_empty
             output << "/>"
             return
           end
@@ -44,26 +51,41 @@ module Moxml
           mixed = has_text && has_elements
 
           # Handle children based on content type
-          unless node.children.empty?
+          all_children_empty = node.children.empty? && !(entity_refs && !entity_refs.empty?)
+          unless all_children_empty
             @level += @indentation.length unless mixed
 
-            node.children.each_with_index do |child, _index|
-              # Skip insignificant whitespace
-              next if child.is_a?(::REXML::Text) &&
-                child.to_s.strip.empty? &&
-                !(child.next_sibling.nil? && child.previous_sibling.nil?)
-
-              # Indent non-text nodes in non-mixed content
-              # if !mixed && !child.is_a?(::REXML::Text)
-              #   output << ' ' * @level
-              # end
-
-              write(child, output)
-
-              # Add newlines between elements in non-mixed content
-              # if !mixed && !child.is_a?(::REXML::Text) && index < node.children.size - 1
-              #   output << "\n"
-              # end
+            if entity_refs && !entity_refs.empty? && child_sequence
+              # Interleave native children with entity refs using tracked sequence
+              eref_idx = 0
+              native_idx = 0
+              child_sequence.each do |type|
+                case type
+                when :native
+                  if native_idx < node.children.size
+                    child = node.children[native_idx]
+                    native_idx += 1
+                    next if child.is_a?(::REXML::Text) &&
+                      child.to_s.strip.empty? &&
+                      !(child.next_sibling.nil? && child.previous_sibling.nil?)
+                    write(child, output)
+                  end
+                when :eref
+                  if eref_idx < entity_refs.size
+                    write(entity_refs[eref_idx], output)
+                    eref_idx += 1
+                  end
+                end
+              end
+            else
+              node.children.each_with_index do |child, _index|
+                # Skip insignificant whitespace
+                next if child.is_a?(::REXML::Text) &&
+                  child.to_s.strip.empty? &&
+                  !(child.next_sibling.nil? && child.previous_sibling.nil?)
+
+                write(child, output)
+              end
             end
 
             # Reset indentation for closing tag in non-mixed content

data/lib/moxml/adapter/headed_ox.rb

@@ -60,10 +60,10 @@ module Moxml
         #
         # This overrides the Ox adapter's xpath method which uses locate().
         #
-        # @param [Moxml::Node] node Starting node (wrapped Moxml node)
+        # @param node Starting node (native or wrapped)
         # @param [String] expression XPath expression
         # @param [Hash] namespaces Namespace prefix mappings
-        # @return [Moxml::NodeSet, Object] Query results
+        # @return [Array, Object] Native node array or scalar value
         def xpath(node, expression, namespaces = {})
           # If we receive a native node, wrap it first
           # Document#xpath passes @native, but our compiled XPath needs Moxml nodes
@@ -85,16 +85,33 @@ module Moxml
           # Execute on the node (now guaranteed to be wrapped Moxml node)
           result = proc.call(node)
 
-          # Wrap Array results in NodeSet, return other types directly
+          # Return native arrays for Node#xpath to wrap, scalars directly.
+          # The adapter contract: xpath() returns Array<native> | scalar.
           case result
           when Array
-            # Deduplicate by native object identity to handle descendant-or-self
-            # which may yield the same native node multiple times
-            nodeset = NodeSet.new(result, node.context)
-            nodeset.uniq_by_native
+            # XPath engine returns wrapped Moxml::Node objects.
+            # Extract native nodes and deduplicate by object identity.
+            native_nodes = result.map { |n| n.is_a?(Moxml::Node) ? n.native : n }
+            seen = {}
+            native_nodes.select do |native|
+              id = native.object_id
+              if seen[id]
+                false
+              else
+                seen[id] = true
+              end
+            end
           when NodeSet
-            # Deduplicate NodeSet results as well
-            result.uniq_by_native
+            # NodeSet from intermediate evaluation - extract natives and deduplicate
+            seen = {}
+            result.to_a.map(&:native).select do |native|
+              id = native.object_id
+              if seen[id]
+                false
+              else
+                seen[id] = true
+              end
+            end
           else
             # Scalar values (string, number, boolean) - return as-is
             result
@@ -113,10 +130,10 @@ module Moxml
         # @param [Moxml::Node] node Starting node
         # @param [String] expression XPath expression
         # @param [Hash] namespaces Namespace prefix mappings
-        # @return [Moxml::Node, Object, nil] First result or nil
+        # @return [Object, nil] First native node or scalar value
         def at_xpath(node, expression, namespaces = {})
           result = xpath(node, expression, namespaces)
-          result.is_a?(NodeSet) ? result.first : result
+          result.is_a?(Array) ? result.first : result
         end
 
         # Check if XPath is supported