moxml 0.1.14 → 0.1.16

data/TODO.remaining/2-entity-restoration-model-driven.md

@@ -0,0 +1,169 @@
+# TODO 2: Model-Driven Entity Restoration
+
+## Problem
+
+The `restore_entities` feature in `DocumentBuilder` is hardcoded to only handle
+the 5 standard XML entities (amp, lt, gt, quot, apos). It ignores the
+EntityRegistry entirely — despite EntityRegistry knowing 2125+ entities from
+the W3C HTML/MathML set. This means non-standard entities like ` `,
+`©`, `—` are never restored, which is the core round-trip problem
+that motivated the entire entity feature.
+
+Additionally, the restoration logic lives in DocumentBuilder with hardcoded
+knowledge that belongs in the model layer.
+
+## Current State (verified)
+
+`lib/moxml/document_builder.rb:80-110` — `restore_entities_in_text`:
+```ruby
+entity_chars = {
+  "<" => "lt", ">" => "gt", "&" => "amp",
+  '"' => "quot", "'" => "apos",
+}
+```
+
+This is a hardcoded lookup that duplicates knowledge already in EntityRegistry.
+It only triggers for characters `<`, `>`, `&`, `"`, `'` — the regex guard
+`/[<>&"']/` on line 73 prevents it from ever seeing characters like U+00A0
+(non-breaking space, `&nbsp;`).
+
+**Critical**: Because only Nokogiri has `create_native_entity_reference`
+(see TODO 1), `restore_entities` raises `NotImplementedError` on all other
+adapters even for the 5 standard entities.
+
+## XML Entity Model
+
+XML has a clear entity model:
+
+1. **5 predefined entities** (amp, lt, gt, quot, apos) — always available per
+   XML spec. These characters MUST be entity-encoded in certain contexts
+   (e.g., `<` and `&` in text content).
+
+2. **DTD-declared entities** — declared via `<!ENTITY name "value">` in the
+   document's DOCTYPE internal subset or external subset.
+
+3. **API-supplied entities** — registered by the user via
+   `EntityRegistry.register` or `entity_provider` callback.
+
+4. **Bundled detection set** — the W3C HTML/MathML entities bundled in
+   `data/w3c_entities.json`. These are not "declared" in any DTD but are
+   recognized by Moxml for restoration purposes.
+
+The EntityRegistry already knows about categories 1, 3, and 4. Category 2
+(DTD parsing) is future work.
+
+## Design: Model-Driven Restoration
+
+EntityRegistry should be THE source of truth for "should this character become
+an entity reference?" The restoration policy should be:
+
+```ruby
+# In EntityRegistry (or a cooperating policy object)
+STANDARD_CODEPOINTS = [0x26, 0x3C, 0x3E, 0x22, 0x27].freeze  # amp, lt, gt, quot, apos
+
+def should_restore?(codepoint, config:)
+  name = primary_name_for_codepoint(codepoint)
+  return false unless name
+
+  # 1. The 5 standard XML entities are ALWAYS restored.
+  #    These are syntactically required — the XML wouldn't be well-formed
+  #    without encoding them.
+  return true if STANDARD_CODEPOINTS.include?(codepoint)
+
+  # 2. Non-standard entities: only if restore_entities is enabled.
+  return false unless config.restore_entities
+
+  # 3. In the future, strict vs lenient mode will gate this further.
+  #    Strict: only if declared in DTD (not yet implemented).
+  #    Lenient: any known entity name.
+  true
+end
+```
+
+### Changes to DocumentBuilder
+
+Replace the hardcoded hash with delegation to the registry:
+
+```ruby
+def visit_text(node)
+  prepared = adapter.prepare_for_new_document(node, @current_doc.native)
+  content = adapter.text_content(node)
+
+  if should_restore_entities?(content)
+    restore_entities_in_text(content)
+  else
+    @node_stack.last&.add_child(Text.new(prepared, context))
+  end
+end
+
+private
+
+def should_restore_entities?(content)
+  return false unless context.config.restore_entities
+  # Scan for any character that the registry knows about
+  content.to_s.chars.any? { |c| context.entity_registry.should_restore?(c.ord, config: context.config) }
+end
+
+def restore_entities_in_text(content)
+  parent = @node_stack.last
+  return unless parent
+
+  content.to_s.chars.each do |char|
+    codepoint = char.ord
+    name = context.entity_registry.primary_name_for_codepoint(codepoint)
+
+    if context.entity_registry.should_restore?(codepoint, config: context.config)
+      entity_node = adapter.create_entity_reference(name)
+      parent.add_child(EntityReference.new(entity_node, context))
+    else
+      text_node = adapter.create_text(char)
+      parent.add_child(Text.new(text_node, context))
+    end
+  end
+end
+```
+
+**Note**: This splits each text node into per-character nodes. For documents
+with few entity references, this creates unnecessary overhead. A future
+optimization should buffer consecutive non-entity characters into a single
+text node.
+
+### Performance Optimization (deferred)
+
+Instead of character-by-character processing:
+1. Scan the text for characters that have entity names in the registry
+2. Split only at those positions, keeping runs of plain characters together
+3. This reduces node count dramatically for typical documents
+
+```ruby
+def restore_entities_in_text(content)
+  parent = @node_stack.last
+  return unless parent
+
+  buffer = +""
+  content.to_s.chars.each do |char|
+    codepoint = char.ord
+    name = context.entity_registry.primary_name_for_codepoint(codepoint)
+
+    if name && context.entity_registry.should_restore?(codepoint, config: context.config)
+      # Flush buffer before entity
+      if !buffer.empty?
+        parent.add_child(Text.new(adapter.create_text(buffer), context))
+        buffer.clear
+      end
+      parent.add_child(EntityReference.new(adapter.create_entity_reference(name), context))
+    else
+      buffer << char
+    end
+  end
+  # Flush remaining buffer
+  if !buffer.empty?
+    parent.add_child(Text.new(adapter.create_text(buffer), context))
+  end
+end
+```
+
+## Files to Modify
+
+- `lib/moxml/entity_registry.rb` — add `should_restore?` method
+- `lib/moxml/document_builder.rb` — replace hardcoded entity_chars with registry-driven logic

data/TODO.remaining/3-entity-reference-test-coverage.md

@@ -0,0 +1,170 @@
+# TODO 3: EntityReference Test Coverage
+
+## Problem
+
+There are zero tests for EntityReference node behavior, zero tests for
+entity round-trip preservation, and zero adapter-level tests for entity
+reference creation or serialization. Only `EntityRegistry` has tests
+(`spec/moxml/entity_registry_spec.rb`).
+
+This means the entire EntityReference feature is untested — including the
+`restore_entities` config, `create_entity_reference` factory, `visit_entity_reference`
+in DocumentBuilder, and the `entity_reference` Builder DSL method.
+
+## Required Test Coverage
+
+### 1. EntityReference Node Tests
+
+**File**: `spec/moxml/entity_reference_spec.rb`
+
+```ruby
+RSpec.describe Moxml::EntityReference do
+  # Test per adapter (use shared examples)
+  %i[nokogiri].each do |adapter|  # expand as adapters gain support
+    context "with #{adapter} adapter" do
+      let(:ctx) { Moxml.new(adapter) }
+
+      it "creates an entity reference node" do
+        doc = ctx.create_document
+        ref = doc.create_entity_reference("nbsp")
+        expect(ref).to be_a(Moxml::EntityReference)
+        expect(ref.name).to eq("nbsp")
+      end
+
+      it "has empty text content" do
+        doc = ctx.create_document
+        ref = doc.create_entity_reference("amp")
+        expect(ref.text).to eq("")
+        expect(ref.content).to eq("")
+      end
+
+      it "serializes to entity syntax" do
+        doc = ctx.create_document
+        ref = doc.create_entity_reference("mdash")
+        expect(ref.to_xml).to eq("—")
+      end
+
+      it "is recognized as entity_reference type" do
+        doc = ctx.create_document
+        ref = doc.create_entity_reference("copy")
+        expect(ref.entity_reference?).to be true
+      end
+
+      it "survives add_child and retrieval" do
+        doc = ctx.create_document
+        root = doc.create_element("p")
+        doc.root = root
+        ref = doc.create_entity_reference("nbsp")
+        root.add_child(ref)
+        expect(root.children.first).to be_a(Moxml::EntityReference)
+        expect(root.children.first.name).to eq("nbsp")
+      end
+
+      it "validates entity reference name" do
+        doc = ctx.create_document
+        expect {
+          doc.create_entity_reference("123invalid")
+        }.to raise_error(Moxml::ValidationError)
+      end
+    end
+  end
+end
+```
+
+### 2. Builder DSL Tests
+
+**File**: `spec/moxml/builder_spec.rb` (add to existing or create new section)
+
+```ruby
+it "creates entity references via DSL" do
+  doc = Moxml::Builder.new(ctx).build do
+    element("p") { entity_reference("nbsp") }
+  end
+  expect(doc.root.children.first).to be_a(Moxml::EntityReference)
+  expect(doc.to_xml).to include(" ")
+end
+```
+
+### 3. Restore Entities Integration Tests
+
+**File**: `spec/moxml/adapter/entity_restoration_spec.rb` (shared examples)
+
+```ruby
+RSpec.shared_examples "entity restoration" do |adapter_name|
+  context "with #{adapter_name}" do
+    let(:ctx) { Moxml.new(adapter_name, restore_entities: true) }
+
+    it "restores standard XML entities" do
+      doc = ctx.parse("<p>a &amp; b</p>")
+      output = doc.to_xml
+      expect(output).to include("&amp;")
+    end
+
+    it "restores non-standard entities from registry" do
+      # nbsp (U+00A0) is in the bundled W3C entity set
+      doc = ctx.parse("<p>\u00A0</p>")
+      output = doc.to_xml
+      expect(output).to include("&nbsp;")
+    end
+
+    it "preserves entity syntax through round-trip" do
+      doc = ctx.parse("<p>&nbsp;&copy;&mdash;</p>")
+      output = doc.to_xml
+      reparsed = ctx.parse(output)
+      # Text content should be identical after round-trip
+      expect(reparsed.root.text).to eq(doc.root.text)
+    end
+
+    it "does not restore entities when restore_entities is false" do
+      ctx_no_restore = Moxml.new(adapter_name, restore_entities: false)
+      doc = ctx_no_restore.parse("<p>a &amp; b</p>")
+      output = doc.to_xml
+      # Standard entities may still appear as &amp; due to XML escaping,
+      # but no EntityReference nodes should be created
+      expect(doc.root.children).not_to include(a_kind_of(Moxml::EntityReference))
+    end
+  end
+end
+```
+
+### 4. Cross-Adapter Consistency Tests
+
+**File**: `spec/consistency/entity_reference_consistency_spec.rb`
+
+Verify that EntityReference behavior is consistent across all adapters that
+support it:
+- Same entity name produces same serialization
+- Same text content after round-trip
+- Children enumeration includes EntityReference nodes
+
+### 5. EntityRegistry.should_restore? Tests
+
+**File**: Add to `spec/moxml/entity_registry_spec.rb`
+
+```ruby
+describe "#should_restore?" do
+  it "always restores the 5 standard XML entities" do
+    registry = described_class.new
+    config = Moxml::Config.new(:nokogiri)
+    expect(registry.should_restore?(0x26, config: config)).to be true  # amp
+    expect(registry.should_restore?(0x3C, config: config)).to be true  # lt
+  end
+
+  it "restores non-standard entities only when restore_entities is true" do
+    registry = described_class.new
+    config_on = Moxml::Config.new(:nokogiri)
+    config_on.restore_entities = true
+    config_off = Moxml::Config.new(:nokogiri)
+    config_off.restore_entities = false
+
+    expect(registry.should_restore?(0xA0, config: config_on)).to be true   # nbsp
+    expect(registry.should_restore?(0xA0, config: config_off)).to be false
+  end
+end
+```
+
+## Dependencies
+
+- TODO 1 must be partially complete (at least one adapter working) before
+  adapter-level tests can pass
+- TODO 2 must be complete before non-standard entity restoration tests can pass

data/TODO.remaining/4-lenient-entities-mode.md

@@ -0,0 +1,106 @@
+# TODO 4: Lenient Entities Mode
+
+## Problem
+
+XML only defines 5 predefined entities (amp, lt, gt, quot, apos). Any other
+entity must be declared in a DTD. However, real-world XML documents frequently
+use HTML entities (` `, `©`) without DTD declarations — particularly
+office documents (OOXML/ODF) and legacy systems.
+
+Currently Moxml has no way to configure whether undeclared entities should be
+preserved. The `restore_entities` flag is a boolean that enables restoration
+for all known entities from the registry. There is no distinction between
+"only DTD-declared" (strict) and "any recognized" (lenient).
+
+## Design
+
+### Config Option
+
+Add `entity_restoration_mode` to Config with two values:
+
+- `:strict` (default) — Only restore entities that are declared in the DTD
+  internal subset. The 5 standard XML entities are always restored regardless
+  (they are implicitly declared per XML spec). DTD parsing is prerequisite.
+
+- `:lenient` — Restore any character that has a known entity name in the
+  EntityRegistry. This covers the bundled W3C HTML/MathML set (2125 entities)
+  plus any user-registered entities. No DTD required.
+
+This replaces the boolean `restore_entities` which becomes a derived property:
+- `restore_entities = true` + `entity_restoration_mode = :lenient` → restore all known
+- `restore_entities = true` + `entity_restoration_mode = :strict` → restore only declared
+- `restore_entities = false` → don't restore any
+
+### EntityRegistry Enhancement
+
+```ruby
+class EntityRegistry
+  def should_restore?(codepoint, config:)
+    name = primary_name_for_codepoint(codepoint)
+    return false unless name
+
+    # Standard XML entities always restored (XML well-formedness requirement)
+    return true if standard_entity?(codepoint)
+
+    # Must have restoration enabled
+    return false unless config.restore_entities
+
+    case config.entity_restoration_mode
+    when :lenient
+      # Any known entity
+      true
+    when :strict
+      # Only if declared in DTD (future: check DTD declarations)
+      # For now, fall back to lenient behavior until DTD parsing is implemented
+      true
+    else
+      false
+    end
+  end
+
+  def standard_entity?(codepoint)
+    STANDARD_ENTITIES.value?(codepoint)
+  end
+end
+```
+
+### User-Supplied Entities
+
+Users can supply entities through three mechanisms:
+
+1. **EntityRegistry.register** — programmatic registration:
+   ```ruby
+   context = Moxml.new(:nokogiri)
+   context.entity_registry.register({ "myentity" => 0xABCD })
+   ```
+
+2. **entity_provider callback** — for custom/external entity sources:
+   ```ruby
+   Moxml.new(:nokogiri) do |c|
+     c.entity_load_mode = :custom
+     c.entity_provider = -> { { "myentity" => 0xABCD } }
+   end
+   ```
+
+3. **Bundled W3C set** — loaded by default in `:required` mode (2125 entities
+   from HTML/MathML/ISO sets). Controlled by `entity_load_mode` config.
+
+None of these require DTD. They are model-level knowledge in the EntityRegistry.
+
+### DTD-Declared Entities (Future)
+
+Strict mode's full value requires parsing DTD entity declarations from
+`<!DOCTYPE ... [ <!ENTITY name "value"> ]>`. This is a separate feature
+(external to this TODO). Until then, strict mode behaves like lenient mode.
+
+## Files to Modify
+
+- `lib/moxml/config.rb` — add `entity_restoration_mode` attribute
+- `lib/moxml/entity_registry.rb` — add `should_restore?`, `standard_entity?`
+- `lib/moxml/document_builder.rb` — use `should_restore?` from registry (ties into TODO 2)
+
+## Dependencies
+
+- TODO 2 (model-driven restoration) should be done first so the policy is
+  centralized in EntityRegistry
+- TODO 1 (adapter support) should be done first so entities can actually be created

data/TODO.remaining/5-fixture-integrity.md

@@ -0,0 +1,65 @@
+# TODO 5: Fixture Integrity and CI Validation
+
+## Problem
+
+The Metanorma bilingual presentation fixture was previously corrupted (error
+messages appended after valid XML). It was truncated to fix the corruption,
+but the truncated version has not been verified against the upstream source.
+
+Additionally, there is no automated validation of XML fixtures in CI — a
+corrupted fixture could be introduced and not caught until round-trip tests
+fail with confusing errors.
+
+## Remaining Tasks
+
+### 1. Verify Bilingual Fixture Against Upstream
+
+The file `spec/fixtures/round-trips/metanorma/bilingual.presentation.xml`
+was truncated from 111,606 lines to fix corruption. Need to:
+
+- Obtain a clean copy from the Metanorma project
+- Compare with the current truncated version (21,211 lines — different from
+  the 55,802 lines mentioned in the original TODO, suggesting further changes)
+- Confirm no data loss occurred in truncation
+
+### 2. Add CI Fixture Validation
+
+Add a Rake task or RSpec test that validates all XML fixtures are well-formed
+before running round-trip tests. This prevents silent corruption.
+
+**Option A**: Rake task using `xmllint`:
+```ruby
+# In Rakefile
+namespace :spec do
+  task :validate_fixtures do
+    errors = []
+    Dir.glob("spec/fixtures/**/*.xml").each do |path|
+      output = `xmllint --noout "#{path}" 2>&1`
+      errors << "#{path}: #{output}" unless $?.success?
+    end
+    raise "Invalid fixtures:\n#{errors.join("\n")}" unless errors.empty?
+  end
+end
+task spec: ["spec:validate_fixtures"]
+```
+
+**Option B**: RSpec test:
+```ruby
+# spec/integration/fixture_validation_spec.rb
+RSpec.describe "XML fixtures" do
+  Dir.glob("spec/fixtures/**/*.xml").each do |path|
+    it "#{path} is valid XML" do
+      ctx = Moxml.new(:nokogiri)
+      expect { ctx.parse(File.read(path)) }.not_to raise_error
+    end
+  end
+end
+```
+
+Option A is preferred — `xmllint` is stricter and catches issues that
+lenient parsers might silently accept.
+
+## Files to Create/Modify
+
+- `Rakefile` — add `spec:validate_fixtures` task
+- Verify/replace `spec/fixtures/round-trips/metanorma/bilingual.presentation.xml`

data/TODO.remaining/6-ox-element-ordering-bug.md

@@ -0,0 +1,36 @@
+# TODO 6: Ox Adapter Element Ordering Bug
+
+## Problem
+
+When round-tripping certain XML fixtures through the Ox adapter, child elements
+are produced in a different order compared to Nokogiri, Oga, and REXML. This
+causes cross-adapter consistency failures for `elements_with_attributes`
+comparisons.
+
+The semantic equivalence check (double round-trip) still passes, so the
+document content is correct — only the ordering is wrong.
+
+## Current State
+
+Suppressed in `spec/consistency/round_trip_spec.rb:332` via
+`KNOWN_ELEMENT_ORDERING_ISSUES` set. Affected fixture/adapter combinations:
+
+```
+niso-jats/element_citation.xml  nokogiri <-> ox
+niso-jats/element_citation.xml  ox <-> oga
+niso-jats/element_citation.xml  rexml <-> ox
+```
+
+## Investigation Needed
+
+- Determine whether Ox's DOM building reorders nodes or if the issue is in
+  Moxml's tree traversal during serialization.
+- Check if Ox's `Ox::Element#nodes` preserves insertion order.
+- Compare Ox's native serialization (`Ox.dump`) with Moxml's custom serializer
+  to narrow down where the reorder happens.
+
+## Files
+
+- `spec/consistency/round_trip_spec.rb` — suppression set
+- `lib/moxml/adapter/ox.rb` — serialization path
+- `lib/moxml/adapter/customized_ox/` — wrapper classes involved in tree walk

data/TODO.remaining/7-headed-ox-limitations.md

@@ -0,0 +1,95 @@
+# TODO 7: HeadedOx Adapter Limitations (15 Skipped Tests)
+
+## Problem
+
+HeadedOx (Ox + pure-Ruby XPath engine) has 15 skipped tests representing 7
+distinct limitation areas. Some require upstream Ox gem enhancements; others
+need investigation or Moxml-side fixes.
+
+Full details in `docs/_pages/headed-ox-limitations.adoc`.
+
+## Limitation Areas
+
+### 7a. XPath `@*` Attribute Wildcard (3 tests)
+
+The XPath parser does not support wildcard in the attribute axis.
+
+**Tests:**
+- `spec/moxml/xpath/compiler_spec.rb:156` — descendant-or-self wildcards
+- `spec/moxml/xpath/compiler_spec.rb:192` — attribute axis wildcards
+- `spec/moxml/xpath/axes_spec.rb:225` — attribute + predicate combinations
+
+**Workaround:** Use `element.attributes.values` via Ruby enumeration.
+
+### 7b. Namespace Methods (4 tests)
+
+Ox does not expose namespace information through its public API. The adapter
+cannot implement `node.namespace`, `node.namespaces`, or namespace inheritance.
+
+**Tests:**
+- `spec/integration/shared_examples/edge_cases.rb:93` — default namespace changes
+- `spec/integration/shared_examples/edge_cases.rb:119` — recursive namespace defs
+- `spec/integration/shared_examples/edge_cases.rb:139` — namespace-prefixed attr access
+- `spec/integration/shared_examples/integration_workflows.rb:83` — complex namespaces
+
+**Requires:** Ox gem API enhancement (namespace accessors on `Ox::Element`).
+
+### 7c. Text Content from Nested XPath Results (4 tests)
+
+Accessing text content from child elements of XPath result nodes returns empty
+strings. Likely a node wrapping or text node handling issue in HeadedOx.
+
+**Tests:**
+- `spec/moxml/adapter/headed_ox_spec.rb:74` — string functions in predicates
+- `spec/moxml/adapter/headed_ox_spec.rb:82` — position functions
+- `spec/moxml/adapter/headed_ox_spec.rb:304` — last() function
+- `spec/integration/shared_examples/node_wrappers/node_behavior.rb:113` — XPath text access
+
+**Needs:** Investigation — check node wrapping and text node registration.
+
+### 7d. CDATA `]]>` Escaping (2 tests)
+
+Ox serializes CDATA sections as-is without splitting on `]]>` sequences, which
+violates the XML spec.
+
+**Tests:**
+- `spec/integration/shared_examples/edge_cases.rb:39`
+- `spec/integration/shared_examples/node_wrappers/cdata_behavior.rb:44`
+
+**Requires:** Ox gem enhancement or Moxml-side serialization override.
+
+### 7e. Parent Node Setter (1 test)
+
+Ox has no native method to change a node's parent after creation.
+
+**Test:**
+- `spec/integration/shared_examples/integration_workflows.rb:126`
+
+**Requires:** Ox gem reparenting API or workaround via remove + re-add.
+
+### 7f. Namespace-Aware XPath with Predicates (1 test)
+
+Queries like `//xmlns:item[@id="123"]` return empty results under HeadedOx.
+
+**Test:**
+- `spec/integration/shared_examples/integration_workflows.rb:63`
+
+**Needs:** Investigation — check namespace resolution in predicate context.
+
+### 7g. Wildcard Element Counting (1 test)
+
+`//*` returns a different count (6) vs Nokogiri (7+), likely due to Ox's DOM
+structure.
+
+**Test:**
+- `spec/moxml/xpath/compiler_spec.rb:156`
+
+**Impact:** Low — real-world queries typically use specific element names.
+
+## Files
+
+- `docs/_pages/headed-ox-limitations.adoc` — full documentation
+- `lib/moxml/adapter/headed_ox.rb`
+- `lib/moxml/adapter/ox.rb`
+- `lib/moxml/xpath/` — pure-Ruby XPath engine
+- All spec files listed above