moxml 0.1.9 → 0.1.11

data/README.adoc

@@ -26,6 +26,8 @@ Key features:
 
 == Supported XML libraries
 
+=== General
+
 Moxml supports the following XML libraries:
 
 REXML:: https://github.com/ruby/rexml[REXML], a pure Ruby XML parser
@@ -110,7 +112,9 @@ section.
 NOTE: HeadedOx provides full XPath 1.0 support via a pure Ruby XPath engine
 layered on top of Ox's C parser. See HeadedOx documentation for details.
 
-NOTE: Ox/HeadedOx SAX: Only core events supported (start_element, end_element, characters, errors). No separate CDATA, comment, or processing instruction events.
+NOTE: Ox/HeadedOx SAX: Only core events supported (start_element, end_element,
+characters, errors). No separate CDATA, comment, or processing instruction
+events.
 
 == Adapter comparison
 
@@ -356,7 +360,7 @@ NOTE: Ox/HeadedOx SAX: Only core events supported (start_element, end_element, c
 | ✅ Full
 | ✅ Full
 | ✅ Full
-| ⚠️ Limited^4^
+| ✅ Full
 | ✅ Full
 | ✅ Full
 
@@ -400,13 +404,12 @@ NOTE: Ox/HeadedOx SAX: Only core events supported (start_element, end_element, c
 | ✅ Yes
 | ✅ Yes
 |===
-
++
 ^1^ Ox/HeadedOx: Text node replacement may fail in some cases due to internal node structure +
 ^2^ Ox: `//book[@id]` works (returns all book elements), but doesn't filter by attribute existence +
 ^3^ HeadedOx: Full XPath 1.0 with all 27 functions and 6 axes. Pure Ruby XPath engine on Ox's C parser. 99.20% pass rate. See link:docs/headed-ox.adoc[] +
 ^4^ Ox: Use `.find { |el| el["id"] == "123" }` instead of XPath attribute value predicates +
-^5^ LibXML: DOCTYPE parsing works, serialization is limited (no round-trip preservation) +
-^6^ HeadedOx limitations: Namespace introspection and 7 axes not implemented. See link:docs/HEADED_OX_LIMITATIONS.md[]
+^5^ HeadedOx limitations: Namespace introspection and 7 axes not implemented. See link:docs/HEADED_OX_LIMITATIONS.md[]
 
 === Adapter selection guide
 
@@ -453,10 +456,13 @@ NOTE: Ox/HeadedOx SAX: Only core events supported (start_element, end_element, c
 * Need more XPath capabilities than standard Ox provides
 * Memory efficiency is important but XPath features are required
 
-CAUTION: Ox's custom XPath engine supports common patterns but may not handle
+CAUTION: Ox's custom XPath engine supports common patterns but cannot handle
 complex XPath expressions. Test thoroughly if your use case requires advanced
 XPath.
 
+TODO: We should throw errors when unsupported XPath features are used with Ox
+or HeadedOx to prevent silent failures.
+
 
 == Getting started
 
@@ -586,7 +592,81 @@ book.add_child(doc.create_comment('Book details'))
 title = doc.create_element('title')
 title.text = 'Ruby Programming'
 book.add_child(title)
+
+# Add entity reference (for declared entities)
+book.add_child(doc.create_entity_reference('mdash'))
+----
+
+=== Entity References
+
+Moxml supports `EntityReference` nodes for preserving entity syntax in XML documents. This enables round-trip preservation of entity references like ` `, `©`, and custom entities defined in the DOCTYPE.
+
+[source,ruby]
+----
+# Create entity reference programmatically
+ref = doc.create_entity_reference('nbsp')
+element.add_child(ref)
+
+# Or using the builder pattern
+doc = Moxml::Builder.new(Moxml.new).build do
+  element 'text' do
+    entity_reference 'ndash'
+    entity_reference 'copy'
+  end
+end
+----
+
+**Parsing and Round-Trip:**
+
+When parsing XML with declared entities, Moxml preserves entity references:
+
+[source,ruby]
+----
+# Parse document with custom entity
+xml = <<-XML
+<!DOCTYPE root [<!ENTITY nbsp " "> ]>
+<root>hello&nbsp;world</root>
+XML
+
+doc = Moxml.new(:nokogiri).parse(xml)
+doc.to_xml  # => preserves &nbsp; entity reference
+----
+
+**Adapter Notes:**
+
+- *Nokogiri*: Preserves custom declared entities as `EntityReference` nodes
+- *Ox, Oga*: These adapters resolve entities during parsing and do not expose entity reference nodes. Use Nokogiri or LibXML for entity preservation.
+
+**Entity Loading Configuration:**
+
+Moxml provides configurable entity loading with four modes to balance between functionality, performance, and security:
+
+[source,ruby]
 ----
+# Default: Load all W3C entities (HTML + MathML + ISO entity sets)
+# Raises error if entity data is unavailable
+context = Moxml.new
+
+# Optional: Load entities if available, silently skip if not
+context = Moxml.new do |config|
+  config.entity_load_mode = :optional
+end
+
+# Disabled: No entity loading (fastest, for controlled XML sources)
+context = Moxml.new do |config|
+  config.entity_load_mode = :disabled
+end
+
+# Custom: Load entities from your own source
+context = Moxml.new do |config|
+  config.entity_load_mode = :custom
+  config.entity_provider = -> { MyEntitySource.all_entities }
+end
+----
+
+The entity data comes from the W3C XML Core WG Character Entities specification (HTMLMathML set), bundled locally in `data/w3c_entities.json` for offline capability. Set the `MOXML_ENTITY_DEFINITIONS_PATH` environment variable to use a custom entity data source.
+
+For backward compatibility, `config.load_external_entities = false` maps to `:disabled` mode, and `config.load_external_entities = true` maps to `:required` mode.
 
 === Fluent interface API
 
@@ -643,6 +723,23 @@ For complete SAX documentation including all handler types, event methods, adapt
 
 For complete node API reference including traversal methods, manipulation, queries, type checking, and node information, see link:docs/_pages/node-api-reference.adoc[Node API Reference].
 
+=== Node identity
+
+Moxml provides a consistent `#identifier` method across all node types to safely identify nodes:
+
+[source,ruby]
+----
+element = doc.at_xpath("//book")
+puts element.identifier  # => "book"
+
+attr = element.attribute("id")
+puts attr.identifier     # => "id"
+----
+
+The `#identifier` method returns the primary identifier for each node type (tag name for elements, attribute name for attributes, target for processing instructions, or `nil` for content nodes).
+
+IMPORTANT: Always use type-safe patterns when working with mixed node types. See the link:docs/_guides/node-api-consistency.adoc[Node API Consistency Guide] for complete documentation on safe coding patterns, API surface by node type, and migration guidelines.
+
 
 == Advanced features
 
@@ -694,7 +791,8 @@ rescue Moxml::Error => e
 end
 ----
 
-For complete error class hierarchy, error types, best practices, and debugging techniques, see link:docs/_pages/error-handling.adoc[Error Handling Guide].
+For complete error class hierarchy, error types, best practices, and debugging
+techniques, see link:docs/_pages/error-handling.adoc[Error Handling Guide].
 
 
 == Configuration
@@ -717,22 +815,53 @@ context = Moxml.new do |config|
 end
 ----
 
-For all configuration options, adapter selection, serialization options, and environment-based configuration, see link:docs/_pages/configuration.adoc[Configuration Guide].
+=== Namespace URI validation
+
+Moxml validates namespace URIs against
+https://www.rfc-editor.org/rfc/rfc3986[RFC 3986] by default, as required by the
+https://www.w3.org/TR/xml-names/[W3C Namespaces in XML] specification.
+
+For documents that use non-standard namespace identifiers, a lenient mode is
+available:
+
+[source,ruby]
+----
+# Strict mode (default) — rejects invalid URIs per RFC 3986
+context = Moxml.new do |config|
+  config.namespace_uri_mode = :strict
+end
+
+# Lenient mode — accepts any string as a namespace URI
+context = Moxml.new do |config|
+  config.namespace_uri_mode = :lenient
+end
+----
+
+For all configuration options, adapter selection, serialization options, and
+environment-based configuration, see
+link:docs/_pages/configuration.adoc[Configuration Guide].
 
 
 
 == Thread safety
 
-For complete information on thread-safe patterns, context management, and concurrent processing, see the link:docs/_pages/thread-safety.adoc[Thread Safety Guide].
+For complete information on thread-safe patterns, context management, and
+concurrent processing, see the link:docs/_pages/thread-safety.adoc[Thread Safety
+Guide].
 
 
 == Performance considerations
 
-For detailed performance optimization strategies, memory management best practices, and efficient querying patterns, see the link:docs/_pages/performance.adoc[Performance Considerations Guide].
+For detailed performance optimization strategies, memory management best
+practices, and efficient querying patterns, see the
+link:docs/_pages/performance.adoc[Performance Considerations Guide].
 
 == Best practices
 
-For comprehensive best practices covering XPath queries, adapter selection, error handling, namespace handling, memory management, thread safety, performance optimization, and testing strategies, see link:docs/_pages/best-practices.adoc[Best Practices Guide].
+For comprehensive best practices covering XPath queries, adapter selection,
+error handling, namespace handling, memory management, thread safety,
+performance optimization, and testing strategies, see
+link:docs/_pages/best-practices.adoc[Best Practices Guide].
 
 
 == Specific adapter limitations
@@ -756,11 +885,13 @@ The Ox adapter provides maximum parsing speed but has XPath limitations.
 doc.xpath("//book").find { |book| book["id"] == "123" }
 ----
 
-For complete Ox adapter documentation including all limitations and workarounds, see link:docs/_pages/adapters/ox.adoc[Ox Adapter Guide].
+For complete Ox adapter documentation including all limitations and workarounds,
+see link:docs/_pages/adapters/ox.adoc[Ox Adapter Guide].
 
 === HeadedOx adapter
 
-The HeadedOx adapter combines Ox's fast C-based XML parsing with Moxml's comprehensive pure Ruby XPath 1.0 engine.
+The HeadedOx adapter combines Ox's fast C-based XML parsing with Moxml's
+comprehensive pure Ruby XPath 1.0 engine.
 
 **Status:** Production-ready v1.2 (99.20% pass rate, 1,992/2,008 tests)
 
@@ -796,12 +927,6 @@ For complete HeadedOx documentation including architecture, XPath capabilities,
 
 ==== LibXML adapter
 
-*DOCTYPE Limitations:*
-
-* DOCTYPE parsing works
-* DOCTYPE round-trip preservation is limited
-* DOCTYPE cannot be reliably re-serialized after parsing
-
 *Performance:*
 
 * Serialization speed: ~120 ips (slower than target)
@@ -817,9 +942,151 @@ limitations. Use these adapters when you need full XPath and namespace support.
 
 
 
+== Round-trip XML Testing
+
+Moxml includes comprehensive round-trip testing to verify that XML documents remain
+semantically equivalent when parsed and serialized across different adapters.
+
+=== Purpose
+
+Round-trip testing ensures:
+
+* **Cross-adapter compatibility** - XML parsed with one adapter (e.g., Nokogiri) can be
+  serialized and re-parsed with another adapter (e.g., Oga) while preserving content
+* **Structural fidelity** - Element names, attributes, and document structure are maintained
+* **Content preservation** - Text content and entity references survive multiple parse/serialize cycles
+* **Double round-trip verification** - Source → Target → Source sequences produce semantically
+  equivalent output
+
+=== Test Fixtures
+
+Round-trip tests use real-world XML documents organized into collections:
+
+**rfcxml** - IETF RFC documents in XML format. These provide complex, standards-compliant
+XML with mixed content, namespaces, and attributes. The collection includes:
+
+* Large documents (500KB-2.4MB) for stress testing
+* Rich metadata and cross-references
+* Various XML schema patterns
+
+**metanorma** - Metanorma document processing XML. These test:
+
+* Document structure preservation
+* Nested elements and complex hierarchies
+* Standard XML vocabularies
+
+**niso-jats** - NISO Journal Article Tag Suite XML. These provide:
+
+* Scholarly publishing XML schemas
+* Rich bibliographic metadata
+* Mixed content models
+
+=== Running Round-trip Tests
+
+[source,bash]
+----
+# Run all round-trip tests
+bundle exec rake spec:consistency
+
+# Exclude REXML for larger fixtures (faster, REXML is pure Ruby)
+MOXML_ROUNDTRIP_REXML_MAX_SIZE=0 bundle exec rake spec:consistency
+
+# Adjust the per-example timeout (default: 120 seconds)
+MOXML_ROUNDTRIP_TIMEOUT=300 bundle exec rake spec:consistency
+----
+
+REXML is a pure Ruby XML parser and becomes very slow on large documents (500KB+).
+By default, REXML adapter pairs are skipped for fixtures exceeding 500KB. All other
+adapters (Nokogiri, Oga, Ox) are tested against every fixture.
+
+=== Test Mechanics
+
+For each fixture, tests run across all adapter pairs (4 adapters = 12 combinations):
+
+1. Parse with source adapter
+2. Serialize to XML string
+3. Parse serialized output with target adapter
+4. Compare semantic equivalence (element names, attributes, text content)
+
+A "double round-trip" test additionally verifies: Source → Target → Source → Target
+produces consistent results.
+
+NOTE: REXML is excluded from adapter pairs for fixtures larger than 500KB (configurable
+via `MOXML_ROUNDTRIP_REXML_MAX_SIZE`). This is because REXML is pure Ruby and cannot
+parse large XML documents in a practical timeframe. A per-example timeout
+(`MOXML_ROUNDTRIP_TIMEOUT`, default 120s) prevents tests from hanging indefinitely.
+
+=== Ox Adapter Element Ordering Caveat
+
+The Ox adapter produces elements in a different order than other adapters for certain
+fixtures with complex nested structures (e.g., `element_citation.xml`,
+`collection1nested.xml`, `pnas_sample.xml`). This causes the `elements_with_attributes`
+comparison to fail with "Array length mismatch" even though the semantic equivalence
+check (double round-trip) passes.
+
+Round-trip tests automatically skip the `elements_with_attributes` comparison for these
+known Ox ordering issues. The `ruby-versions` CI job tests only Nokogiri and Oga adapters;
+the `nokogiri-ox` and `nokogiri-rexml` CI jobs test Ox and REXML respectively but are
+marked as **experimental** since these adapters lack full XML feature support:
+
+* **Ox**: Lacks proper namespace support, XPath with predicates, and uses a custom
+  `locate()` method instead of standard XPath
+* **REXML**: Pure Ruby, exponential time complexity with document size, impractical for
+  documents over ~500KB
+
+For production use, prefer Nokogiri or Oga which provide complete XML conformance.
+
+To run tests with a specific adapter set locally:
+
+[source,bash]
+----
+# Nokogiri + Oga only (fast, full test suite)
+MOXML_ROUNDTRIP_ADAPTERS=nokogiri,oga bundle exec rspec spec/consistency/ --tag round_trip
+
+# Nokogiri × Ox only (experimental)
+MOXML_ROUNDTRIP_ADAPTERS=nokogiri,ox MOXML_ROUNDTRIP_TIMEOUT=300 bundle exec rspec spec/consistency/ --tag round_trip
+
+# Nokogiri × REXML only (experimental, small fixtures due to exponential complexity)
+MOXML_ROUNDTRIP_ADAPTERS=nokogiri,rexml MOXML_ROUNDTRIP_TIMEOUT=300 MOXML_ROUNDTRIP_REXML_MAX_SIZE=50000 bundle exec rspec spec/consistency/ --tag round_trip
+----
+
+=== Why Semantic Equivalence?
+
+While a pure round-trip test with raw XML comparison would be ideal, different XML adapters
+have fundamentally different philosophies for handling:
+
+* **Element ordering** - Some preserve document order, others sort alphabetically
+* **Whitespace handling** - Some normalize spaces, others preserve exactly
+* **Attribute representation** - Different data structures for the same attributes
+* **Text extraction** - Varying approaches to concatenating text content
+
+Instead of raw comparison, Moxml implements semantic equivalence testing that focuses on
+meaningful XML structure and content:
+
+[source,ruby]
+----
+# Element name must match
+expect(target_element.name).to eq(source_element.name)
+
+# Attributes must be semantically equivalent
+expect(target_attributes).to eq(source_attributes)
+
+# Text content must be preserved (whitespace-normalized)
+expect(normalized_text(target)).to eq(normalized_text(source))
+
+# Document structure (element count) must match
+expect(doc.xpath("//*").size).to eq(original.xpath("//*").size)
+----
+
+This approach tolerates adapter-specific serialization differences while ensuring
+the actual XML content remains intact.
+
+
 == Development and testing
 
-For complete information on development setup, testing strategies, benchmarking, and coverage reporting, see the link:docs/_guides/development-testing.adoc[Development and Testing Guide].
+For complete information on development setup, testing strategies, benchmarking,
+and coverage reporting, see the
+link:docs/_guides/development-testing.adoc[Development and Testing Guide].
 
 == Contributing
 

data/Rakefile

@@ -30,6 +30,17 @@ namespace :spec do
     t.pattern = "spec/consistency/**/*_spec.rb"
   end
 
+  namespace :consistency do
+    desc "Run round-trip tests for a specific fixture category (CATEGORIES=metanorma,rfcxml,niso-jats)"
+    task :by_category do
+      categories = ENV.fetch("CATEGORIES", "").split(",").map(&:strip)
+      abort "Usage: CATEGORIES=metanorma,rfcxml rake spec:consistency:by_category" if categories.empty?
+
+      include_filters = categories.map { |c| "--tag fixture_category:#{c}" }.join(" ")
+      sh "bundle exec rspec spec/consistency/ --tag round_trip #{include_filters}"
+    end
+  end
+
   desc "Run example tests"
   RSpec::Core::RakeTask.new(:examples) do |t|
     t.pattern = "spec/examples/**/*_spec.rb"