moxml 0.5.12 → 0.5.13

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7f702d6c51c5df49bdb2bf0ac5189d3fe0716d94093ced7decdb743dcf87fa5d
-  data.tar.gz: 56316b016c8865d6290033873477d9dcc92b04e1e16ae77716626398bd976ad3
+  metadata.gz: 6f890c98078986e7be37d59762c08c6387cb173a7260212be428c705e043bd1a
+  data.tar.gz: 6a5dae862bb917c2906581412c33f0cd9f4506112ba5f24a146feed9f2dc0371
 SHA512:
-  metadata.gz: a1e503460e2b5d645cd4147d4164d2a67c254e8e2174b145fbf4f17dab222d1eae03216b7a35dca18a800684c3c904daf9633140f262195f212d9dd235ccc675
-  data.tar.gz: 4cf7957e370339230fc4d184a7fd6b9312bd3f5e8f9e843984ed4e56967e33597d18abef6479de69cde47af781a65703427349cdaff16eaf317cdb39ba8cd135
+  metadata.gz: d8acaee20452b9d74ba026cc4fce32d6a8911af51afee523bf474e831786d9e71c31e089360d023c17a62c8610081a8ad4d55b774261279e56966942ab26b040
+  data.tar.gz: c9e1d227bca2d9f7f5771604de905b5efec5a61451e01b69919ec315484673cafadbe65345423fd65633fcdc1d8e8197de58d6822f5bc42a6fa6fe253ec4b7f3

data/docs/_pages/adapter-protocol.adoc

@@ -0,0 +1,113 @@
+---
+title: Adapter protocol
+nav_order: 46
+---
+
+= The adapter protocol
+
+The seam between moxml's wrapper layer (`Moxml::Node` and friends) and the
+engine adapters. Every adapter is a class under `Moxml::Adapter` implementing
+class methods over its library's native nodes; wrappers dispatch through
+`context.config.adapter` and never touch a native directly.
+
+The contract is pinned by the shared example groups under
+`spec/moxml/adapter/shared_examples/` (the adapter contract) and the
+consistency suite — behavior there is authoritative when this page and the
+code disagree.
+
+== Method groups
+
+Parse and document lifecycle::
+`parse(xml, options, context)`, `create_document`, `create` factories
+(`create_native_element`, `create_native_text`, `create_native_cdata`,
+`create_native_comment`, `create_native_processing_instruction`,
+`create_native_declaration`, `create_native_doctype`,
+`create_native_entity_reference`, `create_native_namespace`),
+`free_document` (#134; no-op on GC-managed engines).
+
+Tree navigation::
+`children`, `parent`, `next_sibling`, `previous_sibling`, `root`,
+`set_root`, `document`, `duplicate_node`. `children` on a document
+lists document-level parts in document order where the engine exposes
+them (Nokogiri-shaped contract).
+
+Node typing and names::
+`node_type`, `node_name`, `set_node_name`, `attribute_name`.
+
+Attributes::
+`get_attribute` / `get_attribute_value` (the raw layer beneath
+`Moxml::AttributeResolver` — expanded-name resolution lives above the
+adapter), `set_attribute_value`, `remove_attribute`,
+`patch_node`/`unpatch_node` for engines needing wrapper natives
+(libxml, ox).
+
+Namespaces::
+`namespace`, `namespace_definitions` (the element's own declarations),
+`in_scope_namespaces`, `set_namespace`, `remove_namespace`.
+
+Mutation::
+`add_child`, `add_next_sibling`, `add_previous_sibling`, `remove`,
+`replace`, `replace_children`. Return contract: methods that can swap
+the native (`set_attribute_name`, `set_namespace`, adoptions) return
+the tracked native; wrappers re-key through `refresh_native!`.
+
+Serialization::
+`serialize(node, options)`. Options: `indent`, `encoding`,
+`declaration`, `no_declaration`, `expand_empty`, `line_ending`.
+Entity-marker restoration is owned by the wrapper layer
+(`Node#to_xml`), guarded by `entity_bearing?`.
+
+XPath::
+`xpath(expression, namespaces)`. Engines without a native engine use
+moxml's pure-Ruby XPath 1.0 engine (`Moxml::XPath`).
+
+== Capability predicates
+
+The wrapper layer probes adapters rather than assuming:
+
+`patches_children?`::
+True when `children` results need per-child `patch_node` rewriting
+(libxml, ox). Others skip the identity map over every child list.
+
+`wrappers_recyclable?`::
+False for engines that mint fresh Ruby objects per access (libxml) —
+the context's wrapper identity map opts out.
+
+`entity_bearing?(native)`::
+Whether the subtree can contain entity markers; gates the
+post-serialize restore scan.
+
+`bulk_materialize?`::
+Whether the adapter offers a bulk path for `Moxml::Materializer`;
+`materialize_records` yields flattened records, returning nil falls
+back to the generic wrapper walk.
+
+Constants like `DOC_NODE_SUPPORTED` / `DTDATTR_SUPPORTED` /
+`TRAVERSE_SUBTREE_BOUNDED` (Leptris) follow the same idea: one probe
+per engine capability, computed at load.
+
+== Return shapes
+
+* Mutations return the tracked native (or nil when not applicable) —
+  never a wrapper.
+* `children` returns an array of natives; the wrapper layer builds the
+  `NodeSet`.
+* `xpath` returns `Array<native>` or a scalar for non-node results.
+* Marker-bearing text (leptris) is split by the adapter into
+  text/entity-reference pseudo-natives before the wrapper sees it.
+
+== Writing an adapter
+
+Subclass `Moxml::Adapter::Base`, implement the groups above, and
+include the shared examples:
+
+[source,ruby]
+----
+RSpec.describe Moxml::Adapter::Mine do
+  it_behaves_like "xml adapter"
+  it_behaves_like "adapter contract"
+end
+----
+
+The contract suite is the test surface: if it passes, the wrapper
+layer's behavior is covered.

data/docs/_pages/adapters/index.adoc

@@ -14,15 +14,21 @@ how to choose the right adapter for your needs.
 === What are adapters?
 
 Moxml uses an adapter pattern to provide a unified interface over different
-XML processing libraries. Each adapter wraps a specific XML library (Nokogiri,
-LibXML, Oga, REXML, or Ox) and provides consistent behavior through Moxml's
-API.
+XML processing libraries. Each adapter wraps a specific XML library (Leptris,
+Nokogiri, LibXML, Oga, REXML, or Ox) and provides consistent behavior through
+Moxml's API. See link:../adapter-protocol[the adapter protocol] for the seam
+itself.
 
 === Available adapters
 
+link:leptris[Leptris adapter]::
+Pure-C99 XML engine, the default on CRuby when the installed binding
+supports programmatic documents — fastest on every measured operation,
+with native XPath 1.0, SAX, and deterministic memory release.
+
 link:nokogiri[Nokogiri adapter]::
-Industry standard XML library with excellent performance and full XPath 1.0
-support. Recommended for most use cases.
+The industry-standard XML library and the fallback default when Leptris
+is not installed. Excellent performance with full XPath 1.0 support.
 
 link:libxml[LibXML adapter]::
 Alternative to Nokogiri using native libxml2 bindings. Excellent performance

data/docs/_pages/adapters/leptris.adoc

@@ -0,0 +1,69 @@
+---
+title: Leptris
+parent: Adapters
+nav_order: 1
+---
+
+== Leptris adapter
+
+Leptris is a pure-C99 XML 1.0 parser with an FFI binding (the
+https://github.com/leptris/leptris-ruby[leptris-ruby] gem). It is the
+preferred default on CRuby: fastest on every measured moxml operation,
+with a native XPath 1.0 engine, SAX, and canonicalization.
+
+== Default resolution
+
+Leptris becomes the default when the installed binding supports
+programmatic document construction (`Leptris::XML::Document.create`,
+leptris >= 1.3). Otherwise the default falls back to Nokogiri. Under
+Opal the default is rexml (stock oga requires its C extension and
+libleptris does not build for Opal).
+
+[source,ruby]
+----
+Moxml.new                     # => Leptris context when capable
+ctx.config.adapter_name        # => :leptris / :nokogiri / :rexml (Opal)
+----
+
+== Parse options
+
+`readonly: true`::
+The binding memoizes every read and refuses mutations — the
+parse-and-read lifecycle for multi-pass consumers (comparison, diff,
+signatures). Repeat attribute reads on a large element drop by
+orders of magnitude. Builder documents stay read-write.
+
+`dtdattr: true`::
+Materialize DTD `<!ATTLIST>` defaults. Off by default, matching
+libxml2/Nokogiri/REXML semantics (libleptris >= 1.9.8).
+
+`strict: true`::
+Raise `Moxml::ParseError` on malformed input instead of returning an
+empty document.
+
+== Lifecycle
+
+`Document#free` releases the C tree deterministically — batch
+workloads parsing thousands of documents otherwise hold native memory
+until GC finalizers run. Access after `#free` raises the binding's
+use-after-free error.
+
+== Document model
+
+Documents expose the libxml2-shaped document node: prolog PIs and
+comments, the root element, epilog PIs and comments, in document
+order (leptris >= 1.9.26). The DOCTYPE round-trips with its internal
+subset. See link:../conversion-apis[Conversion and memory lifecycle APIs]
+for `materialize` — the flattened-record conversion path with a bulk
+implementation over one `leptris_node_traverse` call.
+
+== Known limitations (tracked upstream)
+
+* Tab indentation (`indent_text`) needs a serialize-time parameter in
+  libleptris; post-hoc rewriting would corrupt mixed content.
+* Document-level PIs are read-only natives (`target=`/`data=`/
+  `unlink` rejected); write-through and removal are pending C
+  surface (leptris/leptris#612).
+* A PI preceding the DOCTYPE reorders on serialization — libleptris
+  normalizes DOCTYPE-first and the document node does not record its
+  relative position.

data/docs/_pages/adapters/libxml.adoc

@@ -1,7 +1,7 @@
 ---
 title: LibXML
 parent: Adapters
-nav_order: 2
+nav_order: 4
 ---
 
 == LibXML adapter

data/docs/_pages/adapters/nokogiri.adoc

@@ -1,7 +1,7 @@
 ---
 title: Nokogiri
 parent: Adapters
-nav_order: 1
+nav_order: 3
 ---
 
 == Nokogiri adapter

data/docs/_pages/conversion-apis.adoc

@@ -0,0 +1,78 @@
+---
+title: Conversion and memory lifecycle APIs
+nav_order: 45
+---
+
+= Conversion and Memory Lifecycle APIs
+
+Scanned streaming (`materialize`), deterministic release (`Document#free`), and parse-lifecycle options (`readonly:`, `dtdattr:`) — the seams for conversion-heavy and batch workloads.
+
+== Bulk materialization — `materialize` (issue #132)
+
+Flattened, post-order records for a subtree with no `Moxml::Node` allocation — the conversion path for consumers that build their own tree from a document:
+
+[source,ruby]
+----
+context.materialize(xml) { |record| ... }   # parse + flatten
+document.materialize.each                    # Enumerator, root subtree
+any_element.materialize                      # any subtree
+----
+
+Record shape (all seven keys on every record):
+
+[source,ruby]
+----
+{ kind: :element|:text|:cdata|:comment|:processing_instruction|:entity_reference,
+  qname: "tag"|"pi-target"|nil,
+  prefix: "p"|nil,
+  namespace_uri: "urn:x"|nil,
+  namespaces: [[prefix, uri], ...],              # element's OWN declarations; nil prefix = default
+  attributes: [[name, value, namespace_uri, prefix], ...],
+  text: String|nil,
+  depth: Integer }
+----
+
+* Post-order: children arrive before their parent — build with a stack.
+* `Document#materialize` yields exactly the root subtree (issue #140); document-level parts (prolog/epilog PIs and comments, DOCTYPE) stay enumerable via `children`.
+* The leptris adapter has a bulk path (one `leptris_node_traverse` call); every other adapter uses the generic wrapper walk. Both emit identical streams — the parity is spec-pinned.
+* Entity-bearing documents fall back to the generic walk so entity references still split.
+
+== Deterministic release — `Document#free` (issue #134)
+
+Adapters backed by C trees hold native memory until a GC finalizer runs. Batch workloads (parse → convert → discard, thousands of documents) release it deterministically:
+
+[source,ruby]
+----
+doc = context.parse(xml)
+doc.free            # C tree released now; attachments swept
+doc.root            # => raises the engine's use-after-free error (leptris)
+----
+
+`#free` is a no-op on GC-managed engines (nokogiri, oga, ox, rexml). Normally-scoped documents keep working via the finalizer either way. Measured with GC disabled, 1000 × ~115KB parses: +1.2GB RSS without `#free` vs +121MB with it.
+
+== Parse-lifecycle options
+
+Both options pass through `Context#parse` to the engine and are no-ops where the concept does not apply:
+
+* `readonly: true` (issue #133, leptris) — memoized reads, refused mutations. The parse-and-read lifecycle for multi-pass consumers (comparison, diff, signature): repeat attribute reads on a 100-attribute element drop from ~99µs to ~0.15µs (680×). Builder documents stay read-write.
+
+* `dtdattr: true` (libleptris 1.9.8+, leptris/leptris#606) — materialize DTD `<!ATTLIST>` defaults. Off by default, matching libxml2/Nokogiri/REXML semantics:
+
+[source,ruby]
+----
+context.parse(dtd_xml)                     # plain: ATTLIST defaults excluded
+context.parse(dtd_xml, dtdattr: true)      # opt-in: attr="default" materialized
+----
+
+== Pretty-print parity (issue #129)
+
+Through the leptris adapter, byte-identical output to raw Nokogiri:
+
+[source,ruby]
+----
+context.parse(src).to_xml(
+  indent: 2, declaration: true, expand_empty: false, encoding: "UTF-8",
+) == Nokogiri::XML(src).to_xml(indent: 2, encoding: "UTF-8")   # => true
+----
+
+Known residual divergences (engine-level, tracked upstream): tab indentation needs a serialize-time parameter in libleptris, and a PI preceding the DOCTYPE reorders (the engine normalizes DOCTYPE-first).

data/docs/_pages/index.adoc

@@ -28,6 +28,12 @@ Compare features, performance, and compatibility across adapters.
 link:configuration[Configuration]::
 Configure Moxml for your application's requirements.
 
+link:conversion-apis[Conversion and memory lifecycle APIs]::
+Bulk materialization, deterministic document release, and parse-lifecycle options for conversion-heavy workloads.
+
+link:adapter-protocol[The adapter protocol]::
+The seam between the wrapper layer and engine adapters — method groups, capability predicates, return shapes, and how to add an adapter.
+
 link:error-handling[Error handling]::
 Comprehensive guide to Moxml's error classes and debugging.
 

data/lib/moxml/adapter/leptris/document_parts.rb

@@ -0,0 +1,208 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    class Leptris
+      module DocumentParts
+        # Issue #134: deterministic release of the C tree. The binding
+        # clears its wrapper cache and raises UseAfterFreeError on
+        # later access; moxml-side attachments for the document are
+        # swept too (the context wrapper identity map self-cleans via
+        # its size valve).
+        DOCUMENT_ATTACHMENT_KEYS = %i[
+          entity_markers doc_pi_nodes declaration doctype
+          had_source_declaration document_text
+        ].freeze
+
+        def free_document(native)
+          DOCUMENT_ATTACHMENT_KEYS.each { |key| attachments.delete(native, key) }
+          native.free
+          nil
+        end
+
+        def assemble_document_children(doc)
+          children = []
+
+          native_doctype = doc.doctype
+          children << native_doctype if native_doctype
+
+          doctype_wrapper = attachments.get(doc, :doctype)
+          children << doctype_wrapper if doctype_wrapper
+
+          if DOC_NODE_SUPPORTED
+            # The libxml2-model document node lists prolog PIs/comments,
+            # the root, and epilog PIs/comments in document order —
+            # the Nokogiri-shaped contract, epilog anchoring included
+            # (issue #130). Built (programmatic) documents are not yet
+            # fully reflected by the node (binding gap: an attached
+            # root does not appear); document_node_children answers
+            # nil there for the legacy parts path.
+            doc_children = document_node_children(doc)
+            if doc_children
+              children.concat(doc_children)
+            else
+              children.concat(document_pi_nodes(doc))
+              children << doc.root if doc.root
+            end
+          else
+            # Legacy path: document-level PIs live outside the element
+            # tree in a flat pre-root list (libleptris < 1.9.7 C
+            # model); epilog anchoring is not representable there.
+            children.concat(document_pi_nodes(doc))
+
+            children << doc.root if doc.root
+          end
+
+          texts = attachments.get(doc, :document_text)
+          children.concat(texts) if texts
+          children
+        end
+
+        # Document-level PI pseudo-nodes, materialized once from the C
+        # list and cached per document: children and serialization read
+        # the same objects, so wrapper mutations round-trip. Build the
+        # cache BEFORE appending a PI with add_pi, or the C-side
+        # addition would be double-counted.
+        # The document node's children, or nil when the node does not
+        # reflect reality: parsed documents always list the root
+        # element among their children, but programmatically built
+        # ones do not (binding gap) — those keep the legacy parts
+        # path.
+        def document_node_children(doc)
+          doc_children = doc.children.to_a
+          has_root = doc_children.any?(::Leptris::XML::Element)
+          return doc_children if has_root
+          return doc_children if doc.root.nil?
+
+          nil
+        end
+
+        def document_pi_nodes(doc)
+          attachments.get(doc, :doc_pi_nodes) || begin
+            nodes = doc.processing_instructions.map do |(target, data)|
+              CustomizedLeptris::DocumentPI.new(target, data, doc)
+            end
+            attachments.set(doc, :doc_pi_nodes, nodes)
+            nodes
+          end
+        end
+
+        def add_document_child(doc, child)
+          case child
+          when CustomizedLeptris::Declaration
+            child.parent_doc = doc
+            attachments.set(doc, :declaration, child)
+          when CustomizedLeptris::Doctype
+            child.parent_doc = doc
+            attachments.set(doc, :doctype, child)
+          when ::Leptris::XML::DocType
+            raise Moxml::DocumentStructureError.new(
+              "libleptris does not support attaching a native DocType to a document",
+            )
+          when ::Leptris::XML::Element
+            doc.root = child
+          when ::Leptris::XML::ProcessingInstruction
+            document_pi_nodes(doc)
+            doc.add_pi(child.target, child.content.to_s)
+            document_pi_nodes(doc) << CustomizedLeptris::DocumentPI.new(
+              child.target, child.content.to_s, doc
+            )
+          when CustomizedLeptris::DocumentPI
+            document_pi_nodes(doc)
+            doc.add_pi(child.target, child.data)
+            document_pi_nodes(doc) << child
+          when ::Leptris::XML::Text
+            texts = attachments.get(doc, :document_text) || []
+            texts << child
+            attachments.set(doc, :document_text, texts)
+            child
+          else
+            raise Moxml::DocumentStructureError.new(
+              "Unsupported document child: #{child.class}",
+            )
+          end
+          child
+        end
+
+        # Documents compose from their parts: the native serializer
+        # only walks the root subtree, so declaration, DOCTYPE, PIs and
+        # document-level text are assembled around it explicitly.
+        def serialize_document(doc, options)
+          # Nokogiri's document shape: every top-level part is
+          # newline-terminated, at any indent — declaration, DOCTYPE,
+          # document PIs, the root element, trailing newline after it.
+          # Document-level text is content, not structure: no added
+          # newline.
+          parts = []
+
+          include_decl = !options[:no_declaration] && options.fetch(:declaration) do
+            document_has_declaration?(doc)
+          end
+          if include_decl
+            declaration = attachments.get(doc, :declaration)
+            parts << (declaration ? declaration.to_xml : default_declaration_xml(doc, options)) << "\n"
+          end
+
+          doctype = attachments.get(doc, :doctype)
+          parts << doctype.to_xml << "\n" if doctype
+
+          native = native_doctype_xml(doc)
+          parts << native << "\n" if native
+
+          if DOC_NODE_SUPPORTED
+            # The libxml2-model document node: prolog PIs/comments,
+            # the root, epilog PIs/comments — in document order, so
+            # epilog parts serialize after the root (issue #130).
+            doc_children = document_node_children(doc)
+            if doc_children
+              doc_children.each { |child| parts << raw_serialize(child, options) << "\n" }
+            else
+              document_pi_nodes(doc).each { |pi| parts << pi.to_xml << "\n" }
+              parts << raw_serialize(doc.root, options) << "\n" if doc.root
+            end
+          else
+            document_pi_nodes(doc).each { |pi| parts << pi.to_xml << "\n" }
+
+            parts << raw_serialize(doc.root, options) << "\n" if doc.root
+          end
+
+          texts = attachments.get(doc, :document_text)
+          texts&.each { |text| parts << XmlEmitter.escape_text(text.content.to_s) }
+
+          parts.join
+        end
+
+        def native_doctype_xml(doc)
+          dt = doc.doctype
+          return nil unless dt
+
+          subset = dt.internal_subset if dt.class.method_defined?(:internal_subset)
+          XmlEmitter.doctype_xml(dt.root_name, dt.public_id, dt.system_id, subset)
+        end
+
+        def default_declaration_xml(doc, options)
+          encoding = options[:encoding] || doc.encoding
+          encoding = "UTF-8" if encoding.to_s.empty?
+          XmlEmitter.declaration_xml("1.0", encoding, nil)
+        end
+
+        def document_has_declaration?(native)
+          return false unless native.is_a?(::Leptris::XML::Document)
+
+          return true if attachments.get(native, :declaration)
+
+          attachments.get(native, :had_source_declaration) ? true : false
+        end
+
+        def marker_text_for(parent, name)
+          return nil unless parent.is_a?(::Leptris::XML::Element)
+
+          marker = "#{Entity::MARKER}#{name};"
+          parent.children.to_a.find do |child|
+            child.is_a?(::Leptris::XML::Text) && child.content == marker
+          end
+        end
+      end
+    end
+  end
+end

data/lib/moxml/adapter/leptris/markers.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+module Moxml
+  module Adapter
+    class Leptris
+      module Markers
+        # Marker presence is a document-level fact: parse records it,
+        # the ER builder path flips it, and the split/restore scans
+        # consult it. Customized natives only exist as split products
+        # of marker-bearing text, so they always report true.
+        def entity_bearing?(native)
+          case native
+          when CustomizedLeptris::Declaration, CustomizedLeptris::Doctype,
+             CustomizedLeptris::EntityReference, CustomizedLeptris::TextSegment,
+             CustomizedLeptris::DocumentPI
+            true
+          else
+            doc = native.document
+            doc.nil? || attachments.get(doc, :entity_markers) != false
+          end
+        end
+
+        # Expand marker-bearing text nodes into the child sequence the
+        # moxml contract exposes: text, EntityReference, text, ...
+        def split_entity_markers(natives, parent)
+          result = []
+          natives.each do |child|
+            # FFI Text#content returns a fresh unfrozen BINARY string:
+            # retag in place (dup would be a throwaway allocation), but
+            # before include? — BINARY.include? with the UTF-8 marker raises
+            if child.is_a?(::Leptris::XML::Text)
+              content = child.content
+              content.force_encoding("UTF-8")
+            end
+            if content&.include?(Entity::MARKER)
+              content.scan(/([^#{Entity::MARKER}]*)(?:#{Entity::MARKER}([\w.:-]+);)?/o) do
+                text_part = Regexp.last_match(1)
+                name = Regexp.last_match(2)
+                result << CustomizedLeptris::TextSegment.new(text_part, parent) unless text_part.empty?
+                result << CustomizedLeptris::EntityReference.new(name) if name
+              end
+            else
+              result << child
+            end
+          end
+          result
+        end
+      end
+    end
+  end
+end