moxml 0.5.30 → 0.5.32

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b9c5b752aba6a52719fbeb794be70155e0fc52a553889e424b3cfbc0a4afa324
-  data.tar.gz: ee7a443060fc562236ccfb135c509ae65e51ba3ec930cd8a39cb3a917f5e41ed
+  metadata.gz: 627111c496efca4687eabeeb78989d611d3d308adc5ed32443c7058da51c57c9
+  data.tar.gz: 86d1cc6c471d07084e0d2e0de9412b286b5486b202f9fe677b7417f514c1fe55
 SHA512:
-  metadata.gz: d92bc0324a80312556a369b6b1ce7b1619c13ef23fac2212c3148211deb7c786543b83568ea6a1d312df9e09189f1d8feabf8b5d44840618852124d4c5550dd7
-  data.tar.gz: 9a0b511e096230051cd1e92ae3e36603b02ddfe2f2c03cfaf6d8072b21defda2e65938c4f9963516d3d3696a44a7689d98cc68244bd60e26672c49e04d292079
+  metadata.gz: f1108c98dc8af92913605dd08b84eb50bbfec92a5dd286e15656c3ecd57937ce56bc186c9393a740f6bd8a8316f06fb059db226f20ffc9f0b84cfb433e1a532d
+  data.tar.gz: 02d109a32659b30fc34ac5d6d3f01bc0d32d9e1ca6a38cfdc2fe3880a02c779ce5ff41fa2f0eda04adc2a9ccd6702f372d29b4d2a6fe33faeb0456cb8852d26b

data/docs/_pages/adapter-protocol.adoc

@@ -29,6 +29,13 @@ whitespace-only text nodes where the engine has the flag
 (#147; recover-mode diagnostics — `[]` on clean parses; engines with
 an error channel map it here, e.g. Nokogiri's `doc.errors`).
 
+HTML mode::
+`parse_html(html, options, context)` — tolerant HTML4/5 parsing into
+the standard DOM, where the engine offers it (leptris >= 1.9.80 via
+engine #659; Nokogiri via libxml2's HTML parser). The base class
+raises `Moxml::AdapterError` — adapters without an HTML mode keep
+that default.
+
 Tree navigation::
 `children`, `parent`, `next_sibling`, `previous_sibling`, `root`,
 `set_root`, `document`, `duplicate_node`. `children` on a document

data/docs/_pages/adapters/leptris.adoc

@@ -51,6 +51,74 @@ libxml2/Nokogiri/REXML semantics (libleptris >= 1.9.8).
 Raise `Moxml::ParseError` on malformed input instead of returning an
 empty document.
 
+== HTML parsing
+
+`Context#parse_html(html)` (leptris >= 1.9.80; engine
+leptris/leptris/#659) parses tolerant HTML4/5 into the standard XML
+DOM: implied end tags (`<li>a<li>b`), void elements (`<br>`, `<img>`),
+raw-text script/style, case-insensitive lowercased names and
+attributes, minimized/boolean attributes, and the HTML named-entity
+table. `html`/`head`/`body` are synthesized when the input implies
+them (no empty head for headless fragments); tables get no implied
+`tbody`. Malformed input degrades to text — the parse never fails;
+only an entirely empty result raises `Moxml::ParseError`.
+
+[source,ruby]
+----
+doc = Moxml.new(:leptris).parse_html(%(<ul><li>a<li>b</ul>))
+doc.xpath("//li").map(&:text)   # => ["a", "b"]
+----
+
+Serialization is XML-shaped: text content is escaped (`a &lt; b`),
+so script bodies with markup characters round-trip through a strict
+XML reparse. The adapter stands the entity-marker pipeline down for
+HTML parses — the engine decodes HTML entities directly into text.
+
+Measured on 1.9.105+ (named-entity decoding fixed, engine #848):
+parse 2.5-3.5x, serialize 4.1x, xpath 6-29x faster than Nokogiri —
+uniformly, including entity-laden pages.
+
+Since 1.9.118/1.9.121 the engine also handles WHATWG foreign
+content — embedded SVG and MathML in HTML parse into the tree
+(preserving `foreignObject` subtrees that Nokogiri drops), the
+source HTML doctype round-trips, and `template`/`colgroup` place
+correctly. Foreign-content camelCase names are currently
+lowercased (engine conformance note leptris/leptris#1013).
+
+Adapters without an engine HTML mode (Ox, Oga, REXML, LibXML) raise
+`Moxml::AdapterError`; Nokogiri parses through libxml2's HTML mode.
+
+== Streaming iteration — `iterparse`
+
+`Context#iterparse(xml, mode:)` / `iterparse_file(path, mode:)`
+expose the engine's incremental parser (libleptris v1.6, #586):
+completed elements yield while parsing, prior subtrees release, and
+memory stays bounded by the largest subtree — `iterparse_file`
+streams the file C-side. `:top_level` yields the root's children;
+`:full_document` yields every element in completion (post-order)
+order.
+
+Measured on a 21MB / 108k-element file: **+20MB peak RSS** (full
+parse: +169MB; Nokogiri Reader: +62MB), bare iteration ~97ms on
+leptris >= 1.9.105 (Nokogiri Reader: 179ms) — ~1.8x faster at a
+third of the memory. Attribute reads on yielded elements improved
+~16x on current builds (the parentless-read cost filed in
+leptris-ruby#152 is fixed).
+
+The yielded elements are parentless (`#document` nil) and valid
+only inside the block — the subtree is released when the iterator
+advances. Since binding 1.9.105, access after the iteration raises
+`UseAfterFreeError` instead of crashing (leptris-ruby#152). Adapters without an engine iterator raise
+`Moxml::AdapterError`.
+
+[source,ruby]
+----
+Moxml.new(:leptris).iterparse_file("huge.xml") do |record|
+  process(record.at_attrs, record.children.map(&:text))
+end
+----
+
+== Lifecycle
 == Lifecycle
 
 `Document#free` releases the C tree deterministically — batch

data/docs/_pages/conversion-apis.adoc

@@ -89,6 +89,24 @@ context.parse(dtd_xml)                     # plain: ATTLIST defaults excluded
 context.parse(dtd_xml, dtdattr: true)      # opt-in: attr="default" materialized
 ----
 
+== HTML input — `parse_html`
+
+Web-scraped and HTML-fragment inputs parse tolerantly into the
+standard DOM (leptris >= 1.9.80 via engine #659; Nokogiri via
+libxml2's HTML mode; other adapters raise `Moxml::AdapterError`):
+
+[source,ruby]
+----
+doc = context.parse_html(%(<ul><li>a<li>b</ul>))
+doc.xpath("//li").map(&:text)   # => ["a", "b"]
+----
+
+Implied end tags, void elements, lowercased names, boolean
+attributes, and HTML named entities decode at parse; serialization
+is XML-shaped. See the leptris adapter page for the semantics
+matrix. On leptris >= 1.9.105 every HTML operation — including
+entity-laden pages — measures 2.5-4x faster than Nokogiri.
+
 == Pretty-print parity (issue #129)
 
 Through the leptris adapter, byte-identical output to raw Nokogiri:

data/docs/_pages/performance.adoc

@@ -56,6 +56,148 @@ Consumers on CRuby 3.3+ (canon's pretty-print pipeline, batch
 conversion) should treat it as the default; moxml's own CI
 performance gates run with YJIT enabled.
 
+=== Wrapper retention — the identity map is weak
+
+`Context` keeps a native→wrapper identity map so repeated traversals
+hand back the same wrapper. Since 0.5.31 it is an
+`ObjectSpace::WeakMap`: entries die with their native, so
+parse-and-drop workloads release wrappers, natives, and (once the
+binding's finalizer runs) the C subtrees. Measured walking and
+dropping 220 documents (~600 elements each) with one long-lived
+Context: **+1.9MB RSS growth vs +41MB before**, zero wrapper
+retention beyond the binding's own constant one-document cache —
+and no identity loss, which the previous 65,536-entry wholesale
+clear traded away when it fired.
+
+=== Measuring moxml — methodology and traps
+
+Every number in this page follows the same discipline; contributors
+benchmarking moxml should too. The traps below all produced wrong
+conclusions at least once.
+
+*Timing:*
+
+- min-of-N (>= 10 samples) around loops, never single-shot averages;
+  `Process.clock_gettime(Process::CLOCK_MONOTONIC)`; `GC.start`
+  before each round.
+- Time LOOPS, not single calls: single sub-microsecond operations
+  read as 0 on some clocks (macOS CLOCK_MONOTONIC granularity).
+- Gate on load average — shared machines under build load (load 30+)
+  swing identical runs 2x. Wait for a calm window (< 5).
+- One heavy benchmark at a time; concurrent suites and benchmarks
+  have crashed this machine.
+- dup benchmarks must free each copy — native trees accumulate
+  otherwise and the numbers degrade monotonically (that drift is a
+  leak signal, not noise).
+
+*Allocation counts (GC.stat(:total_allocated_objects)) are
+load-immune — use them when the machine is busy:*
+
+- WARM THE PATH FIRST: the first call through any code path pays
+  autoloads (adapter, xpath engine, serialize modules — dozens of
+  classes and regexps). A cold first run charges those to whichever
+  variant ran first and fabricates "superadditive" overhead.
+- Keep the fixture shape identical across compared cells: parent
+  vs leaf elements allocate differently in every engine.
+- Negative deltas are impossible — if you see one, the harness is
+  broken, not the code.
+
+*Correctness under load:* race repros and corruption checks need
+hundreds of iterations — spot checks of 30 have missed 5-30% races
+on this codebase.
+
+=== leptris 1.9.105 numbers (reference table)
+
+XML parse 2.4-2.8x, document serialize 2.4x, element queries 32x,
+attribute-node queries 1.4x (native since 1.9.105), HTML parse
+2.5-3.5x including entity-laden pages (engine #848 fixed), HTML
+serialize 4.1x, streaming iteration ~97ms/21MB at +21MB RSS (~1.8x Nokogiri
+Reader), C14n 1.0 native on leptris >= 1.9.121 (engine #919 fixed)
+at 1.8x+ Nokogiri through the full document path.
+
+SAX (calm-machine, verified): the raw binding parses 1.19x faster
+than Nokogiri; through moxml's unified handler the API contract
+(per-event attribute Hash + namespace separation) nets 0.74x interp
+/ 0.93x YJIT.
+
+=== Element serialization
+
+Bulk `element.to_xml` (per-element, no options) was wrapper-tax
+bound — the raw engine serializes 3x faster than Nokogiri but the
+Ruby option/guard layers spent 4.6x the C call's cost per element.
+The argless path is now allocation-free (frozen prebuilt defaults),
+the element face skips the declaration fetch, UTF-8 requests ride
+the binding's shared default options, the trailing-newline strip is
+version-gated (engine fixed in 1.9.42), and `entity_bearing?` memoizes
+against an adapter generation counter. Measured per element:
+3-child **0.74x -> 1.0x** (parity — fixed guard tax on tiny output),
+21-child **1.7x**; document-level serialize stays 2.4x.
+
+=== Held-document footprint
+
+Four lifecycles, measured (RSS, isolated processes; leptris 1.9.80
+vs Nokogiri/libxml2):
+
+| lifecycle | moxml(leptris) | nokogiri | |
+|---|---|---|---|
+| bare tree (parse+hold, unaccessed) | **203 B/node** | 408 B/node | engine tree 2.0x smaller |
+| parse + walk + drop | leak-free (weak registry) | — | `#free` for deterministic release |
+| peak transient, 3.6MB doc (parse+pretty-print) | +48.8MB | +55.6MB | **0.88x smaller peak** |
+| held + walked (37KB doc) | 1186 kB/doc | 754 kB/doc | 1.57x larger — binding wrapper layer |
+
+The one regression is held-and-walked: the binding's per-node Ruby
+wrappers (element 80 B + a dedicated 88 B `FFI::Pointer` + cache
+slot ≈ 170+ B/node) outweigh even the C tree; upstream
+leptris-ruby#147 tracks it with a TypedData fix path. Guidance for
+held workloads: keep documents you will not re-traverse in the
+parse-and-read lifecycle (`readonly:`, `#free`), and prefer
+`materialize` for one-shot conversion.
+
+=== Programmatic construction
+
+Building documents node-by-node is the one surface where the C
+engine's lead does not carry through: each create/attach/attribute
+write crosses the FFI seam, and Nokogiri's C-extension calls are
+cheaper per crossing. Measured end to end:
+
+| builder shape | interp | YJIT |
+|---|---|---|
+| node churn (1052 tiny nodes — worst case) | 0.40x | 0.78x |
+| payload-realistic (attrs + long text) | 0.58x | **~1.0x (parity)** |
+
+Run builders with YJIT (`RUBY_YJIT_ENABLE=1`, see above): it
+compiles moxml's wrapper layer away (-47% on the churn shape) while
+leaving Nokogiri's thin C layer unchanged. The remaining churn-shape
+gap is the binding's own floor — even a zero-overhead wrapper
+measures 0.90x under YJIT (leptris-ruby#149 tracks it, with a
+batch-create proposal to collapse the crossings).
+
+=== Bulk construction — `append_xml`
+
+`element.append_xml(fragment)` appends a raw fragment's top-level
+nodes by parsing it and attaching the subtrees — the engine's C
+parser does the node construction in one crossing instead of an FFI
+round trip per create/attach/attribute call:
+
+| 300-node subtree | time | vs per-node |
+|---|---|---|
+| per-node build (leptris) | 2065µs | — |
+| **append_xml (leptris)** | **1019µs** | **2.0x** |
+| per-node build (nokogiri) | 1746µs | append_xml is 1.7x faster |
+
+The fragment must be namespace-self-contained and well-formed as
+the content of one wrapper element. Supported on leptris, nokogiri,
+oga, and rexml; the Ox adapter raises (its customized node wrappers
+do not survive cross-document attachment).
+
+=== Lazy result sets
+
+`xpath` results and `children` node sets allocate wrapper slots on
+first access, not at construction — `.size`/`.empty?`/`.first` on a
+1000-node result no longer allocate the 1000-slot wrapper array
+(that path also skips the binding's per-node wrapper materialization
+entirely; see `LazyNodeSet` under the leptris adapter).
+
 === Prefer bulk paths for per-node conversion
 
 Per-node Ruby iteration pays the wrapper tax per element. When the

data/lib/compat/opal/moxml_boot.rb

@@ -12,6 +12,7 @@ require "moxml/xml_utils"
 require "moxml/xml_utils/encoder"
 require "moxml/node"
 require "moxml/node_set"
+require "moxml/lazy_node_set"
 require "moxml/document"
 require "moxml/element"
 require "moxml/attribute"

data/lib/moxml/adapter/base.rb

@@ -28,6 +28,35 @@ module Moxml
           )
         end
 
+        # Streaming incremental parse (leptris engine): yields each
+        # completed element while the parse runs, releasing prior
+        # subtrees — memory bounded by the largest subtree, not the
+        # document. Adapters without an engine iterator raise.
+        def iterparse(_xml, _mode = :top_level, _context = nil)
+          raise Moxml::AdapterError.new(
+            "Streaming iteration is not supported by the #{name.split('::').last} adapter",
+            adapter: name, operation: "iterparse",
+          )
+        end
+
+        def iterparse_file(_path, _mode = :top_level, _context = nil)
+          raise Moxml::AdapterError.new(
+            "Streaming file iteration is not supported by the #{name.split('::').last} adapter",
+            adapter: name, operation: "iterparse_file",
+          )
+        end
+
+        # Tolerant HTML4/5 parsing into the standard DOM (engine
+        # issue leptris/leptris#659): implied end tags, void elements,
+        # case-insensitive lowercased names, the HTML named-entity
+        # table. Adapters whose engine has an HTML mode override this.
+        def parse_html(_html, _options = {}, _context = nil)
+          raise Moxml::AdapterError.new(
+            "HTML parsing is not supported by the #{name.split('::').last} adapter",
+            adapter: name, operation: "parse_html",
+          )
+        end
+
         def parse(_xml, _options = {}, _context = nil)
           raise Moxml::NotImplementedError.new(
             "parse not implemented",
@@ -181,6 +210,41 @@ namespace_validation_mode: :strict)
           false
         end
 
+        # Whether add_child can keep tracking the same native —
+        # adapters that may recreate the node on attach (libxml's
+        # doc.root=) override to false so the wrapper refresh path
+        # stays armed.
+        def native_identity_stable?
+          false
+        end
+
+        # Whether a BARE-name attribute READ addresses only the
+        # no-namespace attribute (qualified-name semantics) — the
+        # gate for Element#[]'s fast path (bare_attr_value).
+        # Differs per engine: rexml's bare read returns a namespaced
+        # sibling's value; oga's raw values need resolver-only
+        # marker restoration.
+        def bare_get_qname_safe?
+          false
+        end
+
+        # Whether set_attribute with a BARE name behaves as a
+        # qualified-name write: replaces only the no-namespace
+        # attribute and never touches a namespaced p:<local> sibling.
+        # Verified per engine; oga's repeated bare writes diverge, so
+        # it stays false there and assign keeps the full resolve.
+        def bare_set_qname_safe?
+          false
+        end
+
+        # Generation of adapter-level state that cached serialize
+        # decisions depend on (leptris: the entity-marker document
+        # flag). Bumping invalidates wrapper-level memos; adapters
+        # whose answers are static keep the constant zero.
+        def serialize_generation
+          0
+        end
+
         # Whether the subtree at native can contain entity markers.
         # Marker-tracking adapters override this so the post-serialize
         # restore can skip its full-output scans on marker-free

data/lib/moxml/adapter/leptris/document_parts.rb

@@ -64,6 +64,14 @@ module Moxml
             texts << child
             attachments.set(doc, :document_text, texts)
             child
+          when ::Leptris::XML::Comment
+            # The tree model supports document comments (they parse
+            # and serialize, libleptris 1.9.3 #578) but the engine
+            # has no add entry yet (leptris/leptris#1032).
+            raise Moxml::NotImplementedError.new(
+              "Adding document-level comments requires an engine entry (leptris/leptris#1032)",
+              feature: "add_document_child", adapter: "Leptris",
+            )
           else
             raise Moxml::DocumentStructureError.new(
               "Unsupported document child: #{child.class}",

data/lib/moxml/adapter/leptris/markers.rb

@@ -15,8 +15,11 @@ module Moxml
              CustomizedLeptris::DocumentPI
             true
           else
+            # Parentless elements come only from Iterparse —
+            # engine parsed, outside the marker pipeline: never
+            # bearing.
             doc = native.document
-            doc.nil? || attachments.get(doc, :entity_markers) != false
+            doc.nil? ? false : attachments.get(doc, :entity_markers) != false
           end
         end
 

data/lib/moxml/adapter/leptris/serialize.rb

@@ -33,13 +33,53 @@ module Moxml
           # Entity restoration belongs to the wrapper layer
           # (Node#to_xml runs adapter.restore_entities for every
           # adapter); doing it here scanned the output a second time.
-          xml = normalize_serialization(raw_serialize(node, options), options)
+          xml = if native_expand?(node, options)
+                  opts = options.dup
+                  opts[:__expand_handled_natively] = true
+                  normalize_serialization(raw_serialize(node, opts), opts)
+                else
+                  normalize_serialization(raw_serialize(node, options), options)
+                end
           # The binding's FFI strings come back binary-tagged; the
           # engine encoded the bytes per this option, so tag them.
           xml.force_encoding(options[:encoding]) if options[:encoding]
           xml
         end
 
+        # Element-face trailing-newline strip: engine fix landed in
+        # 1.9.42; armed only on older floor bindings.
+        TRAILING_NL_STRIP_ACTIVE =
+          Gem::Version.new(::Leptris::VERSION) < Gem::Version.new("1.9.42")
+
+        # The binding's element face with all-default options — the
+        # frozen splat keeps the hot argless path allocation-free.
+        # Native expand-empty (engine #882, libleptris 1.9.95,
+        # bindings 1.9.144): empty elements emit <a></a> through a
+        # C-side ext entry — the Ruby full-output regex rewrite and
+        # its "/>" probe scan drop out of the element path. The
+        # document face does not expose the option yet; documents
+        # keep the Ruby pass.
+        EXPAND_EMPTY_NATIVE =
+          Gem::Version.new(::Leptris::VERSION) >= Gem::Version.new("1.9.144")
+
+        ELEMENT_DEFAULT_KWARGS = { indent: 0, no_decl: true, encoding: nil }.freeze
+        ELEMENT_EXPAND_KWARGS =
+          if EXPAND_EMPTY_NATIVE
+            { indent: 0, no_decl: true, encoding: nil, expand_empty: true }.freeze
+          else
+            ELEMENT_DEFAULT_KWARGS
+          end.freeze
+
+        # Elements (not documents) with expand_empty and NO
+        # indent-unit string: the C ext entry handles expansion. The
+        # element unit serializer does not take the flag — those keep
+        # the Ruby pass.
+        def native_expand?(node, options)
+          EXPAND_EMPTY_NATIVE && options[:expand_empty] &&
+            !node.is_a?(::Leptris::XML::Document) &&
+            !options[:indent_text].is_a?(String)
+        end
+
         def raw_serialize(node, options)
           # CDATA must precede Text in this chain: CDATA < Text in the
           # binding, so a Text branch first would swallow CDATA nodes.
@@ -60,23 +100,33 @@ module Moxml
             return serialize_document(node, options)
           end
 
-          include_decl = options.fetch(:declaration) do
-            options[:no_declaration] ? false : document_has_declaration?(node)
-          end
-          kwargs = {
-            indent: options.fetch(:indent, 0),
-            no_decl: !include_decl,
-            encoding: options[:encoding],
-          }
-          if INDENT_UNIT_SUPPORTED && options[:indent_text].is_a?(String)
-            kwargs[:indent_text] = options[:indent_text]
-          end
-          xml = node.to_xml(**kwargs)
-          # Element output always ends with the close tag — but the
-          # engine's serializer appends a stray trailing newline when
-          # the element's last text child is non-ASCII (fixed engine
-          # side in 1.9.42; kept for older floor bindings).
-          xml.sub(/\n+\z/, "")
+          # Element serialization never emits a declaration — the C
+          # element serializer ignores the flag (verified
+          # byte-identical); skipping the declaration fetch avoids a
+          # document attachment walk per element serialize. Likewise
+          # an unset encoding serializes UTF-8 — byte-identical to an
+          # explicit "UTF-8" — and the binding then reuses its shared
+          # DEFAULT_OPTIONS instead of rebuilding an options struct
+          # per call; the wrapper force-tags the string either way.
+          indent = options.fetch(:indent, 0)
+          encoding = options[:encoding] == "UTF-8" ? nil : options[:encoding]
+          native_expand = native_expand?(node, options)
+          xml = if indent.zero? && encoding.nil?
+                  node.to_xml(**(native_expand ? ELEMENT_EXPAND_KWARGS : ELEMENT_DEFAULT_KWARGS))
+                else
+                  kwargs = { indent: indent, no_decl: true, encoding: encoding }
+                  kwargs[:expand_empty] = true if native_expand
+                  if INDENT_UNIT_SUPPORTED && options[:indent_text].is_a?(String)
+                    kwargs[:indent_text] = options[:indent_text]
+                  end
+                  node.to_xml(**kwargs)
+                end
+          # Element output always ends with the close tag — but older
+          # engines append a stray trailing newline when the element's
+          # last text child is non-ASCII (fixed engine side in
+          # 1.9.42). The strip stays armed only below that version —
+          # a regex sub per element serialize is measurable in bulk.
+          TRAILING_NL_STRIP_ACTIVE ? xml.sub(/\n+\z/, "") : xml
         end
 
         # A bare ampersand — not starting a named or numeric entity
@@ -102,7 +152,9 @@ module Moxml
           # The libxml2-layout serializer (>= 1.9.42) keeps attribute
           # apostrophes literal; older engines escaped them.
           needs_apos = !LIBXML2_LAYOUT_PARITY && xml.include?("&apos;")
-          needs_expand = options[:expand_empty] && xml.include?("/>")
+          needs_expand = options[:expand_empty] &&
+            !options[:__expand_handled_natively] &&
+            xml.include?("/>")
           # Corruption guards: the 1-char ampersand probe is ~1µs
           # (memchr-class); the raw-< scan runs only on builds that
           # still carry the parse race (leptris-ruby#131).