moxml 0.5.29 → 0.5.31

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: df62f6f6a27c7ff63c2893ae7d85547d09645d059b187fd4235111c18ad2467f
-  data.tar.gz: a9c0e9ee0a3f7f2c8775ba5dfa49563e048ef3e2aac506eaef9135c2a2390139
+  metadata.gz: 32f1034201de38bed35a6b1496d10f96a580f0e4c92517c2253ded0631331439
+  data.tar.gz: bb3d697a73fbac699e00b51905ef446f9837b9ee649785bb1ed4df2027870e78
 SHA512:
-  metadata.gz: 705a551074ea195562d7af4b726a388e8c3517ec4366cf9dff3d3b0828a79f46b6a50d79501d9c5ed1df871855f90bc5d5b69b35e22eb006b558e26c0da5626b
-  data.tar.gz: 2e5b4b7da9018d41434c23c70ea4ced99e626ef813dc92b6d2e1df8d9f8f81e89ac536dd52f684bd95de9fcb610ba673eaa9914de5048a491ee0f217bdfa1975
+  metadata.gz: 3a854ab201001d88494c37d79b163bbf4246880702f5c7d61cf52fd41128459be2a1e8c4601c894852b6aa73e33efca3c471ebfdbf9a126120afa698ab28aff1
+  data.tar.gz: 531a60b0d52e84f4f0df5a026d51b5eeed850b56da1c1eea23297c2c2fb331efa4047a9dc73ef21a430998035aad7a58fe8adb7905047f2001f5b941d07353fc

data/docs/_pages/adapter-protocol.adoc

@@ -29,6 +29,13 @@ whitespace-only text nodes where the engine has the flag
 (#147; recover-mode diagnostics — `[]` on clean parses; engines with
 an error channel map it here, e.g. Nokogiri's `doc.errors`).
 
+HTML mode::
+`parse_html(html, options, context)` — tolerant HTML4/5 parsing into
+the standard DOM, where the engine offers it (leptris >= 1.9.80 via
+engine #659; Nokogiri via libxml2's HTML parser). The base class
+raises `Moxml::AdapterError` — adapters without an HTML mode keep
+that default.
+
 Tree navigation::
 `children`, `parent`, `next_sibling`, `previous_sibling`, `root`,
 `set_root`, `document`, `duplicate_node`. `children` on a document

data/docs/_pages/adapters/leptris.adoc

@@ -51,6 +51,67 @@ libxml2/Nokogiri/REXML semantics (libleptris >= 1.9.8).
 Raise `Moxml::ParseError` on malformed input instead of returning an
 empty document.
 
+== HTML parsing
+
+`Context#parse_html(html)` (leptris >= 1.9.80; engine
+leptris/leptris/#659) parses tolerant HTML4/5 into the standard XML
+DOM: implied end tags (`<li>a<li>b`), void elements (`<br>`, `<img>`),
+raw-text script/style, case-insensitive lowercased names and
+attributes, minimized/boolean attributes, and the HTML named-entity
+table. `html`/`head`/`body` are synthesized when the input implies
+them (no empty head for headless fragments); tables get no implied
+`tbody`. Malformed input degrades to text — the parse never fails;
+only an entirely empty result raises `Moxml::ParseError`.
+
+[source,ruby]
+----
+doc = Moxml.new(:leptris).parse_html(%(<ul><li>a<li>b</ul>))
+doc.xpath("//li").map(&:text)   # => ["a", "b"]
+----
+
+Serialization is XML-shaped: text content is escaped (`a &lt; b`),
+so script bodies with markup characters round-trip through a strict
+XML reparse. The adapter stands the entity-marker pipeline down for
+HTML parses — the engine decodes HTML entities directly into text.
+
+Measured on 1.9.105 (named-entity decoding fixed, engine #848):
+parse 2.5-3.5x, serialize 4.1x, xpath 6-29x faster than Nokogiri —
+uniformly, including entity-laden pages.
+
+Adapters without an engine HTML mode (Ox, Oga, REXML, LibXML) raise
+`Moxml::AdapterError`; Nokogiri parses through libxml2's HTML mode.
+
+== Streaming iteration — `iterparse`
+
+`Context#iterparse(xml, mode:)` / `iterparse_file(path, mode:)`
+expose the engine's incremental parser (libleptris v1.6, #586):
+completed elements yield while parsing, prior subtrees release, and
+memory stays bounded by the largest subtree — `iterparse_file`
+streams the file C-side. `:top_level` yields the root's children;
+`:full_document` yields every element in completion (post-order)
+order.
+
+Measured on a 21MB / 108k-element file: **+20MB peak RSS** (full
+parse: +169MB; Nokogiri Reader: +62MB), bare iteration ~97ms on
+leptris >= 1.9.105 (Nokogiri Reader: 179ms) — ~1.8x faster at a
+third of the memory. Attribute reads on yielded elements improved
+~16x on current builds (the parentless-read cost filed in
+leptris-ruby#152 is fixed).
+
+The yielded elements are parentless (`#document` nil) and valid
+only inside the block — the subtree is released when the iterator
+advances. Since binding 1.9.105, access after the iteration raises
+`UseAfterFreeError` instead of crashing (leptris-ruby#152). Adapters without an engine iterator raise
+`Moxml::AdapterError`.
+
+[source,ruby]
+----
+Moxml.new(:leptris).iterparse_file("huge.xml") do |record|
+  process(record.at_attrs, record.children.map(&:text))
+end
+----
+
+== Lifecycle
 == Lifecycle
 
 `Document#free` releases the C tree deterministically — batch

data/docs/_pages/conversion-apis.adoc

@@ -89,6 +89,24 @@ context.parse(dtd_xml)                     # plain: ATTLIST defaults excluded
 context.parse(dtd_xml, dtdattr: true)      # opt-in: attr="default" materialized
 ----
 
+== HTML input — `parse_html`
+
+Web-scraped and HTML-fragment inputs parse tolerantly into the
+standard DOM (leptris >= 1.9.80 via engine #659; Nokogiri via
+libxml2's HTML mode; other adapters raise `Moxml::AdapterError`):
+
+[source,ruby]
+----
+doc = context.parse_html(%(<ul><li>a<li>b</ul>))
+doc.xpath("//li").map(&:text)   # => ["a", "b"]
+----
+
+Implied end tags, void elements, lowercased names, boolean
+attributes, and HTML named entities decode at parse; serialization
+is XML-shaped. See the leptris adapter page for the semantics
+matrix. On leptris >= 1.9.105 every HTML operation — including
+entity-laden pages — measures 2.5-4x faster than Nokogiri.
+
 == Pretty-print parity (issue #129)
 
 Through the leptris adapter, byte-identical output to raw Nokogiri:

data/docs/_pages/performance.adoc

@@ -56,6 +56,148 @@ Consumers on CRuby 3.3+ (canon's pretty-print pipeline, batch
 conversion) should treat it as the default; moxml's own CI
 performance gates run with YJIT enabled.
 
+=== Wrapper retention — the identity map is weak
+
+`Context` keeps a native→wrapper identity map so repeated traversals
+hand back the same wrapper. Since 0.5.31 it is an
+`ObjectSpace::WeakMap`: entries die with their native, so
+parse-and-drop workloads release wrappers, natives, and (once the
+binding's finalizer runs) the C subtrees. Measured walking and
+dropping 220 documents (~600 elements each) with one long-lived
+Context: **+1.9MB RSS growth vs +41MB before**, zero wrapper
+retention beyond the binding's own constant one-document cache —
+and no identity loss, which the previous 65,536-entry wholesale
+clear traded away when it fired.
+
+=== Measuring moxml — methodology and traps
+
+Every number in this page follows the same discipline; contributors
+benchmarking moxml should too. The traps below all produced wrong
+conclusions at least once.
+
+*Timing:*
+
+- min-of-N (>= 10 samples) around loops, never single-shot averages;
+  `Process.clock_gettime(Process::CLOCK_MONOTONIC)`; `GC.start`
+  before each round.
+- Time LOOPS, not single calls: single sub-microsecond operations
+  read as 0 on some clocks (macOS CLOCK_MONOTONIC granularity).
+- Gate on load average — shared machines under build load (load 30+)
+  swing identical runs 2x. Wait for a calm window (< 5).
+- One heavy benchmark at a time; concurrent suites and benchmarks
+  have crashed this machine.
+- dup benchmarks must free each copy — native trees accumulate
+  otherwise and the numbers degrade monotonically (that drift is a
+  leak signal, not noise).
+
+*Allocation counts (GC.stat(:total_allocated_objects)) are
+load-immune — use them when the machine is busy:*
+
+- WARM THE PATH FIRST: the first call through any code path pays
+  autoloads (adapter, xpath engine, serialize modules — dozens of
+  classes and regexps). A cold first run charges those to whichever
+  variant ran first and fabricates "superadditive" overhead.
+- Keep the fixture shape identical across compared cells: parent
+  vs leaf elements allocate differently in every engine.
+- Negative deltas are impossible — if you see one, the harness is
+  broken, not the code.
+
+*Correctness under load:* race repros and corruption checks need
+hundreds of iterations — spot checks of 30 have missed 5-30% races
+on this codebase.
+
+=== leptris 1.9.105 numbers (reference table)
+
+XML parse 2.4-2.8x, document serialize 2.4x, element queries 32x,
+attribute-node queries 1.4x (native since 1.9.105), HTML parse
+2.5-3.5x including entity-laden pages (engine #848 fixed), HTML
+serialize 4.1x, streaming iteration ~97ms/21MB at +21MB RSS (~1.8x Nokogiri
+Reader), C14n 1.0 native on leptris >= 1.9.121 (engine #919 fixed)
+at 1.8x+ Nokogiri through the full document path.
+
+SAX (calm-machine, verified): the raw binding parses 1.19x faster
+than Nokogiri; through moxml's unified handler the API contract
+(per-event attribute Hash + namespace separation) nets 0.74x interp
+/ 0.93x YJIT.
+
+=== Element serialization
+
+Bulk `element.to_xml` (per-element, no options) was wrapper-tax
+bound — the raw engine serializes 3x faster than Nokogiri but the
+Ruby option/guard layers spent 4.6x the C call's cost per element.
+The argless path is now allocation-free (frozen prebuilt defaults),
+the element face skips the declaration fetch, UTF-8 requests ride
+the binding's shared default options, the trailing-newline strip is
+version-gated (engine fixed in 1.9.42), and `entity_bearing?` memoizes
+against an adapter generation counter. Measured per element:
+3-child **0.74x -> 1.0x** (parity — fixed guard tax on tiny output),
+21-child **1.7x**; document-level serialize stays 2.4x.
+
+=== Held-document footprint
+
+Four lifecycles, measured (RSS, isolated processes; leptris 1.9.80
+vs Nokogiri/libxml2):
+
+| lifecycle | moxml(leptris) | nokogiri | |
+|---|---|---|---|
+| bare tree (parse+hold, unaccessed) | **203 B/node** | 408 B/node | engine tree 2.0x smaller |
+| parse + walk + drop | leak-free (weak registry) | — | `#free` for deterministic release |
+| peak transient, 3.6MB doc (parse+pretty-print) | +48.8MB | +55.6MB | **0.88x smaller peak** |
+| held + walked (37KB doc) | 1186 kB/doc | 754 kB/doc | 1.57x larger — binding wrapper layer |
+
+The one regression is held-and-walked: the binding's per-node Ruby
+wrappers (element 80 B + a dedicated 88 B `FFI::Pointer` + cache
+slot ≈ 170+ B/node) outweigh even the C tree; upstream
+leptris-ruby#147 tracks it with a TypedData fix path. Guidance for
+held workloads: keep documents you will not re-traverse in the
+parse-and-read lifecycle (`readonly:`, `#free`), and prefer
+`materialize` for one-shot conversion.
+
+=== Programmatic construction
+
+Building documents node-by-node is the one surface where the C
+engine's lead does not carry through: each create/attach/attribute
+write crosses the FFI seam, and Nokogiri's C-extension calls are
+cheaper per crossing. Measured end to end:
+
+| builder shape | interp | YJIT |
+|---|---|---|
+| node churn (1052 tiny nodes — worst case) | 0.40x | 0.78x |
+| payload-realistic (attrs + long text) | 0.58x | **~1.0x (parity)** |
+
+Run builders with YJIT (`RUBY_YJIT_ENABLE=1`, see above): it
+compiles moxml's wrapper layer away (-47% on the churn shape) while
+leaving Nokogiri's thin C layer unchanged. The remaining churn-shape
+gap is the binding's own floor — even a zero-overhead wrapper
+measures 0.90x under YJIT (leptris-ruby#149 tracks it, with a
+batch-create proposal to collapse the crossings).
+
+=== Bulk construction — `append_xml`
+
+`element.append_xml(fragment)` appends a raw fragment's top-level
+nodes by parsing it and attaching the subtrees — the engine's C
+parser does the node construction in one crossing instead of an FFI
+round trip per create/attach/attribute call:
+
+| 300-node subtree | time | vs per-node |
+|---|---|---|
+| per-node build (leptris) | 2065µs | — |
+| **append_xml (leptris)** | **1019µs** | **2.0x** |
+| per-node build (nokogiri) | 1746µs | append_xml is 1.7x faster |
+
+The fragment must be namespace-self-contained and well-formed as
+the content of one wrapper element. Supported on leptris, nokogiri,
+oga, and rexml; the Ox adapter raises (its customized node wrappers
+do not survive cross-document attachment).
+
+=== Lazy result sets
+
+`xpath` results and `children` node sets allocate wrapper slots on
+first access, not at construction — `.size`/`.empty?`/`.first` on a
+1000-node result no longer allocate the 1000-slot wrapper array
+(that path also skips the binding's per-node wrapper materialization
+entirely; see `LazyNodeSet` under the leptris adapter).
+
 === Prefer bulk paths for per-node conversion
 
 Per-node Ruby iteration pays the wrapper tax per element. When the

data/lib/compat/opal/moxml_boot.rb

@@ -12,6 +12,7 @@ require "moxml/xml_utils"
 require "moxml/xml_utils/encoder"
 require "moxml/node"
 require "moxml/node_set"
+require "moxml/lazy_node_set"
 require "moxml/document"
 require "moxml/element"
 require "moxml/attribute"

data/lib/moxml/adapter/base.rb

@@ -28,6 +28,35 @@ module Moxml
           )
         end
 
+        # Streaming incremental parse (leptris engine): yields each
+        # completed element while the parse runs, releasing prior
+        # subtrees — memory bounded by the largest subtree, not the
+        # document. Adapters without an engine iterator raise.
+        def iterparse(_xml, _mode = :top_level, _context = nil)
+          raise Moxml::AdapterError.new(
+            "Streaming iteration is not supported by the #{name.split('::').last} adapter",
+            adapter: name, operation: "iterparse",
+          )
+        end
+
+        def iterparse_file(_path, _mode = :top_level, _context = nil)
+          raise Moxml::AdapterError.new(
+            "Streaming file iteration is not supported by the #{name.split('::').last} adapter",
+            adapter: name, operation: "iterparse_file",
+          )
+        end
+
+        # Tolerant HTML4/5 parsing into the standard DOM (engine
+        # issue leptris/leptris#659): implied end tags, void elements,
+        # case-insensitive lowercased names, the HTML named-entity
+        # table. Adapters whose engine has an HTML mode override this.
+        def parse_html(_html, _options = {}, _context = nil)
+          raise Moxml::AdapterError.new(
+            "HTML parsing is not supported by the #{name.split('::').last} adapter",
+            adapter: name, operation: "parse_html",
+          )
+        end
+
         def parse(_xml, _options = {}, _context = nil)
           raise Moxml::NotImplementedError.new(
             "parse not implemented",
@@ -181,6 +210,41 @@ namespace_validation_mode: :strict)
           false
         end
 
+        # Whether add_child can keep tracking the same native —
+        # adapters that may recreate the node on attach (libxml's
+        # doc.root=) override to false so the wrapper refresh path
+        # stays armed.
+        def native_identity_stable?
+          false
+        end
+
+        # Whether a BARE-name attribute READ addresses only the
+        # no-namespace attribute (qualified-name semantics) — the
+        # gate for Element#[]'s fast path (bare_attr_value).
+        # Differs per engine: rexml's bare read returns a namespaced
+        # sibling's value; oga's raw values need resolver-only
+        # marker restoration.
+        def bare_get_qname_safe?
+          false
+        end
+
+        # Whether set_attribute with a BARE name behaves as a
+        # qualified-name write: replaces only the no-namespace
+        # attribute and never touches a namespaced p:<local> sibling.
+        # Verified per engine; oga's repeated bare writes diverge, so
+        # it stays false there and assign keeps the full resolve.
+        def bare_set_qname_safe?
+          false
+        end
+
+        # Generation of adapter-level state that cached serialize
+        # decisions depend on (leptris: the entity-marker document
+        # flag). Bumping invalidates wrapper-level memos; adapters
+        # whose answers are static keep the constant zero.
+        def serialize_generation
+          0
+        end
+
         # Whether the subtree at native can contain entity markers.
         # Marker-tracking adapters override this so the post-serialize
         # restore can skip its full-output scans on marker-free

data/lib/moxml/adapter/leptris/markers.rb

@@ -15,8 +15,11 @@ module Moxml
              CustomizedLeptris::DocumentPI
             true
           else
+            # Parentless elements come only from Iterparse —
+            # engine parsed, outside the marker pipeline: never
+            # bearing.
             doc = native.document
-            doc.nil? || attachments.get(doc, :entity_markers) != false
+            doc.nil? ? false : attachments.get(doc, :entity_markers) != false
           end
         end
 

data/lib/moxml/adapter/leptris/serialize.rb

@@ -40,6 +40,15 @@ module Moxml
           xml
         end
 
+        # Element-face trailing-newline strip: engine fix landed in
+        # 1.9.42; armed only on older floor bindings.
+        TRAILING_NL_STRIP_ACTIVE =
+          Gem::Version.new(::Leptris::VERSION) < Gem::Version.new("1.9.42")
+
+        # The binding's element face with all-default options — the
+        # frozen splat keeps the hot argless path allocation-free.
+        ELEMENT_DEFAULT_KWARGS = { indent: 0, no_decl: true, encoding: nil }.freeze
+
         def raw_serialize(node, options)
           # CDATA must precede Text in this chain: CDATA < Text in the
           # binding, so a Text branch first would swallow CDATA nodes.
@@ -60,23 +69,31 @@ module Moxml
             return serialize_document(node, options)
           end
 
-          include_decl = options.fetch(:declaration) do
-            options[:no_declaration] ? false : document_has_declaration?(node)
-          end
-          kwargs = {
-            indent: options.fetch(:indent, 0),
-            no_decl: !include_decl,
-            encoding: options[:encoding],
-          }
-          if INDENT_UNIT_SUPPORTED && options[:indent_text].is_a?(String)
-            kwargs[:indent_text] = options[:indent_text]
-          end
-          xml = node.to_xml(**kwargs)
-          # Element output always ends with the close tag — but the
-          # engine's serializer appends a stray trailing newline when
-          # the element's last text child is non-ASCII (fixed engine
-          # side in 1.9.42; kept for older floor bindings).
-          xml.sub(/\n+\z/, "")
+          # Element serialization never emits a declaration — the C
+          # element serializer ignores the flag (verified
+          # byte-identical); skipping the declaration fetch avoids a
+          # document attachment walk per element serialize. Likewise
+          # an unset encoding serializes UTF-8 — byte-identical to an
+          # explicit "UTF-8" — and the binding then reuses its shared
+          # DEFAULT_OPTIONS instead of rebuilding an options struct
+          # per call; the wrapper force-tags the string either way.
+          indent = options.fetch(:indent, 0)
+          encoding = options[:encoding] == "UTF-8" ? nil : options[:encoding]
+          xml = if indent.zero? && encoding.nil?
+                  node.to_xml(**ELEMENT_DEFAULT_KWARGS)
+                else
+                  kwargs = { indent: indent, no_decl: true, encoding: encoding }
+                  if INDENT_UNIT_SUPPORTED && options[:indent_text].is_a?(String)
+                    kwargs[:indent_text] = options[:indent_text]
+                  end
+                  node.to_xml(**kwargs)
+                end
+          # Element output always ends with the close tag — but older
+          # engines append a stray trailing newline when the element's
+          # last text child is non-ASCII (fixed engine side in
+          # 1.9.42). The strip stays armed only below that version —
+          # a regex sub per element serialize is measurable in bulk.
+          TRAILING_NL_STRIP_ACTIVE ? xml.sub(/\n+\z/, "") : xml
         end
 
         # A bare ampersand — not starting a named or numeric entity
@@ -86,17 +103,17 @@ module Moxml
         RAW_AMP_RE = /&(?!#{Entity::NAME_PATTERN};|#\d+;|#x[0-9A-Fa-f]+;)/
 
         # A raw "<" in text position — the engine intermittently
-        # loses the escape when parsing under allocation pressure
-        # (leptris-ruby#131), so a decoded &#x3c; serializes bare and
-        # reparsing truncates at it. Valid markup NEVER puts "<"
-        # before a non-name, non-delimiter character, so this is
-        # deterministic in markup segments (CDATA is literal and
-        # excluded by the segment walker). One class-anchored scan:
-        # multi-char literal probes cost ~30x a single regex pass on
-        # CRuby (measured include?("< ") at 31µs vs this regex at
-        # 64µs for 14 probes' worth of coverage on 32KB).
+        # lost the escape when parsing under allocation pressure
+        # (leptris-ruby#131), so a decoded &#x3c; serialized bare and
+        # reparsing truncated at it. Fixed in leptris 1.9.80 (verified
+        # 0/200 on the churn repro); the guard stays armed below that
+        # version and for any future build that regresses — the
+        # parse-under-pressure spec in leptris_spec fails loudly if a
+        # fixed-version build corrupts while the guard is stood down.
         RAW_LT_RE = /<(?![A-Za-z_:\/?!])/
         RAW_LT_TRIGGER_RE = /<[^A-Za-z_:\/?!]/
+        RAW_LT_GUARD_ACTIVE =
+          Gem::Version.new(::Leptris::VERSION) < Gem::Version.new("1.9.80")
 
         def normalize_serialization(xml, options)
           # The libxml2-layout serializer (>= 1.9.42) keeps attribute
@@ -104,10 +121,10 @@ module Moxml
           needs_apos = !LIBXML2_LAYOUT_PARITY && xml.include?("&apos;")
           needs_expand = options[:expand_empty] && xml.include?("/>")
           # Corruption guards: the 1-char ampersand probe is ~1µs
-          # (memchr-class); both regexes only run when their cheap
-          # necessary condition holds.
+          # (memchr-class); the raw-< scan runs only on builds that
+          # still carry the parse race (leptris-ruby#131).
           needs_amp = xml.include?("&") && xml.match?(RAW_AMP_RE)
-          needs_lt = xml.match?(RAW_LT_TRIGGER_RE)
+          needs_lt = RAW_LT_GUARD_ACTIVE && xml.match?(RAW_LT_TRIGGER_RE)
           return xml unless needs_apos || needs_expand || needs_amp || needs_lt
 
           out = +""