makiri 0.16.0 → 0.17.0.rc2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 80c74b7d335f5f5f93f8476a8f99a8918560df8cf75f43a5a71d9b132e892873
-  data.tar.gz: d6593459917f22ceb301093d17b981a597d8be7d975d1fab1e4142f6cf17fcbc
+  metadata.gz: dc64f5f05d50f24d910c10e0fc9083f90db3739e73f2ab83d801e8775f2a62f4
+  data.tar.gz: 46db21d534703f776deeeff576a612e810f6926ceb41f335c1c42097edd472c9
 SHA512:
-  metadata.gz: efa98c3ca322d6a3ee141428c1f5e9c3e5b46fba9dde4d7ac75d7dcfdf27d4dd083ef46eed5fb8db21685396d86caeb58e1755f9dd5014d396674cc149378df9
-  data.tar.gz: 836d97359577dfcd4bd60a8de0dfcf355c596b95fa7fffd5e56a0a355bb4267f3141bf8786f4f905596d79bef44721eeb3bfe4f25fa97e489c5aa3319c9f7655
+  metadata.gz: 07ff20653102efa921a4079b10ead383074a7f4a3a97c1cd84134ffd344b30f61fe5f19d094a45be7b5cd514e91f029cf02c5bc44098d9a408f62b1c1551d66b
+  data.tar.gz: d241369c7b6e7a2be95827e0724bdcc0f58ce79cbbbe1d3937bc6708ee0d438af966ae5eccc9319e2fb940638e80faed7564808c51b032b0877bf179223d7e02

data/CHANGELOG.md

@@ -1,5 +1,211 @@
 # Changelog
 
+## [Unreleased]
+
+## [0.17.0.rc2] - 2026-10-24
+
+### Added
+
+* HTML documents are decoded as browsers decode them: the HTML Standard's
+  encoding sniffing algorithm with the Encoding Standard's decoders. A
+  binary (ASCII-8BIT) or US-ASCII String is read in the encoding its byte
+  order mark or `<meta>` declares, or as UTF-8 when it is valid UTF-8, or
+  else in a detected encoding (Shift_JIS, EUC-JP, ISO-2022-JP, ...); a
+  `<meta>` past the first 1024 bytes re-reads the document in the encoding
+  it names. `Makiri::HTML(src, encoding: "shift_jis")` gives the encoding
+  outright - read as a String tagged with it is read, so
+  `encoding: Encoding::UTF_32LE` works - and `Document#encoding` returns
+  the one used (it was always `"UTF-8"`).
+
+### Changed
+
+* Ruby 3.3 or newer is required (3.2 reached its end of life in March 2026).
+* Arguments are no longer converted with `to_s` / `to_str`. A string
+  argument (an attribute value, text content, markup, a variable value, a
+  namespace URI) must be a `String`, and a name (an attribute or element
+  name, a namespace prefix, a selector or an XPath expression) a `String` or
+  a `Symbol`; anything else raises `TypeError` ("... call #to_s to convert
+  it"), where `node["n"] = 1` used to store `"1"`. `Makiri.HTML` /
+  `Makiri.XML` still read an IO and convert their source with `String()`,
+  and `XML::Builder` still converts with `to_s`. An index or a count
+  (`NodeSet#[]`, `#child_at`, `to_xml(indent:)`) must likewise be an
+  `Integer`; `to_int` is not called.
+* A binary or US-ASCII String holding invalid UTF-8 is no longer read as
+  UTF-8 with U+FFFD replacements: it is sniffed, and read in the encoding a
+  detector guesses - which, for short text or text with little variety,
+  can be the wrong one. A single invalid byte is enough: UTF-8 with one
+  stray byte, or cut off inside a character, is read whole as a legacy
+  encoding (windows-1252, say), as a browser reads such a local file. Pass
+  `encoding:` when it is known, `"utf-8"` (or a UTF-8 String) for the old
+  reading.
+* A String in a legacy encoding is decoded with the Encoding Standard's
+  decoder for it rather than Ruby's transcoder, as browsers decode it:
+  `ISO-8859-1` reads as windows-1252 (`\x80` is `€`), and Shift_JIS's
+  `0x8160` is U+301C where Ruby's `Windows-31J` made U+FF5E.
+* Selectors, XPath expressions, names, attribute values and text read a
+  String in its own encoding, as parsing does: one that is not UTF-8,
+  US-ASCII or binary is converted to UTF-8 first, and refused with
+  `Makiri::Error` when it cannot be. Its bytes were read as UTF-8, so a
+  Windows-1252 `"\xC3\xA9"` was stored as `é` and a UTF-16 `"a"` as `"a\0"`.
+* `Makiri::XML` reads an encoding declaration by its IANA charset name or
+  alias in any case (`latin1`, `csISOLatin1`), as XML 1.0 has it, and
+  raises `Makiri::XML::SyntaxError` for a name that is neither (`utf8`);
+  such a declaration was ignored, or reported as conflicting with the
+  String's encoding.
+* `Makiri::XML` `#ancestors` returns a NodeSet, as HTML's does; it returned
+  an Array.
+* `Node#traverse` lists the subtree once, natively, before the first node is
+  yielded: about five times faster on a large tree, and a block that adds or
+  removes nodes no longer changes which nodes are visited.
+
+### Removed
+
+* XPath custom-function handlers. `Node#xpath(expr, handler)`,
+  `#at_xpath(expr, handler)` and `XPathContext#evaluate(expr, handler)` raise
+  `ArgumentError`, and a call to a function XPath 1.0 does not define is an
+  `XPath::SyntaxError` (it was a `Makiri::Error` when the expression was
+  evaluated). No Ruby code runs during an evaluation any more, so mutating a
+  document is never refused "while evaluating XPath over it".
+
+### Performance
+
+* A node wrapper keeps its data in its own object (an embedded TypedData,
+  Ruby 3.3+): wrapping a node for the first time is ~25-45 ns cheaper, and
+  a fresh document's query or walk 10-30% faster.
+* A document's indexes are built when a query first needs them, each in
+  one walk. An XPath that names no tag (`/html/body`) no longer builds the
+  tag index; the text index waits until `#text` calls have walked half the
+  document, so a first `doc.text` is twice as fast and a first small
+  `li.text` builds nothing; and the order XPath sorts by is kept with the
+  document, so a union over 40,000 nodes sorts in 0.7 ms instead of 13 ms.
+* A NodeSet of up to eight nodes holds them in itself, with no allocation
+  of its own: `#children` went from ~185 to ~120 ns.
+* A reader that returns a String (`#text`, `#name`, `#[]`) builds it with
+  no intermediate copy (`#text` ~124 -> ~112 ns), and `Document#text` joins
+  its text with fixed-size copies (20 KB: 9.2 -> 6.7 us).
+
+### Fixed
+
+* HTML `add_previous_sibling`, `add_next_sibling` and `replace` given the
+  node itself leave it where it is, as the DOM does and `Makiri::XML`
+  already did; they raised "cannot insert a node into its own subtree".
+* `Makiri::XML`'s `content=` on a DocumentFragment replaces its children,
+  and on a Document or a DocumentType does nothing, as the DOM's
+  `textContent` does and HTML already did; both raised.
+* `remove` / `unlink` on an HTML Document raise "cannot remove the document
+  node", as on an XML one; it returned the document unchanged.
+* A UTF-8 byte order mark at the start of a document is no longer kept as a
+  U+FEFF character in the text.
+* A refused `Makiri::XML` insert of a node from another document (onto a
+  Text node, say) no longer leaves the copy it made in the target document,
+  where each refused call grew it and used up its node budget.
+* XPath over HTML keeps a plain attribute of an element created in the
+  XMLNS namespace on `@*` and `[@name]`; it was taken for a namespace
+  declaration and skipped.
+* XPath's `id()` with many tokens indexes the document's IDs once instead
+  of searching it per token: 8,000 IDs over 8,000 elements raised
+  `Makiri::XPath::LimitExceeded`.
+* `==` / `eql?` compare the nodes' Documents as well, so nodes of two
+  documents can no longer compare equal after the per-document counter wraps.
+* Importing an XML processing instruction whose data holds `?>` into an
+  HTML document raises `ArgumentError`; it reported "out of memory".
+* An HTML Attr serializes to the empty string, as an XML Attr does and as
+  DOM Parsing and Firefox have it: `attr.to_s`, `puts attr` and `NodeSet#to_s`
+  over attributes raised `Makiri::Error` ("HTML serialization failed").
+* Under a default namespace (the ns Hash's `"xmlns"`), a compound with no
+  type selector (`.x`, `[a]`) matches only that namespace's elements, as if
+  `*` were written (Selectors 4); it matched elements in any namespace.
+* An undeclared namespace prefix inside `:is()` or `:where()` drops that
+  alternative alone, as browsers do: `:is(a, z|a)` matches `a` instead of
+  raising `Makiri::CSS::SyntaxError`.
+
+## [0.17.0.rc1] - 2026-10-22
+
+### Added
+
+* `Makiri::Lexbor::CSS.parse_stylesheet` returns the declarations in an
+  at-rule's block (`@font-face`, `@page`, ...) as its `:declarations`; they
+  were dropped.
+
+### Changed
+
+* HTML is parsed by Makiri's own Rust port of Lexbor's tokenizer and tree
+  builder, into a DOM of Makiri's own, and the CSS selector and stylesheet
+  parsers are ported too. A parsed document takes about 40% less memory.
+* Installing from source needs Rust 1.84 or newer (and libclang), and no
+  longer CMake: the gem carries none of Lexbor's sources, only its LICENSE
+  and NOTICE.
+* `Makiri::XML`'s `#css`, `#at_css` and `#matches?` run on the same matcher
+  as HTML's and answer as a browser's `querySelectorAll` does in an XML
+  document, 10-20x faster on compound selectors (`.item`, `ul li.item`).
+  This changes some answers:
+  * a bare type selector (`p`) matches that name in any namespace;
+  * the root element's namespace declarations are no longer picked up: a
+    prefix must be given in the ns Hash, whose `"xmlns"` entry is the
+    default namespace (on HTML too);
+  * a selector run on a node may match through ancestors outside it
+    (`node.css("body p")`); only the node's descendants are returned;
+  * `[a=v i]`, `:nth-child(... of S)`, and the form-state and link
+    pseudo-classes on XHTML elements are supported;
+  * `||` and `:current()` raise `Makiri::CSS::SyntaxError`, and an over-long
+    chain `Makiri::Error` (it was `Makiri::XPath::LimitExceeded`).
+* HTML `css(selector, ns)` resolves namespace prefixes: `svg|path` is the SVG
+  `path`, `|p` an element in no namespace, `[x|href]` the attribute in `x`'s
+  namespace. A prefix the Hash does not bind raises
+  `Makiri::CSS::SyntaxError`; prefixes used to be ignored.
+* HTML parsing follows the HTML Standard where Lexbor did not. This affects
+  malformed or unusual markup only: `<frameset>`, `<form>` and tables inside
+  `<template>`, `<![CDA` or `<!` at the end of the input, the adoption
+  agency, `<textarea>`, `<input type=HIDDEN>` and `<image>` in a table,
+  `<sup>` in SVG or MathML, and a repeated attribute name (the first is
+  kept). Fragment parsing (`#inner_html=`, `#outer_html=`, `Node#parse`,
+  `context:` given an element) takes the context element's quirks mode,
+  namespace and ancestor `<form>` into account, as browsers do.
+* CSS selectors are parsed as Selectors 4 says: `:has()` inside `:has()` and
+  an invalid argument to `:has()` make the selector invalid, an empty
+  `:is()` / `:where()` is valid and matches nothing, and `#1a` is not an ID
+  selector.
+* `parse_stylesheet` reads names and values by one rule for every property: a
+  declaration name is lower-cased unless it is a custom property, an at-rule
+  name is lower-cased, and a value is the text as written with whitespace
+  runs collapsed (`color: RED` stays `RED`; it came back as `red`).
+  `:specificity` is counted as Selectors 4 does (`:is()` / `:not()` /
+  `:has()` take their most specific argument).
+
+### Removed
+
+* The 10,000-`<option>` limit on a `<select>`: a select with any number of
+  options parses in linear time, as browsers parse it.
+
+### Fixed
+
+* `parse_stylesheet` keeps a rule such as `a:hover { ... }` inside `@media`
+  or `@supports`; it was read as a declaration, which dropped that rule and
+  every one after it in the block. A comment between a declaration's name
+  and its `:` no longer drops the declarations that follow.
+* A namespaced type selector in the middle of a compound (`.x|a`) raises
+  `Makiri::CSS::SyntaxError`, as browsers do.
+* `:any-link` / `:link` match only an HTML `a` or `area`, or an SVG `a`, with
+  an `href` (`<link href>` and `<map>` matched). `:read-write` /
+  `:read-only` follow the HTML Standard (a checkbox is read-only, an element
+  inside `contenteditable` is read-write), and `:empty` ignores processing
+  instructions.
+* HTML's `:nth-child` family counts element siblings only:
+  `html:first-child` matched nothing after `<!doctype html>`.
+* An attribute keeps the namespace it was made in: `set_attribute_ns` in the
+  element's own namespace replaces the attribute it set before instead of
+  adding a second, and an attribute removed from its element keeps its
+  `namespace_uri`.
+* An attribute name is serialized in the case it was set (`e["viewBox"]` on
+  an element in no namespace was written lower-cased).
+* A void element's `#inner_html` is empty even when it has children, and a
+  prefixed HTML, SVG or MathML element is written by its local name.
+* `Node#line` of a void element fostered out of a table is its line, not nil.
+* Many namespaces created with `create_element_ns` no longer make lookups
+  quadratic.
+* `HTML::Document#import_node` given a document raises "cannot import a
+  document; use #dup", as `#clone_node` does.
+
 ## [0.16.0] - 2026-10-10
 
 ### Added

data/NOKOGIRI_DIFFERENCES.md

@@ -41,11 +41,20 @@ what browsers do - rather than libxml2. Detailed, test-backed notes live in
     and in XPath's data model. An `xmlns` on an HTML element is an ordinary
     attribute and stays visible.
 
-* A `StandardError` raised by a custom-function handler becomes `Makiri::Error`
-  ("handler raised: <message>"), with the handler's exception as its `#cause`
-  * Nokogiri re-raises the handler's exception itself. Anything that is not a
-    `StandardError` (`Interrupt`, `SystemExit`, Timeout's exception) and a
-    `throw` reach the caller unchanged in both.
+* Arguments are not converted
+  * Nokogiri calls `to_s` on an attribute value, text content or a variable
+    value (`node["n"] = 1`). Makiri takes a `String` for those, and a `String`
+    or a `Symbol` for a name; anything else is a `TypeError`. The parse
+    entries still take an IO or anything `String()` converts, and
+    `XML::Builder` converts with `to_s` as Nokogiri's does.
+  * An index is an `Integer`: `node_set[1.5]` and a Range with a Float end
+    are a `TypeError`, where Nokogiri (as `Array#[]`) truncates them.
+
+* There are no custom XPath functions
+  * Nokogiri answers a function XPath does not define by calling a method of
+    a handler object (`xpath("my-fn(.)", handler)`). Makiri runs no Ruby while
+    it evaluates: a call to a function XPath 1.0 does not define is an
+    `XPath::SyntaxError`, and a handler argument is an `ArgumentError`.
 
 * A number literal is read as the nearest double; libxml2's own reader is not
   correctly rounded for a literal with more digits than a double holds, so
@@ -149,7 +158,7 @@ what browsers do - rather than libxml2. Detailed, test-backed notes live in
 * `<?php ... ?>` in HTML input is a **ProcessingInstruction** node; `#to_html`
   writes it back as `<?php ... ?>`, and `<?` at end of input is ignored.
   * The HTML Standard added processing-instruction tokens and tree-construction
-    rules for them; Makiri follows them through Lexbor. `Nokogiri::HTML5` (gumbo)
+    rules for them, and Makiri's parser follows them. `Nokogiri::HTML5` (gumbo)
     still produces the older bogus comment (`<!--?php ... ?-->`), and
     `Nokogiri::HTML` (libxml2) its own comment.
 
@@ -158,17 +167,39 @@ what browsers do - rather than libxml2. Detailed, test-backed notes live in
   Nokogiri's. `inner_html=`, `outer_html=` and `Node#parse` take no keyword.
   `Nokogiri::HTML` (libxml2) instead stops at depth 256 and returns the
   truncated document.
-* More than 10,000 `<option>`s in one `<select>` raise `Makiri::Error`;
-  Nokogiri has no such limit.
+
+* A document's bytes are decoded by the HTML Standard's **encoding sniffing
+  algorithm**, with the Encoding Standard's decoders, and `Document#encoding`
+  is the encoding they were read in (`"Shift_JIS"`, `"windows-1252"`).
+  * A binary (ASCII-8BIT) or US-ASCII String with no `<meta>` charset is read
+    as UTF-8 when it is valid UTF-8 and is otherwise detected (Shift_JIS,
+    EUC-JP, ISO-2022-JP, ...), as browsers read a local file. One invalid
+    byte - a stray one, or a character cut off at the end - rules UTF-8 out,
+    and the whole document is read in the detected encoding; pass
+    `encoding: "utf-8"` for input known to be UTF-8, and only the bad bytes
+    become U+FFFD. `Nokogiri::HTML5` reads it as windows-1252, so valid
+    UTF-8 and Shift_JIS alike come out garbled.
+  * A `<meta>` charset past the first 1024 bytes re-reads the document in that
+    encoding, as browsers do; `Nokogiri::HTML5` ignores it.
+  * A String with an encoding is read in that encoding, and `encoding:` gives
+    one explicitly; a `<meta>` cannot override either. Both use the Encoding
+    Standard's decoders rather than Ruby's transcoders, so `ISO-8859-1` reads
+    as windows-1252 (`\x80` is `€`) and Shift_JIS's `0x8160` is U+301C, as in
+    browsers.
+  * `Nokogiri::HTML5`'s `#encoding` is always `"UTF-8"`.
 
 ## HTML mutation
 
+* `Node#traverse` lists the subtree before the first node is yielded
+  * Nokogiri reads each node's children as it reaches it, so a block that
+    adds a node below or after the current one can have it visited. Makiri
+    visits the nodes there were when `traverse` was called, in the same
+    order (each node after its children).
 * There is no `Node#name=` / `#node_name=` (on HTML or XML nodes). Nokogiri
   renames a node in place and keeps its identity; the DOM has no rename, and
-  Lexbor keeps elements such as `<template>`, `<option>` and `<style>` in
-  structs of their own, so a node cannot change its tag safely. Create an
-  element of the new name, move the attributes and children, and `replace`
-  the old one (see CHANGELOG.md).
+  an element's tag decides what it is (a `<template>` owns its contents), so a
+  node cannot change its tag. Create an element of the new name, move the
+  attributes and children, and `replace` the old one (see CHANGELOG.md).
 * Element and attribute names follow the WHATWG DOM's rules
   * `create_element`, `[]=` and `set_attribute_ns` raise `ArgumentError` for a
     name the DOM refuses - one holding whitespace, `/`, `>` (or `=` for an
@@ -224,26 +255,19 @@ what browsers do - rather than libxml2. Detailed, test-backed notes live in
   * One exception to the DOM: an HTML attribute named `xml:lang` (in no
     namespace) becomes the XML namespace's `xml:lang`, as the XML reader reads
     that name, so XHTML-style HTML still writes as XML.
-* A known gap, in Lexbor's tag table: an HTML document that already holds a
-  parsed element named with a colon (`<x:y>`, one local name) and then
-  receives, by `import_node` from another document, a prefixed element
-  written the same way (`x:y` from XML: prefix `x`, local name `y`) re-points
-  the table's entry for that spelling - the parsed element then no longer
-  matches CSS `x\:y`. Copies within one document do not touch the table.
 * An XML element with a prefix, imported into HTML, keeps it (`h:div` in
   XHTML has the local name `div`). Two readers then disagree, as they do in
-  browsers: CSS's `div` matches it (Lexbor matches the local name), XPath's
-  `//div` does not (an HTML element's name test reads its qualified name). And
-  Lexbor's HTML serializer writes the prefix (`<h:div>`), where the HTML
-  standard writes the local name, so the HTML does not re-parse to the same
-  element.
+  browsers: CSS's `div` matches it (the matcher compares the local name),
+  XPath's `//div` does not (an HTML element's name test reads its qualified
+  name). `#to_html` writes the local name (`<div>`), as the HTML standard
+  does.
 
 ## CSS
 
 * Most jQuery/Nokogiri CSS extensions are not supported (`:gt`, `:lt`, `:eq`, `:first`, ...)
-  * Makiri uses Lexbor's selector engine, which is standards-based apart from one
-    text-containment extension. Use XPath (`xpath("//p[contains(., 'x')]")`) or
-    Enumerable (`css('li')[1]`) for the rest.
+  * Makiri's selector engine is standards-based. Use XPath
+    (`xpath("//p[contains(., 'x')]")`) or Enumerable (`css('li')[1]`) for the
+    rest.
     Standard Level-4 selectors (`:is` / `:where` / `:has`) are supported; some of which Nokogiri rejects.
   * There is no text-containment pseudo-class. Nokogiri's `:contains()` is
     not supported, and neither is Lexbor's `:lexbor-contains()`: it raises
@@ -255,33 +279,45 @@ what browsers do - rather than libxml2. Detailed, test-backed notes live in
   selector) is supported and correct on both HTML and XML - the "type" is the
   element's own expanded name.
   * Nokogiri (XML and HTML5) mistranslates these to first-/only-child
-    (`//*[position()=1]` / `//*[last()=1]`), so it under-matches; Makiri matches
-    Lexbor's HTML matcher.
-* HTML CSS takes a `{prefix => uri}` Hash (`css(selector, ns)`) but does not
-  resolve prefixes against it: Lexbor's matcher matches a prefixed type selector
-  loosely, so `svg|path`, `|path` and `path` all find the SVG element whatever
-  is bound. `Nokogiri::HTML5` honours the binding (a wrong URI finds nothing).
-  * The Hash is accepted and unused rather than refused, so the `css(selector,
-    ns)` a caller writes for both representations works. Use `#xpath`, where a
-    prefix IS resolved against the bindings, when the namespace matters.
-  * `Makiri::XML` resolves CSS prefixes properly - it lowers the selector to the
-    XPath engine, which registers the bindings.
-  * The same holds for attribute selectors: HTML `[|href]` (no namespace) and
-    plain `[href]` also find an SVG `xlink:href`, which `Nokogiri::HTML5`
-    does not.
-    `Makiri::XML` reads `[|a]` as the no-namespace attribute.
+    (`//*[position()=1]` / `//*[last()=1]`), so it under-matches; Makiri does
+    not.
+* CSS resolves a namespace prefix against the `{prefix => uri}` Hash
+  (`css(selector, ns)`), on HTML and XML alike: `svg|path` is the SVG `path`
+  once `svg` is bound, `|path` an element in no namespace, and a prefix the
+  Hash does not bind raises `Makiri::CSS::SyntaxError`. `[x|href]` finds the
+  XLink `href` once `x` is bound to XLink. The Hash's `"xmlns"` entry is the
+  default namespace, as CSS `@namespace` without a prefix sets one: a type or
+  universal selector without a prefix is then in it.
+  * Without a default, a bare type selector matches its name in any namespace
+    (`path` finds the SVG element, `entry` an Atom entry), as a browser's
+    `querySelectorAll` does. `Nokogiri::XML` binds a bare name to the root's
+    default namespace when no Hash is given, and to no namespace once one is;
+    pass `"xmlns" => uri` (or `"xmlns" => ""`) for that reading.
+  * `Makiri::XML` does not collect the document's own declarations: a prefix
+    is declared by the Hash or not at all. `Nokogiri::XML` collects the
+    root's.
+  * A bare `*` under a default namespace is in it, as CSS has it;
+    `Nokogiri::XML` matches any element.
+  * Up to 0.16 HTML accepted the Hash and ignored every prefix, and
+    `Makiri::XML` collected the root's declarations as Nokogiri does.
 * A selector under a node matches the way `Element#querySelectorAll` does in a
-  browser, not scoped to that node, on HTML: `at_css("#c").css("div p")` finds a
-  `p` inside `#c` when `#c` is itself a `div`, since the selector is matched
-  against the whole document and only the results are kept to descendants.
-  `Nokogiri::HTML5` and `Nokogiri::XML` scope the selector to the node (`#c`
-  cannot be the `div`), and so does `Makiri::XML`, which lowers the selector to
-  an XPath from the node. Lexbor has no `:scope`; for a scoped match on HTML,
-  use XPath from the node (`xpath(".//div//p")`).
-* The attribute case modifiers (`[a="x" i]`, `[a="x" s]`): HTML supports both,
-  through Lexbor's matcher. `Makiri::XML` accepts `s` (case-sensitive, which XML
-  values are anyway) and refuses `i` with `Makiri::CSS::SyntaxError`. Nokogiri
-  refuses both on either representation.
+  browser, not scoped to that node, on HTML and XML alike:
+  `at_css("#c").css("div p")` finds a `p` inside `#c` when `#c` is itself a
+  `div`, since the selector is matched against the whole tree and only the
+  results are kept to descendants. `Nokogiri::HTML5` and `Nokogiri::XML` scope
+  the selector to the node (`#c` cannot be the `div`), and so did
+  `Makiri::XML` up to 0.16. There is no `:scope`; for a scoped match, use XPath
+  from the node (`xpath(".//div//p")`).
+* The attribute case modifiers (`[a="x" i]`, `[a="x" s]`) are supported on
+  both representations. Nokogiri refuses both on either.
+* The form-state and link pseudo-classes (`:checked`, `:disabled`,
+  `:enabled`, `:required`, `:optional`, `:placeholder-shown`, `:link`, ...)
+  answer on `Makiri::XML` too, for XHTML elements, as browsers do in an XML
+  document; on any other element they match nothing. `Nokogiri::XML` refuses
+  them.
+* A selector `Makiri` cannot run (the column combinator `||`, `:current()`)
+  raises `Makiri::CSS::SyntaxError`, as a browser's `querySelectorAll` throws a
+  SyntaxError, on both representations.
 * `#matches?` answers for a DETACHED node (`document.create_element("p")
   .matches?("p")` is true, on both representations). Nokogiri raises
   `NoMethodError` there - it implements `#matches?` as a search from
@@ -294,7 +330,9 @@ what browsers do - rather than libxml2. Detailed, test-backed notes live in
   therefore matches no type selector.
   * `Nokogiri::HTML5` is case-sensitive on HTML elements too, so `LI` does not
     match `<li>` there. `Makiri::XML`'s `#css` is case-sensitive, as XML names
-    are.
+    are - an XHTML element's included, as in a browser's XML document - and so
+    are its attribute values, the HTML Standard's case-insensitive list of
+    attributes applying only to an HTML document.
 
 ## Serialization
 
@@ -310,6 +348,10 @@ what browsers do - rather than libxml2. Detailed, test-backed notes live in
   disabled (its children are elements, not raw text) and escaping is what makes
   that round-trip; `Nokogiri::HTML5` writes it literally and contradicts its own
   parser there.
+* An Attr serializes to the empty string (`attr.to_s`, `#to_html`,
+  `NodeSet#to_s` over attributes), as DOM Parsing's algorithm and Firefox
+  have it, in HTML and XML. Nokogiri's `Attr#to_s` is the value (`"a"`) and
+  its `#to_html` the markup (`id="a"`); use `#value` for the value.
 
 ## Text input (mutation APIs)
 

data/NOTICE

@@ -8,6 +8,57 @@ This product includes software developed by:
   Licensed under the Apache License, Version 2.0
   https://lexbor.com / https://github.com/lexbor/lexbor
 
+  The HTML tokenizer and tree builder under ext/makiri/rust/src/html/ and the
+  CSS syntax, selector and stylesheet parser under
+  ext/makiri/rust/src/selectors/ are Rust ports of Lexbor's, as are the HTML
+  CSS matcher under ext/makiri/rust/src/css_match/ and the tag and
+  attribute tables generated from Lexbor's. The extension links no Lexbor
+  code; Lexbor's LICENSE and NOTICE are kept under vendor/lexbor/ in the gem.
+
+* encoding_rs (linked into the native extension)
+  Copyright Mozilla Foundation
+  Licensed under the Apache License, Version 2.0 (at the licensee's option,
+  also the MIT license)
+  https://github.com/hsivonen/encoding_rs
+
+  encoding_rs includes data derived from the data files supplied with the
+  WHATWG Encoding Standard, under this license:
+
+    Copyright © WHATWG (Apple, Google, Mozilla, Microsoft).
+
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions are
+    met:
+
+    1. Redistributions of source code must retain the above copyright
+       notice, this list of conditions and the following disclaimer.
+
+    2. Redistributions in binary form must reproduce the above copyright
+       notice, this list of conditions and the following disclaimer in the
+       documentation and/or other materials provided with the distribution.
+
+    3. Neither the name of the copyright holder nor the names of its
+       contributors may be used to endorse or promote products derived from
+       this software without specific prior written permission.
+
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+    IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+    THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+    PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+    CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+    EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+    PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+    PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+    LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+    SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+* chardetng (linked into the native extension)
+  Copyright Mozilla Foundation
+  Licensed under the Apache License, Version 2.0 (at the licensee's option,
+  also the MIT license)
+  https://github.com/hsivonen/chardetng
+
 The XPath 1.0 engine under ext/makiri/rust/src/xpath/ is an original
 implementation written for this project. No code is derived from libxml2 or
 libxslt.

data/README.md

@@ -2,34 +2,38 @@
 
 Makiri is a Ruby library for parsing and querying HTML and XML documents.
 
-It uses [Lexbor](https://lexbor.com/) for HTML parsing and CSS selector matching, and includes a built-in native XPath 1.0 engine and XML 1.0 parser.
-Makiri does not depend on libxml2.
+It parses HTML with a Rust port of [Lexbor](https://lexbor.com/)'s HTML5 parser, and has its own CSS selector matcher, XPath 1.0 engine and XML 1.0 parser.
+Makiri does not depend on libxml2, and links no C library at all.
 
 > [!WARNING]
 > Status: early release. APIs and behavior may change before v1.0.
 
 ## What / Why
 
-Makiri uses Lexbor for HTML5 parsing and CSS selector support, and implements
-XPath 1.0 evaluation in its own native engine, with no libxml2 dependency.
-
-* HTML5 parsing via [Lexbor](https://lexbor.com)
-  * Makiri uses Lexbor as the parsing backend and provides a Ruby-facing DOM/query layer.
-* CSS selector support via Lexbor
-  * Supports Lexbor-backed standard CSS selector querying, including `:is`/`:where`/`:has`
+Makiri is one Rust extension: an HTML5 parser ported from Lexbor, CSS
+selectors, an XPath 1.0 engine and an XML 1.0 parser, with no libxml2
+dependency.
+
+* HTML5 parsing
+  * A Rust port of Lexbor's tokenizer and tree builder, into a compact DOM of
+    Makiri's own. It is held to the vendored Lexbor (which the gem does not
+    ship) and to the html5lib tests.
+* CSS selectors
+  * Selectors 4, including `:is`/`:where`/`:not`/`:has` and `:nth-child(... of S)`,
+    matched by one engine for HTML and XML and checked against Chromium,
+    Firefox and WebKit's `querySelectorAll`.
 * Native XPath 1.0 engine
   * XPath is parsed and evaluated by Makiri's own engine, written from scratch.
   * Makiri does not depend on libxml2 for parsing, DOM representation, or XPath evaluation.
 * Native XML 1.0 parser
   * A strict, non-validating, fail-closed parser with its own node arena (not
-    Lexbor's HTML DOM), queried through the same native XPath engine, with
+    the HTML DOM), queried through the same native XPath engine, with
     in-place tree edits (attributes, content, remove).
   * Conformance is held by the W3C XML Conformance Test Suite, an XPath
     differential, and property-based testing vs Nokogiri (see below).
 * Bounded, fail-closed execution
   * XPath evaluation is bounded by per-evaluation limits on work, memory, and recursion.
-  * HTML parsing bounds the tree depth (`max_tree_depth:`, default 400) and the
-    `<option>`s per `<select>` (10,000).
+  * HTML parsing bounds the tree depth (`max_tree_depth:`, default 400).
   * Ownership and borrowing are kept explicit across layers, with owned/borrowed
     string types and verified text at engine boundaries.
   * Programmatic invalid input, limit violations, allocation failures, and unsupported constructs
@@ -50,7 +54,7 @@ doc = Makiri::HTML(<<~HTML)
   </body></html>
 HTML
 
-# CSS selectors (Lexbor's selector parser, Makiri's own matcher)
+# CSS selectors
 doc.css("a").map { |a| a["href"] }      # => ["/a", "/b"]
 doc.at_css("p.lead").text               # => "Hello"
 
@@ -64,9 +68,14 @@ link = doc.at_css("a")
 link["href"]                            # => "/a"
 link.parent.name                        # => "div"
 
-# Source location (reconstructed from the tokenizer, no Lexbor patches)
+# Source location (the line of the element's start tag)
 doc.at_css("p").line                    # => 3
 
+# Bytes are decoded as a browser decodes them: BOM, <meta charset>, else
+# UTF-8 or a detected legacy encoding (or say which with encoding:)
+Makiri::HTML(File.binread("legacy.html")).encoding  # => "Shift_JIS"
+Makiri::HTML(bytes, encoding: "euc-jp")
+
 # Nesting deeper than 400 elements raises Makiri::Error (Nokogiri's default);
 # max_tree_depth: raises the limit, and a negative value disables it
 Makiri::HTML(deep_html, max_tree_depth: 2000)
@@ -109,9 +118,10 @@ el = doc.at_xpath("//a:entry", ns)
 el.local_name                                  # => "entry"
 el.namespace_uri                               # => "http://www.w3.org/2005/Atom"
 
-# CSS selectors work too (lowered to the native XPath engine): a bare type
-# selector binds to the document's default namespace, so this just works.
+# CSS selectors work too, as in a browser: a bare type selector matches its
+# name in any namespace, so this just works.
 doc.css("entry").length                        # => 2
+doc.css("a|entry", ns).length                  # => 2 (a prefix the Hash binds)
 doc.css("feed > entry").map { |e| e.at_css("title").text }  # => ["Hello", "World"]
 
 # Serialize back to XML
@@ -239,11 +249,14 @@ The HTML XPath and CSS suites are differentials against `Nokogiri::HTML5`
 HTML5, so the DOM is isomorphic and results are compared node-for-node. HTML
 parsing itself is checked against the WHATWG html5lib-tests corpus, and
 XPath-over-HTML semantics additionally against browsers via a WPT port.
+HTML parsing and CSS selectors are also run in Chromium, Firefox and WebKit,
+which are the reference where they and Nokogiri disagree.
 See also [`spec/conformance/README.md`](spec/conformance/README.md).
 
 | Suite | Input | Oracle | `rake` task |
 |---|---|---|---|
 | HTML parsing | HTML | WHATWG html5lib-tests (expected-tree corpus) | `conformance:html5` |
+| HTML parsing | HTML | Chromium, Firefox, WebKit (Playwright) | `conformance:browsers` |
 | XPath 1.0 | HTML | `Nokogiri::HTML5` (libxml2 XPath) — differential | `conformance:xpath` |
 | XPath over HTML | HTML | browsers (WPT `domxpath`, hand-ported; runs under `rake spec`) | — |
 | CSS selectors | HTML | `Nokogiri::HTML5#css` — differential | `conformance:css` |
@@ -251,15 +264,19 @@ See also [`spec/conformance/README.md`](spec/conformance/README.md).
 | XPath 1.0 | XML | `Nokogiri::XML` — differential | `conformance:xpath_xml` |
 | Parsed tree (property-based) | XML | `Nokogiri::XML` — differential | `conformance:xml_pbt` |
 | CSS selectors | XML | `Nokogiri::XML` — differential | `conformance:css_xml` |
+| CSS selectors | HTML, XML | Chromium, Firefox, WebKit `querySelectorAll` (Playwright) | `conformance:css_browsers` |
 | XML Builder | Ruby DSL | `Nokogiri::XML::Builder` — differential | `conformance:builder` |
 
 ## Requirements
 
-* CRuby 3.2 or newer.
-* A Rust toolchain (stable) with `cargo` - the extension is a Rust crate.
+* CRuby 3.3 or newer.
+* Rust 1.84 or newer with `cargo` - the extension is a Rust crate. (Binary
+  gems are published for the common platforms, which need no toolchain.)
 * libclang (clang's development library): rb-sys runs bindgen at build time and
   reads Ruby's headers through it.
-* CMake and a C toolchain, to build the vendored Lexbor static library.
+
+Development also needs CMake and a C toolchain, to build the vendored Lexbor
+that the Rust port is tested against (`rake lexbor`).
 
 ## Build (development)