semdiff-html 0.1.0__tar.gz

semdiff_html-0.1.0/.gitignore

@@ -0,0 +1,23 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+
+# Environments
+.venv/
+venv/
+
+# Tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+
+# Corpus capture staging (D-025: only admitted pairs go under corpus/)
+.captures/
+
+# Local secrets / logs
+.env
+.env.*
+*.log

semdiff_html-0.1.0/FINDINGS.md

@@ -0,0 +1,40 @@
+# Findings
+
+Observations made during tasks that were deliberately **not** acted on (CLAUDE_CODE_INSTRUCTIONS §3 rule 1). Each needs a decision before anyone acts on it.
+
+| # | Found during | Finding |
+|---|---|---|
+| F-001 | T-00 | `extruct` transitively installs `requests` and `urllib3` (via `mf2py` and `pyrdfa3`) plus `rdflib`, `beautifulsoup4`, `html5lib`, `lxml_html_clean`. **Corrected at T-30:** that attribution is incomplete -- `requests` is a direct dependency in `extruct`'s own metadata, not only one of `mf2py`'s and `pyrdfa3`'s, so three declared paths lead to it; of those, `mf2py` is the one that imports it *eagerly* (`pyrdfa3` and `rdflib` declare it without importing it). It remains transitive from SemDiff's side, which is what the sentence after this one means. NFR-2 says "no HTTP client" in the core install; the HTTP client arrives as a transitive dependency, not a direct one. Nothing in `src/semdiff/` will call it (hard rule 3), but the install footprint contradicts the letter of NFR-2. Options: accept and document; or call `extruct` syntax parsers selectively and vendor/replace `mf2py`/`pyrdfa3` usage later. **Resolved at D-047:** `extruct` is removed from the core install and will be declared by the release that first imports it; NFR-2 is restated as a principle rather than a package list. The second option above -- calling `extruct`'s syntax parsers selectively to avoid `mf2py`/`pyrdfa3` -- is measured and **not available**: importing any `extruct` submodule runs the package `__init__`, which imports the microformat extractor and loads `requests` with it. The install footprint and the condition this places on T-38 are recorded in D-047. |
+| F-002 | T-00 | `git config user.email` is unset globally on this machine. Irrelevant until the first commit. |
+| F-003 | T-02 | Naming inconsistency in ARCHITECTURE §Cross-cutting / T-02: `ParseError` and `InputTooLargeError` carry an `Error` suffix, `AlignmentBudgetExceeded` does not. Kept verbatim per D-010; a rename would be a public-API change. |
+| F-004 | T-03 | The sdist includes `docs/`, `research/`, and `FINDINGS.md` (hatchling default). Harmless, but decide what the published sdist should contain before the first PyPI upload. **Resolved at D-050.** Measured on a real build before deciding: 37 files, `docs/` (8), `research/` (1), `FINDINGS.md`, `scripts/capture.py`, `.gitattributes` and `.gitignore`, with `tests/`, `corpus/` and `.github/` already excluded. The rule is the code plus the evidence the README cites -- it sends readers to `FINDINGS.md` and `docs/DECISIONS.md`, so those stay -- and the things that are neither are now excluded: `.gitattributes`, `scripts/` (whose capture tool fetches URLs, stdlib only, never a hidden dependency) and the working agreement. `.gitignore` is listed too but ships regardless -- hatchling force-includes the VCS ignore files it used for its own exclusions, which the artifact showed and the configuration did not. 37 files before, 34 after. `tests/` and `corpus/` stay out with the downstream-testability cost stated, and `scripts/check_sdist.py` asserts all of it on every CI run. |
+| F-005 | T-13 | FR-5's illustrative Emotion pattern `css-[a-z0-9]{6,}` misses real 5-char hashes (`css-eu4da` on mui.com); the shipped rule uses `{5,}`. FR-5's CSS-modules example `[A-Za-z]+_[a-z0-9]{5,}` misses Docusaurus' 4-char mixed-case suffixes (`_O4bj`); the shipped rule requires a 4–8 char suffix with a digit or mixed case. Docs example regexes should be updated to match the shipped ruleset. |
+| F-006 | T-14 | Stripping a hashed `id` leaves its references (`for=`, `aria-labelledby=`, `aria-controls=`, `href="#…"`) pointing at nothing. Harmless for diffing (both snapshots are treated identically) but T-18 `REPLACE_WITH_PLACEHOLDER` should apply the same placeholder to `id` and its referencing attributes so the link survives; T-15 tokens family should consider React `useId` values in those referencing attributes too. **Closed at T-18 (D-043):** the id rules now rewrite the hashed span to a constant sentinel and run over `for`, `form`, `list`, `headers`, the ARIA id-reference attributes and `href` fragments, so an id and the references to it come out identical without the engine keeping any mapping. The residual gaps are in F-029. |
+| F-007 | T-15 | `token.session_input` (entropy on hidden-input values) strips ant.design's 15 `<input type=hidden name=parameters>` values — lz-string-compressed demo source posted to CodeSandbox/CodePen, not tokens. Encoded payloads are indistinguishable from session tokens by shape. Deterministic and harmless for the noise-only gate (the visible code block still diffs), but a consumer relying on hidden payload changes would need to disable `token.session_input`. Documented limitation candidate for T-65. |
+| F-008 | T-16 | docs.python.org (corpus fixture `docs-python-org-001`) rebuilds daily and prints `Last updated on Sep 17, 2026 (02:47 UTC).` — a human-readable date + `(HH:MM UTC)`, which is neither ISO-8601 nor relative phrasing, so FR-9 as written does not cover it. This pair will fail the T-21 gate on that footer alone unless FR-9 is extended with a conservative "human date + clock time (+ zone)" pattern. Decide at T-21 review, not before. **Resolved (D-041):** FR-9 extended and `timestamp.human_datetime` shipped; `corpus/noise_only/docs-python-org-001` now normalizes identically and is the regression test. |
+| F-009 | T-17 | Full normalization (parse + all five families) takes 54–520 ms on the five T-04 captures (docs.python.org 110 KB: 519 ms, 978 whitespace applications) — 1.5–5× over the NFR-3 sub-100 ms target for the parse+normalize path. Likely cost: `Locator.for_node` builds a positional CSS path per application (a parent walk with sibling counting). Profile before T-19/T-21 and consider lazy or cheaper locators; not a T-17 correctness issue. |
+| F-010 | T-19 | `canonical.whitespace` collapses **non-ASCII** whitespace: its `_WS` pattern is `\s+`, which is Unicode-aware in Python, so U+00A0 (`&nbsp;`), U+2009 and friends count as collapsible layout. The HTML spec collapses only ASCII whitespace; `&nbsp;` is content. Consequence: `normalize("<p>a&nbsp;b</p>") == normalize("<p>a b</p>")`, so a real nbsp/space edit is invisible to the diff. **Resolved in the T-19 change set (D-035):** `_WS` is now `[\t\n\f\r ]+` and `canonical.whitespace` is at `version = 2`. |
+| F-011 | T-19 | `semdiff.normalize` (the function) shadows the `semdiff.normalize` **subpackage** attribute, exactly as `semdiff.parse` already shadows `semdiff/parse.py`. `from semdiff.normalize import X`, `importlib.import_module` and deep submodule imports all still work, but `import semdiff.normalize as x` binds the *function*, and pytest's string-target `monkeypatch.setattr("semdiff.normalize.registry....")` no longer resolves (two tests now pass the module object instead). Pre-existing pattern, required by FR-10 + D-030; the alternative is renaming the subpackage to `semdiff._normalize`. |
+| F-012 | T-19 | Performance baseline for F-009, measured through the public API on the five T-04 captures (best of 5, 3.12): 122 / 171 / 203 / 282 / 550 ms total, of which parse is only 2.5–11.9 ms. **>95% of the cost is the rule engine**, not parsing — so F-009's suspicion (`Locator.for_node` per application, 249–1652 applications per page) is where to look. Still 1.2–5.5x over NFR-3. **Corrected at T-23:** re-measuring the same five captures on a quiet machine gives 32 / 43 / 53 / 129 / 171 ms — 3–4x faster with *more* rules running and no change to `Locator` or the engine hot path, so the T-19 figures were inflated by machine load and are not a usable baseline. Three of five captures now sit inside the NFR-3 100 ms budget; ant.design (576 KiB) and docs.python.org remain over. Any future optimization work must benchmark against a quiet machine and report a distribution, not a single run. |
+| F-013 | T-04 | **Hashed asset URLs are the dominant real-world deploy noise and no family covers them.** On the 2026-09-25 re-capture they are the *only* difference in docusaurus.io (two `<script src>` bundle hashes; the rest of the document is byte-identical) and the main one in mui.com (3 chunk hashes, the Next.js `buildId` in two `src`s and again inside `__NEXT_DATA__`, plus one `<link rel="preload">` reordered). docs.python.org adds the query-string variant (`pydoctheme.css?v=4365c8fe` → `?v=c60eaebf`) and ant.design the same in `/umi.da948370.js`. FR-5 covers `class`, FR-6 covers `id`, FR-7 covers tokens — nothing covers `src`/`href`/`srcset`. Blocks the noise-only admission of two otherwise-perfect pairs (Q-23). **Addressed in T-23** (FR-44, D-036/D-037): docusaurus.io now normalizes identically; mui.com does not, because its `buildId` also sits inside `__NEXT_DATA__` (protected) and one preload link is reordered — both deliberately out of scope. |
+| F-014 | T-04 | ant.design emits `<meta name="build-time" content="1789367183442">` and `<meta name="build-hash" content="db0488c1…">`, both changing every deploy. These are a timestamp and a hash in **attribute** position; FR-9's rules are TEXT-target only and FR-6 is scoped to `id`, so neither fires. Same shape as F-008 (a timestamp the family does not reach), different position. Confirmed at T-23: the same two values are also embedded in an inline `window.__UMI_METADATA_LOADER_DATA__` script, which D-029 protects, so even a `<meta>` rule would leave ant.design's pair differing. |
+| F-015 | T-04 | storybook.js.org streams its React Server Component payload as ~400 `<script>self.__next_f.push([1,"…"])</script>` calls, and the *chunk boundaries* shift between deploys even where the payload is equivalent: 418 of 1582 normalized tokens differ. Script content is protected by design (D-029), so normalization can never neutralize this. Any RSC/streamed-payload site is structurally unsuitable as a noise-only fixture unless the corpus rule allows ignoring `<script>` bodies — which would weaken the protection that makes JSON-LD extraction trustworthy. |
+| F-016 | T-04 | A 7-day interval on actively maintained sites produces **real** content changes, not just noise: ant.design bumped a version string (`6.6.4` → `6.6.5`), docs.python.org reworded a paragraph and added a `<div class="admonition note">` block, storybook.js.org's GitHub star counter moved (`91,045` → `91,144`). Three of the five pairs are disqualified from `noise_only` on content grounds alone. A noise-only corpus needs either a much shorter interval (hours, catching a redeploy without an edit) or pages that are not living documentation. **Superseded in part by F-018/F-019:** the shorter-interval half of this guess is wrong; the lever is deploy frequency, not interval length. |
+| F-017 | T-21 experiment | **React Suspense / SSR error-boundary markers vary between renders of the same build.** docusaurus.io served `<!--$!--><template data-dgst=""></template><!--/$-->` around two code blocks on 2026-09-26 and no Suspense markers at all on 2026-09-25 — same build id, same visible text, two extra elements. These mark Suspense boundaries that *errored* during streaming SSR, so they are neither deploy noise nor content: the same page renders differently request to request. Affects any streamed-SSR React site. Observed only — **no normalization rule** (D-039 keeps it a structural difference). |
+| F-018 | T-21 experiment | **"Low churn" is the wrong corpus selection criterion.** Six pages chosen for permissive licensing and stable text (pytest, PEP 8, Requests, Vite, Svelte, Tailwind docs) produced **zero** byte changes at 12.4 h and zero back-to-back — a page that never rebuilds never generates noise either. The productive profile is the opposite pairing: frozen text on a frequently-deployed site (versioned docs, archived posts, API reference for an old release, on projects that merge daily). Measured cadences: docusaurus.io deployed twice in 9 days; ant.design / mui.com / storybook.js.org moved over 7 days but not over 1; the six above did not move in 12.4 h. |
+| F-019 | T-21 experiment | **No per-request noise exists on any source tested.** Back-to-back fetches 5–7 s apart were byte-identical on 11 of 11 pages (the five T-04 URLs, plus the six above). Not one nonce, CSRF token, request id or relative timestamp varied. Sub-hour capture intervals therefore cannot produce a pair from this kind of source, whatever the page: they can only catch per-request variation, and there is none. The token family (T-15) still matters for sites that do carry such values — none of these do. |
+| F-020 | T-21 prep | `timestamp.human_datetime` (D-041) requires the clock time to be **parenthesised**, because that is the only form evidence exists for (`Sep 25, 2026 (15:12 UTC)`). The common unparenthesised variants — `Sep 25, 2026 at 15:12 UTC`, `25 Sep 2026 15:12 UTC`, `2026/09/25 15:12 UTC` — are deliberately not matched. Extend only when a capture shows one; the parentheses are doing real work as a false-positive fence, so dropping them needs its own negative-case calibration. |
+| F-021 | T-21 round 2 | **Visible build-identity strings are a third rejection class**, distinct from content edits and from deploy noise: the page states which build it is, in content position. Three shapes in one round -- the CPython patch version in body text (`3.14.7 Documentation` -> `3.14.8`, and `3.12.14` -> `3.12.15`, disqualifying all seven docs.python.org pairs at once), the rustdoc nightly identity in text and in an attribute (`1.101.0-nightly (75a75c3e0 2026-09-26)` -> `(21b707e3f 2026-09-30)`), and `<meta name=generator content="Starlight v0.42.4">` -> `v0.42.5`. Same shape as F-014's `build-time`/`build-hash` metas, but in text rather than attribute position, and it takes out a whole host at a time rather than a single pair. Left open as Q-26; no family added. |
+| F-022 | T-21 round 2 | **Scoped-style hashes can live in attribute *names*, where no rule can reach them.** vitest.dev (Vue SFC scoped styles) rebuilt `data-v-b483c686` -> `data-v-3d33a1ae` and friends across 372 of its 2329 elements, with zero text and zero element changes. Every `Target` in the rule model addresses an attribute *value*, a node or text; none addresses an attribute name, so this is unreachable by construction, not merely unimplemented. Any Vue-SFC-rendered site is therefore structurally unsuitable as a noise-only fixture, the same way F-015 rules out streamed-RSC sites. |
+| F-023 | T-21 round 2 | Two further generator-specific noise shapes outside FR-5/FR-6's patterns: Astro's `astro-<hash>` scoped classes (`astro-k3jukadp` -> `astro-vfz2b2k4` on starlight.astro.build and tauri.app, document-wide) and VitePress' regenerated tab ids on rollupjs.org, which churn in three attributes at once (`id=tab-NdIXQNq` -> `tab-__KmlXq`, with the matching `name=group-*` and `for=`). The VitePress case also moves `window.__VP_HASH_MAP__` inside a protected script. Both appeared only alongside disqualifying changes, so neither is worth a rule on this evidence; note that FR-6 covers `id` alone, so the `name`/`for` half would survive it anyway (cf. F-006). |
+| F-024 | T-21 round 2 | **The round-2 interval slipped to 4 days 5 h and F-016 reasserted itself immediately.** CPython shipped 3.14.8 and 3.12.15 inside the window, which disqualified the seven docs.python.org pairs -- the host that had been the corpus' best verified source. Their deploy noise was present and clean (the `?v=` cache busters rotated, `pydoctheme.css?v=c60eaebf` -> `?v=48df1187`, and the footer timestamp moved), so a window without a point release would plausibly have yielded all seven. 24 hours is not the conservative choice, it is the operative one: the probability of catching a real edit scales with the interval, and one release burns a whole host. |
+| F-025 | T-21 round 4 | **The Q-26 build-identity shape recurs reliably, which makes the open question costly rather than academic.** `doc.rust-lang.org/nightly/std/option/enum.Option.html` produced a clean build-identity-only pair in two of three rounds -- round 3 `(21b707e3f 2026-09-30)` -> `(0abfedbc7 2026-10-02)` and round 4 `(0abfedbc7 2026-10-02)` -> `(db8f076d2 2026-10-03)`, each time one text node and one `data-rustdoc-version` value out of 4940 text nodes and 5415 attribute values, with zero element or attribute-name movement. Rustdoc nightly is therefore a dependable daily source of exactly one noise shape, and Q-26 is the only thing standing between it and the corpus. The sibling page (`Vec`) is not usable the same way: its source-link line numbers shift whenever the upstream file moves. Recorded as evidence for Q-26; **no family added**, per the T-21 collection constraint. |
+| F-026 | T-21 round 4 | **tauri.app is the project's only real-world witness for the token family (FR-7), and it is permanently unusable as a fixture.** Its Netlify Core Web Vitals beacon carries `data-netlify-cwv-token="<JWT>"`, whose payload embeds `site_id`, `account_id` and `deploy_id`; the token rotated on every deploy across rounds 2, 3 and 4, each time as the *only* difference in the document (1090 elements, 310 text nodes, 1783 attribute values otherwise identical). This qualifies F-019's "no per-request noise exists on any source tested": none of the sources carries *per-request* noise, but at least one carries a genuine *per-deploy* token. The pair is rejected on sensitive-material grounds (committing a signed token would trip secret scanners, and corpus/README.md records the rejection), so FR-7 still has no corpus coverage from real captures -- only the synthetic cases in the unit tests. Worth stating plainly in T-65's documented limitations. |
+| F-027 | T-21 round 4 checkpoint | **selectolax 1.0.0 breaks `canonical.attr_order` on foreign content, and the dependency was unpinned.** CI installs the newest wheel (`pip install -e ".[dev]"`, no lock), so the 1.0.0 release turned every build red on a test that had been green since T-13: `<svg width="18" viewBox="0 0 16 16" fill="none">` comes back unsorted instead of `fill`, `viewBox`, `width`. Local 0.4.11 still sorts it, so the break is purely the upstream version. Everything else passed on 1.0.0 -- 512 of 513 tests, including all 17 corpus fixtures on Linux -- so the blast radius looks confined to attribute handling on foreign elements. **Deferred deliberately:** `selectolax<1.0` pins CI back to deterministic behaviour (NFR-1) rather than changing the normalization engine mid-collection, which could move normalized output for fixtures already admitted. Adapting to 1.0.0 -- and deciding whether the parser version belongs in `config_hash` -- is its own task before the pin is lifted. **Resolved at T-25 (D-046).** The cause is narrower than "attribute handling": 1.0.0 reports a spec-adjusted foreign-content attribute key in its adjusted spelling (`viewBox`), where 0.4.x lowercased it, so `_serialized_names`' case-sensitive guard rejected every `<svg>` and the rule declined them — it never misbehaved, it switched itself off. The guard now lowercases both sides and the rewrite looks the key up by the spelling the parser actually used. Measured across all 40 snapshots: 0.4.11, 0.4.13 and 1.0.0 now produce byte-identical normalized output and identical application records, and 0.4.13's output is unchanged from before the fix, so no rule version or `config_hash` moved. The pin is lifted (`selectolax>=0.4.11`) and CI runs 0.4.13 and 1.0.0 explicitly plus a weekly unpinned job. Two residues: `Locator.node_hash` still differs on foreign content between the lines (464 of 23,204 records — D-046 rider 2 declined to lowercase names inside D-044's fingerprint), and 1.0.0's narrowed `css_first` return type broke `mypy --strict` in 86 places across seven *test* files (none in `src/`), fixed by routing those queries through `tests/nodes.py`. |
+| F-028 | T-22 | **The ROADMAP's framing of T-22 overstated the baselines, and the measured story is different.** The pitch was "the standard tools report hundreds of changes and this reports none". Measured on the 20 noise-only pairs: raw `difflib` flags **20 of 20** pairs but only 1-3 changed units each, text-based `difflib` and `lxml.html.diff` flag **6 of 20**, and SemDiff flags **2 of 20** (both the FR-44 base64url gap). So the defensible claim is a false-positive *rate* of 20/20 -> 2/20, not a change *volume* of hundreds -> none. Two further corrections the measurement forced: the text-shaped tools are quiet on the 14 asset-hash pairs only because they never read `<head>` or a `<script src>` (the same hash on an `<img src>` is flagged, and a meaningful `href` change is invisible to text diffing), and `lxml.html.diff` passes a body `<script src>` change through unannotated, reporting no change while the attribute changed. ROADMAP wording corrected; the numbers and their limits are published in `docs/BASELINE.md`. **Updated after D-042:** extending FR-44 to base64url hashes took SemDiff from 2/20 to **0/20**, so the published claim is now 20/20 against 0/20. The correction this finding records — rate, not volume — stands unchanged. |
+| F-029 | T-18 | **What the placeholder cannot preserve.** Three limits, all accepted rather than engineered around, because none can produce a false positive — both snapshots collapse the same way — and each costs only recall. (1) Two distinct hashes *without a stable prefix* become the same text, so a reference re-pointed from `_R_aaa_` to `_R_bbb_` is invisible; a stable prefix (`tab-__semdiff__` vs `panel-__semdiff__`) is what keeps the common cases apart, and `test_two_distinct_hashes_without_a_stable_prefix_collide` pins the behaviour. (2) A reference whose value matches no id pattern is left alone, so it still dangles after the id it names is placeholdered — the patterns, not a document-wide mapping, decide what is an id. (3) `id.entropy` is off `href`: `EntropyMatcher` works on the longest alphanumeric run of the whole value and cannot be fenced to a fragment, so an entropic id in `href="#…"` is not rewritten, while the same id in `id=` and `for=` is. A page genuinely containing the literal `__semdiff__` would also mask a change there; the corpus contains none. |
+| F-030 | T-18 | **`Locator.node_hash` is order-dependent, so the fused attribute pass can report a different hash than one pass per rule.** `_node_hash` hashes `node.html`, an element's whole serialized subtree, so the fingerprint recorded for a node depends on whether its *descendants* have already been rewritten. Rule-major execution (one pass per rule) finishes a rule over the whole document before the next starts; the fused pass (optimization #3, `4f24bfc`) visits a node once and runs the run's rules there, so a node whose descendant is rewritten by an *earlier* rule in the same run gets a different `node_hash` in the two engines. Reproduced with two STRIP rules and nested elements: 1 of 2 records drifts when the descendant's rule sorts first. Not observable on the corpus — 0 of 40 snapshots, 23,204 records — and `tests/corpus/test_fused_attribute_equivalence.py` is green, but the invariant claimed in `_apply_attribute_rules`' docstring ("attribute rules are node-local") is true of the tree and false of this one piece of provenance. A T-18 bug briefly made it observable (a `class` value with trailing whitespace reported a no-op change on a descendant, disturbing 24 records across 4 snapshots); the bug is fixed, the mechanism is not. Three ways out, all needing a decision: hash only the node's own start tag (makes the two engines agree by construction and arguably a better node identifier, but changes every published `node_hash`); accept and document the fused engine's definition; or stop fusing. **Resolved at T-18 by D-044 (option 1):** an element's fingerprint is now `repr((tag, sorted attribute pairs))`, a text node's its text, a comment's its own markup. The reproduction that showed the drift — two STRIP rules over nested elements, the descendant's rule sorting first — is at 0 of 2 records, and the corpus stays at 0 of 40 snapshots across 23,204 records. `tests/test_locator.py` and `test_nested_elements_get_the_same_provenance_from_either_engine` pin it; restoring the subtree hash fails five of those tests. Published `node_hash` values all changed, which is the price the decision records. |
+| F-031 | T-18 | **Lexbor keeps an HTML `<template>`'s contents in a detached fragment, so the `template` entry in `PROTECTED_TEXT_PARENTS` only bites in foreign content.** For an HTML template, `child` is `None`, a CSS query inside it finds nothing, `text()` is empty and `traverse(include_text=True)` yields no node for it; serialization (`html` / `inner_html`) is the only way to that text. A TEXT rule therefore cannot reach it whether or not the tag is protected — the content survives because it is unreachable, not because it is declined, which made the original template protection test vacuous. Inside `<svg>` or `<math>` the name carries no special parsing rule, so the children are ordinary traversable nodes and the protection does real work; the protection test and a new whitespace test now use that shape, and dropping `template` from the set fails both (it failed neither before). Two consequences: on the lxml backend (T-11, deferred to Phase 2c at D-048) template contents are ordinary children -- measured, not predicted: `lxml.html` reports one child and `inside` as its text (F-034) -- so the HTML case becomes reachable there and the protection must be re-tested; and any family that needs to see inside an HTML template on selectolax would have to re-parse `inner_html`, which nothing does today. |
+| F-032 | T-27 | **`LexborNode.__eq__` is proportional to the serialized subtree, which is why provenance cannot be memoized by node — and `mem_id` is the way round it.** Measured: `__hash__` is the underlying DOM node's address (`hash(node) == node.mem_id` for every node checked, 5,411 distinct hashes for 5,411 nodes) and costs ~0.04 us, so a node-keyed dict *hashes* cheaply; `__eq__` does not. Two wrappers for the same node compare equal, but the comparison costs **765 us for the root `<html>` of a 58 KB document** against **0.6 us for a 4-byte leaf**. The hypothesis that fits those two points — that equality serializes and compares the subtree — is not confirmed against selectolax's source and is not needed: what is measured is that the cost scales with subtree size. The consequence is measured directly. A dict or set keyed by *element* nodes pays that comparison whenever a probe lands on an occupied bucket, so 5,411 membership tests cost 0.21 ms when the probes are the same wrapper objects and **17-19.5 ms** when they are wrappers from a second traversal — the normal case in this engine, where each pass makes its own. Memoizing `_css_path` keyed by the node is therefore correct but slower: 0 mismatches over every element of a document, **25x faster once warm** (27.3 -> 1.1 ms) and **5.5x slower cold** (27.3 -> 149.2 ms), because the cache never warms inside one pass; end to end it made `apply_rules` about 7x slower. The same cache keyed by `mem_id` — an `int`, so `__eq__` is never reached — is 1.09-1.23x *faster* across two runs, and is what T-27 implements. Two standing limits on that key: it is an address, so a cache must not outlive a pass that can destroy an element (no builtin rule does; `DROP_NODE` is available to user rules under FR-11, and `_apply_node` is given no cache), and `mem_id` is a selectolax implementation detail with no documented stability guarantee, so it carries the same parser-version exposure T-25 dealt with — `tests/corpus/parser_contract.json` is what would catch it moving. This also explains why `_protected_text_nodes`' node-keyed set is affordable: its keys are text nodes, whose subtrees are empty. All timings are from one Windows laptop that was not idle, on Python 3.12.10 and selectolax 0.4.11; treat the ratios as the result. |
+| F-033 | T-29 | **What the network gate proves, and the three routes that go around it.** `tests/test_network_isolation.py` blocks every standard-library outbound entry point -- `socket.socket.connect`, `connect_ex`, `sendto`, `sendmsg`, plus `socket.create_connection`, `socket.getaddrinfo` and `socket.gethostbyname` -- recording each blocked call *and* raising a `BaseException` subclass, then runs the public path under that guard: `normalize()` over all 40 corpus snapshots, over a document baiting a fetch at every reference a browser would follow (external entity, `meta refresh`, `base href`, stylesheet, preload, `@import`, `script src`, `img src`/`srcset`, `iframe`, `object`, `embed`, `video poster`/`source`), `parse()` with a `url=` argument, and the CLI with `--report`. **Measured: zero attempts on any of them.** Four deliberate network calls inserted into `normalize()` and `parse()` are each caught, including one swallowed by `except BaseException: pass` -- that mutant fails on the recorded list alone, where a guard that only raised would have reported nothing, which is why the list exists. **What it is not:** evidence over the inputs exercised, not a proof over all inputs or all execution paths. The patches sit at Python level, so three routes bypass them -- a call straight into `_socket`, an `os`/`subprocess` shell-out, and a socket opened by a C extension inside its own code, which patches at Python level cannot observe at all. That third route describes where the patches sit, not any particular extension: selectolax is simply the only C extension on this path, and nothing measured here is an observation that it opens a socket. **No evidence suggests SemDiff takes any of the three:** none of the 19 modules under `src/semdiff/` imports `socket`, `_socket`, `ssl`, `http`, `urllib` or an HTTP client (audited at T-29; `socket` and `urllib.parse` do reach `sys.modules` because pydantic imports them, which is exactly why the gate watches calls rather than imports), and nothing shells out. The three are limits of the measuring instrument, not suspicions about the code; closing them needs an OS-level sandbox in CI -- a syscall filter or a network namespace -- which is a decision, not a task. **NFR-8's extruct clause is separately unexercised.** It requires "no outbound requests from extruct paths"; no module under `src/semdiff/` imports `extruct`, and since D-047 it is not in the core install at all, so there is no extruct path to run and the clause is vacuously satisfied rather than verified. `test_extruct_is_not_on_the_public_path_yet` pins that absence in a subprocess, so it fails the day T-38 wires extruct in -- at which point the gate must be extended over the extraction path and this finding revisited. |
+| F-034 | D-048 | **What the two parser backends actually agree on, measured on the corpus -- and it is not equivalence.** selectolax 0.4.11 against lxml 6.1.3 (libxml2 2.11.9), all 40 snapshots, through each parser's own API. **Structure agrees:** the list of `nth-child` element paths, in document order, is identical on 40/40, so `Locator.css` would carry over. **Attributes agree except in one way:** 150 elements across the corpus differ, and every difference is the same one -- Lexbor reports a boolean attribute as `defer=""`, lxml as `defer="defer"`. No name and no non-boolean value differs. That feeds D-044's fingerprint directly, so a cross-backend `node_hash` cannot match, the same shape D-046 found between parser *versions*. On the selectolax 1.0.x line, where Lexbor reports `viewBox` (D-046) and lxml reports `viewbox` (measured here), foreign-content attribute *names* would diverge too, so cross-backend attribute agreement is itself a function of which selectolax is installed. **Serialization agrees nowhere:** 0/40 round-trip byte-identical. Measured causes on controlled shapes -- lxml drops the doctype, drops an empty `<head>`, writes a boolean attribute bare, decodes `&nbsp;` to U+00A0, and lowercases `viewBox` in the markup where Lexbor keeps the adjusted spelling. **The models differ underneath:** lxml has no text nodes at all (`div.text`, `span.tail`), and a comment is an element whose `.tail` carries the text after it, where Lexbor exposes `-text` and `-comment` nodes; an HTML `<template>`'s contents are ordinary children on lxml and unreachable on Lexbor (F-031). **Cost, measured:** interleaved A/B over the 40 snapshots, 7 rounds, Lexbor median 42.6 ms against lxml 127.0 ms (1.06 ms vs 3.18 ms per snapshot, 2.98x) -- same direction as FR-2's cited 2.39s/9.09s. **Safety:** no Python-level connection attempt from `lxml.html.document_fromstring`, `etree.HTMLParser()` or `HTMLParser(no_network=False)` on a bait document carrying an external-entity DTD, `<base>`, `<link rel=stylesheet>`, meta-refresh and `<img>`, all at `192.0.2.1`; F-033's limits apply unchanged, so this says nothing about libxml2's own C-level fetching, which T-11 would have to gate separately. **None of this is a claim of equivalence**: it says where the two backends already match and where they cannot without new code, and which reading of "equivalent" Q-08 still has to choose. |

semdiff_html-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 AmitKumar-24
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

semdiff_html-0.1.0/PKG-INFO

@@ -0,0 +1,254 @@
+Metadata-Version: 2.5
+Name: semdiff-html
+Version: 0.1.0
+Summary: Semantic HTML change detection: two snapshots in, typed change objects out.
+Project-URL: Homepage, https://github.com/AmitKumar-24/semdiff
+Project-URL: Repository, https://github.com/AmitKumar-24/semdiff
+Project-URL: Issues, https://github.com/AmitKumar-24/semdiff/issues
+Author: AmitKumar-24
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: change-detection,diff,html,normalization,scraping,web-monitoring
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: pydantic>=2
+Requires-Dist: selectolax>=0.4.11
+Provides-Extra: all
+Requires-Dist: sentence-transformers; extra == 'all'
+Requires-Dist: trafilatura; extra == 'all'
+Provides-Extra: content
+Requires-Dist: trafilatura; extra == 'content'
+Provides-Extra: dev
+Requires-Dist: lxml; extra == 'dev'
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Provides-Extra: ml
+Requires-Dist: sentence-transformers; extra == 'ml'
+Description-Content-Type: text/markdown
+
+# SemDiff
+
+Semantic HTML change detection. Two HTML snapshots in, typed change objects out
+(`price_change`, `availability_change`, …) with confidence and provenance — instead of a raw
+text or DOM diff. Deterministic core, no ML required, built to be the diff backend inside
+monitoring tools and scraping pipelines.
+
+**Status: pre-alpha.** Diffing is not implemented yet. What works today is the layer
+underneath it — normalization — which is useful on its own.
+
+## Normalization
+
+Most "changes" a page monitor reports are not changes: a framework re-hashed its CSS class
+names, a CSRF token rotated, a relative timestamp ticked over, a template re-indented.
+`semdiff.normalize()` removes that noise and canonicalizes the tree, so two renders of an
+unchanged page compare equal — with `==`, or with whatever differ you already use.
+
+```python
+import semdiff
+
+semdiff.normalize(old_html) == semdiff.normalize(new_html)   # True if nothing real changed
+```
+
+```python
+semdiff.normalize('<div class="card css-1dbjc4n"><!-- r4 --><p>Updated 3 minutes ago</p></div>')
+# '<html><head></head><body><div class="card"><p>Updated</p></div></body></html>'
+```
+
+```python
+semdiff.normalize(html, config=None, *, encoding=None) -> str
+```
+
+Accepts `str` or `bytes` (decoded by declared encoding → BOM → `<meta charset>` → UTF-8) and
+returns the canonical serialization. Raises `ParseError` on empty input, `InputTooLargeError`
+above the configured ceiling (10 MiB by default), and `ValueError` if the config names a rule
+that does not exist.
+
+### What it removes
+
+| Family | Examples |
+|---|---|
+| Dynamic classes | Emotion `css-1dbjc4n`, styled-components `sc-bdfBwQ`, CSS Modules `toggle_bT41`, Ant Design `acss-q7dsfq` |
+| Hashed ids | `react-root-7a3b2c`, React `useId` (`:R2m:`), high-entropy generated ids |
+| Tokens | `nonce` attributes, CSRF meta tags and hidden inputs, session-shaped attribute values |
+| Timestamps | ISO-8601 datetimes, "3 minutes ago", "just now", "updated yesterday", build stamps like `Sep 25, 2026 (15:12 UTC)` |
+| Canonicalization | comments, ASCII-whitespace collapse, attribute order, void/boolean-attribute spelling |
+
+Microdata, RDFa and JSON-LD carriers are protected: no rule touches `itemprop`, `itemtype`,
+`about`, `typeof` and their kin, and text inside `<script>`, `<style>` and `<template>` is
+left alone. `&nbsp;` is content, not layout, and survives. Every rule is individually
+toggleable and reports what it changed.
+
+```python
+from semdiff import Config, NormalizationConfig
+
+config = Config(normalization=NormalizationConfig(disabled_rules=frozenset({"timestamp.relative_ago"})))
+semdiff.normalize(html, config)
+```
+
+### Command line
+
+```
+semdiff normalize page.html                  # normalized HTML on stdout
+cat page.html | semdiff normalize            # or from stdin
+semdiff normalize page.html -o clean.html    # or straight to a file
+semdiff normalize page.html --report         # which rules fired, on stderr
+semdiff normalize page.html --disable-rule timestamp.relative_ago --disable-rule class.emotion
+semdiff normalize page.html --config semdiff.json
+```
+
+`--report` writes to stderr, so stdout stays byte-identical with or without it:
+
+```
+$ semdiff normalize page.html --report > clean.html
+parser selectolax/1.0.0
+6 rules fired, 10 applications
+  class.emotion           1
+  id.hex_suffix           1
+  timestamp.relative_ago  1
+  asset.dotted_hash       1
+  canonical.comments      1
+  canonical.whitespace    5
+```
+
+Exit codes: `0` success, `2` bad command line, `3` parse error, `4` input too large,
+`5` configuration error, `6` I/O error, `1` anything else.
+
+### Guarantees
+
+- **Deterministic.** Same input, same config and same parser build, same output — no clock,
+  no network, no randomness. `Config.config_hash` covers the config and the ruleset version;
+  the parser is the one input it does not cover, so `semdiff.parser_fingerprint()` names it
+  (`selectolax/1.0.0`) and `--report` prints it. Normalized output is identical across the
+  selectolax versions CI covers; a `Locator.node_hash` on foreign content is not (F-027).
+- **Idempotent.** `normalize(normalize(x)) == normalize(x)`.
+- **Safe.** Input is untrusted: parse-only, no script execution, no external entity
+  resolution, no outbound requests.
+
+## Limitations
+
+What SemDiff does not do, stated plainly: the failure modes a library leaves undocumented are
+the ones that cost it trust. Each item names the evidence behind it — `FINDINGS.md` for the
+observations, `docs/DECISIONS.md` for the decisions that accepted them.
+
+### Not implemented
+
+- **One parser backend.** `parser="lxml"` raises `NotImplementedError`. selectolax (Lexbor) is
+  the only backend; the lxml adapter is T-11, deferred out of v0.1 to sit beside the provenance
+  work that wants it, with no equivalence between the two backends claimed (D-048).
+- **Rules are toggleable, not extensible.** Any shipped rule can be turned off or on through
+  config, but no config field adds one. A custom rule means reaching past the public API into
+  the registry, which is not a supported surface yet.
+- **`normalize()` does not report what it changed.** It returns a string and nothing else. The
+  rules that fired come from `apply_rules()`, or on stderr from `semdiff normalize --report`.
+- **Slower than the 100 ms per-pair target.** The published benchmark measures a median of
+  373 ms per pair against 125 ms for `lxml.html.diff` — about 3x, bought for twenty fewer false
+  positives. The numbers, the machine and the method are in `docs/BASELINE.md`.
+
+### What normalization cannot reach
+
+- **Hashes that live in attribute names.** Every rule target addresses an attribute's value, a
+  node or text; none addresses a name. Vue SFC scoped-style markers (`data-v-b483c686`)
+  therefore survive, and they rebuild on every deploy (F-022).
+- **Anything inside a protected carrier.** Text in `<script>` and `<style>` is never touched,
+  which is what keeps JSON-LD and other structured data trustworthy (D-029). The cost: a
+  streamed React Server Component payload whose chunk boundaries shift between deploys reads as
+  a change that no rule can remove (F-015).
+- **The contents of an HTML `<template>`.** Lexbor keeps them in a detached fragment, so no
+  rule reaches them on the selectolax backend — they survive because they are unreachable, not
+  because they are declined (F-031).
+- **Stamps that name the build itself.** A nightly version string in body text, or ant.design's
+  `<meta name="build-time">` and `<meta name="build-hash">`, is left alone. Whether a
+  build-identity string is noise at all is still an open question (Q-26, F-014).
+- **A human datetime whose clock time is not parenthesised.** `Sep 25, 2026 (15:12 UTC)` is
+  neutralized; `Sep 25, 2026 at 15:12 UTC` and `2026/09/25 15:12 UTC` are not. The parentheses
+  are doing real work as a false-positive fence, so widening the pattern needs its own
+  calibration against negative cases (F-020).
+
+### Accepted recall costs
+
+Each of these costs recall and never precision — both snapshots collapse the same way, so none
+of them can invent a change — and each was accepted rather than engineered around.
+
+- **Two hashes with no stable prefix collide.** Both become `__semdiff__`, so a reference
+  re-pointed from `_R_aaa_` to `_R_bbb_` is invisible. A stable prefix is what keeps the common
+  cases apart: `tab-__semdiff__` and `panel-__semdiff__` still differ (F-029).
+- **`id.entropy` does not reach inside an `href`.** It scores the longest alphanumeric run of a
+  whole attribute value and cannot be fenced to the fragment after `#`, so an entropic id is
+  rewritten in `id=` and `for=` and left alone in `href="#…"` (F-029).
+- **`token.session_input` strips encoded hidden-input payloads.** A compressed payload in a
+  hidden input has the same shape as a session token — ant.design's CodeSandbox demo sources
+  are stripped as tokens. If you depend on hidden-input values, disable the rule (F-007).
+
+### Where the evidence is thin
+
+- **Three noise families have no real-world coverage.** The corpus is 14 `asset_hash` and
+  6 `timestamp` pairs. Real captures offered no `dynamic_class`, `hashed_id` or `token` churn to
+  admit, so those three families rest on unit tests alone (F-019, F-023, F-026).
+- **Recall is gated on synthetic mutations.** The gate edits those same twenty captures in
+  memory, one surgical change at a time, and asserts each edit survives normalization as
+  itself. It cannot show how normalization behaves on the changes a real deploy happens to
+  make; a corpus of captured real changes does not exist yet (D-045).
+- **Determinism is qualified by the parser build.** Same input, same `config_hash`, same parser
+  build, same output. Normalized output is identical across the selectolax versions CI covers,
+  but a `Locator.node_hash` on foreign content is not: 464 of 23,204 application records differ
+  between the 0.4 and 1.0 lines (D-046).
+
+## Install
+
+```
+pip install semdiff-html
+```
+
+The distribution is `semdiff-html`, because `semdiff` on PyPI is an unrelated project. What
+you write is unaffected — the import package and the command are both `semdiff`:
+
+```python
+import semdiff
+semdiff.normalize(html)
+```
+
+```
+semdiff normalize page.html
+```
+
+To work on it instead:
+
+```
+pip install -e ".[dev]"
+```
+
+Requires Python 3.11+. Runtime dependencies: `selectolax`, `pydantic`. `lxml` is needed
+only by the baseline comparison in the test suite, so it lives in `[dev]`.
+Two further extras are declared for layers that are not built yet — `semdiff-html[content]`
+(trafilatura scoping) and `semdiff-html[ml]`, with `[all]` their union — so installing one
+today adds a dependency and no capability. The names are fixed now because a requirements
+file cannot be renamed retroactively (D-050).
+Both selectolax lines are supported and tested: CI runs 0.4.13 and 1.0.0 against every
+Python version, plus a weekly job on whatever is newest.
+
+## Development
+
+```
+pytest -q
+mypy --strict src/semdiff tests scripts
+```
+
+CI runs both on Python 3.11, 3.12 and 3.13, against selectolax 0.4.13 and 1.0.0.
+
+## Documentation
+
+`docs/` holds the design: `DOMAIN_MODEL.md` (authoritative), `ARCHITECTURE.md`,
+`REQUIREMENTS.md`, `ROADMAP.md`, and `DECISIONS.md` for the decision log and open questions.
+`FINDINGS.md` records observations that were deliberately not acted on.
+
+## License
+
+Apache-2.0.