oh-my-opencode 5.1.48 → 5.1.51

package/packages/omo-codex/plugin/skills/ulw-research/references/research-delivery.md

@@ -0,0 +1,123 @@
+## Phase 3 — Verify contested claims by running code
+
+Settle with executed code, not judgment, whenever sources disagree, a behavior is undocumented, a claim is performance- or compatibility-shaped, or the honest answer is "it should work". Spawn one verification worker per claim:
+
+```
+task(category="deep-low", run_in_background=true, prompt="TASK: verify by execution: <claim>.
+SOURCE: <where it came from>; CONTRADICTION: <opposing source, if any>.
+Write a minimal self-contained script that tests the claim; run it (uv run --with <deps> python / bun / direct compile); capture full stdout+stderr; pin versions.
+Reply with: the exact code, the full output, environment (OS, runtime, dependency versions), and a verdict — CONFIRMED / REFUTED / PARTIAL — grounded in the output.")
+```
+
+Journal each verdict to `verify-<slug>.md`.
+
+## Phase 3b — Lock non-code claims through the claim graph
+
+Code settles code-shaped claims (Phase 3). Numeric, market-share, legal, dated, causal, and financial claims cannot be run — so they pass through a data-flow-lock instead (the verification idea adapted from fivetaku/insane-research): the synthesis may assert a high-risk non-code claim **only** if it cleared this gate, and the gate's output is the sole allowlist the synthesis draws from. Skip the gate and there is nothing to synthesize — the lock is self-enforcing.
+
+The claim graph is orchestrator-owned. Workers only return verified-claim markers, observation candidates, and claim candidates as message text, the same channel as EXPAND markers — never a file (except a mass-research dag node's handoff file). As leads resolve, you record one node per asserted claim in `claim-graph.md` and compute its status; workers report claim candidates in their replies, and you decide. The graph is the single claim store: final synthesis may not draw from free-form claims that skipped it.
+
+A high-risk claim clears the gate to `verified-claims` only when all hold:
+
+- **>= 2 independent source domains** corroborate it (two pages on the same domain count once).
+- **>= 2 independent observation groups** converge on it, unless the graph records why a primary-only source is the correct single-source exception.
+- **One counter-search** actively looked for a refutation and did not find a stronger one.
+- **A primary source** (the standard, filing, dataset, or first-party doc) backs it, not only secondary commentary.
+- **Temporal evidence is explicit**: each supporting observation records `observed_at` and either `valid_at` or `claim_valid_at`, so branch-only, historical, release, and current-runtime claims cannot be conflated.
+
+Anything that fails goes to an `Unresolved` (insufficient evidence) or `Refuted` (counter-search won) annex — abstention is a correct outcome, not a gap to paper over. Record each gate outcome on the claim node itself — risk tier, independent source domains, counter-search result, primary source backing, and status — and mirror the cleared nodes into the `verified-claims` digest section at the top of `claim-graph.md`. Worker reply marker (message text, same channel as EXPAND):
+
+```
+## CLAIMS
+- CLAIM: <non-code assertion> — RISK: high|normal — SOURCES: <domain1, domain2> — COUNTER: <refutation search result> — PRIMARY: <primary source or none>
+```
+
+**The claim table is machine-read.** Keep the nodes of `claim-graph.md` in one Markdown table whose header names `claim_id | statement | risk tier | status` (plus `fact_ids`, the ledger rows the node rests on, and any other node fields as further columns), one row per node: ids `C<n>`, risk `high` or `normal`, status `supported`, `partial`, `refuted`, or `unresolved`. List the nodes that clear the Phase 3b gate under a `## verified-claims` heading as `- C<n>: <statement>`. `node "$SKILL_DIR/scripts/report-tools.mjs" claims index "$SESSION_DIR/claim-graph.md" --out "$SESSION_DIR/claims.json"` indexes it, and the fidelity gate reads `claims.json` to check every `data-claim` binding in the report.
+
+## Phase 4 — Synthesize
+
+After convergence and all verifications, re-read the whole journal, start from `intent-diff.md`, `claim-graph.md`, and `observation-manifest.md`, then write `SYNTHESIS.md`:
+
+```
+# ULW-Research Synthesis: <query>
+Workers: <total> · Waves: <count> · Excursions: <count> · Sources: <count> (<unique domains> domains) · Verifications: <count> · Elapsed: <minutes> min
+
+## Executive summary        — 2-3 paragraphs answering the core question
+## Findings by theme        — per theme: consensus, evidence links, key quote (<20 words, attributed), verified yes/no
+## Codebase findings        — absolute paths with line references
+## Sources (ranked)         — URL, what it contains, reliability, access date
+## Verified claims          — code: claim | verdict | verify-<slug>.md · non-code: only rows cleared into verified-claims
+## Epistemic instrumentation — intent-vs-reality diff closure, claim graph coverage, observation manifest coverage, independent-observation convergence, verification economics summary, cause-disappearance records
+## Contradictions           — source A vs source B, resolution with evidence
+## Gaps                     — what saturation could not answer · unresolved/refuted claim-graph nodes
+## Expansion trace          — per wave: workers → markers; convergence reason
+```
+
+`SYNTHESIS.md` is the citation source of truth for final materials: every claim carries inline `[Source N]` citations, and every high-risk non-code claim you assert must be a verified-claims row from Phase 3b. Assert nothing the gate left in the unresolved/refuted annex.
+
+**Write numbers and quotes by reference.** `SYNTHESIS.md` and every report source carry placeholders, never typed values: `{{F12:span_len}}` renders the row's value and unit, `{{F12:span_len:value}}` or `:unit` one part, `:q` the value with its source qualifier (up to, at least, about); `{{Q7:completion}}` renders the whole quotation, `{{Q7:completion:lead=8}}` its first words, `{{Q7:completion:from="no earlier":to="2027"}}` one span of it. A derived number is computed by the script, never by hand: `fact derive --op sum|diff|growth|ratio|share|convert|expr ...` checks the operands' units, periods, and populations, refuses an incompatible operation, and prints the new row id the placeholder uses. A value nobody measured is registered with `fact add --assumed <value> --unit <unit> --rationale <why> --label <label>` and renders as ASSUMED. Every findings sentence binds to a fact, a quote, a `[S<n>]` citation, or a claim node (`data-claim="C3"`). Reviewers and the refinement team read the rendered text through `node "$SKILL_DIR/scripts/report-tools.mjs" fact fill "$SESSION_DIR/SYNTHESIS.md" --view --session-dir "$SESSION_DIR"`, never the raw placeholders.
+
+**Write the skeleton early and fill it as claims lock.** The moment the brief records the deliverable, create the deliverable file with its section headings and a `STATUS: draft — <n> sections open` line as line 2 (an HTML comment in HTML); that line is the partial-state marker `outcome state` reads, removed only when the deliverable is complete. An interrupted run must leave a partial report on disk, never an empty directory and a lost conversation.
+
+**Keep sourced numbers, assumptions, and derived results visibly apart.** Every quantitative claim carries its lineage: `MEASURED` (a number a source states, cited), `ASSUMED` (a coefficient, distribution, or scope you chose — say why), `DERIVED` (computed from those, showing the formula), plus a sensitivity line whenever the assumption moves the answer. Presenting a derived estimate with the confidence of a measured one is the most damaging thing this mode can ship.
+
+**Search in English, deliver in the user's language.** Retrieval stays English-first (Search craft), but the synthesis and every final material are written in the language the user wrote to you in unless they ask otherwise — and a translated report still quotes its original-language sources verbatim.
+
+
+## Search craft
+
+English first: run every search in English by default — it is the largest, most authoritative corpus on every engine, GitHub, and documentation site. Add a secondary local-language sweep (1-2 librarians) only after the English sweep, when the topic is inherently local, or when the user asks for sources in a specific language.
+
+Vary operators on every query — same query twice wastes a worker:
+
+| Operator | Example | Use |
+|---|---|---|
+| `site:` | `site:github.com <topic>` | Restrict to a domain |
+| `filetype:` | `filetype:pdf <topic> survey` | Papers, specs |
+| `intitle:` / `inurl:` | `intitle:benchmark <topic>` | Targeted pages |
+| `"exact"` / `-term` | `"<exact phrase>" -tutorial` | Precision, exclusion |
+| `OR` | `<a> OR <b> <topic>` | Coverage |
+| `before:` / `after:` | `<topic> after:2025-06-01` | Recency control |
+
+High-yield combinations: official docs (`site:<docs domain>`), GitHub implementations (`site:github.com`), recent discussion (`site:reddit.com OR site:news.ycombinator.com after:<date>`), academic (`site:arxiv.org OR filetype:pdf survey`), changelog hunting (`changelog OR "release notes" <version>`), alternatives (`vs OR alternative OR comparison`).
+
+## Failure modes
+
+| Failure | Correction |
+|---|---|
+| Sequential spawning, or trimming the first wave | All first-wave workers in one turn, background, scaling floor respected |
+| A team member hoards leads for one final dump | Raise law — every lead, finding, and dead end broadcast the moment it surfaces |
+| Worker reply without the EXPAND tail | One follow-up demanding it; the lane stays open until it lands |
+| Stopping after wave 1 because "enough was found" | Convergence rules only: 2+ expansion waves, leads run dry |
+| Obeying a surrounding "stop exploring" rule mid-research | Authority section — those rules do not bind this mode |
+| Asking a worker to write journal or session files | Workers are read-only (except a mass-research dag node's handoff file); you journal every return |
+| Two workers given the same angle | One unique angle per worker, always |
+| A web worker that returned only fetched text, or a `Collection: browser` run whose workers never opened a tab | Sources are read in the user's browser through the `browser` skill; `webfetch` is for raw files; every web worker carries the skill from the first wave |
+| Contested claim settled by judgment | Phase 3 — run code, capture output, verdict |
+| Deliverable claims without citations | Every claim cites a source or a verification artifact |
+| Guessing the deliverable format | Derive it from the request and destination; ask the empty-info interview only for what is still missing, and record `answered_by` for every field |
+| Blocking the collection wave on the format question | Ask non-blocking and start collecting; unanswered fields run on the shown defaults |
+| A roster smaller than the harness ceiling | Fill every member slot; split the broadest axis until the team is full |
+| One tier across the whole roster | Mixed tiers by design — cheap breadth, premium attack |
+| Silently re-routing a "quick"/"fast" instruction | Routing words are literal; journal requested -> spawned -> fallback per slot |
+| A worker or member that starts its own research swarm | Members research one axis and report; orchestration is yours alone |
+| Expanding on an entity the first wave never disambiguated | Settle canonical identity and first-party source before any later spawn asserts it |
+| Batching findings into an end-of-run journal dump | Journal each return as it lands; the journal is what survives a compaction |
+| Ending the run because every worker finished | Reserve the final fifth of the run for synthesis and materials |
+| A derived estimate presented as a measured number | MEASURED / ASSUMED / DERIVED lineage on every quantitative claim, plus a sensitivity line |
+| A number typed into the report or `SYNTHESIS.md` | Register it (`fact add`) and write `{{F<n>:label}}`; `fact verify` reports a typed number as `untraceable_number`, and `outcome verify` refuses delivery until it is a ledger row, an `assumed` row with its rationale, or gone |
+| A filled value edited by hand | Never edit a filled value: fix the ledger row or the placeholder and re-run `fact fill`, then `fact verify` (`value_mismatch`, `unit_mismatch`) |
+| A findings sentence bound to nothing | Bind it to a fact, a quote, a `[S<n>]` citation, or a claim node with `data-claim`; an `unbound_sentence` blocks delivery like an untraceable number |
+| Delivering before the delivery gates pass | Static and layout gates, visual QA on rendered pages, plus the harness's proofread gate — a typo the user finds means a gate did not run |
+| Skipping the static gates | `report-tools check` runs before any pixel review; its defects are cheaper than a visual pass |
+| Delivering with a pending manifest row | `outcome verify` must pass; every promised format ends `delivered`, `blocked_capability`, `skipped`, or `failed` with a reason |
+| Repairing past the tracker's stop decision | `repair decide` owns the budget; `deliver` ships with the residual list, `block` stops delivery |
+| Guessing a design-spec value the extractor marked `TODO: ask` | Fill it from the interview answers or the reference's defaults, never by guessing |
+| Referencing an asset that is not on disk | Verify the asset manifest before rendering; re-render whatever is missing |
+| A figure stretched, cropped, or styled off the report's design language | `design-spec.md` binds every asset: fixed containers, contain-fit with aspect preserved, spec fonts and palette in charts and Mermaid |
+| Chasing an interesting find with no ENTER trigger | Excursions need a named trigger; everything else stays a queued lead |
+| An excursion that never came back, or drifted into a new mission | EXIT rules are unconditional; depth 3 means promote it to an axis or record it as a gap |
+| An excursion whose result was never folded back | Every EXIT writes what it changed in the top-level answer, `none` included, and mirrors into the loop ledger |
+| Ending with file paths while an inline HTML page tool is available | Show the report inline first (the gated `report.html` or its inline edition), then the paths and the summary |
+| A visual reproduction checked only against itself | Reference compare: real source frames next to renders at the same moments, a critique per pair, fixes, and the pairs in the report |
+| Delivering without the closing briefing | Paste the block `outcome briefing` prints; never compute or recall its numbers by hand |

package/packages/omo-codex/plugin/skills/ulw-research/scripts/claim-kinds.mjs

@@ -4,7 +4,7 @@
 // emits it as the data-claim-kind attribute on the element it renders, and the fidelity gate
 // (fidelity-claims.mjs) checks that every marked element cites a row the kind accepts. An
 // unmarked element is a plain fact or quote and is checked only by the value and quote checks.
-// The one lexical piece is `unmarked`: a few first-person run patterns that raise a minor warning
+// The one lexical piece is `unmarked`: first-person run and check patterns that raise a minor warning
 // when a sentence reads like the kind but carries no mark; they never raise a major. The table in
 // references/fidelity-claim-kinds.md is rendered from CLAIM_KINDS (`node claim-kinds.mjs --render-doc`).
 // Pure; node: builtins only. Korean phrases are \u escapes.
@@ -23,6 +23,32 @@ const KO_WE_RUN = "(?:\uC2E4\uD589(?:\uD588|\uD558\uC5EC|\uD574)|\uB3CC[\uB838\u
 const KO_RUN = "(?:\uC2E4\uD589|\uB3CC[\uB9AC\uB824\uB838\uB9B0\uB9B4])"
 const KO_SUBJECT = "[^\\s,.;!?]*(?:[\uAC00\uC774\uC740\uB294]|\uAED8\uC11C)(?=\\s)"
 
+// One cue list; the rule text below is built from it. A check verb fires only in first person with
+// a result object (a result noun or a digit) in its clause, and never when that clause names a document.
+// The document guard ends where the clause ends (a comma, a semicolon, "as", a relative pronoun; in Korean a
+// comma, a semicolon, cheoreom, deusi, gati), and a document word followed by a number is a locator ("page 3").
+const EN_RUN = ["ran", "run", "re-ran", "re-run", "executed", "verified by"]
+const EN_CHECK = ["verified", "checked", "confirmed", "measured", "tested", "double-checked", "cross-checked"]
+const EN_RESULT = ["value", "values", "result", "results", "figure", "figures", "number", "numbers", "total", "totals", "count", "counts", "measurement", "measurements", "reading", "readings", "output", "outputs", "span", "length", "rate", "rates", "latency", "throughput", "score", "scores", "benchmark", "benchmarks", "metric", "metrics", "data", "estimate", "estimates", "speed", "duration", "size", "ratio", "growth"]
+const EN_DOCUMENT = ["report", "reports", "filing", "filings", "document", "documents", "paper", "papers", "article", "articles", "publication", "statement", "disclosure", "website", "page", "docs", "documentation"]
+// KO_CHECK = hwagin, geomjeung, cheukjeong, jeomgeom (+ haet/hayeot/hayeo/hae/han); KO_RESULT = gyeolgwa, gap, suchi, gil-i, hapgye,
+// biyul, sokdo, jipyo, deiteo; KO_DOCUMENT = bogoseo, munseo, gongsi, nonmun, gisa, jaryo.
+const KO_CHECK = ["\uD655\uC778", "\uAC80\uC99D", "\uCE21\uC815", "\uC810\uAC80"]
+const KO_DONE = ["\uD588", "\uD558\uC600", "\uD558\uC5EC", "\uD574", "\uD55C"]
+const KO_RESULT = ["\uACB0\uACFC", "\uAC12", "\uC218\uCE58", "\uAE38\uC774", "\uD569\uACC4", "\uBE44\uC728", "\uC18D\uB3C4", "\uC9C0\uD45C", "\uB370\uC774\uD130"]
+const KO_DOCUMENT = ["\uBCF4\uACE0\uC11C", "\uBB38\uC11C", "\uACF5\uC2DC", "\uB17C\uBB38", "\uAE30\uC0AC", "\uC790\uB8CC"]
+const any = (words) => words.map((word) => word.replace(/-/g, "-?").replace(/ /g, "\\s+")).join("|")
+const EN_WE = "(?:\\b[Ww]e|\\bI)(?:'ve|'d)?\\s+(?:(?:have|had|also|then|first|just|again|independently|manually|directly)\\s+)*"
+const CLAUSE = "(?:[^.;!?]|\\.\\d)*?"
+const EN_BOUNDARY = ["as", "which", "who", "whom", "whose", "where"]
+const KO_BOUNDARY = ["\uCC98\uB7FC", "\uB4EF\uC774", "\uAC19\uC774"]
+const EN_GUARD = `(?:(?!,|\\b(?:${any(EN_BOUNDARY)})\\b)[^.;!?]|\\.\\d)*?`
+const KO_GUARD = `(?:(?!,|${any(KO_BOUNDARY)})[^.;!?]|\\.\\d)*?`
+const EN_OBJECT = `(?=${CLAUSE}(?:\\b(?:${any(EN_RESULT)})\\b|\\d))(?!${EN_GUARD}\\b(?:${any(EN_DOCUMENT)})\\b(?!\\s*\\d))`
+const KO_OBJECT = `(?=${CLAUSE}(?:${any(KO_RESULT)}|\\d))(?!${KO_GUARD}(?:${any(KO_DOCUMENT)})(?!\\s*\\d))`
+const escapes = (text) => text.replace(/[^\x00-\x7f]/g, (c) => `\\u${c.charCodeAt(0).toString(16).toUpperCase().padStart(4, "0")}`)
+const list = (words) => escapes(words.join(", "))
+
 /**
  * kind -> { prefix: placeholder prefix (null = the default form), placeholder: an example,
  *   rowKinds: ledger row kinds it accepts, sourceKinds: meta.json kinds the snapshot holding the
@@ -50,11 +76,13 @@ export const CLAIM_KINDS = Object.freeze({
 		note: "The cited row itself must have a locator artifact inside `sources/<source_id>/` whose `meta.json` kind is `command`; another command row in the sentence never counts. Whether it is the right command for the claim is the meaning verifier's call (minor), not this gate's.",
 		unmarked: {
 			code: "verification_claim_unmarked",
-			rule: "a first-person run: we/I (have) ran, run, executed, verified by (not ran out of); by running after we/our in the sentence; \\uC6B0\\uB9AC or \\uC800\\uD76C as the subject word (+ \\uAC00, \\uB294, or \\uD300\\uC774), then \\uC2E4\\uD589\\uD588, \\uC2E4\\uD589\\uD558\\uC5EC, \\uC2E4\\uD589\\uD574, \\uB3CC\\uB838, or \\uB3CC\\uB824 in the clause with no third-party subject between; \\uC9C1\\uC811 with no third-party subject, then \\uC2E4\\uD589 or \\uB3CC\\uB9AC in the clause",
+			rule: `a first-person run: we/I (have) ${list(EN_RUN)} (not ran out of), or by running after we/our in the sentence; a first-person check: we/I (have) ${list(EN_CHECK)} (not checked in/into/out) with a result object in the clause (${list(EN_RESULT)}, or a digit) and no document named before the clause ends (${list(EN_DOCUMENT)}; a comma, a semicolon, or ${list(EN_BOUNDARY)} ends it, and a document word followed by a number is a locator); \\uC6B0\\uB9AC or \\uC800\\uD76C as the subject word (+ \\uAC00, \\uB294, or \\uD300\\uC774), then \\uC2E4\\uD589\\uD588, \\uC2E4\\uD589\\uD558\\uC5EC, \\uC2E4\\uD589\\uD574, \\uB3CC\\uB838, or \\uB3CC\\uB824 in the clause with no third-party subject between, or then ${list(KO_CHECK)} + ${list(KO_DONE)} with a result object in the clause (${list(KO_RESULT)}, or a digit) and no document before the clause ends (${list(KO_DOCUMENT)}; a comma, a semicolon, or ${list(KO_BOUNDARY)} ends it, and a document word followed by a number is a locator); \\uC9C1\\uC811 with no third-party subject, then \\uC2E4\\uD589 or \\uB3CC\\uB9AC in the clause; a curly apostrophe reads as a straight one`,
 			patterns: [
-				/\b(?:we|I)(?:'ve)?\s+(?:(?:have|had|also|then|first|just|again)\s+)*(?:ran|run|re-?ran|re-?run|executed|verified\s+by)\b(?!\s+out\s+of\b)/i,
-				/\b(?:we|our)\b.*?\bby\s+running\b/i,
+				new RegExp(`${EN_WE}(?:${any(EN_RUN)})\\b(?!\\s+out\\s+of\\b)`),
+				new RegExp(`${EN_WE}(?:${any(EN_CHECK)})\\b(?!\\s+(?:in|into|out)\\b)${EN_OBJECT}`),
+				/\b(?:[Ww]e|[Oo]ur)\b.*?\bby\s+running\b/,
 				new RegExp(`${KO_WE_SUBJECT}(?:(?!${KO_SUBJECT})[^\\s,.;!?]+\\s+)*?${KO_WE_RUN}`),
+				new RegExp(`${KO_WE_SUBJECT}${KO_OBJECT}(?:(?!${KO_SUBJECT})[^\\s,.;!?]+\\s+)*?(?:${any(KO_CHECK)})(?:${any(KO_DONE)})`),
 				new RegExp(`(?:^|[,.;!?])\\s*(?:(?!${KO_SUBJECT})[^\\s,.;!?]+\\s+)*\uC9C1\uC811\\s*(?:[^\\s,.;!?]+\\s*)??${KO_RUN}`),
 			],
 		},
@@ -86,7 +114,7 @@ export function unmarkedRequires(row) {
 /** The kinds whose unmarked patterns match the sentence. */
 export function unmarkedKinds(text) {
 	return Object.entries(CLAIM_KINDS)
-		.filter(([, entry]) => entry.unmarked?.patterns.some((re) => re.test(text)))
+		.filter(([, entry]) => entry.unmarked?.patterns.some((re) => re.test(String(text).replace(/\u2019/g, "'"))))
 		.map(([name]) => name)
 }
 

package/packages/omo-codex/plugin/skills/ulw-research/scripts/contracts.mjs

@@ -31,7 +31,7 @@ export const DEFECT_CODES = deepFreeze({
 	en_dash_in_prose: { severity: "minor", integrity: false, hint: "Use a hyphen for compounds; keep the en dash only between numbers." },
 	heading_too_long: { severity: "minor", integrity: false, hint: "Shorten the heading to one line (26 Korean characters or 12 English words)." },
 	heading_generic: { severity: "minor", integrity: false, hint: "Make the heading carry the section's claim instead of a generic label." },
-	unsourced_number: { severity: "major", integrity: false, hint: "Tag the number MEASURED / ASSUMED / DERIVED or cite its source in the same paragraph." },
+	unsourced_number: { severity: "major", integrity: false, hint: "Tag the number MEASURED / ASSUMED / DERIVED or cite its source in the same paragraph; under Sources only citation-field numbers and names (a hash, #12, GPT-4, Q3, 10-Q) are exempt, so quote a reference title that carries a code or a figure, or wrap it in <cite>: an unquoted one is checked, and 10-K / 8-K stay numerals because K is a multiplier. A slashed N/M reads as a month/day, so a fraction or a score (2/3, 7/12) under Sources is not checked (#11386): write it as a percent or tag it." },
 	chart_without_figure: { severity: "major", integrity: false, hint: "Wrap the image or chart in a figure container with a caption." },
 	chart_svg_no_text: { severity: "major", integrity: false, hint: "Give the chart a title, axis labels, units, and value labels as SVG text." },
 	chart_svg_missing_labels: { severity: "minor", integrity: false, hint: "Add the missing title, axis ticks, or unit label to the chart." },
@@ -58,12 +58,14 @@ export const DEFECT_CODES = deepFreeze({
 	pointer_label_mismatch: { severity: "blocker", integrity: true, hint: "The placeholder's label is not the row's label; check the id, then the label." },
 	quote_anchor_unresolved: { severity: "blocker", integrity: true, hint: "A from or to anchor is missing or occurs more than once in the quote; pick unique anchor words." },
 	unit_duplicated: { severity: "blocker", integrity: true, hint: "The unit after the placeholder repeats the row's unit; use :value or remove the unit." },
+	conversion_altered_number: { severity: "blocker", integrity: true, hint: "The source conversion altered this number (sources/S<n>/conversion.json lists it); snapshot a verified text of the source over it with `source snapshot --id S<n> --text <verified.txt> --replace` (refused while an active row cites S<n>; then retract those rows or use a new id), then re-add the fact." },
 	polish_diff_fail: { severity: "blocker", integrity: true, hint: "The polish pass changed numbers, tickers, table rows, claim ids, markup, or hidden content; revert to the pre-polish artifact and apply the language fixes only." },
 	fidelity_interview_incomplete: { severity: "blocker", integrity: true, hint: "A fidelity field is pending or missing in brief.md; answer the open question or record its default, then rerun." },
 	derived_not_allowed: { severity: "blocker", integrity: true, hint: "The brief allows no derived or assumed numbers; cite a source row directly or change the brief with the requester." },
 	source_outside_floor: { severity: "blocker", integrity: true, hint: "The row's source is outside the source floor the brief records; use a source the brief allows or remove the fact." },
 	brief_changed: { severity: "blocker", integrity: true, hint: "The edition's fill map was not written under the current ## Fidelity block (it changed after fill, or the map predates the binding); re-run fact fill, then fact verify." },
-	snapshot_hash_mismatch: { severity: "blocker", integrity: true, hint: "The source snapshot changed after it was pinned; re-snapshot the source and re-register its rows." },
+	snapshot_hash_mismatch: { severity: "blocker", integrity: true, hint: "The source snapshot changed after it was pinned or cannot be read; restore it, or run source refresh to re-anchor its rows." },
+	snapshot_not_raw: { severity: "blocker", integrity: true, hint: "Measured rows need the original bytes: snapshot the source with --url, --html, or --raw <bytes> --text <extraction>; converted-only sources support warned quotations only." },
 	chart_value_untraceable: { severity: "blocker", integrity: true, hint: "A chart value, tick, or data file does not match its ledger-backed data source; re-render the chart from the exported data." },
 	quote_unbound: { severity: "blocker", integrity: true, hint: "A direct quotation has no ledger quote row; register it with fact quote and insert it through its placeholder." },
 	edition_unverified: { severity: "blocker", integrity: true, hint: "The delivered file has no matching fact verify record; run fact verify on that edition and record it with --verified-by." },
@@ -82,13 +84,18 @@ export const DEFECT_CODES = deepFreeze({
 	polish_hedging_decreased: { severity: "major", integrity: false, hint: "The polish pass removed qualifiers; restore them." },
 	polish_text_churn: { severity: "major", integrity: false, hint: "More than half the text changed during polish; the pass composed instead of proofreading." },
 	snapshot_mismatch: { severity: "major", integrity: false, hint: "The operands come from different collection windows; use rows from one window or state the windows separately." },
+	operand_timestamp_missing: { severity: "major", integrity: false, hint: "Record the operand's observed_at timestamp before checking its collection window." },
+	operand_missing: { severity: "major", integrity: false, hint: "A windowed operation names an operand that is not in the fact ledger; restore the row or re-derive from existing rows." },
 	join_integrity_fail: { severity: "major", integrity: false, hint: "A table row combines facts from different source observations; keep each row to one observation or split it." },
 	quote_stale_version: { severity: "major", integrity: false, hint: "The quoted source revision differs from the revision the report targets; re-snapshot the target revision or date the quote." },
 	quote_fuzzy_match: { severity: "minor", integrity: false, hint: "The source text is OCR-noisy; confirm the snapshot and prefer the raw document." },
+	relocated: { severity: "minor", integrity: false, hint: "After source refresh the exact quote and context sit at one new byte range; review the old and new offsets (verification never rewrites the row)." },
+	quote_converted_only: { severity: "minor", integrity: false, hint: "The quote comes from a converted-only source with no raw bytes; re-snapshot the original with --raw and --text, or keep the warned quote." },
 	name_outside_quote: { severity: "minor", integrity: false, hint: "A name in the sentence is not in the cited quote; add the source or remove the name." },
 	verification_claim_unmarked: { severity: "minor", integrity: false, hint: "The sentence reads as a check we ran but carries no verified mark; mark it {{V:F<n>:label}} on a recorded command-output row, or reword it." },
 	scope_widened_warn: { severity: "minor", integrity: false, hint: "The sentence uses a broader scope word than the quote; confirm the scope or narrow it." },
 	particle_unverified: { severity: "minor", integrity: false, hint: "The particle after this value could not be checked; read it aloud and fix it by hand if wrong." },
+	unit_unbound: { severity: "minor", integrity: false, hint: "Use the full fact placeholder, or write the unit right after :value (with or without a space; a Korean particle may follow), as its :unit element, or in the matching table column header." },
 	fallback_without_entity: { severity: "minor", integrity: false, hint: "The replacement quote has no entity for attribution; set one on the row." },
 	meaning_drift_warn: { severity: "minor", integrity: false, hint: "A separate reader found the sentence changes scope, subject, or certainty; rewrite from the quote." },
 	polish_edit_unlocated: { severity: "minor", integrity: false, hint: "The edit's before-text was not found exactly once; re-propose against the current text." },
@@ -98,15 +105,17 @@ export const DEFECT_CODES = deepFreeze({
 
 /** Codes only the fidelity verifier raises (fact ledger, fill, verify, meaning, polish). */
 export const FIDELITY_CODES = Object.freeze([
+	"unit_unbound",
 	"unresolved_placeholder", "value_mismatch", "unit_mismatch", "quote_not_verbatim", "claim_status_invalid",
 	"incompatible_operands", "pointer_label_mismatch", "quote_anchor_unresolved", "unit_duplicated", "polish_diff_fail",
 	"fidelity_interview_incomplete", "derived_not_allowed", "source_outside_floor", "brief_changed", "snapshot_hash_mismatch",
+	"conversion_altered_number",
 	"chart_value_untraceable", "quote_unbound", "edition_unverified", "untraceable_number", "unsupported_verification", "claim_kind_mismatch",
 	"conflicting_readings", "hedge_dropped", "direction_flipped", "causality_added", "qualifier_dropped", "unbound_sentence",
 	"meaning_drift_confirmed", "meaning_unverified", "polish_edit_rejected", "polish_hedging_decreased", "polish_text_churn", "snapshot_mismatch",
-	"join_integrity_fail", "quote_stale_version", "quote_fuzzy_match", "name_outside_quote", "verification_claim_unmarked", "scope_widened_warn",
+	"operand_timestamp_missing", "operand_missing", "join_integrity_fail", "quote_stale_version", "quote_fuzzy_match", "name_outside_quote", "verification_claim_unmarked", "scope_widened_warn",
 	"particle_unverified", "fallback_without_entity", "meaning_drift_warn", "polish_edit_unlocated", "polish_link_changed",
-	"polish_name_changed",
+	"polish_name_changed", "snapshot_not_raw", "relocated", "quote_converted_only",
 ])
 const FIDELITY_SET = new Set(FIDELITY_CODES)
 

package/packages/omo-codex/plugin/skills/ulw-research/scripts/conversion-probe.mjs

@@ -0,0 +1,229 @@
+// conversion-probe.mjs - numeric multiset differences after source conversion.
+// Inline markup may split one number; superscripts must not merge into it. Numbers are read with the
+// one grouped-number grammar (number-grammar.mjs). A cell, row, list-item, block or line break is a
+// hard boundary: a space never groups digits across it. A sign counts when no Latin, Greek or
+// Cyrillic letter, digit or underscore touches it (an en dash before a digit is a minus too), and a
+// "-" or U+2212 counts after ONE such letter ("\u0394\u22125") or after an exponent ("1e-5"); a run of
+// two or more ("cite_note-7", "AB-12", "UTF-8", "2019-2023") is an identifier or range and carries no
+// negative. Markdown markup a converter writes beside a sign never decides it: the closer run before it
+// (read with the CommonMark inline grammar in markdown-closers.mjs: emphasis, code, strike, titled and
+// parenthesised link and image targets, reference endings, backslash-escaped closers) is skipped to
+// the first character it closes ("**2019**-2023" and '[1990](/wiki/1990 "1990")-2000' are ranges), and
+// an opener run between the sign and its digits ("-**5**") keeps the sign. A raw-side character
+// reference (`−`, ` `) is decoded on BOTH sides with the one entities.mjs table, so a
+// verbatim copy probes clean. Raw inline HTML a converter leaves in the Markdown (pandoc's gfm writer
+// emits `<sup>`, `<sub>`, `<img ... width=320>`, `<span>`, `<br>`) is read with the html-lite tag
+// reader: sup, sub and break tags are boundaries, as on the raw side, and any other tag or comment is
+// dropped with its attributes, so it is a closer or opener like Markdown markup. Digits a converter
+// writes out from a link target, an image source, alt text, or a title cover a converted number only
+// inside Markdown link or image syntax (`](...)` with any title and balanced parentheses,
+// `![...](...)`, `<https://...>`, `[ref]: url "title"` with the title on the same or the next line),
+// never in prose; a footnote
+// definition's text ("[^1]: ...") is prose. Converter syntax is not a number: a line-start list marker
+// ("1. ", "2) ", "(3) ", indented or after "> ") is exempt only while the source has an <ol> item with
+// that number and the markers so far do not outnumber the <ol> items (so lazy "1. 1. 1." numbering
+// over three items passes), a numeric reference label ("[x][1]" with its "[1]: url") only while both
+// halves are present, and a footnote definition label ("[^1]:") only while its "[^1]" is used;
+// otherwise each is a candidate. A converter's hard wrap may split a space-grouped number over one line
+// break; the two halves read as one number only when the raw side holds that number and not both
+// halves, so raw "12 345 and 12 and 345" wrapped inside "12 345" reports both halves (it fails safe).
+// The built-in snapshot writes no attribute text, syntax, wraps or tags, and its text is already
+// decoded, so it gets none of these allowances. Constructs outside this grammar are declared out of
+// scope in references/fact-ledger.md: they report as altered (the safe direction).
+
+import { existsSync, readFileSync } from "node:fs"
+import { join } from "node:path"
+
+import { CliError } from "./cli-support.mjs"
+import { decodeEntities, parseHtml, PROSE_BREAKS, readTag } from "./html-lite.mjs"
+import { escapedAt, inlineLinkSpans, stripClosers } from "./markdown-closers.mjs"
+import { NUMBER_SOURCE, plainDecimal } from "./number-grammar.mjs"
+
+const LETTER = "\\p{Script=Latin}\\p{Script=Greek}\\p{Script=Cyrillic}"
+const WORD = new RegExp(`[${LETTER}\\p{N}_]`, "u")
+const ALPHA = new RegExp(`[${LETTER}]`, "u")
+const EXPONENT = new RegExp(`(?<![${LETTER}\\p{N}_.,])\\d+(?:[.,]\\d+)?[eE]$`, "u")
+/** A dash, then any Markdown opener run, then a number; signed() decides whether the dash is a minus. */
+const NUMBERS = new RegExp(`(?:([-\\u2212\\u2013])([*_~\`[]*))?(${NUMBER_SOURCE})`, "gu")
+/** Sup, sub, datetime chunks and every break join no neighbour: neither a digit, a separator, nor whitespace. */
+const BOUNDARY = "\u0001"
+const CARRIED = ["href", "src", "alt", "title"]
+const LIST_MARKER = /(?:^|\u0001)(?: ?>)* ?(?:\d+[.)]|\(\d+\)) /gu
+const REF_USE = /\]\[(\d+)\]/gu
+const REF_DEF = /(?:^|\u0001)(?: ?>)* ?\[(\d+)\]:/gu
+const NOTE_USE = /\[\^(\d+)\](?!:)/gu
+const NOTE_DEF = /(?:^|\u0001)(?: ?>)* ?\[\^(\d+)\]:/gu
+const TITLE = `(?:"[^"\u0001]*"|'[^'\u0001]*'|\\([^)\u0001]*\\))`
+/** A reference definition's title may sit alone on the next line (only whitespace after it). */
+const REF_DEFINITION = `(?:^|\u0001) ?\\[(?!\\^)[^\\]\u0001]+\\]: ?[^\\s\u0001]*(?: ${TITLE}| ?\u0001 ?${TITLE}(?= ?(?:\u0001|$)))?`
+const LINK_SYNTAX = new RegExp(`!\\[[^\\]\u0001]*\\](?:\\([^)\u0001]*\\))?|\\]\\([^)\u0001]*\\)|<https?:\\/\\/[^>\\s\u0001]+>|${REF_DEFINITION}`, "gu")
+const TAG_NAME = /^[a-z][a-z0-9-]*$/u
+/** What a tag leaves in the numeric text on either side: sup, sub and every break join no neighbour. */
+const edgeOf = (tag) => (tag === "sup" || tag === "sub" || PROSE_BREAKS.has(tag) ? BOUNDARY : "")
+
+/** Prose text, attribute text and <ol> item numbers of an HTML document; a non-HTML document is prose only. */
+function numericText(source, html) {
+	if (!html) return { prose: decodeEntities(source), carried: "", listed: [] }
+	const chunks = []
+	const carried = []
+	const listed = []
+	const walk = (node) => {
+		if (node.tag === "#text") { chunks.push(node.text.replace(/\s+/gu, " ")); return }
+		for (const name of CARRIED) if (node.attrs?.[name] !== undefined) carried.push(node.attrs[name])
+		if (node.tag === "ol") {
+			const start = /^\s*\d+\s*$/u.test(node.attrs.start ?? "") ? Number(node.attrs.start) : 1
+			node.children.filter((child) => child.tag === "li").forEach((_, index) => listed.push(String(start + index)))
+		}
+		const edge = edgeOf(node.tag)
+		chunks.push(edge)
+		if (node.tag === "time" && node.attrs.datetime !== undefined) chunks.push(`${BOUNDARY}${node.attrs.datetime}${BOUNDARY}`)
+		for (const child of node.children) walk(child)
+		chunks.push(edge)
+	}
+	walk(parseHtml(source))
+	return { prose: chunks.join(""), carried: carried.join(BOUNDARY), listed }
+}
+
+/** Raw inline HTML in converted Markdown: sup, sub and break tags become boundaries, other tags and comments go. */
+function inlineHtml(text) {
+	let out = ""
+	let kept = 0
+	for (let lt = text.indexOf("<"); lt !== -1; lt = text.indexOf("<", Math.max(lt + 1, kept))) {
+		const comment = text.startsWith("<!--", lt)
+		const name = lt + (text[lt + 1] === "/" ? 2 : 1)
+		if (escapedAt(text, lt) || (!comment && !/[a-z]/iu.test(text[name] ?? ""))) continue
+		const tag = comment ? null : readTag(text, name)
+		const end = comment ? text.indexOf("-->", lt + 4) + 3 : tag.end
+		if (end <= lt || (tag && !(TAG_NAME.test(tag.name) && text[end - 1] === ">")) || /\n[ \t\r]*\n/u.test(text.slice(lt, end))) continue
+		out += text.slice(kept, lt) + (tag ? edgeOf(tag.name) : "")
+		kept = end
+	}
+	return out + text.slice(kept)
+}
+
+/**
+ * Start offsets of numeric reference labels whose use ("][1]") and definition ("[1]:") are both present,
+ * and of footnote definition labels ("[^1]:") whose "[^1]" is used (the use itself mirrors the source).
+ */
+function referenceLabels(flat) {
+	const uses = [...flat.matchAll(REF_USE)]
+	const defs = [...flat.matchAll(REF_DEF)]
+	const used = new Set(uses.map((match) => match[1]))
+	const defined = new Set(defs.map((match) => match[1]))
+	const paired = [...uses.filter((match) => defined.has(match[1])), ...defs.filter((match) => used.has(match[1]))]
+	const noted = new Set([...flat.matchAll(NOTE_USE)].map((match) => match[1]))
+	const notes = [...flat.matchAll(NOTE_DEF)].filter((match) => noted.has(match[1]))
+	return new Set([...paired, ...notes].map((match) => match.index + match[0].search(/\d/u)))
+}
+
+/** Whether the dash at `at` is a minus: Markdown closers (and a backslash escaping the dash) before it are skipped, as is an opener run before a lone letter. */
+function signed(flat, at, dash) {
+	const before = stripClosers(flat.slice(Math.max(0, at - 512), escapedAt(flat, at) ? at - 1 : at))
+	const last = before.at(-1) ?? ""
+	if (!WORD.test(last)) return true
+	if (dash === "\u2013") return false
+	if (EXPONENT.test(before)) return true
+	return ALPHA.test(last) && !WORD.test(before.slice(0, -1).replace(/[*_~`[]+$/u, "").at(-1) ?? "")
+}
+
+/**
+ * Line breaks become boundaries, then whitespace runs one space; `linked` marks tokens inside link
+ * syntax, `marker` a line-start list marker, `label` a paired reference label, `gap` the text since
+ * the previous number.
+ */
+function tokensOf(text, { links = false } = {}) {
+	const flat = text.replace(/\r\n?|\n|\f/gu, BOUNDARY).replace(/\s+/gu, " ")
+	const spans = links ? [...[...flat.matchAll(LINK_SYNTAX)].map((match) => [match.index, match.index + match[0].length]), ...inlineLinkSpans(flat)] : []
+	const markers = new Set([...flat.matchAll(LIST_MARKER)].map((match) => match.index + match[0].search(/\d/u)))
+	const labels = referenceLabels(flat)
+	let previousEnd = 0
+	return [...flat.matchAll(NUMBERS)].map((match, n) => {
+		const sign = match[1] !== undefined && signed(flat, match.index, match[1])
+		const at = sign ? match.index : match.index + match[0].length - match[3].length
+		const gap = flat.slice(previousEnd, at)
+		previousEnd = match.index + match[0].length
+		return {
+			n,
+			gap,
+			text: sign ? match[0] : match[3],
+			key: `${sign ? "-" : ""}${plainDecimal(match[3])}`,
+			linked: spans.some(([start, end]) => at >= start && at < end),
+			marker: markers.has(at),
+			label: labels.has(at),
+		}
+	})
+}
+
+/** The comparison key of one number token, so other tools can match a converted number against a probe record. */
+export const numberKey = (token) => tokensOf(String(token))[0]?.key ?? null
+
+/** The numbers in `texts` that the source's conversion probe recorded as altered (sources/<id>/conversion.json). */
+export function alteredIn(sourceDir, texts) {
+	const record = join(sourceDir, "conversion.json")
+	let altered = []
+	if (existsSync(record)) {
+		try {
+			altered = JSON.parse(readFileSync(record, "utf8")).conversion_altered_numbers ?? []
+		} catch (error) {
+			throw new CliError(`${record} is not a readable conversion record (${error.message}); snapshot the source again`)
+		}
+	}
+	const keys = new Set(altered.map(numberKey))
+	return texts.flatMap((text) => tokensOf(String(text ?? ""))).filter((token) => keys.has(token.key)).map((token) => token.text)
+}
+
+/** Rejoin a grouped number a hard wrap split over one line break, when only the joined reading is in the raw counts. */
+function joinWrapped(tokens, raw) {
+	const out = []
+	for (const token of tokens) {
+		const prev = out.at(-1)
+		const wrapped = prev?.n === token.n - 1 && /^ ?\u0001 ?$/u.test(token.gap) && !prev.linked && !token.linked
+		const [joined, ...more] = wrapped ? tokensOf(`${prev.text} ${token.text}`) : []
+		if (joined && more.length === 0 && raw.has(joined.key) && !(raw.has(prev.key) && raw.has(token.key))) {
+			out[out.length - 1] = { ...joined, n: token.n, text: `${prev.text}\n${token.text}` }
+			continue
+		}
+		out.push(token)
+	}
+	return out
+}
+
+const countOf = (tokens) => tokens.reduce((counts, token) => counts.set(token.key, (counts.get(token.key) ?? 0) + 1), new Map())
+
+/** The tokens `counts` cannot cover; each covered token uses up one count. */
+function uncovered(tokens, counts) {
+	return tokens.filter((token) => {
+		const remaining = counts.get(token.key) ?? 0
+		if (remaining === 0) return true
+		counts.set(token.key, remaining - 1)
+		return false
+	})
+}
+
+/**
+ * Preserve token spelling in diagnostics; the raw side is read as HTML, the converted side as Markdown
+ * with raw inline tags, both decoded with one entity table. `attributes: false`
+ * (the built-in snapshot) gives attribute digits no cover, converter syntax no exemption and no wrap rejoin. An exempt
+ * marker or label is not a converted number at all: it neither covers a raw number nor counts as altered.
+ */
+export function probeConversion(raw, converted, { attributes = true } = {}) {
+	const source = numericText(raw, /<[a-z!/]/i.test(raw))
+	const before = tokensOf(source.prose)
+	const listed = new Set(source.listed)
+	let items = attributes ? source.listed.length : 0
+	const syntax = tokensOf(attributes ? decodeEntities(inlineHtml(converted)) : converted, { links: attributes })
+		.filter((token) => !(attributes && token.label) && !(token.marker && listed.has(token.key) && items-- > 0))
+	const after = attributes ? joinWrapped(syntax, countOf(before)) : syntax
+	const counts = countOf(before)
+	const extra = new Set(uncovered(after.filter((token) => !token.linked), counts))
+	for (const token of tokensOf(source.carried)) counts.set(token.key, (counts.get(token.key) ?? 0) + 1)
+	for (const token of uncovered(after.filter((token) => token.linked), counts)) extra.add(token)
+	const lost = uncovered(before, countOf(after)).map((token) => token.text)
+	return { altered: after.filter((token) => extra.has(token)).map((token) => token.text), lost, raw_count: before.length, converted_count: after.length }
+}
+
+export function runConversionProbe(argv) {
+	if (argv.length !== 2) throw new CliError("usage: source probe-conversion <raw-file> <converted-file>")
+	const result = probeConversion(readFileSync(argv[0], "utf8"), readFileSync(argv[1], "utf8"))
+	return { json: result, summary: `source probe-conversion: ${result.altered.length} altered, ${result.lost.length} lost`, exitCode: result.altered.length === 0 ? 0 : 1 }
+}