hyperresearch 0.8.7__tar.gz → 0.9.0__tar.gz

{hyperresearch-0.8.7 → hyperresearch-0.9.0}/CHANGELOG.md

@@ -1,5 +1,108 @@
 # Changelog
 
+## [Unreleased]
+
+## [0.9.0] - 2026-07-23
+
+### Coverage before elegance: the synthesizer stops trading substance for prose
+
+A rerun of a comparison query scored below its own older baseline on comprehensiveness and insight, and reading both reports side by side showed why: same word count, spent differently. The humanization work (primers, calm citations, one committed thesis) had quietly taught the synthesizer to dissolve a systematic ten-axis comparison into an elegant narrative and to compress a fully developed quantitative mechanism down to a one-line mention. Cleaner to read, and worse on exactly the axes a "compare X, Y, Z" prompt is graded on. The prompts were conflating two different edits: cutting prose, which is right, and cutting substance, which is not.
+
+- **The synthesizer now treats coverage and mechanism depth as content, not optional structure.** On a compare or survey task, every decision-relevant dimension the corpus names gets explicit coverage, and a mechanism the sources develop (a named decomposition, a formula with its terms, a causal chain with its numbers) gets developed in the report rather than gestured at. The rule it now follows: elegance is spent on the words between points, never on the number of points.
+- **"Selectivity" is now defined precisely.** The anti-sprawl guidance used to read as license to drop points; it now says selectivity means choosing which sources to cite for a point, not which points to make. Dropping a comparison axis or compressing a mechanism is a coverage gap, not economy.
+- **The length pass cuts prose, never points.** When trimming to the word ceiling, redundancy and filler go first; a comparison dimension, a developed mechanism, a counterargument, and a load-bearing primary source are off limits. If prose cuts do not get under the ceiling, the report has too many words per point, not too many points.
+- **Two adversarial backstops.** The depth critic now flags a developed quantitative mechanism compressed to a bare mention as the highest-insight loss a draft can take. The instruction critic gains a comparison-axis coverage check (register-independent, since a comparison prompt needs its axes covered whatever the register) that names dropped or compressed dimensions.
+
+### Run levers: auto-selected register, domain notes, and inference depth
+
+The pipeline had one hard-coded voice (evaluative-argumentative), so a "teach me X" or "survey the landscape" query got an opinionated verdict report. The Q62 register experiment showed the judge prices register heavily (+2.5 RACE from register alone), so register is now a run-time lever instead of a constant.
+
+- **Three levers, auto-selected in step 1** and written into the decomposition: `register` (`teach` / `survey` / `analyze` / `advocate`, classified from the query's verb shape, defaulting to `analyze` — today's behavior — unless the signal is strong; explicit user directives always win), freeform `domain_notes` (sourcing strategy, evidence norms, recency window), and `inference_depth` (`surface` / `standard` / `deep` — the rabbithole dial; step 4 may upgrade it after seeing the actual corpus via `hpr levers set <tag> inference_depth=deep --rerender`).
+- **`hpr levers render <tag>`** materializes the levers into four role-scoped shim files (`shims/{research,drafting,critics,polish}.md`) that spawn templates paste VERBATIM into subagent prompts — the orchestrator never composes posture text. Shims compose additively (register block + domain block + depth block), and division of labor is strict: profiles own every number, levers own posture only (drift-proofed by a test that rejects numeric budget ranges in shim text).
+- **The critics and polish auditor are register-aware**, so the pipeline can't undo its own mode: in survey/teach register the dialectic critic flags unfair representation instead of missing commitment, the instruction critic stops demanding rankings the prompt never asked for, and the polish auditor's hedge-striking stands down. In advocate register all three tighten instead. The cite-checker and ship gate receive NO shim — verification never softens by mode.
+- **Graceful degradation everywhere:** agents proceed with today's defaults when no directives block arrives, lever-less runs skip the new `levers-rendered` verify check, and `run status` surfaces the chosen levers so a misclassification is visible before hours of pipeline run on it.
+
+### Ship gate enforcement: `run finish` (lessons from the first premier run)
+
+The first end-to-end premier benchmark run exposed both prose-only gates failing exactly the way prose gates fail: the orchestrator never invoked `run verify` (a 25,647-word report shipped against a 16K ceiling), and when lint flagged 24 hallucinated/mangled quotes it wrote itself a "false positives" memo and shipped anyway.
+
+- **`hpr run finish <tag>`** is the new terminal gate and the ONLY path to manifest status `done`. It runs the full verification battery and flips the run to `done` on pass or `blocked (verify)` on fail, recording the verdict in the manifest either way. The router's final gate now centers on it, with explicit no-override language (gate errors are fixed by changing the report, never re-interpreted), a bounded 3-round fix loop, and a new invariant: a run is complete only when `finish` reports `passed: true`.
+- **`verify_run` now includes the blocking content lints** (`quote-integrity`, `retracted-citations`) in-process, so one command carries the whole verdict and there is no seam where a failing rule can be run separately and argued with.
+- **Length enforcement moved upstream too.** Step 11 gains a mechanical word-count gate with the one permitted fix (a single synthesizer compression respawn, cheapest at that moment); the synthesizer's word-target table is now rendered from the profile (it was hardcoded to full-gear numbers, so premier's 8-16K target never reached the prompt) with the high end stated as a hard ceiling; the polish auditor strips quotation marks from non-verbatim rhetorical framing before the gate ever sees them.
+
+### Report register: pedagogy primers, calm citations, and de-AI'd prose
+
+- **Pedagogy primers.** A four-report comparison on bench Q62 (benchmark reference vs. two pipeline generations) showed the judge's only consistent losses were pedagogy and audience adaptation, never coverage or insight: the reference teaches each concept before judging it; our reports open expert-dense. Every major body section now starts with a 3-5 sentence plain-language primer (what the thing is, how it works, why it matters here) before the analysis. Written by the synthesizer (pass-1 requirement + pass-2 structural gate), enforced by a new instruction-critic check (`missing-section-primers`).
+- **Calm citation style.** Stacked brackets (`[7][8][79]`) made claim-dense prose read like a parts list. New style: one citation point per sentence at the sentence end, multiple sources grouped in a single bracket (`[7, 12]`, capped at 3), same-source sentence runs consolidated to one marker, and number-bearing or quoting sentences always keeping their own anchor so cite-check pair verification stays exact. All four mechanical consumers understand grouped markers (cite-check triage splits them into per-source pairs; `run verify` citation density counts cited sources rather than brackets, so grouping never lowers measured density; the quote/numeric lint strips them; style-preservation accepts them). The polish auditor's one permitted citation edit is merging an adjacent stack into a grouped bracket, numbers verbatim.
+- **Register discipline, distilled from a humanization ruleset.** The synthesizer's pass-2 audit and the polish auditor now target the two loudest machine-writing tells in past reports: meta-descriptive text (narrating what the report or section is doing instead of saying it; self-describing prose; section-number cross-references) is deleted on sight, and hedging is restricted to unverified specifics — provenance-stated scoping stays, but conclusions the report argues for are asserted bare, and hedge-stacks ("may potentially indicate") always collapse. Dramatic standalone kicker sentences are rationed to one per section, and the primers double as rhythm breaks so density stays readable.
+
+### Per-agent models are now real config; dollar-cost claims removed
+
+- **ModelMap wired into agent rendering.** The profile's per-agent model map existed but was decorative — every installed agent's `model:` frontmatter line was hardcoded in `hooks.py`, so overriding models via a profile silently did nothing. All 16 agent templates now carry `model: << p.models.X >>`, rendered from the profile at install time, and `ModelMap` gained the two missing agents (`cite_checker`, `browser_fetcher`) plus validation (non-empty; aliases or full model IDs both accepted). Full flexibility per agent: `[profile.full]` + `models = { fetcher = "haiku" }` swaps every fetcher to Haiku on the next install/`profile use`; unspecified agents keep their defaults. Defaults are unchanged (verified byte-identical by the goldens).
+- **Model names left the prose.** Agent descriptions and skill text no longer claim "Runs on Sonnet/Opus" (or Sonnet-specific context-window sizes) — the rendered `model:` line is the single source of truth, so a model override can never be contradicted by stale prose. Install-action labels and code comments were de-modeled to match. Drift-proofed by tests: any literal `model:` line in an agent template, or any "Runs on <model>" claim in rendered output, fails the suite.
+- **Dollar-cost estimates removed from the product** (the local benchmark harness keeps its billing-aware cost math). On subscription billing, Claude Code's `cost_usd` is an API-equivalent valuation, not a charge — stating costs as prices contradicted how most users run the pipeline. Gone: the `cost_estimate` profile field and its four builtin values, the router tier table's cost column, `profile list/use` cost output, gap-fetch's "+$1-3 per run", and the source-analyst's "$2-5 per spawn" block (now "Effort discipline"). Time estimates remain. The opt-in budget governor (`run init --budget`) stays, relabeled as a ceiling on *estimated API-equivalent* spend; `run status/report` spend lines now say "API-equiv". A rendered-prompt test rejects any future `$N-M` range.
+
+### Benchmark harness: gear-aware, one-command, and citation-correct
+
+- **Fixed: wrapped runs shipped citation-free reports.** The step-1 skill has always documented "the benchmark harness sets `inline` via wrapper_contract" — but the harness never wrote that file. Every wrapped run therefore fell back to the `wikilink` default, shipping vault-internal `[[note-id]]` markers that `evaluate.py` strips before grading, so reports reached the FACT citation evaluator with **zero verifiable URLs**. Confirmed on the existing fleet: `runs_layercake/query_62` is 7,932 words with 0 numbered citations, 0 URLs. The harness now writes `research/wrapper_contract.json` (`citation_style: inline` + required terminal sections), the pipeline prompt states the requirement explicitly, and `citation-style-preservation` + `quote-integrity` joined the post-run validation rules so a regression is caught per query.
+- **Harness modernized for V8.** The pipeline prompt invoked `/research-layercake` (retired), and the startup banner checked for V7 `layercake-*` skills expecting 14 — now `/hyperresearch` and 18 `hyperresearch-*` step skills. The timeout was hardcoded at 3600s, which would have silently timed out (and discarded) every premier run; it is now gear-scaled (1h at `full`, 6h at `premier`).
+- **Gear-aware fleets.** Scale comes from the installed gear, not a flag — `_setup_run_dir` copies the project root's rendered `.claude/` + `config.toml` into every run dir. Non-`full` gears now get their own runs dir (`bench/runs_layercake-premier/`) and tag suffix so fleets never overwrite each other; `evaluate.py` gained a matching `--gear` flag (it previously hardcoded `runs_layercake` and stripped the tag by a fixed length, which a gear suffix breaks).
+- **`bench/compare.py`** — head-to-head fleet comparison on shared queries only: RACE sub-scores where graded, plus structural metrics (words, citations, citation density, unique URLs, vault sources, cost, duration) that need no API key. Flags the citation confound when comparing against pre-fix fleets.
+- **`bench/run_premier.py`** — one command: preflight (CLIs, queries, gear validity, grading keys) → set + verify gear → cost confirmation → run → RACE/FACT evaluation → comparison.
+
+### Scale gears: premier profile + `hpr profile use`
+
+- **New built-in `premier` profile** — the flat pipeline at ~2× scale: 100–130 sources (min 90), 80–160 planned searches, 14–18 wave-1 fetchers, 10 loci with a doubled depth budget (80), and a widened downstream funnel (claims 150–220, must-read 50–70, 8–16K words, 120–220 citations, raised critic caps) so the extra corpus actually reaches the page instead of stranding in the vault. Estimate: ~3–5 hours.
+- **Gears vs tiers, made explicit.** `light` and `dissertation` are run-time *tiers* (auto-classified / opt-in per query); `full` and `premier` are install-time *gears* — the profile whose numbers are rendered into the skill and agent prompts. The router now carries a "Scale gear (tier ≠ gear)" section, and the width-sweep's full-tier numbers follow the gear (`p.*`) instead of being pinned to the `full` profile (byte-identical under the default gear, verified by goldens).
+- **`hpr profile use <name>`** — the one-command gear shift: validates the profile, re-renders every installed skill/agent prompt, and persists the choice under `[pipeline] profile` in config.toml so later bare `install` runs (e.g. after upgrades) keep the gear. Refuses `light`/`dissertation` with an explanation (they're tiers). `hpr profile list` now shows descriptions, source targets, time estimates, and marks the current gear.
+- **Fix: `[profile.*]` overlays survive config saves.** `VaultConfig.save()` previously dropped user-defined profile tables — any config write (e.g. the crawl4ai auto-setup) silently destroyed custom pipeline profiles. Overlays now round-trip verbatim, including nested inline tables and array-of-array values.
+
+### Phase 5 (2.0 roadmap): verification — the layer that makes it trustworthy
+
+- **Cite-check (new step 14.5, full + dissertation).** Every citation is verified as a citation-sentence BINDING before ship. `hpr citecheck extract` parses (sentence, citation) pairs for both citation styles and mechanically auto-passes pairs whose numbers/wording the claims table confirms; dangling citations (resolving to no vault note) are instant critical findings. The sampled remainder (100% of number-bearing sentences, deterministic sampling for the rest — resume-safe, no RNG) goes to the new `hyperresearch-cite-checker` agent, whose verdicts default skeptical (`supported / partially-supported / unsupported / wrong-source`); findings feed a second, small tool-locked patcher pass. 18 step skills now.
+- **Three verification lint rules.** `quote-integrity` (error): every quoted span ≥5 words must exist verbatim in a vault note — hallucinated quotes cannot ship. `numeric-consistency` (warning): report numbers untraceable to claims or cited-note bodies are flagged for verification. `retracted-citations` (error): citing a retracted source blocks the gate unless the citation itself acknowledges the retraction (sometimes the retraction IS the story).
+- **Ship-time retraction sweep.** `hpr sources retractions --tag` re-checks every DOI-bearing note fresh (cache-bypassing), so a retraction published yesterday is caught today — including on vault sources being REUSED from prior runs.
+- **Independence audit.** `hpr sources independence` clusters derivative sources (canonical-URL identity with tracking-param stripping, near-duplicate bodies via MinHash Jaccard, shared wire-service boilerplate keyed on the body opening — outlets retitle, the wire text doesn't change) and scores members `1/cluster_size`. Step 3's consensus rule now counts independence-weighted voices, so five syndicated copies of one press release are ONE vote, not five.
+- **Run telemetry + verification battery.** `hpr run report [--all]` rolls up per-step wall-time, spend, and event counts from the manifest + events log — the feedback loop for tuning profile constants against observed yield. `hpr run verify` is the CI-able structural gate (report exists, required headings, length in profile range ±20%, citation density ≥1.5/1000 chars, tier artifacts present, scaffold-leak check, cite-check criticals resolved; exit 1 on failure) — wired into the router's final integrity gate alongside the new lint rules.
+- **Bench note.** The phase plan's nightly bench-smoke CI workflow was NOT created: `bench/` turns out to be entirely gitignored (local-only), so a workflow referencing it would be broken by construction. `hpr run verify` is the shipped, CI-able equivalent; local bench scratch (`_*.py` etc.) was tidied into `bench/archive/`.
+
+### Phase 4 (2.0 roadmap): the browser lane — blocked fetches escalate instead of dying
+
+- **Escalation queue (schema v10).** Fetches that hit login walls or bot/captcha walls are no longer discarded — the fetch gate queues them in a new race-safe `escalations` table (SQLite atomic claim semantics; the phase doc's JSON-file design was upgraded deliberately) with reason, utility score, and provenance. Error codes distinguish `AUTH_REQUIRED_ESCALATED` / `JUNK_ESCALATED` so fetcher agents don't retry. Content-quality junk (404s, empty pages) still dies — a 404 in Chrome is still a 404. Policy under `[chrome]`: `enabled`, `escalation_utility_threshold` (low-value blocked URLs are abandoned, the lane is serial and precious), `max_items_per_run`.
+- **`hpr escalation` CLI** — `list/add/claim/complete-via-ingest/human/retry/abandon`. `claim` is atomic under concurrent claimers; `ingest` is the one-shot completion (writes the vault note with `fetch_provider: chrome`, records the source row, syncs, resolves the item) so the browser agent can't half-finish bookkeeping.
+- **`hyperresearch-browser-fetcher` agent** — drains the queue by driving the user's real Chrome browser via Claude-in-Chrome (batched ToolSearch load, new tab always, one instance at a time). Playbook: infinite scroll caps, SPA expansion, PDF-viewer text layers, screenshot transcription for chart-heavy pages, and a Google Scholar lane (`reason: scholar_search` items carry a query; results ingest as structured notes, high-citation hits re-queue). **Hard scope boundary, stated in the prompt and enforced by the workflow: CAPTCHAs, 2FA, and logins are never solved automatically — items go to `needs_human`.**
+- **Human-in-the-loop checkpoint.** All `needs_human` items are consolidated into ONE user prompt at a natural pause point (never per-URL interruptions); after the human completes challenges in their own browser, `escalation retry` + one more drain. Non-interactive runs record `run block --on human-challenges` and continue with everything else; `hpr run status` shows queue depth and needs_human counts.
+- **Graceful degradation.** Without the Claude-in-Chrome extension the queue simply accumulates (visible in `run status`); the floor is the pre-4.0 status quo. Session handoff (Chrome cookies → crawl4ai profile) was evaluated and deliberately NOT implemented: HttpOnly cookies aren't scriptable from page JS, so the honest v1 is the Chrome lane itself plus the existing `hyperresearch setup` guided-login flow.
+
+### Phase 3 (2.0 roadmap): dissertation scale — run isolation, manifest, chapters
+
+- **Per-run workspaces.** Every run-scoped pipeline artifact (scaffold, decomposition, loci, comparisons, critic findings, patch/polish logs, temp scratch, canonical query) now lives under `research/runs/<vault_tag>/` — concurrent and sequential runs can never collide (closes the parallel-run race flagged in 0.8.6). Vault notes stay global; final reports stay at `research/notes/final_report_<tag>.md`. Sync never ingests run workspaces; lint rules resolve artifacts run-aware with full legacy flat-path fallback for pre-3.0 vaults; `vault-tag` collision checks cover run dirs.
+- **Run manifest + explicit resume.** `hpr run init/status/resume/abort/step/spend/event` — `run.json` records per-step + per-chapter status, spend counters, and a heartbeat; `hpr run resume` returns the exact next step and Skill invocation (replacing artifact-scan recovery, which remains as fallback). `run status` flags possibly-stalled runs.
+- **Budget governor.** `run init --budget <usd>` sets a hard ceiling; crossing it flips the run to `blocked (budget)`. The router instructs shrinking fan-out near the ceiling instead of silently skipping profile-mandated steps.
+- **`dissertation` profile + chaptered execution.** New built-in profile (opt-in only, never auto-classified): 250–450 sources across 4–10 chapters, each chapter running the proven 40–80-source pipeline envelope (steps 2–10 loop per chapter, ≤2 in flight), global reconcile/synthesis on top, 25K–80K-word chaptered output, scaled critic caps, multi-hour pacing. New step skill `hyperresearch-1-5-chapter-partition` (17 step skills now).
+- **Literature-review matrix + meta-analysis substrate.** `hpr claims matrix` generates the per-source review table (tier, venue, citations, quality, key finding) from the claims table; `hpr claims targets` groups claims by stance_target across sources with stance splits and source-attributed numbers for comparison tables.
+
+### Phase 2 (2.0 roadmap): the source-ranking engine
+
+Source quality becomes a persistent, queryable property instead of ephemeral prompt prose (schema v9, additive-only).
+
+- **Per-source quality scores.** New note columns: `doi`, `utility_score`, `citation_count`, `venue`, `is_retracted` (frontmatter-mirrored — markdown stays truth) plus derived `authority_score`, `centrality_score`, `independence`, `quality_score` (DB-cache, recomputed, survive re-sync).
+- **`hpr sources score`** — enriches DOI-bearing notes from OpenAlex/Semantic Scholar (citation counts, venue, **retraction flags**), cached in a new `api_cache` table (`[ranking] api_cache_ttl_days`, default 30). `hpr sources backfill-doi` regexes the back-catalog; new fetches capture DOIs/arXiv ids automatically, and `fetch --utility-score` persists the step-2 utility score that was previously discarded after fetch selection.
+- **`hpr graph rank`** — pure-Python PageRank over the link + provenance-breadcrumb graph; centrality here means "many independent research chains converged on this source". Also recomputed during `repair`.
+- **Composite `quality_score`** — renormalized weighted blend of tier weight, utility, citation-authority percentile (vault-relative, log-scaled), and centrality; retracted sources floored at 0.05. Weights configurable under `[ranking]`.
+- **`hpr search --ranked`** — folds `quality_score` into FTS relevance (`(0.5 + quality)` multiplier; unscored notes stay neutral). Default search behavior unchanged.
+- **Claims table** — `hpr claims ingest/list/search` persists fetcher-extracted `claims-*.json` into queryable `claims` + `claims_fts` tables keyed to source notes (idempotent by content hash). This is the substrate for phase-5 cite-checking.
+- **Semantic search (embeddings table revived)** — `hpr embed sync` + `hpr search --semantic` (RRF hybrid with FTS). Provider-pluggable under `[embeddings]`: `none` (default — zero API keys required), `voyage`, `openai`. Brute-force cosine, no vector-DB dependency.
+- **Pipeline integration** — step 2 gains step 2.7 (persist ranking signals after the last wave; retracted sources flagged before they can anchor a locus); fetcher batches carry utility scores; step 10 curates from `search --ranked` instead of orchestrator intuition.
+
+### Phase 1 (2.0 roadmap): config extraction + pipeline profiles
+
+- **Every behavioral constant is now config** (`docs/roadmap-2.0/phase-1-config-profiles.md` WS1). New `.hyperresearch/config.toml` sections: `[fetch]` (page/PDF timeouts, smart-wait polling, visible-browser domain list, image timeout), `[junk]` (content gates, binary-garbage ratio, extra signal lists), `[assets]` (max images, min image bytes), `[dedup]` (MinHash/LSH parameters, threshold), `[lint]` (extract-coverage and stale-review thresholds), plus `[search]` output defaults (`default_limit`, `chars_per_token`, `snippet_len`). All defaults reproduce prior behavior.
+- **BEHAVIOR CHANGE — PDF downloads verify TLS by default.** The PDF fetch path previously hardcoded `verify=False`, silently disabling certificate verification. New `[fetch] pdf_verify_tls` defaults to `true` (secure). Set it to `false` explicitly for cert-broken mirrors you trust.
+- **Pipeline profiles** (`hyperresearch profile list/show/validate`). Every research-scale knob — source gates, fetcher fan-out, loci caps, depth budgets, draft counts, word targets, critic caps, per-agent models — lives in a named, validated profile. Built-in `full` and `light` reproduce the shipped V8 values exactly; users override keys or define new profiles (`[profile.dissertation] extends = "full"`) in config.toml.
+- **Skill/agent prompts are now templates rendered at install.** `hyperresearch install --profile <name>` renders the 17 skills and 15 agent prompts from the chosen profile (custom `<< >>` Jinja delimiters — prompt-native `{{...}}` placeholders and JSON braces pass through untouched). Rendered files carry a `rendered from profile "..."` provenance header after the frontmatter. Golden tests pin the `full`-profile render byte-for-byte against the pre-template prompts, so profile/template drift is a test failure, not a silent prompt change.
+- **Width-sweep consistency fix** (roadmap phase-0 WS5, pulled forward): the three contradictory full-tier source-target statements (40-100 / 40–80 / 55–80) are unified to the profile value (55–80); the light target (12–20 vs 15–25) and the tier-table fetchers-per-wave (8–12 vs 10–12) are likewise unified to the table/Wave-1 values.
+
 ## [0.8.7] - 2026-07-18
 
 Community-fix release: five contributed PRs plus two maintainer follow-ups, closing #32, #33, #35, #37, and #39.

{hyperresearch-0.8.7 → hyperresearch-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hyperresearch
-Version: 0.8.7
+Version: 0.9.0
 Summary: Claude Code harness for disciplined, adversarially-audited deep research with full provenance.
 Project-URL: Homepage, https://github.com/jordan-gibbs/hyperresearch
 Project-URL: Repository, https://github.com/jordan-gibbs/hyperresearch
@@ -69,14 +69,24 @@ Description-Content-Type: text/markdown
 
 ---
 
-**Hyperresearch turns Claude Code into a deep research agent. and currently leads the DeepResearch-Bench RACE leaderboard (benchmarked internally).** A tier-adaptive 16-step pipeline produces adversarially-audited reports with full source provenance. Every fetched source lands in a persistent, searchable vault that compounds across sessions.
+**Hyperresearch turns Claude Code into a deep research agent: one that currently leads the DeepResearch-Bench RACE leaderboard (benchmarked internally).** A tier-adaptive 16-step pipeline takes one prompt and produces an adversarially-audited report with full source provenance. Every source it reads lands in a persistent, searchable vault, so each session starts smarter than the last.
 
 <p align="center">
-  <img src="assets/benchmark.png" alt="DeepResearch-Bench top-5 — hyperresearch leads the chart ahead of Grep Deep Research, Cellcog Max, nvidia-aiq, Gemini Deep Research, and OpenAI Deep Research" width="780">
+  <img src="assets/benchmark.png" alt="DeepResearch-Bench top-5 hyperresearch leads the chart ahead of Grep Deep Research, Cellcog Max, nvidia-aiq, Gemini Deep Research, and OpenAI Deep Research" width="780">
 </p>
 
 <p align="center"><sub>Forward-looking projection from a stratified pilot against the DeepResearch-Bench leaderboard snapshot (https://huggingface.co/spaces/muset-ai/DeepResearch-Bench-Leaderboard). Third party validation is pending.</sub></p>
 
+## Why it wins
+
+- **250+ sources in a single run.** The `premier` scale profile targets 100–130 in the width sweep alone; citation chasing and gap-fill fetches more than double what actually lands in the corpus.
+- **Every citation is verified before the report ships.** A skeptical cite-checker audits whether each cited source actually supports its sentence. Hallucinated quotes and unacknowledged retractions are hard blocks at the gate.
+- **Syndication doesn't count as consensus.** An independence audit clusters derivative copies, so five reprints of one press release argue with the weight of one source.
+- **Adversarial by construction.** Four critics attack every draft in parallel, and a tool-locked patcher can only apply surgical edits. It physically cannot rewrite the report.
+- **Nothing is thrown away.** Every source lands in a searchable markdown-plus-SQLite vault that your next session reuses before it fetches anything new.
+- **Crashed runs resume.** Each run keeps a manifest; `run resume` picks up at the exact step where it died.
+- **Scales from 30 minutes to a dissertation.** Bounded queries auto-route to a 5-step fast path. Opt-in dissertation runs write 25K–80K words across chapters, from 300–450 sources.
+
 ## Install
 
 ```bash
@@ -86,7 +96,7 @@ pip install hyperresearch && hyperresearch install
 
 Then `/hyperresearch <anything>` in Claude Code.
 
-> Python 3.11–3.13. (3.14 not yet supported — use `pyenv install 3.13`, `uv venv -p 3.13`, or `py -3.13 -m venv .venv`.)
+> Python 3.11–3.13. (3.14 not yet supported. Use `pyenv install 3.13`, `uv venv -p 3.13`, or `py -3.13 -m venv .venv`.)
 >
 > Power users: `hyperresearch install --global` makes `/hyperresearch` reachable from every Claude Code session anywhere, at the cost of ~15 lines in every session's system reminder. Per-project install (above) keeps unrelated CC sessions clean.
 
@@ -94,12 +104,13 @@ Then `/hyperresearch <anything>` in Claude Code.
 
 ## The 16-step research pipeline
 
-The entry skill is a thin router. It bootstraps the canonical research query, then invokes one step skill per pipeline phase via Claude Code's `Skill` tool. Each step's procedure is loaded fresh into context only when needed defeating context-rot problems that makes long pipelines silently drop steps.
+The entry skill is a thin router. It pins down the canonical research query, then invokes one step skill per phase via Claude Code's `Skill` tool. Each step's procedure loads into context only when that step actually runs. That's what stops a long pipeline from quietly dropping steps as its context rots.
 
 | # | Step | What it does | Tiers |
 |---|---|---|---|
-| 1 | Decompose | Canonical query → atomic items + coverage matrix + tier classification | both |
-| 2 | Width sweep | Multi-perspective search plan + parallel fetcher waves (Haiku) | both |
+| 1 | Decompose | Canonical query → atomic items + coverage matrix + tier classification | all |
+| 1.5 | Chapter partition | Group atomic items into 4–10 chapters; steps 2–10 then loop per chapter | dissertation |
+| 2 | Width sweep | Multi-perspective search plan + parallel fetcher waves | all |
 | 3 | Contradiction graph | Pair contradictions across the corpus into ranked clusters | full |
 | 4 | Loci analysis | Two parallel loci-analysts → scored loci with source budgets | full |
 | 5 | Depth investigation | K parallel depth-investigators → interim notes with committed positions | full |
@@ -107,35 +118,49 @@ The entry skill is a thin router. It bootstraps the canonical research query, th
 | 7 | Source tensions | Extract expert disagreements → source-tensions.json | full |
 | 8 | Corpus critic | "What source would overturn this?" + targeted gap-fill fetch | full |
 | 9 | Evidence digest | Top claims + verbatim quotes → evidence-digest.md | full |
-| 10 | Triple draft | Per-angle source curation + 3 parallel draft sub-orchestrators (light: single draft) | both |
+| 10 | Triple draft | Per-angle source curation + 3 parallel draft sub-orchestrators (light: single draft) | all |
 | 11 | Synthesize | Plan + outline + spawn synthesizer subagent → final_report.md | full |
 | 12 | Critics | 4 adversarial critics in parallel → findings JSONs | full |
 | 13 | Gap-fetch | Targeted fetch wave for critic-identified vault gaps | full |
 | 14 | Patcher | Surgical Edit hunks applied to draft (tool-locked Read+Edit) | full |
-| 15 | Polish | Hygiene + filler pass (tool-locked Read+Edit subagent) | both |
-| 16 | Readability audit | Recommender writes JSON suggestions; orchestrator selectively applies | both |
+| 14.5 | Cite-check | Verify citation-sentence bindings; skeptical LLM spot-check; second surgical patch pass | full |
+| 15 | Polish | Hygiene + filler pass (tool-locked Read+Edit subagent) | all |
+| 16 | Readability audit | Recommender writes JSON suggestions; orchestrator selectively applies | all |
 
-### Depth Modes 
+### Tiers and gears: the two scale levers
 
-In your prompt, you can request one of two tiers and the rest of the pipeline scales accordingly. Full mode is default.
+**Tiers** route per query. Step 1 auto-classifies `light` vs `full`. `dissertation` is opt-in only; ask for it in your prompt.
 
-| Tier | Steps that run | Typical time |
+| Tier | What runs | Typical time |
 |---|---|---|
-| `light` | bounded factual queries, surveys, comparisons — 1 → 2 → 10 → 15 → 16 | ~30–40 min |
-| `full` | deep argumentative analysis with adversarial review — all 16 steps | ~1.5–2.5 hours |
+| `light` | bounded factual queries, surveys, comparisons: 1 → 2 → 10 → 15 → 16 | ~30–40 min |
+| `full` (default) | deep argumentative analysis with adversarial review: all 16 steps + cite-check | ~1.5–2.5 h at `full` gear |
+| `dissertation` | chaptered mega-runs: 300–450 sources across 4–10 chapters, 25K–80K words | ~4–8 hours |
+
+**Gears** set the scale of the standard pipeline: the source targets, depth budgets, and word targets rendered into the step skills.
+
+```bash
+hyperresearch profile list           # all profiles + descriptions + current gear
+hyperresearch profile use premier    # 100–130 sources, doubled depth budget (~3–5 h)
+hyperresearch profile use full       # back to the 55–80-source baseline
+```
+
+The gear persists per project and survives reinstalls. Custom gears: define `[profile.<name>]` in `.hyperresearch/config.toml` (any knob: source targets, loci caps, draft counts, word targets, per-agent models) and `profile use <name>`.
 
 ### The two load-bearing principles
 
 1. **Patch, never regenerate.** After step 11 produces the synthesized report (or step 10 for light tier), the only modifications are surgical Edit hunks. The patcher and polish auditor are tool-locked to `[Read, Edit]` at the Claude Code allowlist level so they physically cannot Write a new draft. Per-hunk caps make "just rewrite it" mechanically impossible. Critic findings that don't fit a small hunk escalate as structural issues.
 
-2. **Canonical research query is gospel.** The verbatim user prompt is persisted to `research/query-<vault_tag>.md` once and re-read by every subsequent step and every spawned subagent. Wrapper requirements (save paths, citation format, terminal sections) are a separate contract.
+2. **Canonical research query is gospel.** The verbatim user prompt is persisted to `research/runs/<vault_tag>/query.md` once and re-read by every subsequent step and every spawned subagent. Wrapper requirements (save paths, citation format, terminal sections) are a separate contract.
 
 ### Subagent roster
 
-| Agent | Model | Role |
+Models are profile config, not hardcode. The table shows the shipped defaults, and you can override any of them in `.hyperresearch/config.toml`: `[profile.full]` with `models = { fetcher = "haiku" }` swaps every fetcher to Haiku on the next install or `profile use`.
+
+| Agent | Default model | Role |
 |---|---|---|
 | `hyperresearch-fetcher` | Sonnet | URL fetching via crawl4ai; runs 8–12 in parallel per wave |
-| `hyperresearch-source-analyst` | Sonnet (1M ctx) | End-to-end digest of any single long source >5000 words |
+| `hyperresearch-source-analyst` | Sonnet | End-to-end digest of any single long source >5000 words |
 | `hyperresearch-loci-analyst` | Sonnet | Reads the width corpus, returns 1–8 depth loci with rationale |
 | `hyperresearch-depth-investigator` | Sonnet | Investigates one locus, writes one interim note with a committed position |
 | `hyperresearch-corpus-critic` | Sonnet | "What source would overturn the current direction?" pre-draft gap analysis |
@@ -146,14 +171,16 @@ In your prompt, you can request one of two tiers and the rest of the pipeline sc
 | `hyperresearch-width-critic` | Opus | Topical corners the corpus supports but the draft ignores |
 | `hyperresearch-instruction-critic` | Opus | Structural mismatches against the prompt's atomic items |
 | `hyperresearch-patcher` | Opus | Tool-locked `[Read, Edit]`. Applies critic findings as surgical Edit hunks |
+| `hyperresearch-cite-checker` | Sonnet | Skeptically verifies sampled citation-sentence bindings before ship |
 | `hyperresearch-polish-auditor` | Opus | Tool-locked `[Read, Edit]`. Cuts filler, strips hygiene leaks |
 | `hyperresearch-readability-recommender` | Opus | Writes JSON suggestions for paragraph rhythm and list/table conversion |
+| `hyperresearch-browser-fetcher` | Sonnet | Drains the escalation queue by driving your real Chrome (Claude-in-Chrome) |
 
 ---
 
 ## The vault: persistent, searchable, compounding
 
-Hyperresearch is not a one-shot report generator like most other Deep research harnesses. Every fetched source lands in a SQLite-indexed vault that every future research session can reuse.
+Most deep research harnesses are one-shot: report out, everything else discarded. Hyperresearch keeps what it reads. Every fetched source lands in a SQLite-indexed vault that future sessions search before they fetch.
 
 ```bash
 hyperresearch search "ion-trap gate fidelity" -j           # Full-text search
@@ -164,26 +191,61 @@ hyperresearch graph backlinks <id> -j                      # Reverse links
 hyperresearch lint -j                                      # Health check (broken links, missing tags)
 ```
 
-**Markdown is truth, SQLite is cache.** Notes live as plain markdown with YAML frontmatter in `research/notes/`. The SQLite index is fully rebuildable. Delete it and `hyperresearch sync` reconstructs it from the markdown. The vault is inspectable in any editor, version-controllable in git, and readable without the tool installed.
+**Markdown is truth, SQLite is cache.** Notes live as plain markdown with YAML frontmatter in `research/notes/`. The SQLite index is fully rebuildable: delete it and `hyperresearch sync` reconstructs it from the markdown. Open the vault in any editor, version it in git. You don't need the tool installed to read your own research.
 
 **PDFs fetch directly.** `hyperresearch fetch` auto-detects PDF URLs (arXiv, NBER, SSRN, direct `.pdf` links) and extracts full text via pymupdf. Raw PDFs land in `research/raw/<note-id>.pdf` and the note's `raw_file:` frontmatter links back.
 
 **Provenance breadcrumbs.** Every fetched source carries a `--suggested-by` link back to whatever surfaced it. The chain forms a rooted tree from seed fetches; the `provenance` lint rule catches disconnected components.
 
+**Semantic search, if you want it.** `hyperresearch embed sync` populates embeddings (provider-pluggable: `voyage`, `openai`, or the default `none`, which needs zero API keys) and `search --semantic` blends vector similarity with full-text ranking.
+
+---
+
+## Source ranking: quality is persistent, not vibes
+
+Every source accumulates a composite `quality_score` built from source-type tier, fetch-time utility, citation authority (from OpenAlex / Semantic Scholar, including **retraction flags**), and vault PageRank centrality:
+
+```bash
+hyperresearch sources score -j             # Enrich DOI-bearing notes: citations, venue, retractions
+hyperresearch graph rank -j                # PageRank over the link + provenance graph
+hyperresearch search "q" --ranked -j       # Quality-weighted full-text search
+hyperresearch sources independence -j      # Cluster syndicated/derivative copies: 5 copies of one press release = 1 vote
+hyperresearch claims search "q" -j         # Query extracted claims across all sources
+```
+
+Retracted sources are floored to near-zero quality, and a ship-time retraction sweep re-checks every cited DOI fresh, so a retraction published yesterday is caught today. Even on vault sources reused from old runs.
+
+---
+
+## Runs: resumable, budgeted, verified
+
+Every run owns an isolated workspace (`research/runs/<vault_tag>/`) and a manifest. Concurrent runs never collide, and a crashed run resumes exactly where it stopped:
+
+```bash
+hyperresearch run status -j          # Step-by-step status, spend, escalation queue depth
+hyperresearch run resume -j          # Exact next step + Skill invocation to continue
+hyperresearch run report -j          # Per-step wall-time / spend / source-yield telemetry
+hyperresearch run verify <tag> -j    # Ship gate: headings, length, citation density, cite-check resolution
+```
+
+`run init --budget 50` caps estimated API-equivalent spend; crossing the cap blocks the run rather than letting it quietly balloon. And before any report ships, the verification battery runs: **quote-integrity** (every quoted span must exist verbatim in a vault note), **retracted-citations** (citing a retracted source unacknowledged blocks the ship), **numeric-consistency** (numbers untraceable to evidence get flagged), plus the cite-check step's per-citation binding audit.
+
 ---
 
 ## What's structurally enforced
 
-- **Verbatim prompt as gospel** — `scaffold-prompt` lint blocks if the scaffold doesn't open with the user's exact prompt
-- **Locus coverage** — every step 4 locus must have a step 5 interim note; missing interims flag as errors
-- **Patch-only modification** — steps 14, 15, 16 are tool-locked to `[Read, Edit]`. They cannot regenerate the draft
-- **Critical findings never silently skip** — `patch-surgery` lint surfaces any critical finding the patcher couldn't apply
-- **Schema integrity** — `tier`, `content_type`, and `type` are SQLite CHECK-constrained vocabularies; corrupted frontmatter cannot poison the index
-- **Hygiene leaks caught on the way out** — scaffold sections, YAML frontmatter, and prompt echoes are stripped by step 15 before ship
+- **Verbatim prompt as gospel.** `scaffold-prompt` lint blocks if the scaffold doesn't open with the user's exact prompt
+- **Locus coverage.** Every step 4 locus must have a step 5 interim note; missing interims flag as errors
+- **Patch-only modification.** Steps 14, 15, 16 are tool-locked to `[Read, Edit]`. They cannot regenerate the draft
+- **Critical findings never silently skip.** `patch-surgery` lint surfaces any critical finding the patcher couldn't apply
+- **Quoted text must exist.** `quote-integrity` lint blocks any quoted span that doesn't appear verbatim in a vault note; hallucinated quotes cannot ship
+- **Retractions block the ship.** Citing a retracted source without acknowledging the retraction is a hard error at the final gate
+- **Schema integrity.** `tier`, `content_type`, and `type` are SQLite CHECK-constrained vocabularies; corrupted frontmatter cannot poison the index
+- **Hygiene leaks caught on the way out.** Scaffold sections, YAML frontmatter, and prompt echoes are stripped by step 15 before ship
 
 ---
 
-## Authenticated crawling
+## Authenticated crawling + the browser lane
 
 Fetch from LinkedIn, Twitter, paywalled sites or anything you can log into:
 
@@ -193,16 +255,18 @@ hyperresearch setup       # Browser opens. Log into your sites. Done.
 
 LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a visible browser to avoid session kills.
 
+**Blocked fetches escalate instead of dying.** When headless crawling hits a login wall or bot wall mid-run, the URL queues as an escalation (`hyperresearch escalation list -j`). If you have the [Claude-in-Chrome](https://claude.com/chrome) extension, the browser-fetcher agent drains the queue by driving your real, logged-in Chrome. Hard boundary: **CAPTCHAs, 2FA, and logins are never solved automatically.** They're consolidated into one message and handed to you.
+
 ---
 
 ## Academic APIs before web search
 
 For any topic with a research literature, hit academic APIs BEFORE web search. They return citation-ranked canonical papers; web search returns derivative commentary.
 
-- **Semantic Scholar** — `https://api.semanticscholar.org/graph/v1/paper/search`
-- **arXiv** — `https://export.arxiv.org/api/query`
-- **OpenAlex** — `https://api.openalex.org/works`
-- **PubMed** — `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi`
+- **Semantic Scholar:** `https://api.semanticscholar.org/graph/v1/paper/search`
+- **arXiv:** `https://export.arxiv.org/api/query`
+- **OpenAlex:** `https://api.openalex.org/works`
+- **PubMed:** `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi`
 
 After the academic sweep, run web searches for context, news, non-academic angles, and at least one adversarial search ("criticism of X", "limitations of X").
 
@@ -212,7 +276,7 @@ After the academic sweep, run web searches for context, news, non-academic angle
 
 - It doesn't replace your judgment on which sources matter. The agent picks, you steer.
 - It can't fetch what's behind a paywall you haven't logged into.
-- It runs on Anthropic models Opus + Sonnet + Haiku via the subagent roster. Costs scale with tier and corpus size. If anyone wants to port this to Codex, put up a PR! 
+- It runs on Anthropic models via the subagent roster (per-agent assignments come from the profile's model map). Usage scales with tier, gear, and corpus size. If anyone wants to port this to Codex, put up a PR! 
 - The lint gate catches **structural** failures (missing scaffold, broken provenance, unresolved CRITICALs). It cannot guarantee factual accuracy, that's still your call.
 
 ---