hyperresearch 0.2.0__tar.gz → 0.4.0__tar.gz

{hyperresearch-0.2.0 → hyperresearch-0.4.0}/.gitignore

@@ -11,3 +11,6 @@ build/
 .mypy_cache/
 .ruff_cache/
 *.egg
+.env
+bench/
+codex_bench/

hyperresearch-0.4.0/CHANGELOG.md

@@ -0,0 +1,124 @@
+# Changelog
+
+## [0.4.0] - 2026-04-13
+
+### New
+
+- **Request-type classification (Step 0)** — The research workflow now starts by classifying the user's request into one of 7 types (Canonical Knowledge Retrieval, Market / Landscape Mapping, Engineering / Technical How-To, Interpretive / Humanities Analysis, Comparative Evaluation, Emerging / Cutting-Edge Research, Forecast / Strategy / Recommendation) plus a General fallback. Classification happens before any searching and governs the rest of the workflow.
+- **Type-specific parameter blocks** — Each of the 7 types specifies its own source strategy (count + primary/secondary mix), target length, opening-section shape, H2 heading count, analytical mode, and special rules. A humanities analysis wants 6–10 long thematic sections; a market landscape wants 8–14 vendor-cluster sections with a mandatory comparison matrix; a cutting-edge research request wants primary-heavy preprint reading with a "What we don't know yet" section. One workflow, seven parameterizations.
+- **Primary-heavy vs. secondary-heavy source policy** — New explicit axis: Types 1/4/5/6 are primary-heavy (cite originals, engage deeply, prune irrelevant secondary coverage), Types 2/7 are secondary-heavy (triangulate across many descriptions), Type 3 is balanced. Source count is now a function of request type, not topic complexity.
+- **Conceptual scaffold step (before writing)** — Agent must answer four questions in a scratch file before drafting: the hard question, the naive answer, the structural tension, and a dependency-ordered heading sketch. The final report's opening section must be a framework section, not a definition.
+- **Cross-source comparison step** — Before writing the body, agent finds 3–5 places where sources actually disagree and captures short comparison blocks. Sources earn citations by being compared, not listed. These become the backbone of body sections.
+- **Writing-draft hard constraints** — Target 400–600 words per H2, 12–20 H2s on a 10K-word report, never one-section-per-source, every section ends with an analytical beat, comparison tables not fact tables. Type-specific blocks override these (Type 4 Humanities targets 800–1500 words per section across 6–10 sections).
+- **Frontmatter-first note triage (Step 4.5)** — Six-level protocol for reading notes efficiently. Always start with `note list -j` for summaries, use `note show --meta -j` for frontmatter-only reads, `search --include-body --max-tokens 6000 -j` for token-capped multi-note pulls, and **delegate notes with `word_count > 6000` to a fresh Sonnet subagent** with a pointed extraction prompt (~40× context savings per large note). Rely on the summary field first; read the body only when it earns its place.
+- **Type-aware adversarial audit** — The structure-auditor subagent now checks whether the draft honors its declared type's parameter block: thematic sections for Humanities, mandatory comparison matrix for Comparative, "What we don't know yet" for Emerging, a position on winners for Market. Flags every type violation.
+
+### Changed
+
+- **`fit_markdown` via PruningContentFilter** — crawl4ai provider now uses `DefaultMarkdownGenerator` with `PruningContentFilter` so fetched notes contain just the main content, stripping navigation, footers, and sidebar chrome. Both `AsyncWebCrawler.arun()` and the Playwright visible-browser path use the same generator for consistent output. Applied to single fetch, batch fetch, and visible browser paths.
+- **Skip numeric wiki-links in note parser** — `[[100]]`-style citation markers in bibliographies and academic papers are no longer extracted as note references. Avoids thousands of spurious broken-link warnings on papers that use numbered references.
+- **"Over-collect, then prune" reframed as "over-collect, then engage deeply"** — A report built from 30 sources that disagree and force you to take positions is worth more than a report built from 80 sources that each contribute one bullet of description. Collection is a means to an argument, not the goal.
+- **Scaffold and comparison artifacts are ephemeral, NOT hyperresearch notes** — Both the conceptual scaffold and the cross-source comparison blocks live in `/tmp/scaffold.md` or working memory, explicitly not as notes. Protects the research base from pre-writing scratch work.
+
+## [0.3.0] - 2026-04-11
+
+### New
+
+- **Native PDF extraction** — PDFs detected by URL pattern, downloaded directly with httpx, text extracted with pymupdf. No browser needed. arXiv `/abs/` links auto-convert to `/pdf/`.
+- **Raw file storage** — PDF bytes saved to `research/raw/<note-id>.pdf`, linked from note frontmatter via `raw_file:` field. Agent can read the raw PDF directly.
+- **Junk page detection** — `WebResult.looks_like_junk()` catches Cloudflare captchas, error pages, cookie walls, binary garbage, reCAPTCHA, and empty content before saving. Returns `JUNK_CONTENT` error instead of creating useless notes.
+- **Gap analysis step** — after drafting the report, agent re-reads the original query word by word, identifies gaps, and does another full round of research to fill them.
+- **Adversarial audit** — two subagents (comprehensiveness auditor + logic/structure auditor) review the draft in parallel. Runs up to 2 loops. Agent uses wait time productively to improve summaries and tags.
+- **Source checkpoint** — agent must review collected sources before writing any draft. Checks coverage breadth, missing angles, uncited references. Expects 50-100+ sources on complex topics.
+- **Scholarly API guidance** — CLAUDE.md and `/research` skill now encourage use of arXiv, Semantic Scholar, CrossRef, and PubMed APIs for academic research.
+- **Date injection** — today's date injected programmatically into CLAUDE.md at install time.
+- **Multi-round research emphasis** — agent docs stress multiple rounds of search → fetch → follow links, spawning 10-20 fetcher agents per round.
+
+### Changed
+
+- **Agent-driven curation replaces auto-enrich** — removed the keyword-matching `enrich_note_file()` from the fetch pipeline. Fetcher subagents now read content, write real summaries, add meaningful tags, and quality-check each source (deprecating junk/off-topic notes).
+- **Fetcher subagent quality gate** — subagent now checks relevance, content quality, and duplicates. Deprecates bad notes instead of leaving them as drafts.
+- **`_resolve_executable()` prioritizes venv** — checks venv `Scripts/` dir before PATH, preventing system-wide installs from overriding the project's venv.
+- **PDF binary detection improved** — checks for `endstream`, `endobj`, `/FlateDecode`, `%PDF-` markers and non-printable character ratios. Catches binary garbage in both single-fetch and batch-fetch paths.
+- **Junk detection thresholds raised** — empty content threshold: 100→300 chars, cookie page threshold: 500→1500 chars. Added `recaptcha`, `checking your browser`, `verify you are human` to bot detection signals.
+- **SSL verification disabled for PDF downloads** — academic sites often have self-signed certs. `httpx.get(verify=False)` for PDF fetches only.
+- **PDF fetch logging** — `_fetch_pdf` failures now logged via `logging.getLogger("hyperresearch.pdf")` instead of silently returning None.
+- **Fetcher subagent continues on failure** — no longer stops on first fetch error, tries all URLs and reports failures individually.
+
+### Added dependencies
+
+- `pymupdf>=1.24` — PDF text extraction
+- `httpx>=0.27` — direct HTTP downloads for PDFs (bypasses browser)
+
+## [0.2.0] - 2026-04-10
+
+### New
+
+- **`/research` skill** — Scripted deep research workflow as a Claude Code slash command. Clarifies ambiguous requests, searches broadly, fetches aggressively, follows rabbit holes, auto-curates, synthesizes, and presents findings with hub notes
+- **`hyperresearch setup`** — Interactive TUI onboarding: web provider, browser profile selection/creation, agent hooks. Auto-launches on first `install`
+- **`hyperresearch fetch-batch`** — Concurrent multi-URL fetch with batched sync (O(1) syncs instead of O(n))
+- **`hyperresearch link --auto`** — Holistic auto-linking: scans notes for mentions of other notes' titles and appends wiki-links
+- **`hyperresearch assets list/path`** — Browse downloaded screenshots and images
+- **`--save-assets` flag** — Opt-in screenshot + content image download on fetch
+- **`--visible` flag** — Non-headless browser for stubborn auth sites (auto-enabled for LinkedIn, Twitter, Facebook, Instagram, TikTok)
+- **`--max-tokens` on search** — Token budget truncation for context-aware agents
+- **Auto-curation at fetch time** — Notes arrive with auto-generated tags and summaries
+- **MCP write tools** — `fetch_url`, `create_note`, `update_note` (MCP server is now read-write)
+- **MinHash+LSH dedup** — O(n) approximate dedup for large vaults (200+ notes), falls back to brute-force for small vaults
+- **Hub notes auto-surfaced** after research sessions
+- **Synthesis notes** saved as feedback loop (agent Q&A becomes searchable)
+- **`hyperresearch-fetcher` subagent** — Haiku-powered URL fetcher installed to `.claude/agents/`
+- **Login wall detection** — `AUTH_REQUIRED` error instead of saving login page junk
+- **Smart SPA wait** — Polls DOM stability (2s initial + 10s ceiling) instead of fixed delays
+
+### Changed
+
+- **crawl4ai is the sole browser provider** — Removed firecrawl, tavily, trafilatura
+- **crawl4ai v0.8.x API** — AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, arun/arun_many
+- **Authenticated crawling** via crawl4ai browser profiles (`crwl profiles` or setup TUI)
+- **CLI path baked into CLAUDE.md** — Works without venv activation (forward slashes for Windows bash)
+- **Deep research philosophy** — Agent docs say "over-collect, then prune" and "go down rabbit holes"
+- **Windows encoding fix** — `stream.reconfigure(encoding="utf-8")` at startup, no more charmap crashes
+- **Note slugs capped at 80 chars** — Avoids Windows MAX_PATH issues
+- **Anti-bot stealth always on** when crawl4ai is used (no setup question)
+- **Config commands** now support `web.provider`, `web.profile`, `web.magic`
+
+### Removed
+
+- Dead fields: `confidence`, `superseded_by`, `llm_compiled`, `llm_model`, `compile_source`
+- Tag plural normalization (use explicit `tag_aliases` instead)
+- `deprecated-no-successor` and `low-confidence` lint rules
+- Firecrawl, Tavily, Trafilatura web providers
+
+## [0.1.0] - 2026-04-09
+
+Initial release. Forked from [llm-kasten](https://github.com/jordan-gibbs/llm-kasten) and repositioned for agent-driven research workflows.
+
+### New
+
+- **`hyperresearch install`** — One-step setup: init vault + inject agent docs + install PreToolUse hooks for Claude Code, Codex, Cursor, Gemini CLI
+- **`hyperresearch fetch <url>`** — Fetch a URL, extract content, save as a research note with source tracking
+- **`hyperresearch research <topic>`** — Deep research: web search, fetch results, follow links, save as linked notes, generate synthesis MOC
+- **`hyperresearch sources list/check`** — List and query fetched web sources
+- **Web provider plugin system** — Pluggable backends: builtin (stdlib), crawl4ai (local headless browser)
+- **Agent hook system** — PreToolUse hooks that remind agents to check the research base before web searches
+- **Sources table** — URL deduplication, domain tracking, fetch metadata
+- **Extended frontmatter** — `source_domain`, `fetched_at`, `fetch_provider` fields
+- **MCP server** with 10 tools including `check_source` and `list_sources`
+
+### From kasten (the backbone)
+
+- SQLite FTS5 full-text search with BM25 ranking
+- Markdown notes with YAML frontmatter as source of truth
+- `[[wiki-link]]` tracking with backlinks
+- `--json` / `-j` structured output on every command
+- Note lifecycle: draft → review → evergreen → stale → deprecated → archive
+- Auto-sync (mtime + SHA-256 change detection)
+- Agent doc injection (CLAUDE.md, AGENTS.md, GEMINI.md, copilot-instructions.md)
+- Web viewer with force-directed knowledge graph
+- 70 tests
+
+[0.4.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.4.0
+[0.3.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.3.0
+[0.2.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.2.0
+[0.1.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.1.0

hyperresearch-0.4.0/PKG-INFO

@@ -0,0 +1,188 @@
+Metadata-Version: 2.4
+Name: hyperresearch
+Version: 0.4.0
+Summary: Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki.
+Project-URL: Homepage, https://github.com/jordan-gibbs/hyperresearch
+Project-URL: Repository, https://github.com/jordan-gibbs/hyperresearch
+Project-URL: Issues, https://github.com/jordan-gibbs/hyperresearch/issues
+Author-email: Jordan Gibbs <jordan@jordangibbs.xyz>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,cli,knowledge-base,llm,markdown,research,wiki,zettelkasten
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Requires-Python: >=3.11
+Requires-Dist: crawl4ai>=0.4
+Requires-Dist: httpx>=0.27
+Requires-Dist: jinja2>=3.1
+Requires-Dist: platformdirs>=4.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pymupdf>=1.24
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.9.0
+Provides-Extra: all
+Requires-Dist: mcp>=1.6; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pytest-cov>=4.1; extra == 'dev'
+Requires-Dist: pytest>=7.4; extra == 'dev'
+Requires-Dist: ruff>=0.3; extra == 'dev'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.6; extra == 'mcp'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="assets/banner.png" alt="HYPERRESEARCH" width="700">
+</p>
+
+<h3 align="center">Ultra deep, persistent web research for AI coding agents</h3>
+
+<p align="center">
+  <a href="https://pypi.org/project/hyperresearch/"><img src="https://img.shields.io/pypi/v/hyperresearch" alt="PyPI version"></a>
+  <a href="https://pypi.org/project/hyperresearch/"><img src="https://img.shields.io/pypi/pyversions/hyperresearch" alt="Python 3.11+"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/github/license/jordan-gibbs/hyperresearch" alt="License: MIT"></a>
+  <a href="https://github.com/jordan-gibbs/hyperresearch"><img src="https://img.shields.io/github/stars/jordan-gibbs/hyperresearch?style=social" alt="GitHub stars"></a>
+</p>
+
+---
+
+Your AI agent searches the web, finds great sources, synthesizes an answer — then the session ends and everything is gone. Next time, it starts from zero.
+
+**Hyperresearch makes research persist.** Every source your agent finds is fetched with a real headless browser, saved as searchable markdown, and indexed into a knowledge base that compounds across sessions. Hyperresearch can follow maps of links to digest paths of information that current agent web searching isn't capable of.
+
+```bash
+pip install hyperresearch
+hyperresearch install
+```
+
+## What people use it for
+
+- **In-depth topic research** — "Research the latest advances in state space models" → agent fetches 20+ papers, docs, and blog posts, follows citations to primary sources, builds a linked knowledge graph
+- **News tracking over time** — run research sessions weekly on a topic, each session builds on what's already collected, nothing gets re-fetched
+- **State-of-the-art surveys** — "What's the current SOTA for speech recognition?" → agent goes down rabbit holes, collects benchmarks, papers, and implementations
+- **Competitive analysis** — scrape company pages, LinkedIn profiles, product docs, news articles into a persistent, searchable corpus
+- **Due diligence** — aggregate everything about a person, company, or technology from across the web with authenticated crawling (LinkedIn, Twitter, paywalled sites)
+
+## How it works
+
+1. **You ask your agent to research something**
+2. **Agent searches the web** — multiple queries, different angles
+3. **Fetches every source** with a real headless browser (crawl4ai) — JS rendering, bot detection bypass, login-gated content
+4. **Saves each page** as a searchable markdown note with tags, summary, and source tracking
+5. **Follows links** to primary sources — the paper, not the blog post about the paper
+6. **Auto-links related notes** with `[[wiki-links]]` across the knowledge graph
+7. **Synthesizes findings** into a summary note linking all sources
+8. **Next session** — agent checks the KB before searching the web. Knowledge compounds.
+
+```
+your-repo/
+  .hyperresearch/        # Config + SQLite FTS5 index (rebuildable)
+  research/
+    notes/               # Markdown notes — the source of truth
+    index/               # Auto-generated wiki pages
+  CLAUDE.md              # Agent docs (auto-injected)
+```
+
+## Works with every major agent
+
+`hyperresearch install` hooks into your agent in one step:
+
+| Platform | Hook | Trigger |
+|----------|------|---------|
+| **Claude Code** | `.claude/settings.json` + `/research` skill | Before WebSearch, WebFetch |
+| **Codex** | `.codex/hooks.json` | Before Bash |
+| **Cursor** | `.cursor/rules/hyperresearch.mdc` | Always-apply rule |
+| **Gemini CLI** | `.gemini/settings.json` | Before tool calls |
+
+```bash
+hyperresearch install --platform all    # Hook every platform at once
+```
+
+## Key features
+
+- **Real headless browser** — crawl4ai runs local Chromium. Handles JavaScript, bypasses bot detection, renders SPAs. Not a simple HTTP fetch.
+- **Native PDF extraction** — fetches PDFs directly, extracts full text with pymupdf, saves raw files to `research/raw/`. Academic papers, whitepapers, reports — all first-class citizens.
+- **Authenticated crawling** — log into LinkedIn, Twitter, paywalled news. Your sessions persist across fetches.
+- **Agent-driven curation** — fetcher subagents read each source, write real summaries, add meaningful tags, and flag junk. No keyword-matching shortcuts.
+- **Junk detection** — Cloudflare captchas, error pages, cookie walls, binary garbage, login redirects all caught and rejected before saving.
+- **Multi-round deep research** — the agent does multiple rounds of search → fetch → follow links, with a source checkpoint that forces breadth before synthesis.
+- **Gap analysis + adversarial audit** — after drafting, the agent re-reads the original query to find gaps, then spawns two auditor subagents (comprehensiveness + logic) to tear the draft apart. Runs twice.
+- **Cheap parallel fetching** — ships a Haiku-powered subagent that fetches, summarizes, and quality-checks URLs in parallel for pennies. Spawn 10-20 per round.
+- **Scholarly API guidance** — agent docs encourage use of arXiv, Semantic Scholar, CrossRef, and PubMed APIs for academic topics.
+- **`/research` skill** — scripted deep research workflow. Clarifies ambiguous requests, searches broadly, fetches aggressively, follows rabbit holes, audits, synthesizes.
+- **Smart SPA wait** — polls DOM stability instead of fixed delays. Fast pages finish instantly, SPAs get up to 10 seconds.
+- **FTS5 search** — instant full-text search across thousands of notes with BM25 ranking
+- **Knowledge graph** — `[[wiki-links]]`, backlinks, hub detection, auto-linking
+- **MCP server** — 13 tools (read + write) for Claude Desktop, Cursor, or any MCP client
+- **Note lifecycle** — draft → review → evergreen → stale → deprecated → archive
+
+## Commands
+
+```bash
+# Research
+hyperresearch fetch <url> --tag t -j           # Fetch a URL into the KB
+hyperresearch fetch-batch <urls...> -j         # Fetch many URLs at once
+hyperresearch research "topic" --max 5 -j      # Full pipeline: search → fetch → link → synthesize
+
+# Search
+hyperresearch search "query" -j                # Full-text search
+hyperresearch search "query" --max-tokens 8000 # Stay within context budget
+hyperresearch note show <id> -j                # Read a note
+
+# Knowledge graph
+hyperresearch link --auto -j                   # Auto-link related notes
+hyperresearch graph hubs -j                    # Most-connected notes
+hyperresearch graph backlinks <id> -j          # What links to this note
+
+# Manage
+hyperresearch sources list -j                  # Every URL ever fetched
+hyperresearch lint -j                          # Health check
+hyperresearch repair -j                        # Fix links, rebuild indexes
+```
+
+Every command returns `{"ok": true, "data": {...}}` with `-j`.
+
+## Authenticated crawling
+
+Fetch from LinkedIn, Twitter, paywalled sites — anything you can log into:
+
+```bash
+hyperresearch setup       # Browser opens. Log into your sites. Done.
+```
+
+```toml
+# .hyperresearch/config.toml
+[web]
+provider = "crawl4ai"
+profile = "research"
+```
+
+LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a visible browser to avoid session kills.
+
+## Philosophy
+
+- **No LLM calls.** Hyperresearch stores, indexes, and searches. Your agent is the LLM.
+- **Markdown is truth.** Notes are plain files. SQLite is a rebuildable cache.
+- **Over-collect, then prune.** Fetch aggressively. Deprecate what you don't need.
+- **Check before you fetch.** Hooks kill redundant searches across sessions.
+- **Raw content is king.** Save the original with formatting, not a rewritten summary.
+
+## Requirements
+
+- Python 3.11+
+- Windows, macOS, Linux
+
+## License
+
+[MIT](LICENSE)
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=jordan-gibbs/hyperresearch&type=Date)](https://star-history.com/#jordan-gibbs/hyperresearch&Date)

hyperresearch-0.4.0/README.md

@@ -0,0 +1,148 @@
+<p align="center">
+  <img src="assets/banner.png" alt="HYPERRESEARCH" width="700">
+</p>
+
+<h3 align="center">Ultra deep, persistent web research for AI coding agents</h3>
+
+<p align="center">
+  <a href="https://pypi.org/project/hyperresearch/"><img src="https://img.shields.io/pypi/v/hyperresearch" alt="PyPI version"></a>
+  <a href="https://pypi.org/project/hyperresearch/"><img src="https://img.shields.io/pypi/pyversions/hyperresearch" alt="Python 3.11+"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/github/license/jordan-gibbs/hyperresearch" alt="License: MIT"></a>
+  <a href="https://github.com/jordan-gibbs/hyperresearch"><img src="https://img.shields.io/github/stars/jordan-gibbs/hyperresearch?style=social" alt="GitHub stars"></a>
+</p>
+
+---
+
+Your AI agent searches the web, finds great sources, synthesizes an answer — then the session ends and everything is gone. Next time, it starts from zero.
+
+**Hyperresearch makes research persist.** Every source your agent finds is fetched with a real headless browser, saved as searchable markdown, and indexed into a knowledge base that compounds across sessions. Hyperresearch can follow maps of links to digest paths of information that current agent web searching isn't capable of.
+
+```bash
+pip install hyperresearch
+hyperresearch install
+```
+
+## What people use it for
+
+- **In-depth topic research** — "Research the latest advances in state space models" → agent fetches 20+ papers, docs, and blog posts, follows citations to primary sources, builds a linked knowledge graph
+- **News tracking over time** — run research sessions weekly on a topic, each session builds on what's already collected, nothing gets re-fetched
+- **State-of-the-art surveys** — "What's the current SOTA for speech recognition?" → agent goes down rabbit holes, collects benchmarks, papers, and implementations
+- **Competitive analysis** — scrape company pages, LinkedIn profiles, product docs, news articles into a persistent, searchable corpus
+- **Due diligence** — aggregate everything about a person, company, or technology from across the web with authenticated crawling (LinkedIn, Twitter, paywalled sites)
+
+## How it works
+
+1. **You ask your agent to research something**
+2. **Agent searches the web** — multiple queries, different angles
+3. **Fetches every source** with a real headless browser (crawl4ai) — JS rendering, bot detection bypass, login-gated content
+4. **Saves each page** as a searchable markdown note with tags, summary, and source tracking
+5. **Follows links** to primary sources — the paper, not the blog post about the paper
+6. **Auto-links related notes** with `[[wiki-links]]` across the knowledge graph
+7. **Synthesizes findings** into a summary note linking all sources
+8. **Next session** — agent checks the KB before searching the web. Knowledge compounds.
+
+```
+your-repo/
+  .hyperresearch/        # Config + SQLite FTS5 index (rebuildable)
+  research/
+    notes/               # Markdown notes — the source of truth
+    index/               # Auto-generated wiki pages
+  CLAUDE.md              # Agent docs (auto-injected)
+```
+
+## Works with every major agent
+
+`hyperresearch install` hooks into your agent in one step:
+
+| Platform | Hook | Trigger |
+|----------|------|---------|
+| **Claude Code** | `.claude/settings.json` + `/research` skill | Before WebSearch, WebFetch |
+| **Codex** | `.codex/hooks.json` | Before Bash |
+| **Cursor** | `.cursor/rules/hyperresearch.mdc` | Always-apply rule |
+| **Gemini CLI** | `.gemini/settings.json` | Before tool calls |
+
+```bash
+hyperresearch install --platform all    # Hook every platform at once
+```
+
+## Key features
+
+- **Real headless browser** — crawl4ai runs local Chromium. Handles JavaScript, bypasses bot detection, renders SPAs. Not a simple HTTP fetch.
+- **Native PDF extraction** — fetches PDFs directly, extracts full text with pymupdf, saves raw files to `research/raw/`. Academic papers, whitepapers, reports — all first-class citizens.
+- **Authenticated crawling** — log into LinkedIn, Twitter, paywalled news. Your sessions persist across fetches.
+- **Agent-driven curation** — fetcher subagents read each source, write real summaries, add meaningful tags, and flag junk. No keyword-matching shortcuts.
+- **Junk detection** — Cloudflare captchas, error pages, cookie walls, binary garbage, login redirects all caught and rejected before saving.
+- **Multi-round deep research** — the agent does multiple rounds of search → fetch → follow links, with a source checkpoint that forces breadth before synthesis.
+- **Gap analysis + adversarial audit** — after drafting, the agent re-reads the original query to find gaps, then spawns two auditor subagents (comprehensiveness + logic) to tear the draft apart. Runs twice.
+- **Cheap parallel fetching** — ships a Haiku-powered subagent that fetches, summarizes, and quality-checks URLs in parallel for pennies. Spawn 10-20 per round.
+- **Scholarly API guidance** — agent docs encourage use of arXiv, Semantic Scholar, CrossRef, and PubMed APIs for academic topics.
+- **`/research` skill** — scripted deep research workflow. Clarifies ambiguous requests, searches broadly, fetches aggressively, follows rabbit holes, audits, synthesizes.
+- **Smart SPA wait** — polls DOM stability instead of fixed delays. Fast pages finish instantly, SPAs get up to 10 seconds.
+- **FTS5 search** — instant full-text search across thousands of notes with BM25 ranking
+- **Knowledge graph** — `[[wiki-links]]`, backlinks, hub detection, auto-linking
+- **MCP server** — 13 tools (read + write) for Claude Desktop, Cursor, or any MCP client
+- **Note lifecycle** — draft → review → evergreen → stale → deprecated → archive
+
+## Commands
+
+```bash
+# Research
+hyperresearch fetch <url> --tag t -j           # Fetch a URL into the KB
+hyperresearch fetch-batch <urls...> -j         # Fetch many URLs at once
+hyperresearch research "topic" --max 5 -j      # Full pipeline: search → fetch → link → synthesize
+
+# Search
+hyperresearch search "query" -j                # Full-text search
+hyperresearch search "query" --max-tokens 8000 # Stay within context budget
+hyperresearch note show <id> -j                # Read a note
+
+# Knowledge graph
+hyperresearch link --auto -j                   # Auto-link related notes
+hyperresearch graph hubs -j                    # Most-connected notes
+hyperresearch graph backlinks <id> -j          # What links to this note
+
+# Manage
+hyperresearch sources list -j                  # Every URL ever fetched
+hyperresearch lint -j                          # Health check
+hyperresearch repair -j                        # Fix links, rebuild indexes
+```
+
+Every command returns `{"ok": true, "data": {...}}` with `-j`.
+
+## Authenticated crawling
+
+Fetch from LinkedIn, Twitter, paywalled sites — anything you can log into:
+
+```bash
+hyperresearch setup       # Browser opens. Log into your sites. Done.
+```
+
+```toml
+# .hyperresearch/config.toml
+[web]
+provider = "crawl4ai"
+profile = "research"
+```
+
+LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a visible browser to avoid session kills.
+
+## Philosophy
+
+- **No LLM calls.** Hyperresearch stores, indexes, and searches. Your agent is the LLM.
+- **Markdown is truth.** Notes are plain files. SQLite is a rebuildable cache.
+- **Over-collect, then prune.** Fetch aggressively. Deprecate what you don't need.
+- **Check before you fetch.** Hooks kill redundant searches across sessions.
+- **Raw content is king.** Save the original with formatting, not a rewritten summary.
+
+## Requirements
+
+- Python 3.11+
+- Windows, macOS, Linux
+
+## License
+
+[MIT](LICENSE)
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=jordan-gibbs/hyperresearch&type=Date)](https://star-history.com/#jordan-gibbs/hyperresearch&Date)

{hyperresearch-0.2.0 → hyperresearch-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hyperresearch"
-version = "0.2.0"
+version = "0.4.0"
 description = "Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki."
 readme = "README.md"
 license = "MIT"
@@ -28,12 +28,14 @@ dependencies = [
     "pydantic>=2.0",
     "jinja2>=3.1",
     "platformdirs>=4.0",
+    "Crawl4AI>=0.4",
+    "pymupdf>=1.24",
+    "httpx>=0.27",
 ]
 
 [project.optional-dependencies]
-crawl4ai = ["Crawl4AI>=0.4"]
 mcp = ["mcp>=1.6"]
-all = ["hyperresearch[crawl4ai,mcp]"]
+all = ["hyperresearch[mcp]"]
 dev = [
     "pytest>=7.4",
     "pytest-cov>=4.1",

{hyperresearch-0.2.0 → hyperresearch-0.4.0}/src/hyperresearch/__init__.py

@@ -1,3 +1,3 @@
 """Hyperresearch — agent-driven research knowledge base."""
 
-__version__ = "0.2.0"
+__version__ = "0.4.0"

{hyperresearch-0.2.0 → hyperresearch-0.4.0}/src/hyperresearch/cli/fetch.py

@@ -116,6 +116,16 @@ def fetch(
             console.print(f"[red]Auth required:[/] {msg}")
         raise typer.Exit(1)
 
+    # Detect junk pages — captcha, error pages, binary garbage, empty content
+    junk_reason = result.looks_like_junk()
+    if junk_reason:
+        msg = f"Skipped junk content from {url}: {junk_reason}"
+        if json_output:
+            output(error(msg, "JUNK_CONTENT"), json_mode=True)
+        else:
+            console.print(f"[yellow]Skipped:[/] {msg}")
+        raise typer.Exit(1)
+
     # Write note
     note_title = title or result.title or urlparse(url).path.split("/")[-1] or "Untitled"
     domain = result.domain
@@ -140,10 +150,40 @@ def fetch(
         extra_frontmatter=extra_meta,
     )
 
-    # Auto-enrich: add suggested tags and summary before sync
-    from hyperresearch.core.enrich import enrich_note_file
-
-    enrich_note_file(note_path, conn, tags)
+    # Save raw file (PDF, etc.) if present
+    raw_file_path = None
+    if result.raw_bytes and result.raw_content_type:
+        ext_map = {
+            "application/pdf": ".pdf",
+            "image/png": ".png",
+            "image/jpeg": ".jpg",
+            "image/gif": ".gif",
+            "image/webp": ".webp",
+        }
+        ext = ext_map.get(result.raw_content_type, "")
+        if ext:
+            raw_dir = vault.root / "research" / "raw"
+            raw_dir.mkdir(parents=True, exist_ok=True)
+            raw_filename = note_path.stem + ext
+            raw_file = raw_dir / raw_filename
+            raw_file.write_bytes(result.raw_bytes)
+            raw_file_path = f"raw/{raw_filename}"
+
+    # Note: tagging and summarization is the agent's job, not an automatic process.
+    # The agent reads the fetched content and writes meaningful summaries and tags.
+
+    # Add raw_file reference to frontmatter AFTER enrich (enrich rewrites frontmatter)
+    if raw_file_path:
+        note_text = note_path.read_text(encoding="utf-8")
+        if note_text.startswith("---") and "raw_file:" not in note_text:
+            end_idx = note_text.find("---", 3)
+            if end_idx != -1:
+                note_text = (
+                    note_text[:end_idx]
+                    + f"raw_file: {raw_file_path}\n"
+                    + note_text[end_idx:]
+                )
+                note_path.write_text(note_text, encoding="utf-8")
 
     # Sync first so the note exists in the notes table (needed for FK on sources/assets)
     note_id = note_path.stem
@@ -175,6 +215,7 @@ def fetch(
         "path": str(note_path.relative_to(vault.root)),
         "word_count": len(result.content.split()),
         "assets": saved_assets,
+        "raw_file": raw_file_path,
     }
 
     if json_output: