hyperresearch 0.2.0__tar.gz

hyperresearch-0.2.0/.github/workflows/ci.yml

@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: ruff check src/ tests/
+      - run: pytest tests/ -v

hyperresearch-0.2.0/.github/workflows/publish.yml

@@ -0,0 +1,47 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install build tools
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Upload artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+
+    steps:
+      - name: Download artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

hyperresearch-0.2.0/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.venv/
+.idea/
+*.db
+.hyperresearch/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+*.egg

hyperresearch-0.2.0/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+## [0.2.0] - 2026-04-10
+
+### New
+
+- **`/research` skill** — Scripted deep research workflow as a Claude Code slash command. Clarifies ambiguous requests, searches broadly, fetches aggressively, follows rabbit holes, auto-curates, synthesizes, and presents findings with hub notes
+- **`hyperresearch setup`** — Interactive TUI onboarding: web provider, browser profile selection/creation, agent hooks. Auto-launches on first `install`
+- **`hyperresearch fetch-batch`** — Concurrent multi-URL fetch with batched sync (O(1) syncs instead of O(n))
+- **`hyperresearch link --auto`** — Holistic auto-linking: scans notes for mentions of other notes' titles and appends wiki-links
+- **`hyperresearch assets list/path`** — Browse downloaded screenshots and images
+- **`--save-assets` flag** — Opt-in screenshot + content image download on fetch
+- **`--visible` flag** — Non-headless browser for stubborn auth sites (auto-enabled for LinkedIn, Twitter, Facebook, Instagram, TikTok)
+- **`--max-tokens` on search** — Token budget truncation for context-aware agents
+- **Auto-curation at fetch time** — Notes arrive with auto-generated tags and summaries
+- **MCP write tools** — `fetch_url`, `create_note`, `update_note` (MCP server is now read-write)
+- **MinHash+LSH dedup** — O(n) approximate dedup for large vaults (200+ notes), falls back to brute-force for small vaults
+- **Hub notes auto-surfaced** after research sessions
+- **Synthesis notes** saved as feedback loop (agent Q&A becomes searchable)
+- **`hyperresearch-fetcher` subagent** — Haiku-powered URL fetcher installed to `.claude/agents/`
+- **Login wall detection** — `AUTH_REQUIRED` error instead of saving login page junk
+- **Smart SPA wait** — Polls DOM stability (2s initial + 10s ceiling) instead of fixed delays
+
+### Changed
+
+- **crawl4ai is the sole browser provider** — Removed firecrawl, tavily, trafilatura
+- **crawl4ai v0.8.x API** — AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, arun/arun_many
+- **Authenticated crawling** via crawl4ai browser profiles (`crwl profiles` or setup TUI)
+- **CLI path baked into CLAUDE.md** — Works without venv activation (forward slashes for Windows bash)
+- **Deep research philosophy** — Agent docs say "over-collect, then prune" and "go down rabbit holes"
+- **Windows encoding fix** — `stream.reconfigure(encoding="utf-8")` at startup, no more charmap crashes
+- **Note slugs capped at 80 chars** — Avoids Windows MAX_PATH issues
+- **Anti-bot stealth always on** when crawl4ai is used (no setup question)
+- **Config commands** now support `web.provider`, `web.profile`, `web.magic`
+
+### Removed
+
+- Dead fields: `confidence`, `superseded_by`, `llm_compiled`, `llm_model`, `compile_source`
+- Tag plural normalization (use explicit `tag_aliases` instead)
+- `deprecated-no-successor` and `low-confidence` lint rules
+- Firecrawl, Tavily, Trafilatura web providers
+
+## [0.1.0] - 2026-04-09
+
+Initial release. Forked from [llm-kasten](https://github.com/jordan-gibbs/llm-kasten) and repositioned for agent-driven research workflows.
+
+### New
+
+- **`hyperresearch install`** — One-step setup: init vault + inject agent docs + install PreToolUse hooks for Claude Code, Codex, Cursor, Gemini CLI
+- **`hyperresearch fetch <url>`** — Fetch a URL, extract content, save as a research note with source tracking
+- **`hyperresearch research <topic>`** — Deep research: web search, fetch results, follow links, save as linked notes, generate synthesis MOC
+- **`hyperresearch sources list/check`** — List and query fetched web sources
+- **Web provider plugin system** — Pluggable backends: builtin (stdlib), crawl4ai (local headless browser)
+- **Agent hook system** — PreToolUse hooks that remind agents to check the research base before web searches
+- **Sources table** — URL deduplication, domain tracking, fetch metadata
+- **Extended frontmatter** — `source_domain`, `fetched_at`, `fetch_provider` fields
+- **MCP server** with 10 tools including `check_source` and `list_sources`
+
+### From kasten (the backbone)
+
+- SQLite FTS5 full-text search with BM25 ranking
+- Markdown notes with YAML frontmatter as source of truth
+- `[[wiki-link]]` tracking with backlinks
+- `--json` / `-j` structured output on every command
+- Note lifecycle: draft → review → evergreen → stale → deprecated → archive
+- Auto-sync (mtime + SHA-256 change detection)
+- Agent doc injection (CLAUDE.md, AGENTS.md, GEMINI.md, copilot-instructions.md)
+- Web viewer with force-directed knowledge graph
+- 70 tests
+
+[0.2.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.2.0
+[0.1.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.1.0

hyperresearch-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,76 @@
+# Contributing to kasten
+
+Thank you for your interest in contributing.
+
+## Development setup
+
+```bash
+git clone https://github.com/jordan-gibbs/kasten.git
+cd kasten
+python -m venv .venv
+source .venv/bin/activate  # or .venv\Scripts\activate on Windows
+pip install -e ".[dev]"
+```
+
+## Running tests
+
+```bash
+python -m pytest tests/ -v
+```
+
+All tests must pass before submitting a PR. The test suite runs in under 30 seconds.
+
+## Code style
+
+This project uses [ruff](https://docs.astral.sh/ruff/) for linting:
+
+```bash
+ruff check src/
+ruff format src/
+```
+
+Configuration is in `pyproject.toml`. Line length is 100 characters.
+
+## Type checking
+
+```bash
+mypy src/kasten/
+```
+
+Strict mode is enabled in `pyproject.toml`.
+
+## Submitting changes
+
+1. Fork the repository.
+2. Create a branch from `main`.
+3. Make your changes with tests.
+4. Run `ruff check src/` and `python -m pytest tests/`.
+5. Open a pull request against `main`.
+
+Keep PRs focused on a single change. Include a clear description of what changed and why.
+
+## Reporting issues
+
+Open an issue at [github.com/jordan-gibbs/kasten/issues](https://github.com/jordan-gibbs/kasten/issues). Include:
+
+- What you expected to happen
+- What actually happened
+- Steps to reproduce
+- kasten version (`kasten --version`)
+- Python version and OS
+
+## Architecture overview
+
+- `src/kasten/core/` -- vault management, sync engine, frontmatter parsing, SQLite schema
+- `src/kasten/cli/` -- typer CLI commands, output formatting
+- `src/kasten/search/` -- FTS5 search engine, filters, ranking
+- `src/kasten/models/` -- Pydantic models for notes, output envelopes
+- `src/kasten/graph/` -- link parsing (shared patterns)
+- `src/kasten/indexgen/` -- auto-generated index page builder
+- `src/kasten/ingest/` -- file, web, PDF ingestion
+- `src/kasten/compile/` -- LLM compilation pipeline
+- `src/kasten/llm/` -- provider abstraction (OpenAI, Anthropic, Ollama)
+- `src/kasten/serve/` -- lightweight web UI server
+- `tests/` -- pytest test suite
+
+The key design principle: markdown files are the source of truth, SQLite is a derived cache that can be rebuilt at any time with `kasten sync --force`.

hyperresearch-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jordan Gibbs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hyperresearch-0.2.0/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: hyperresearch
+Version: 0.2.0
+Summary: Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki.
+Project-URL: Homepage, https://github.com/jordan-gibbs/hyperresearch
+Project-URL: Repository, https://github.com/jordan-gibbs/hyperresearch
+Project-URL: Issues, https://github.com/jordan-gibbs/hyperresearch/issues
+Author-email: Jordan Gibbs <jordan@jordangibbs.xyz>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,cli,knowledge-base,llm,markdown,research,wiki,zettelkasten
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Requires-Python: >=3.11
+Requires-Dist: jinja2>=3.1
+Requires-Dist: platformdirs>=4.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.9.0
+Provides-Extra: all
+Requires-Dist: crawl4ai>=0.4; extra == 'all'
+Requires-Dist: mcp>=1.6; extra == 'all'
+Provides-Extra: crawl4ai
+Requires-Dist: crawl4ai>=0.4; extra == 'crawl4ai'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pytest-cov>=4.1; extra == 'dev'
+Requires-Dist: pytest>=7.4; extra == 'dev'
+Requires-Dist: ruff>=0.3; extra == 'dev'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.6; extra == 'mcp'
+Description-Content-Type: text/markdown
+
+# Hyperresearch
+
+Agent-driven research knowledge base. Install it, and your AI coding agent can collect, search, and synthesize web research into a persistent, searchable wiki — across sessions.
+
+```bash
+pip install hyperresearch
+hyperresearch install    # init vault + hook your agent
+```
+
+That's it. Your agent now checks the research base before searching the web, saves useful findings automatically, and builds a knowledge graph over time.
+
+## How it works
+
+1. **Agent finds something useful** (via its own web search, browsing, or your input)
+2. **Agent saves it**: `hyperresearch fetch "https://..." --tag ml -j` or `hyperresearch note new "Title" --body-file content.md -j`
+3. **Next time it needs info**, the PreToolUse hook reminds it: *"check hyperresearch first"*
+4. **Agent searches the KB**: `hyperresearch search "attention mechanisms" -j`
+5. **Knowledge compounds** across sessions — no redundant fetches, no lost context
+
+```
+your-repo/
+  .hyperresearch/        # Hidden: config, SQLite index, hook script
+  research/
+    notes/               # Markdown notes (source of truth)
+    index/               # Auto-generated wiki pages
+  CLAUDE.md              # Agent docs (auto-injected)
+```
+
+## Commands
+
+```bash
+# Setup
+hyperresearch install                        # Init + hooks (Claude Code, Cursor, Codex, Gemini)
+hyperresearch install --platform all         # Hook all supported platforms
+
+# Collect
+hyperresearch fetch <url> --tag t -j         # Save a URL as a note
+hyperresearch research "topic" --max 5 -j    # Search → fetch → link → synthesize (needs crawl4ai)
+
+# Search & read
+hyperresearch search "query" -j              # Full-text search
+hyperresearch note show <id> -j              # Read a note
+hyperresearch note list --tag ml -j          # List notes by tag
+
+# Manage
+hyperresearch sources list -j                # What URLs have been fetched
+hyperresearch sources check <url> -j         # Has this URL been fetched?
+hyperresearch repair -j                      # Fix links, promote notes, rebuild indexes
+hyperresearch status -j                      # Vault health overview
+```
+
+Every command returns `{"ok": true, "data": {...}}` with `-j`.
+
+## Agent integration
+
+`hyperresearch install` does three things:
+
+1. **Creates the vault** (`.hyperresearch/` + `research/`)
+2. **Injects usage docs** into CLAUDE.md (or AGENTS.md, GEMINI.md, copilot-instructions.md)
+3. **Installs PreToolUse hooks** that fire before web searches:
+
+| Platform | Hook | Trigger |
+|----------|------|---------|
+| Claude Code | `.claude/settings.json` | Before Glob, Grep, WebSearch, WebFetch |
+| Codex | `.codex/hooks.json` | Before Bash |
+| Cursor | `.cursor/rules/hyperresearch.mdc` | Always-apply rule |
+| Gemini CLI | `.gemini/settings.json` | Before tool calls |
+
+The hook doesn't block — it reminds the agent to check the research base first.
+
+## Web providers
+
+By default, agents use their own web tools (WebSearch, WebFetch) and pipe content into hyperresearch. For JS-rendered pages, blocked sites, or authenticated content, install crawl4ai (local headless Chromium):
+
+```bash
+pip install hyperresearch[crawl4ai]
+crawl4ai-setup                        # Install browser (one-time)
+```
+
+Configure in `.hyperresearch/config.toml`:
+```toml
+[web]
+provider = "crawl4ai"    # or "builtin" (stdlib urllib, no JS)
+profile = ""             # Browser profile name for authenticated crawling (optional)
+magic = false            # Anti-bot stealth mode (recommended for social media)
+```
+
+### Authenticated crawling
+
+Access login-gated content (LinkedIn, Twitter, paywalled sites) by creating a login profile:
+
+```bash
+hyperresearch setup       # Choose option 1 — browser opens, log into your sites, done
+# Or manually:
+crwl profiles             # Create profile, log in, press q when done
+```
+
+```toml
+# .hyperresearch/config.toml
+[web]
+profile = "research"      # Your profile name
+```
+
+## MCP server
+
+For Claude Desktop, Cursor inline, or any MCP-compatible agent:
+
+```bash
+pip install hyperresearch[mcp]
+```
+
+```json
+{"mcpServers": {"hyperresearch": {"command": "hyperresearch", "args": ["mcp"]}}}
+```
+
+10 tools: `search_notes`, `read_note`, `read_many`, `list_notes`, `get_backlinks`, `get_hubs`, `vault_status`, `lint_vault`, `check_source`, `list_sources`.
+
+## Philosophy
+
+- **The agent IS the LLM** — hyperresearch is a dumb tool that stores, indexes, and searches. It never calls an LLM.
+- **Files are truth** — markdown notes survive the tool dying. SQLite is a rebuildable cache.
+- **Agents already have web access** — hyperresearch is where they *store* what they find, not how they find it.
+- **Check before you fetch** — the hook system prevents redundant web searches across sessions.
+
+## Requirements
+
+- Python 3.11+
+- Works on Windows, macOS, Linux
+
+## License
+
+[MIT](LICENSE)