claude-local-docs 1.0.2 → 1.0.13

package/.claude-plugin/marketplace.json

@@ -1,22 +1,42 @@
 {
   "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
   "name": "claude-local-docs",
-  "description": "A local-first Context7 alternative. Indexes your JS/TS project dependency docs locally with a 4-stage RAG pipeline (vector + BM25 + RRF + cross-encoder reranking). No cloud APIs at query time — all models run on your machine.",
+  "description": "Local-first Context7 alternative — indexes JS/TS dependency docs with a 4-stage RAG pipeline. GPU-accelerated via TEI Docker containers.",
   "owner": {
-    "name": "matthew"
+    "name": "matthew",
+    "email": "matteodante@users.noreply.github.com"
+  },
+  "metadata": {
+    "version": "1.0.7"
   },
   "plugins": [
     {
       "name": "claude-local-docs",
-      "description": "A local-first, offline-capable alternative to Context7 for dependency documentation. Reads your package.json, fetches docs (preferring llms.txt), and indexes them with a production-grade 4-stage RAG pipeline: vector search + BM25 keywords + RRF fusion + cross-encoder reranking. All models (nomic-embed-text-v1.5, ms-marco-MiniLM) run locally via ONNX — no cloud API calls at query time, no rate limits, full privacy.",
-      "version": "1.0.0",
+      "description": "Offline-capable documentation search for JS/TS projects. Reads your package.json, fetches docs (preferring llms.txt), and indexes them with a 4-stage RAG pipeline: vector search + BM25 keywords + RRF fusion + cross-encoder reranking. Embeddings and reranking run via TEI (HuggingFace Text Embeddings Inference) Docker containers with auto GPU detection (NVIDIA CUDA, Apple Metal).",
+      "version": "1.0.7",
       "author": {
         "name": "matthew"
       },
       "source": "./",
       "category": "development",
       "license": "MIT",
-      "keywords": ["documentation", "search", "rag", "embeddings", "local-first", "semantic-search", "llms-txt", "context7-alternative", "offline", "vector-search", "bm25", "reranking", "dependency-docs"]
+      "keywords": [
+        "documentation",
+        "search",
+        "rag",
+        "embeddings",
+        "local-first",
+        "semantic-search",
+        "llms-txt",
+        "context7-alternative",
+        "offline",
+        "vector-search",
+        "bm25",
+        "reranking",
+        "dependency-docs",
+        "docker",
+        "tei"
+      ]
     }
   ]
 }

package/.claude-plugin/plugin.json

@@ -1,8 +1,29 @@
 {
   "name": "claude-local-docs",
-  "description": "A local-first Context7 alternative. Indexes JS/TS dependency docs with a 4-stage RAG pipeline (vector + BM25 + reranking). All models run locally — no cloud APIs at query time.",
-  "version": "1.0.0",
+  "version": "1.0.7",
+  "description": "Local-first Context7 alternative — indexes JS/TS dependency docs with a 4-stage RAG pipeline (vector + BM25 + RRF + cross-encoder reranking). Uses TEI Docker containers for GPU-accelerated embeddings and reranking.",
   "author": {
-    "name": "matthew"
-  }
+    "name": "matthew",
+    "url": "https://github.com/matteodante"
+  },
+  "homepage": "https://github.com/matteodante/claude-local-docs",
+  "repository": "https://github.com/matteodante/claude-local-docs",
+  "license": "MIT",
+  "keywords": [
+    "documentation",
+    "search",
+    "rag",
+    "embeddings",
+    "local-first",
+    "semantic-search",
+    "llms-txt",
+    "context7-alternative",
+    "offline",
+    "vector-search",
+    "bm25",
+    "reranking",
+    "dependency-docs",
+    "docker",
+    "tei"
+  ]
 }

package/.mcp.json

@@ -1,6 +1,12 @@
 {
-  "local-docs": {
-    "command": "npx",
-    "args": ["-y", "claude-local-docs"]
+  "mcpServers": {
+    "local-docs": {
+      "command": "npx",
+      "args": ["-y", "claude-local-docs@latest"],
+      "env": {
+        "TEI_EMBED_URL": "http://localhost:39281",
+        "TEI_RERANK_URL": "http://localhost:39282"
+      }
+    }
   }
 }

package/README.md

@@ -1,57 +1,91 @@
 # claude-local-docs
 
-A local-first alternative to Context7 for Claude Code. Indexes your project's dependency documentation locally and provides production-grade semantic search — no cloud APIs at query time, no rate limits, full privacy.
+A local-first alternative to Context7 for Claude Code. Indexes your project's dependency documentation locally and provides production-grade semantic search. Embeddings and reranking run via TEI (HuggingFace Text Embeddings Inference) Docker containers with auto GPU detection.
 
 ## Why not Context7?
 
 | | **claude-local-docs** | **Context7** |
 |---|---|---|
-| **Runs where** | Your machine (ONNX models) | Upstash cloud servers |
+| **Runs where** | Your machine (TEI Docker) | Upstash cloud servers |
 | **Privacy** | Docs never leave your machine | Queries sent to cloud API |
 | **Rate limits** | None | API-dependent |
 | **Offline** | Full search works offline | Requires internet |
+| **GPU accelerated** | NVIDIA CUDA / Apple Metal | N/A |
 | **Search quality** | 4-stage RAG (vector + BM25 + RRF + cross-encoder reranking) | Single-stage retrieval |
 | **Doc sources** | Prefers llms.txt, falls back to official docs | Pre-indexed source repos |
 | **Scope** | Your project's actual dependencies | Any library |
-| **Setup** | `npm install` + `/fetch-docs` | Install plugin |
 | **Monorepo** | Detects pnpm/npm/yarn workspaces, resolves catalogs | N/A |
 
-## How it works
+## Prerequisites
 
-```
-/fetch-docs                        search_docs("how to use useState")
-     │                                       │
-     ▼                                       ▼
- Detect monorepo              ┌─── Vector search (LanceDB) ───┐
- Scan all workspace pkgs      │    nomic-embed-text-v1.5       │
- Resolve catalog: versions    │                                │
-     │                        │                                ├─→ RRF Fusion
-     ▼                        │                                │    (k=60)
- For each runtime dep:        ├── BM25 search (LanceDB FTS) ──┘
-   - Search for llms.txt      │    keyword + stemming            │
-   - Raw fetch (no truncation)│                                  ▼
-   - Chunk + embed + store    │                        Cross-encoder rerank
-                              │                        ms-marco-MiniLM-L-6-v2
-                              │                                  │
-                              └──────────────────────────────────┘
-                                                  │
-                                                  ▼
-                                          Top-K results
-```
+- **Docker** — [Docker Desktop](https://www.docker.com/products/docker-desktop/) for TEI containers
+- **Node.js 20+**
+- **NVIDIA GPU** (optional) — auto-detected, uses architecture-optimized TEI images
+- **Apple Silicon** (optional) — native Metal build via Rust/cargo (no Docker needed)
 
 ## Installation
 
+### As a Claude Code MCP server (recommended)
+
+Add this to your project's `.mcp.json` (or global `~/.claude/mcp.json`):
+
+```json
+{
+  "mcpServers": {
+    "local-docs": {
+      "command": "npx",
+      "args": ["-y", "claude-local-docs@latest"],
+      "env": {
+        "TEI_EMBED_URL": "http://localhost:39281",
+        "TEI_RERANK_URL": "http://localhost:39282"
+      }
+    }
+  }
+}
+```
+
+Then start the TEI containers (clone the repo or download `start-tei.sh` + `docker-compose.yml`):
+
 ```bash
-# Clone into your Claude Code plugins directory
-git clone <repo-url> ~/.claude/plugins/claude-local-docs
+./start-tei.sh
+```
+
+The plugin includes a SessionStart hook that auto-checks TEI health and starts containers if needed.
 
-# Install dependencies and build
-cd ~/.claude/plugins/claude-local-docs
+### Manual / development setup
+
+```bash
+git clone https://github.com/matteodante/claude-local-docs.git
+cd claude-local-docs
 npm install
 npm run build
+
+# Start TEI (auto-detects GPU)
+./start-tei.sh
 ```
 
-Or install as a project-local plugin by cloning into your project and referencing it in your Claude Code settings.
+## How it works
+
+```
+/fetch-docs                        search_docs("how to use useState")
+     |                                       |
+     v                                       v
+ Detect monorepo              +--- Vector search (LanceDB) ---+
+ Scan all workspace pkgs      |    nomic-embed-text-v1.5       |
+ Resolve catalog: versions    |                                |
+     |                        |                                +-> RRF Fusion
+     v                        |                                |    (k=60)
+ For each runtime dep:        +-- BM25 search (LanceDB FTS) --+
+   - Search for llms.txt      |    keyword + stemming            |
+   - Raw fetch (no truncation)|                                  v
+   - Chunk + embed + store    |                        Cross-encoder rerank
+                              |                        ms-marco-MiniLM-L-6-v2
+                              |                          (via TEI :39282)
+                              +----------------------------------+
+                                                  |
+                                                  v
+                                          Top-K results
+```
 
 ## Usage
 
@@ -61,7 +95,7 @@ Or install as a project-local plugin by cloning into your project and referencin
 /fetch-docs
 ```
 
-Claude analyzes your project (including monorepo workspaces), finds all runtime dependencies, searches the web for the best documentation for each one (preferring `llms-full.txt` > `llms.txt` > official docs), and indexes everything locally. Progress is reported one library at a time.
+Claude analyzes your project (including monorepo workspaces), finds all runtime dependencies, searches the web for the best documentation for each one (preferring `llms-full.txt` > `llms.txt` > official docs), and indexes everything locally.
 
 ### 2. Search
 
@@ -79,39 +113,51 @@ Show me the API for zod's .refine()
 - **`get_doc_section`** — Retrieve specific sections by heading or chunk ID
 - **`analyze_dependencies`** — List all deps (monorepo-aware, catalog-resolved, runtime/dev tagged)
 - **`fetch_and_store_doc`** — Fetch a URL and index it directly (no AI truncation)
+- **`discover_and_fetch_docs`** — Auto-discover and index docs for a library (probes npm, llms.txt, GitHub, homepage)
 
-## Search pipeline
-
-This plugin implements a 4-stage advanced RAG pipeline, the current production standard:
+## TEI backend
 
-| Stage | Technology | Purpose |
-|---|---|---|
-| **Vector search** | LanceDB + nomic-embed-text-v1.5 | Semantic similarity (understands meaning) |
-| **BM25 search** | LanceDB native FTS (BM25, stemming, stop words) | Keyword matching (exact terms like `useEffect`) |
-| **RRF fusion** | Reciprocal Rank Fusion (k=60) | Merges both ranked lists, handles different score scales |
-| **Cross-encoder rerank** | ms-marco-MiniLM-L-6-v2 | Rescores top 30 candidates with deep relevance model |
+ML inference runs in TEI (HuggingFace Text Embeddings Inference) containers:
 
-### Why this matters
+| Container | Port | Model | Purpose |
+|---|---|---|---|
+| tei-embed | `:39281` | `nomic-ai/nomic-embed-text-v1.5` | Text embeddings (384-dim Matryoshka) |
+| tei-rerank | `:39282` | `cross-encoder/ms-marco-MiniLM-L-6-v2` | Cross-encoder reranking |
 
-- **Vector-only** search misses exact API names and error codes
-- **Keyword-only** search misses semantic meaning ("state management" won't find "useState")
-- **Hybrid + reranking** catches both, then a cross-encoder picks the truly relevant results
+### Starting TEI
 
-## Models
+```bash
+./start-tei.sh           # Auto-detect GPU
+./start-tei.sh --metal   # Force Apple Metal (native, no Docker)
+./start-tei.sh --cpu     # Force CPU Docker
+./start-tei.sh --stop    # Stop all TEI
+```
 
-All models run locally via ONNX. Downloaded once on first use, then cached.
+Auto-detection selects the optimal backend:
 
-| Model | Size | Purpose |
+| Platform | Backend | Image tag |
 |---|---|---|
-| `nomic-ai/nomic-embed-text-v1.5` | ~270MB | Text embeddings (86% top-5 accuracy, Matryoshka 384-dim) |
-| `Xenova/ms-marco-MiniLM-L-6-v2` | ~23MB | Cross-encoder reranking |
+| NVIDIA RTX 50x0 (Blackwell) | Docker CUDA | `120-1.9` |
+| NVIDIA RTX 40x0 (Ada) | Docker CUDA | `89-1.9` |
+| NVIDIA RTX 30x0 (Ampere) | Docker CUDA | `86-1.9` |
+| Apple Silicon | Native Metal | `cargo install --features metal` |
+| No GPU | Docker CPU | `cpu-1.9` |
 
-## Chunking strategy
+GPU override for NVIDIA:
+```bash
+docker compose -f docker-compose.yml -f docker-compose.nvidia.yml up -d
+```
 
-- Split markdown by headings (`##`, `###`, `####`) preserving the heading path
-- Target ~1500 characters per chunk
-- 10% overlap between chunks to prevent losing context at boundaries
-- Large sections split at paragraph boundaries
+## Search pipeline
+
+4-stage RAG pipeline:
+
+| Stage | Technology | Purpose |
+|---|---|---|
+| **Vector search** | LanceDB + nomic-embed-text-v1.5 via TEI | Semantic similarity (understands meaning) |
+| **BM25 search** | LanceDB native FTS (BM25, stemming, stop words) | Keyword matching (exact terms like `useEffect`) |
+| **RRF fusion** | Reciprocal Rank Fusion (k=60) | Merges both ranked lists, handles different score scales |
+| **Cross-encoder rerank** | ms-marco-MiniLM-L-6-v2 via TEI | Rescores top 50 candidates with deep relevance model |
 
 ## Storage
 
@@ -132,36 +178,36 @@ your-project/.claude/docs/
 | Tool | Description |
 |---|---|
 | `analyze_dependencies` | Monorepo-aware dep analysis: detects workspaces, resolves catalog versions, tags runtime/dev |
-| `store_and_index_doc` | Receive markdown, chunk, embed, store in LanceDB |
-| `fetch_and_store_doc` | Fetch URL directly (raw HTTP, no truncation), then chunk + embed + store |
-| `search_docs` | Full RAG pipeline: vector + BM25 + RRF + rerank |
+| `store_and_index_doc` | Receive markdown, chunk, embed via TEI, store in LanceDB |
+| `search_docs` | Full RAG pipeline: vector + BM25 + RRF + rerank via TEI |
 | `list_docs` | List indexed libraries with metadata |
 | `get_doc_section` | Get specific chunks by library + heading or chunk ID |
+| `fetch_and_store_doc` | Fetch URL directly (raw HTTP, no truncation), then chunk + embed + store |
+| `discover_and_fetch_docs` | Auto-discover docs: probes npm registry, llms.txt URLs, GitHub, homepage HTML. Detects and expands index files |
 
 ## Dependencies
 
-All open source:
-
 | Package | License | Purpose |
 |---|---|---|
 | `@lancedb/lancedb` | Apache 2.0 | Embedded vector database + native FTS |
-| `@huggingface/transformers` | Apache 2.0 | Run ONNX models locally |
 | `@modelcontextprotocol/sdk` | MIT | MCP server framework |
 | `zod` | MIT | Schema validation |
+| `turndown` | MIT | HTML to markdown conversion |
+| `turndown-plugin-gfm` | MIT | GFM support for turndown (tables, strikethrough, etc.) |
 
-No additional dependencies were added for monorepo support or HTTP fetching — everything uses Node built-ins.
+TEI containers (Docker):
+
+| Image | Model | Purpose |
+|---|---|---|
+| `text-embeddings-inference:*` | `nomic-ai/nomic-embed-text-v1.5` | Text embeddings |
+| `text-embeddings-inference:*` | `cross-encoder/ms-marco-MiniLM-L-6-v2` | Cross-encoder reranking |
 
 ## Development
 
 ```bash
 npm run dev    # Watch mode — rebuilds on file changes
 npm run build  # One-time build
-```
-
-### Testing with MCP Inspector
-
-```bash
-npx @modelcontextprotocol/inspector node dist/index.js
+npm test       # Integration test (requires TEI running)
 ```
 
 ## Project structure
@@ -169,25 +215,63 @@ npx @modelcontextprotocol/inspector node dist/index.js
 ```
 claude-local-docs/
 ├── .claude-plugin/
-│   ├── plugin.json         # Plugin manifest
-│   └── marketplace.json    # Marketplace listing
-├── .mcp.json               # MCP server config (stdio transport)
+│   ├── plugin.json           # Plugin manifest
+│   └── marketplace.json      # Marketplace listing
+├── .mcp.json                 # MCP server config (stdio transport)
 ├── commands/
-│   └── fetch-docs.md       # /fetch-docs command — Claude as research agent
+│   └── fetch-docs.md         # /fetch-docs — Claude as research agent
+├── hooks/
+│   └── hooks.json            # SessionStart hook for TEI containers
+├── scripts/
+│   └── ensure-tei.sh         # Idempotent TEI health check + start
+├── docker-compose.yml        # TEI containers (uses ${TEI_TAG})
+├── docker-compose.nvidia.yml # NVIDIA GPU device passthrough
+├── start-tei.sh              # Auto-detect GPU, start TEI
 ├── src/
-│   ├── index.ts            # MCP server entry, 6 tool definitions
-│   ├── indexer.ts           # Chunking + nomic-embed-text-v1.5 embeddings
-│   ├── search.ts            # 4-stage pipeline: vector + BM25 + RRF + rerank
-│   ├── reranker.ts          # Cross-encoder (ms-marco-MiniLM-L-6-v2)
-│   ├── store.ts             # LanceDB storage + metadata persistence
-│   ├── fetcher.ts           # Raw HTTP fetch (no AI truncation)
-│   ├── workspace.ts         # Monorepo detection + pnpm catalog + dep collection
-│   └── types.ts             # Shared TypeScript interfaces
+│   ├── index.ts              # MCP server entry, 7 tool definitions
+│   ├── discovery.ts          # Doc discovery: npm registry, URL probing, index expansion, HTML→markdown
+│   ├── indexer.ts            # Chunking + TEI embeddings
+│   ├── search.ts             # 4-stage pipeline: vector + BM25 + RRF + rerank
+│   ├── reranker.ts           # TEI cross-encoder reranking
+│   ├── store.ts              # LanceDB storage + metadata persistence
+│   ├── fetcher.ts            # Raw HTTP fetch (no AI truncation)
+│   ├── workspace.ts          # Monorepo detection + pnpm catalog
+│   ├── types.ts              # Shared TypeScript interfaces
+│   ├── turndown-plugin-gfm.d.ts  # Type declarations for turndown-plugin-gfm
+│   └── integration.test.ts   # Integration tests (requires TEI running)
 ├── LICENSE
 ├── package.json
 └── tsconfig.json
 ```
 
+## Troubleshooting
+
+### TEI containers not starting
+```bash
+# Check Docker is running
+docker info
+
+# Check container logs
+docker compose logs tei-embed
+docker compose logs tei-rerank
+
+# Restart
+./start-tei.sh --stop && ./start-tei.sh
+```
+
+### Port conflicts
+If 39281/39282 are in use, override via env vars:
+```bash
+TEI_EMBED_URL=http://localhost:49281 TEI_RERANK_URL=http://localhost:49282 node dist/index.js
+```
+
+### Apple Silicon — slow performance
+The default Docker CPU image runs via Rosetta 2. Use native Metal instead:
+```bash
+./start-tei.sh --metal
+```
+Requires Rust (`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`). First build takes a few minutes.
+
 ## License
 
 MIT

package/commands/fetch-docs.md

@@ -1,11 +1,11 @@
 ---
 description: "Fetch and index documentation for all project dependencies"
-allowed-tools: ["mcp__local-docs__analyze_dependencies", "mcp__local-docs__list_docs", "mcp__local-docs__store_and_index_doc", "mcp__local-docs__fetch_and_store_doc", "WebFetch", "WebSearch"]
+allowed-tools: ["mcp__local-docs__analyze_dependencies", "mcp__local-docs__list_docs", "mcp__local-docs__discover_and_fetch_docs", "mcp__local-docs__fetch_and_store_doc", "WebSearch", "WebFetch"]
 ---
 
 # Fetch Documentation for Project Dependencies
 
-You are a documentation research agent. Your job is to find and fetch the best available documentation for each dependency in this project, then store it locally for semantic search.
+You are a documentation indexing agent. Your job is to discover and index the best available documentation for each runtime dependency in this project.
 
 ## Steps
 
@@ -31,32 +31,54 @@ This leaves only **runtime dependencies** that actually need documentation.
 
 Call `list_docs` to see which libraries are already indexed. **Skip** any library that was fetched within the last 7 days unless the user explicitly asks to refresh.
 
-### 4. Fetch Documentation — One Library at a Time
+### 4. Fetch Documentation
 
-Process each remaining library **one at a time** with clear progress reporting. For each library:
+For each remaining library, follow this multi-step strategy. The goal is to find the **best quality** source — `llms-full.txt` > `llms.txt` (expanded index) > homepage HTML > README.
 
-#### a. Search for llms-full.txt (best source)
-1. **WebSearch** for `"{library name} llms-full.txt"`
-2. If you find a direct URL to `llms-full.txt`:
-   - Call **`fetch_and_store_doc`** with the URL (this fetches raw content — no truncation)
-   - Report: `[1/N] library-name — chunks from llms-full.txt (size)`
+#### Step A: Check Known URLs first
 
-#### b. Search for llms.txt (good source)
-If no llms-full.txt found:
-1. **WebSearch** for `"{library name} llms.txt"`
-2. If you find a direct URL to `llms.txt`:
-   - Call **`fetch_and_store_doc`** with the URL
-   - Report: `[2/N] library-name — chunks from llms.txt (size)`
+Before any searching, check if the library is in the **Known URLs Reference** below. If there's a known `llms-full.txt` or `llms.txt` URL, use it directly with `fetch_and_store_doc`. This is the fastest path.
 
-#### c. Fallback: Official docs via WebFetch
-If no llms.txt exists:
-1. **WebSearch** for `"{library name} official documentation"`
-2. **WebFetch** the main documentation page
-3. Call **`store_and_index_doc`** with the fetched content
-4. Report: `[3/N] library-name — chunks from official docs`
+#### Step B: WebSearch for llms.txt
 
-#### d. If all attempts fail
-Report: `[4/N] library-name — SKIPPED (no docs found)` and move on.
+For libraries NOT in the known list, use **WebSearch** to find the actual `llms.txt` or `llms-full.txt` URL. Use queries like:
+
+> `{library-name} llms-full.txt site:{homepage-domain}`
+
+or more broadly:
+
+> `{library-name} llms-full.txt OR llms.txt documentation`
+
+If the search finds a concrete URL to an `llms.txt` or `llms-full.txt` file, pass it directly to **`fetch_and_store_doc`**. Prefer `llms-full.txt` over `llms.txt` when both exist.
+
+**Batch the searches**: Run WebSearch for multiple libraries in parallel (up to 5 at a time) to collect URLs upfront. Then fetch them one by one.
+
+#### Step C: `discover_and_fetch_docs` (automatic probing)
+
+If neither known URLs nor WebSearch found an `llms.txt` URL, call **`discover_and_fetch_docs`**. This tool automatically:
+1. Checks npm registry for `llms`/`llmsFull` fields in package.json (newest convention)
+2. Probes homepage, `docs.{domain}`, `llms.{domain}`, `/docs/` subpath for llms-full.txt/llms.txt
+3. Probes GitHub raw for llms-full.txt/llms.txt on main/master branches
+4. Falls back to README.md from GitHub
+5. Falls back to homepage HTML → markdown conversion
+6. Detects index files and expands them by fetching linked pages
+
+#### Step D: Training data fallback
+
+If all above fail, try **`fetch_and_store_doc`** with documentation URLs you know from your training data (GitHub raw docs, official doc site pages, etc.).
+
+#### Evaluating results
+
+After each library is fetched, check the chunk count:
+- **< 5 chunks**: Very thin. Use WebSearch to find additional doc pages (API reference, guides) and fetch with `fetch_and_store_doc` to supplement.
+- **5-20 chunks**: Acceptable for small libraries.
+- **20+ chunks**: Good coverage.
+
+#### Progress reporting
+
+After each library, report:
+- `[1/N] library-name — X chunks from {source} (size)`
+- `[2/N] library-name — FAILED: {error message}`
 
 ### 5. Final Summary
 
@@ -66,22 +88,113 @@ After processing all libraries, report:
 Done! Indexed X/Y libraries.
 
   react        — 85 chunks (llms-full.txt, 340KB)
-  next         — 120 chunks (llms-full.txt, 510KB)
-  zod          — 45 chunks (llms.txt, 95KB)
-  express      — 30 chunks (official docs)
-  lodash       — SKIPPED (no docs found)
+  next         — 120 chunks (llms.txt-index, expanded 45 pages)
+  zod          — 45 chunks (llms-full.txt, 95KB)
+  express      — 30 chunks (homepage-html)
+  lodash       — FAILED (no docs found)
 
 Total: 280 chunks across 4 libraries.
 Use search_docs to query your documentation.
 ```
 
+## Known URLs Reference
+
+Use these URLs directly with `fetch_and_store_doc` — no searching needed. Prefer `llms-full.txt` when available.
+
+### Frameworks & Core
+
+| Library | Best URL |
+|---|---|
+| react | `https://react.dev/llms.txt` |
+| react-dom | (use react URL above) |
+| next | `https://nextjs.org/docs/llms-full.txt` |
+| nuxt | `https://nuxt.com/llms-full.txt` |
+| svelte | `https://svelte.dev/llms-full.txt` |
+| @sveltejs/kit | `https://svelte.dev/llms-full.txt` |
+| vue | (no official llms.txt — use `discover_and_fetch_docs`) |
+| react-native | `https://reactnative.dev/llms.txt` |
+| expo | `https://docs.expo.dev/llms-full.txt` |
+| hono | `https://hono.dev/llms.txt` |
+| bun | `https://bun.sh/llms.txt` |
+
+### Styling & UI
+
+| Library | Best URL |
+|---|---|
+| tailwindcss | `https://tailwindcss.com/llms.txt` |
+| @shadcn/ui / shadcn | `https://ui.shadcn.com/llms.txt` |
+| @chakra-ui/react | `https://chakra-ui.com/llms-full.txt` |
+| daisyui | `https://daisyui.com/llms.txt` |
+| tamagui | `https://tamagui.dev/llms.txt` |
+| @mantine/core | (check `https://mantine.dev/llms.txt`) |
+| react-native-unistyles | `https://www.unistyl.es/llms.txt` |
+
+### Data & State
+
+| Library | Best URL |
+|---|---|
+| zod | `https://zod.dev/llms-full.txt` |
+| @tanstack/react-query | `https://tanstack.com/query/llms-full.txt` |
+| @tanstack/react-router | `https://tanstack.com/llms.txt` |
+| drizzle-orm | `https://orm.drizzle.team/llms-full.txt` |
+| @prisma/client | `https://prisma.io/docs/llms-full.txt` |
+| convex | `https://docs.convex.dev/llms.txt` |
+
+### Backend & APIs
+
+| Library | Best URL |
+|---|---|
+| stripe | `https://docs.stripe.com/llms.txt` |
+| @supabase/supabase-js | `https://supabase.com/llms.txt` |
+| resend | `https://resend.com/docs/llms-full.txt` |
+| @medusajs/medusa | `https://docs.medusajs.com/llms-full.txt` |
+| better-auth | `https://www.better-auth.com/llms.txt` |
+
+### AI & LLM
+
+| Library | Best URL |
+|---|---|
+| ai (Vercel AI SDK) | `https://sdk.vercel.ai/llms.txt` |
+| @anthropic-ai/sdk | `https://docs.anthropic.com/llms-full.txt` |
+| langchain | `https://js.langchain.com/llms.txt` |
+| @modelcontextprotocol/sdk | `https://modelcontextprotocol.io/llms-full.txt` |
+| mastra | `https://mastra.ai/llms-full.txt` |
+
+### Dev Tools & Infra
+
+| Library | Best URL |
+|---|---|
+| turbo | `https://turbo.build/llms.txt` |
+| @trigger.dev/sdk | `https://trigger.dev/docs/llms-full.txt` |
+| @cloudflare/workers-types | `https://developers.cloudflare.com/llms-full.txt` |
+| @upstash/redis | `https://upstash.com/docs/llms-full.txt` |
+| @netlify/functions | `https://docs.netlify.com/llms.txt` |
+| @liveblocks/client | `https://liveblocks.io/llms-full.txt` |
+
+### Animation
+
+| Library | Best URL |
+|---|---|
+| motion / framer-motion | Special: `https://llms.motion.dev/docs/react-quick-start.md` (or use WebSearch for full index) |
+
+### Notes on special patterns
+
+- **Stripe**: Any Stripe doc page becomes markdown by appending `.md` (e.g. `https://docs.stripe.com/payments.md`)
+- **Motion (Framer Motion)**: Uses `llms.motion.dev` subdomain — `motion.dev/docs/{page}` becomes `llms.motion.dev/docs/{page}.md`
+- **Mintlify-hosted docs**: Sites using Mintlify auto-generate `/llms.txt` and `/llms-full.txt` (Anthropic, Cursor, CrewAI, Pinecone, etc.)
+- **GitBook-hosted docs**: Auto-generates `/llms.txt` since Jan 2025
+- **Nuxt Content docs**: May have separate `https://content.nuxt.com/llms-full.txt`
+- **package.json `llms`/`llmsFull` fields**: Some libraries (like Zod) include doc URLs directly in their npm package metadata — `discover_and_fetch_docs` checks this automatically
+
 ## Critical Rules
 
-- **NEVER write files to the filesystem directly.** Do NOT use the Write tool, Bash tool, or any other method to save documentation content to disk. ALL storage goes through the MCP tools (`fetch_and_store_doc` and `store_and_index_doc`), which save everything inside `.claude/docs/`. No exceptions.
-- **NEVER create markdown files, text files, or any other files** in the project directory. The MCP tools handle all file storage internally.
-- **Use `fetch_and_store_doc` for all llms.txt URLs** — this fetches raw content without AI truncation, preserving full documentation
-- **Use `store_and_index_doc` only for WebFetch fallback** — pass the WebFetch result content directly to the tool, do NOT save it to a file first
-- **One library at a time** — clear progress, no batching
+- **Check known URLs first** — the reference table above is faster and more reliable than searching.
+- **Search second, probe third** — use WebSearch to find llms.txt URLs before falling back to blind URL probing via `discover_and_fetch_docs`.
+- **Prefer `llms-full.txt` over `llms.txt`** — the full version has complete documentation without truncation.
+- **Use `fetch_and_store_doc` when you have a known URL** — from the reference table, WebSearch results, or training data.
+- **Use `discover_and_fetch_docs` when you have no URL** — it will probe common patterns automatically.
+- **Supplement thin results** — if a library has < 5 chunks, search for additional doc pages and fetch them.
+- **NEVER write files to the filesystem directly.** Do NOT use the Write tool, Bash tool, or any other method to save documentation content to disk. ALL storage goes through the MCP tools.
+- **One library at a time for fetching** — clear progress, no batching (but WebSearch can be batched)
 - **Skip dev deps by default** — runtime deps only
-- For scoped packages like `@scope/package`, search for both the full name and just the package part
 - Handle errors gracefully: if a library fails, log it and move to the next one