mcp-researchpowerpack 3.9.5 → 5.0.0

package/README.md

@@ -1,263 +1,171 @@
-<h1 align="center">🔬 MCP Research Powerpack</h1>
+# mcp-researchpowerpack
 
-<p align="center">
-  <strong>Five research tools for AI assistants — search, scrape, mine Reddit, and synthesize with LLMs.</strong>
-</p>
+HTTP MCP server for research. Orientation-first search, Reddit mining, and scraping — all over `/mcp`.
 
-<p align="center">
-  <a href="https://www.npmjs.com/package/mcp-research-powerpack"><img src="https://img.shields.io/npm/v/mcp-research-powerpack.svg?style=flat-square&color=cb3837" alt="npm"></a>
-  <a href="https://www.npmjs.com/package/mcp-research-powerpack"><img src="https://img.shields.io/npm/dm/mcp-research-powerpack.svg?style=flat-square&color=blue" alt="downloads"></a>
-  <a href="https://nodejs.org/"><img src="https://img.shields.io/badge/node-%3E%3D20-93450a.svg?style=flat-square" alt="node"></a>
-  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/license-MIT-grey.svg?style=flat-square" alt="license"></a>
-  <a href="https://modelcontextprotocol.io"><img src="https://img.shields.io/badge/MCP-compatible-5a67d8.svg?style=flat-square" alt="MCP"></a>
-</p>
+Built on [mcp-use](https://github.com/nicepkg/mcp-use). No stdio, HTTP only.
 
-<p align="center">
-  <code>npx mcp-research-powerpack</code>
-</p>
+## tools
 
----
+| tool | what it does | needs |
+|------|-------------|-------|
+| `start-research` | one-time orientation step that unlocks the research workflow for the current conversation/session. Emits the companion `run-research` skill install hint on every boot. | none |
+| `web-search` | parallel Google search across 1–100 queries with URL aggregation, hostname-heuristic `source_type` tagging, and follow-up suggestions. `scope: "reddit"` filters to post permalinks (subreddit homepages dropped). `verbose: true` restores per-row metadata + Signals block. | `SERPER_API_KEY` |
+| `get-reddit-post` | fetch 1–100 Reddit posts with full comment trees. Returns `isError: true` when every URL fails. | `REDDIT_CLIENT_ID` + `REDDIT_CLIENT_SECRET` |
+| `scrape-links` | scrape 1–100 URLs with optional LLM extraction. HTML chrome stripped server-side via Readability. Reddit URLs are rejected with `UNSUPPORTED_URL_TYPE` — use `get-reddit-post`. | `SCRAPEDO_API_KEY` |
 
-An [MCP](https://modelcontextprotocol.io) server that gives Claude, Cursor, Windsurf, and any MCP-compatible AI assistant a complete research toolkit. Google search, Reddit deep-dives, web scraping with AI extraction, and multi-model deep research — all as tools that chain into each other.
+Also exposes `/health`, `health://status`, and two optional MCP prompts: `deep-research` and `reddit-sentiment`.
 
-Zero config to start. Each API key you add unlocks more capabilities.
+## workflow
 
-## Tools
+Call `start-research` once at the beginning of each conversation/session.
 
-| Tool | What it does | Requires |
-|:-----|:-------------|:---------|
-| **`web_search`** | Parallel Google search across 3–100 keywords with CTR-weighted ranking and consensus detection | `SERPER_API_KEY` |
-| **`search_reddit`** | Same search engine filtered to reddit.com — 10–50 queries in parallel | `SERPER_API_KEY` |
-| **`get_reddit_post`** | Fetch 2–50 Reddit posts with full comment trees, smart comment budget allocation | `REDDIT_CLIENT_ID` + `REDDIT_CLIENT_SECRET` |
-| **`scrape_links`** | Scrape 1–50 URLs with JS rendering fallback, HTML→Markdown, optional AI extraction | `SCRAPEDO_API_KEY` |
-| **`deep_research`** | Send questions to research-capable models (Grok, Gemini) with web search, file attachments | `OPENROUTER_API_KEY` |
+It returns the orientation brief that teaches how to route between:
 
-Tools are designed to **chain**: `web_search` → `scrape_links` → `search_reddit` → `get_reddit_post` → `deep_research` for synthesis. Each tool suggests the next logical step in its output.
+- `web-search` (with `scope: "web" | "reddit" | "both"`)
+- `get-reddit-post`
+- `scrape-links`
 
-## Quick Start
+All three gated tools advertise this precondition via `_meta.requires: ["start-research"]` in `tools/list`, so capability-aware clients can skip pre-bootstrap calls.
 
-### Claude Desktop / Claude Code
+Pair the server with the [`run-research`](https://github.com/yigitkonur/skills-by-yigitkonur/tree/main/skills/run-research) skill for the full agentic playbook:
 
-Add to your MCP config (`~/Library/Application Support/Claude/claude_desktop_config.json`):
-
-```json
-{
-  "mcpServers": {
-    "research-powerpack": {
-      "command": "npx",
-      "args": ["-y", "mcp-research-powerpack"],
-      "env": {
-        "SERPER_API_KEY": "your-key-here",
-        "OPENROUTER_API_KEY": "your-key-here"
-      }
-    }
-  }
-}
+```bash
+npx -y skills add -y -g yigitkonur/skills-by-yigitkonur/skills/run-research
 ```
 
-### Cursor
+## quickstart
+
+```bash
+# from npm
+HOST=127.0.0.1 PORT=3000 npx -y mcp-researchpowerpack
+
+# from source
+git clone https://github.com/yigitkonur/mcp-researchpowerpack.git
+cd mcp-researchpowerpack
+pnpm install && pnpm dev
+```
 
-Add to `.cursor/mcp.json` in your project:
+Connect your client to `http://localhost:3000/mcp`:
 
 ```json
 {
   "mcpServers": {
     "research-powerpack": {
-      "command": "npx",
-      "args": ["-y", "mcp-research-powerpack"],
-      "env": {
-        "SERPER_API_KEY": "your-key-here"
-      }
+      "url": "http://localhost:3000/mcp"
     }
   }
 }
 ```
 
-### From Source
-
-```bash
-git clone https://github.com/yigitkonur/mcp-research-powerpack.git
-cd mcp-research-powerpack
-pnpm install && pnpm build
-pnpm start
-```
-
-### HTTP Transport
-
-```bash
-MCP_TRANSPORT=http MCP_PORT=3000 npx mcp-research-powerpack
-```
-
-Exposes `/mcp` endpoint (POST/GET/DELETE with session headers) and `/health`.
-
-## API Keys
-
-Each key unlocks a capability. Missing keys silently disable their tools — the server never crashes.
-
-| Variable | Enables | Free Tier |
-|:---------|:--------|:----------|
-| `SERPER_API_KEY` | `web_search`, `search_reddit` | 2,500 searches/mo — [serper.dev](https://serper.dev) |
-| `REDDIT_CLIENT_ID` + `REDDIT_CLIENT_SECRET` | `get_reddit_post` | Unlimited — [reddit.com/prefs/apps](https://www.reddit.com/prefs/apps) (script type) |
-| `SCRAPEDO_API_KEY` | `scrape_links` | 1,000 credits/mo — [scrape.do](https://scrape.do) |
-| `OPENROUTER_API_KEY` | `deep_research`, LLM extraction | Pay-per-token — [openrouter.ai](https://openrouter.ai) |
-| `CEREBRAS_API_KEY` | Cerebras LLM extraction | — |
-| `USE_CEREBRAS` | Enable Cerebras for extraction (set `true`) | `false` |
-
-## Configuration
-
-Optional tuning via environment variables:
-
-| Variable | Default | Description |
-|:---------|:--------|:------------|
-| `RESEARCH_MODEL` | `x-ai/grok-4-fast` | Primary deep research model |
-| `RESEARCH_FALLBACK_MODEL` | `google/gemini-2.5-flash` | Fallback when primary fails |
-| `LLM_EXTRACTION_MODEL` | `openai/gpt-oss-120b:nitro` | Model for scrape/reddit AI extraction |
-| `DEFAULT_REASONING_EFFORT` | `high` | Research depth: `low`, `medium`, `high` |
-| `DEFAULT_MAX_URLS` | `100` | Max search results per research question (10–200) |
-| `API_TIMEOUT_MS` | `1800000` | Request timeout in ms (default: 30 min) |
-| `MCP_TRANSPORT` | `stdio` | Transport mode: `stdio` or `http` |
-| `MCP_PORT` | `3000` | Port for HTTP mode |
-| `USE_CEREBRAS` | `false` | Set to `true` to use Cerebras for extraction instead of OpenRouter |
-| `CEREBRAS_API_KEY` | — | API key for Cerebras cloud — [cloud.cerebras.ai](https://cloud.cerebras.ai) |
-
-### Cerebras Support
-
-When `USE_CEREBRAS=true` and `CEREBRAS_API_KEY` are set, the `scrape_links` tool uses Cerebras (Z.ai GLM 4.7) for AI content extraction instead of OpenRouter. This provides:
-
-- **Ultra-fast extraction** — Cerebras inference is optimized for speed
-- **Independent from OpenRouter** — extraction works even without `OPENROUTER_API_KEY`
-- **Automatic fallback** — if Cerebras is not configured, falls back to OpenRouter
-
-```bash
-# Enable Cerebras for extraction
-USE_CEREBRAS=true CEREBRAS_API_KEY=your-key npx mcp-research-powerpack
-```
-
-### Network Resilience
-
-All LLM API calls include built-in stability protections:
-
-- **Request deadlines** — hard timeout prevents calls from hanging indefinitely
-- **Stall detection** — if no response arrives within a threshold, the request is aborted and retried
-- **Exponential backoff** — transient failures (429, 5xx) retry with jitter to avoid thundering herd
-- **Connection loss recovery** — network errors (ECONNRESET, ECONNREFUSED) trigger automatic retry
-- **Graceful degradation** — all tools return structured errors instead of crashing
+## config
 
-## How It Works
+Copy `.env.example`, set only what you need. Missing keys don't crash the server — they disable the affected capability with a clear error.
 
-### Search Ranking
+### server
 
-Results from multiple queries are deduplicated by normalized URL and scored using **CTR-weighted position values** (position 1 = 100.0, position 10 = 12.56). URLs appearing across multiple queries get a consensus marker. Frequency threshold starts at ≥3, falls back to ≥2, then ≥1 to ensure results.
+| var | default | |
+|-----|---------|---|
+| `PORT` | `3000` | HTTP port |
+| `HOST` | `127.0.0.1` | bind address |
+| `ALLOWED_ORIGINS` | unset | comma-separated origins for host validation |
+| `MCP_URL` | unset | fallback public MCP URL used by the production origin-protection guard |
+| `REDIS_URL` | unset | Redis-backed MCP sessions, distributed SSE, and workflow state |
 
-### Reddit Comment Budget
+### providers
 
-Global budget of **1,000 comments**, max 200 per post. After the first pass, surplus from posts with fewer comments is redistributed to truncated posts in a second fetch pass.
+| var | enables |
+|-----|---------|
+| `SERPER_API_KEY` | `web-search` (open web + `scope: "reddit"`) |
+| `REDDIT_CLIENT_ID` + `REDDIT_CLIENT_SECRET` | `get-reddit-post` |
+| `SCRAPEDO_API_KEY` | `scrape-links` |
+| `LLM_API_KEY` | AI extraction, search classification, and raw-mode refine suggestions |
 
-### Scraping Pipeline
+### llm (AI extraction + classification)
 
-**Three-mode fallback** per URL: basic → JS rendering → JS + US geo-targeting. Results go through HTML→Markdown conversion (Turndown), then optional AI extraction with a 100K char input cap and 8,000 token output per URL.
+Any OpenAI-compatible provider works — OpenRouter, Cerebras, Together, etc.
 
-### Deep Research
+| var | default | |
+|-----|---------|---|
+| `LLM_API_KEY` | *(required for LLM features)* | API key for the LLM provider |
+| `LLM_BASE_URL` | `https://openrouter.ai/api/v1` | base URL |
+| `LLM_MODEL` | `openai/gpt-5.4-mini` | model identifier |
+| `LLM_MAX_TOKENS` | `8000` | max output tokens |
+| `LLM_REASONING` | `low` | `none` \| `low` \| `medium` \| `high` |
+| `LLM_CONCURRENCY` | `50` | parallel LLM calls |
 
-**32,000 token budget** divided across questions (1 question = 32K, 10 questions = 3.2K each). Gemini models get `google_search` tool access. Grok/Perplexity get `search_parameters` with citations. Primary model fails → automatic fallback to secondary model.
+### evals
 
-### File Attachments
+`pnpm test:evals` writes a JSON artifact to `test-results/eval-runs/<timestamp>.json`.
 
-`deep_research` can read **local files** and include them as context. Files over 600 lines are smart-truncated (first 500 + last 100 lines). Line ranges supported. Line numbers preserved in output.
+When an OpenAI API key is present, it performs a live Responses API + remote MCP evaluation.
+Without an API key, it exits successfully in explicit skip mode and records that skip in the artifact.
 
-## Concurrency
+Useful env vars:
 
-| Operation | Parallel Limit |
-|:----------|:---------------|
-| Web search keywords | 8 |
-| Reddit search queries | 8 |
-| Reddit post fetches per batch | 5 (batches of 10) |
-| URL scraping per batch | 10 (batches of 30) |
-| LLM extraction | 3 |
-| Deep research questions | 3 |
+- `EVAL_MCP_URL`
+- `EVAL_MODEL`
+- `EVAL_API_KEY` or `OPENAI_API_KEY`
 
-All clients use **manual retry with exponential backoff and jitter**. The OpenAI SDK's built-in retry is disabled (`maxRetries: 0`).
+## dev
 
-## Architecture
-
-```
-src/
-├── index.ts                    Entry point — STDIO + HTTP transport, graceful shutdown
-├── worker.ts                   Cloudflare Workers entry (Durable Objects)
-├── config/
-│   ├── index.ts                Env parsing, capability detection, lazy Proxy config
-│   ├── loader.ts               YAML → Zod → JSON Schema pipeline
-│   └── yaml/tools.yaml         Single source of truth for tool definitions
-├── schemas/                    Zod input validation (deep-research, scrape-links, web-search)
-├── tools/
-│   ├── registry.ts             Tool lookup → capability check → validate → execute
-│   ├── search.ts               web_search handler
-│   ├── reddit.ts               search_reddit + get_reddit_post handlers
-│   ├── scrape.ts               scrape_links handler
-│   └── research.ts             deep_research handler
-├── clients/
-│   ├── search.ts               Google Serper API client
-│   ├── reddit.ts               Reddit OAuth + comment tree parser
-│   ├── scraper.ts              Scrape.do client with fallback modes
-│   └── research.ts             OpenRouter client with model-specific handling
-├── services/
-│   ├── llm-processor.ts        Shared LLM extraction (singleton OpenAI client)
-│   ├── markdown-cleaner.ts     HTML → Markdown via Turndown
-│   └── file-attachment.ts      Local file reading with line ranges
-└── utils/
-    ├── retry.ts                Shared backoff + retry constants
-    ├── concurrency.ts          Bounded parallel execution (pMap, pMapSettled)
-    ├── url-aggregator.ts       CTR-weighted scoring + consensus detection
-    ├── errors.ts               Error classification + structured errors
-    ├── logger.ts               MCP logging protocol
-    └── response.ts             Standardized 70/20/10 output formatting
+```bash
+pnpm install
+pnpm dev          # watch mode, serves :3000/mcp
+pnpm typecheck    # tsc --noEmit
+pnpm test         # unit + http integration tests
+pnpm build        # compile to dist/
+pnpm inspect      # mcp-use inspector
 ```
 
-## Deploy
-
-### Cloudflare Workers
+## deploy
 
 ```bash
-npx wrangler deploy
+pnpm build
+pnpm deploy       # manufact cloud
 ```
 
-Uses Durable Objects with SQLite storage. YAML-based tool definitions are replaced with inline definitions since there's no filesystem in Workers.
-
-### npm
-
-Published as [`mcp-research-powerpack`](https://www.npmjs.com/package/mcp-research-powerpack). Binary names: `mcp-research-powerpack`, `research-powerpack-mcp`.
-
-## Development
+Or self-host anywhere with Node 20.19+ / 22.12+:
 
 ```bash
-pnpm install          # Install dependencies
-pnpm dev              # Run with tsx (live TypeScript)
-pnpm build            # Compile to dist/
-pnpm typecheck        # Type-check without emitting
-pnpm start            # Run compiled output
+HOST=0.0.0.0 ALLOWED_ORIGINS=https://app.example.com pnpm start
 ```
 
-### Testing
+## architecture
 
-```bash
-pnpm test:web-search     # Test web search tool
-pnpm test:reddit-search  # Test Reddit search
-pnpm test:scrape-links   # Test scraping
-pnpm test:deep-research  # Test deep research
-pnpm test:all            # Run all tests
-pnpm test:check          # Check environment setup
+```
+index.ts                 server startup, cors, health, shutdown
+src/
+  config/                env parsing, capability detection, lazy proxy config
+  clients/               provider API clients (serper, reddit, scrapedo)
+  prompts/               optional MCP prompts for deep-research and reddit-sentiment
+  tools/
+    registry.ts          registerAllTools() — wires tools to MCP server
+    start-research.ts    workflow orientation entrypoint
+    search.ts            web-search handler
+    reddit.ts            get-reddit-post
+    scrape.ts            scrape-links handler
+    mcp-helpers.ts       response builders (markdown + structured MCP output)
+    utils.ts             shared formatters, token budget allocation
+  services/
+    workflow-state.ts    conversation-aware workflow state with memory/Redis backends
+    llm-processor.ts     AI extraction/synthesis via OpenAI-compatible API
+    markdown-cleaner.ts  HTML/markdown cleanup
+  schemas/               zod v4 input validation per tool
+  utils/
+    workflow-key.ts      workflow identity derivation from user/session context
+    bootstrap-guard.ts   hard gate enforcing start-research first
+    reddit-keyword-guard.ts  one-shot redirect for reddit-first web-search misuse
+    sanitize.ts          strips URL/control-char injection from follow-up suggestions
+    errors.ts            structured error codes (retryable classification)
+    concurrency.ts       pMap/pMapSettled — bounded parallel execution
+    retry.ts             exponential backoff with jitter
+    url-aggregator.ts    CTR-weighted URL ranking for search consensus
+    response.ts          formatSuccess/formatError/formatBatchHeader
+    logger.ts            mcpLog() — stderr-only (MCP-safe)
 ```
 
-## Contributing
-
-1. Fork the repository
-2. Create a feature branch (`git checkout -b feature/amazing-feature`)
-3. Make your changes
-4. Run `pnpm typecheck && pnpm build` to verify
-5. Commit (`git commit -m 'feat: add amazing feature'`)
-6. Push to your branch (`git push origin feature/amazing-feature`)
-7. Open a Pull Request
+Key patterns: capability detection at startup, conversation-aware workflow gating via `start-research`, always-on structured MCP tool output, raw and classified follow-up guidance in `web-search`, bounded concurrency, CTR-based URL ranking, tools never throw (always return `toolFailure`), and structured errors with retry classification.
 
-## License
+## license
 
-[MIT](https://opensource.org/licenses/MIT) © [Yiğit Konur](https://github.com/yigitkonur)
+MIT