smart-web-mcp 0.44.1 → 1.1.0

package/README.md

@@ -2,598 +2,63 @@
 
 # smart-web
 
-Local MCP for agentic web retrieval.
+One local MCP entry point for finding, reading, and using the web.
 
-`smart-web` bundles four public tools in one stdio server:
+**Goal:** finish the user's web task—not merely return a successful request or a browser handoff. Use fast retrieval where it works; continue through the approved local browser-use session when it does not.
 
-- `smartfetch` for direct URLs
-- `smartsearch` for discovery
-- `smartcrawl` for same-site multi-page traversal
-- `smartvertical` for Korean/local structured vertical data such as places, weather, transit, finance, law, commerce, and safety lookups
+## Principles
 
-It is designed for **agents and MCP hosts**, not for manual browser automation. The job is to return structured retrieval output first, then signal when a browser-task runtime would be a better next step.
+- Judge success by the requested content or action, including comments and exact product identity.
+- Measure search relevance, freshness, query constraints, and usable sources—not result count.
+- Prioritize failures from real workflows across sites, not a single showcase domain.
+- Keep output compact, with source evidence, attempt history, and explicit omissions.
+- Isolate fragile site logic; detect drift and repair it through bounded, verified agent runs.
 
-If a page is login-gated, keep the login UI, session persistence, and capture flow in a companion runtime instead of moving that stateful behavior into the `smart-web` core.
+## Status
 
-- npm: <https://www.npmjs.com/package/smart-web-mcp>
-- MCP Registry: `io.github.jojo-labs/smart-web`
-- Issues: <https://github.com/jojo-labs/smart-web/issues>
+The package exposes four retrieval tools plus the bounded `smartbrowser` action tool. The **operator-first revamp is in progress**: broader interactive coverage and automatic repair remain acceptance goals, not a claim of universal support. The legacy web-task-api backend is retired; browser actions use the approved local session. Cookies persist across batches, but tab state only persists within a batch.
 
-If `smart-web` returns reproducible incorrect or misleading output, open a GitHub issue with the exact input, tool arguments, observed output, expected behavior, and version. Skip obvious transient network, auth, or rate-limit failures unless the smart-web classification itself looks wrong.
+Track shipped evidence and remaining work in [the revamp issue](https://github.com/jojo-labs/smart-web/issues/496). Browser-readable content should not require the user to switch tools; genuine login, permission, or unavailable-content limits remain explicit.
 
-## Highlights
+## Quick start
 
-- one local MCP instead of separate search, fetch, crawl, and Korean vertical-data servers
-- explicit routing: URL → `smartfetch`, query → `smartsearch`, known site → `smartcrawl`
-- up-front smartfetch acquisition lanes instead of broad direct → browser retry chains, with `pipeline.acquire.attempt_map` showing seed/direct/relay/archive/browser phases and skip/failure reasons
-- optional Jina Reader relay for weak generic/article fetches when you explicitly enable it, with JSON mode and alternate-link preservation
-- shared document extraction for article-like pages keeps browser primary content separate from raw shell HTML, preserving headings, links, blocks, and Markdown when requested
-- docs export manifests are versioned and include per-document SHA-256 hashes plus resource-handle manifests for stable local Markdown mirrors; resource-handle manifests include both exported documents and metadata handles for the summary/manifest files, and successful export `structuredContent` embeds the same handle manifest for host follow-up calls
-- budgeted `smartfetch` projection records explicit truncation metadata instead of scattering silent hardcoded caps; MCP calls default to the compact preset for token-sensitive agent hosts
-- weak generic pages can recover through JSON-LD or Next.js payload extraction and same-origin RSS/Atom feed discovery before falling back to a handoff
-- optional local adaptive scraper fallback can call a Scrapling-compatible `scrapling` command for AI-targeted extraction on weak JavaScript shells before spending a full Playwright browser pass
-- paywalled article pages prefer lawful archive recovery and search-indexed reference previews before telling the host to escalate elsewhere
-- LinkedIn authwall handling prefers compliant public fallbacks such as archives and search-indexed reference metadata before giving up
-- legal paper fallback for academic URLs: DOI-aware OpenAlex, Unpaywall, Semantic Scholar, CORE discovery, and Europe PMC enrichment plus bioRxiv/medRxiv API fallback when a direct paper page is thin or blocked
-- site-native public search fallbacks cover Reddit (Arctic Shift, live RSS, PullPush, then JSON), GitHub repository discovery/repo-docs queries, npm search/exact packages, Hacker News, Stack Exchange, Wikipedia, Velog, MDN, and crates.io when a matching `site:` query is available
-- YouTube watch, shorts, embed, live, and youtu.be video URLs get best-effort public caption enrichment by default while non-video YouTube pages stay on the generic fetch path
-- structured output with `assessment`, `pipeline`, and partial-result `research_handoff` fields for host-side decision making
-- local-first defaults with privacy-aware search/fetch behavior
-- useful normalization for common public surfaces instead of raw HTML dumps
-- social unavailable shells such as Threads generic join/login pages stay partial and chrome-free instead of masquerading as verified post content
-- blocked commerce URLs can accept up to five contextual `reference_queries`; public catalog metadata is returned only when the resolved product, item, and vendor identifiers all match, so a title hint cannot silently select a different listing
-
-## Tool routing
-
-- `smartfetch` first when the user already gave a URL or shortlink
-- `smartsearch` when the user gave a topic, keywords, or a `site:` query
-- `smartcrawl` when you already know the site and need multiple relevant pages
-
-| Situation                                         | Tool            |
-| ------------------------------------------------- | --------------- |
-| User already gave a URL or shortlink              | `smartfetch`    |
-| User gave a topic, keywords, or `site:` query     | `smartsearch`   |
-| You already know the site and need multiple pages | `smartcrawl`    |
-| User needs Korean/local structured data           | `smartvertical` |
-
-`smart-web` is a retrieval layer. It is not a general-purpose click/type/form-fill browser agent.[^1]
-
-## What the tools return
-
-All public tools are shaped for agent consumption. Common high-signal fields include:
-
-- `assessment`: confidence, block/auth hints, recommended handoff
-- `pipeline`: resolve/acquire/normalize/assess stages, plus policy metadata for relay-style usage and authwall likelihood; `pipeline.acquire.attempt_map` describes the default seed, direct/impit, Jina Reader, archive, and browser phases as succeeded, failed, or skipped
-- `research_handoff`: compact metadata on blocked or partial `smartfetch` results telling downstream research workers and editable report renderers which evidence fields to consume
-- `budget`: output projection metadata showing which fields were truncated and by how much, plus final MCP `response_chars` counts for compact-JSON `structuredContent` and rendered text; use `smartfetch`/`smartsearch`/`smartcrawl`/`smartvertical` `output_budget: "compact"` for token-sensitive MCP hosts, `preferred_budget_chars` for host-budget-aware routing, and `"full"` when larger structured payloads are explicitly needed
-- `evidence.selected_output_budget*`: the selected `smartfetch` budget preset, winning control, and resolver reason so hosts can see which v0.x compatibility hint actually took effect
-- `errors`: structured failure reasons
-- `post`, `thread`, `comments`: normalized content surfaces
-- `assets` and `download`: file-like outputs when present
-
-This lets hosts decide whether deterministic retrieval was good enough or whether they should escalate to an auth-aware companion flow or a broader browser-task runtime.[^2]
-
-For `smartfetch`, MCP `structuredContent` always carries the projected JSON result. MCP calls default to `output_budget: "compact"` so agent hosts do not ingest large page/thread/link payloads unless they ask for them. The projection caps known body/list fields plus arbitrary provider record width, nesting, and aggregate string content, so unknown `post`, thread, or comment fields cannot bypass the preset; `budget.fields` reports omitted nested paths and primary URL/title/author/status/body fields are prioritized. The human-readable `content[0].text` defaults to compact text instead of duplicating that JSON payload, with separate caps for long body text, result, comment, link, and asset sections. Omitted counts are reported in the text view while the projected fields remain available through `structuredContent`; set `format: "json"` only when a host needs JSON repeated in the text channel. Set `output_budget: "balanced"` or `"full"` only for follow-up calls that explicitly need more structured content. For host-side integration, prefer `preferred_budget_chars`; use `context_mode` only as a backward-compatible coarse alias when numeric hints are not supplied. If neither is present, `headroom_tokens` keeps compact output by default and uses `balanced` only when there is enough room. `format: "markdown"` keeps the compact response shape and requests Markdown extraction when available.
-
-When a `smartfetch` result is blocked, partial, or low-confidence, `structuredContent.research_handoff` stays small and metadata-only. It points downstream tools at the source URL and the projected evidence bundle: `pipeline`, `assessment`, `errors`, `assets`, `outbound_links`, and the full projected `structuredContent`. Heavy research agents or editable report/rendering tools can consume that bundle downstream, but `smart-web` itself remains the unified retrieval kernel and does not spawn those tools.
-
-For `smartsearch`, MCP calls now project both `structuredContent` and rendered text through the same compact/balanced/full budget controls. Compact search keeps enough ranked links for first-pass discovery while capping every normalized string field, including query/engine metadata, result titles and URLs, snippets, notes, assessment, and search-to-crawl handoff guidance; `contextMaxCharacters` remains a final text-channel cap for legacy hosts. Each shortened field is recorded under `budget.fields`. Use `output_budget: "full"` only for a known follow-up when the host really wants every returned snippet/result in `structuredContent`.
-
-`smartsearch` reports provider execution separately from search matches under `structuredContent.execution`: the overall `status` is `succeeded` when at least one attempted provider completed, and each entry in `attempts` records its provider, status, result count, and any execution error. If every configured provider fails to execute, MCP returns `isError: true` and the CLI exits non-zero after preserving the structured diagnostics. A successful provider response with zero matching results remains successful even when the compatibility fields stay `engine: "none"` and `results: []`.
-
-For `smartcrawl`, MCP summary calls now project both `structuredContent` and rendered text through compact/balanced/full budget controls, so broad docs/forum crawls do not flood agent context with every page, candidate, note, or error by default. Summary projection caps all normalized titles, URLs, discovery/page metadata, notes, errors, assessment, and handoff strings and records each omission under `budget.fields`. `output_dir` export mode remains handle-first: successful export responses embed `resource_handle_manifest` in `structuredContent`, and export summaries include file:// re-open guidance for the summary, manifest, resource-handle JSON, and first Markdown document so agent hosts can resume from handles without parsing the full JSON first. The resource-handle manifest also carries ordered `recommended_resources` for hosts that need a ready-to-open summary, manifest, and first document without inferring role priority. Blocked exports label those paths as planned/unwritten and include retry guidance instead of presenting false file handles. Set `format: "json"` only for hosts that cannot consume `structuredContent` and intentionally need JSON repeated in the text channel.
-
-For `smartvertical`, MCP calls use the same budget-control precedence as the web tools. Compact mode caps rendered text plus arbitrary structured `result` arrays and long nested strings, and `structuredContent.budget.fields` reports exactly which vertical fields were shortened. Use `output_budget: "balanced"` or a larger `preferred_budget_chars` value when a Korean/local lookup needs more rows or full legal/safety record text.
-
-Use budget presets this way:
-
-- `compact`: default first pass for agent hosts, link triage, social/thread previews, and "is this page enough?" checks.
-- `balanced`: follow-up when the first pass found the right page but omitted useful body, result, comment, or link detail.
-- `full`: explicit retrieval pass for a known relevant URL when the host can afford the larger `structuredContent` payload.
-
-When compact output omits needed content, make a follow-up call for the same URL/query/start site with a larger budget control instead of asking the model to infer from the compact text. Prefer `preferred_budget_chars` when the host knows its available response budget, or use `output_budget: "balanced"` / `"full"` for manual retries.
-
-## Supported high-signal surfaces
-
-- communities: Reddit, DCInside, LinkedIn public posts with public fallback recovery, Algumon
-- search-native surfaces: Reddit, GitHub repository discovery and repo-docs path scopes, npm search and exact package path scopes, Hacker News, Stack Exchange, Wikipedia, Velog, MDN, crates.io
-- competitive-programming surfaces: solved.ac problem/search/profile routes, BOJ problem/workbook/user pages, Codeforces problem/profile pages, AtCoder task/contest pages, QOJ problem pages, Jungol problem pages
-- local map place pages: Naver Map, Kakao Map, and redirected `naver.me` place-share links
-- reference/article pages: NamuWiki, Wikipedia, Naver Blog, Tistory, Velog
-- media/social: YouTube video URLs with best-effort public transcripts, X, Threads, Instagram, Telegram
-- commerce pages: Amazon, Coupang, Danawa, Aladin, AliExpress
-- archives/reference wrappers: Wayback snapshots, archive.md lookup pages, arXiv abstract pages with direct paper links
-- academic paper surfaces: arXiv, PubMed, PMC, bioRxiv, medRxiv, plus legal DOI-linked OA copies surfaced from OpenAlex, Unpaywall, Semantic Scholar, CORE, and Europe PMC when a publisher page is thin or paywalled
-
-When a source stays blocked or under-specified, `smart-web` prefers partial but honest output over pretending to have verified content.
-
-## Install
+Requires Node.js 24 and npm. Start the stdio server:
 
 ```bash
 npx -y smart-web-mcp
 ```
 
-When `smartfetch` or `smartcrawl` needs Playwright-backed page loading, `smart-web` checks whether the bundled Chromium revision is available. If the local Playwright browser cache is missing or outdated, `smart-web` warns at startup and will try `npx playwright install chromium` automatically on first browser use before surfacing a structured error.
-
-## Host setup
-
-### Claude Code
-
-```bash
-claude mcp add smart-web -- npx -y smart-web-mcp
-```
-
-Suggested tool-use profile: default to `smartfetch` without `format` and without a budget control for compact first-pass retrieval; retry with `output_budget: "balanced"` or `"full"` only when the user asks for deeper extraction. If Claude Code exposes a host response budget, send it as `preferred_budget_chars`.
-
-### Codex
-
-```bash
-codex mcp add smart-web -- npx -y smart-web-mcp
-```
-
-Or via `~/.codex/config.toml` (or project-scoped `.codex/config.toml`):
-
-```toml
-[mcp_servers.smart-web]
-command = "npx"
-args = ["-y", "smart-web-mcp"]
-```
-
-Suggested tool-use profile: keep default compact text for normal browsing. For targeted follow-up retrieval, pass `preferred_budget_chars` from the remaining response/tool budget when available; otherwise pass one explicit `output_budget` preset.
-
-### OpenCode
-
-```json
-{
-  "$schema": "https://opencode.ai/config.json",
-  "mcp": {
-    "smart-web": {
-      "type": "local",
-      "command": ["npx", "-y", "smart-web-mcp"],
-      "enabled": true
-    }
-  }
-}
-```
-
-Suggested tool-use profile: call `smartfetch` with no `format` for normal retrieval, `format: "markdown"` only when Markdown extraction is useful, and `format: "json"` only for hosts that intentionally require the full JSON duplicated into `content[0].text`.
-
-### Hermes/Raon
-
-For Hermes/Raon-style hosts, keep the MCP server command standard and set budget controls per tool call:
-
-```json
-{
-  "tool": "smartfetch",
-  "arguments": {
-    "url": "https://example.com/post",
-    "preferred_budget_chars": 40000
-  }
-}
-```
-
-Use `preferred_budget_chars` as the primary host-profile control when Hermes/Raon knows the response budget. Use `headroom_tokens` only when the host has token headroom but no character budget, and use `context_mode` only for older integrations that cannot send numeric budget metadata.
-
-Budget-control precedence is deterministic:
-
-1. `output_budget`
-2. `preferred_budget_chars`
-3. `headroom_tokens`
-4. `context_mode`
-5. compact default
-
-Send only one control when possible. If older host profiles send more than one, `evidence.selected_output_budget_source` reports which control won.
-
-### Custom settings file
-
-To use a non-default settings path, append `--settings-file` to the command:
-
-```bash
-npx -y smart-web-mcp --settings-file /absolute/path/to/smart-web.settings.json
-```
-
-For Claude Code:
-
-```bash
-claude mcp add smart-web -- npx -y smart-web-mcp --settings-file /absolute/path/to/smart-web.settings.json
-```
-
-For OpenCode, add `"args"` after `"command"`:
-
-```json
-{
-  "mcp": {
-    "smart-web": {
-      "type": "local",
-      "command": ["npx", "-y", "smart-web-mcp", "--settings-file", "/absolute/path/to/smart-web.settings.json"],
-      "enabled": true
-    }
-  }
-}
-```
-
-## Configuration
-
-### Settings path
-
-Default: `~/.config/smart-web/settings.json`
-
-Print a template:
-
-```bash
-npx -y smart-web-mcp --print-settings-example
-```
-
-Initialize the default file:
-
-```bash
-npx -y smart-web-mcp --init-settings
-```
-
-`smart-web` treats the settings file as the single runtime config surface.
-
-### Full settings reference
-
-```jsonc
-{
-  // "balanced" (default) or "private"
-  // "private" disables relay-style providers and public search helpers
-  "profile": "balanced",
-
-  "runtime": {
-    // Optional override for staging/export temp files
-    // Default: platform cache root (e.g. ~/.cache/smart-web/tmp)
-    "tempDir": ""
-  },
-
-  "search": {
-    // Self-hosted SearXNG instance URL
-    "searxngBaseUrl": "",
-    "enableSearxng": true
-  },
-
-  "fetch": {
-    // Browser overrides — most users leave these empty
-    "chromeChannel": "",
-    "chromePath": "",
-    // Auto-install Playwright Chromium when missing (default: true)
-    "autoInstallPlaywright": true,
-    // Jina Reader relay — opt-in third-party path for weak article pages
-    "enableJinaReader": false,
-    "jinaReaderBaseUrl": "https://r.jina.ai/",
-    // Academic fallback — legal OA enrichment for paper URLs
-    "enableAcademicFallback": true,
-    "enableOpenAlex": true,
-    "enableEuropePmc": true,
-    "enableBiorxivApi": true,
-    // Unpaywall — requires a contact email
-    "enableUnpaywall": true,
-    "unpaywallEmail": "",
-    // Semantic Scholar — optional API key for higher-rate access
-    "enableSemanticScholar": true,
-    "semanticScholarApiKey": "",
-    // CORE — optional API key for richer search-backed enrichment
-    "enableCoreDiscovery": true,
-    "coreApiKey": "",
-    // FxTwitter — transparent x.com → fxtwitter redirect
-    "enableFxTwitter": true,
-    // Undetected-chromedriver — optional Python Selenium fallback for tough anti-bot pages
-    "enableUndetectedChromedriver": true,
-    "undetectedChromedriverPython": "python3",
-    // Reddit personal retrieval chain (legacy key name retained for compatibility)
-    "enableRedditJson": true,
-    // YouTube video transcript enrichment for public captions (default: true)
-    "enableYoutubeTranscript": true,
-    // Archive fallback — Wayback and archive.md recovery
-    "enableArchiveFallback": true,
-    "enableWayback": true,
-    "enableArchiveMd": true,
-    // Optional local adaptive scraper fallback. This is disabled by default;
-    // install Scrapling separately and opt in only on deployments that want an
-    // extra no-secret local shell/bot-challenge extraction attempt before the
-    // built-in Playwright fallback.
-    "enableAdaptiveScraper": false,
-    "adaptiveScraperCommand": "scrapling",
-    "adaptiveScraperTimeoutMs": 20000,
-    // Output projection budget for smartfetch responses. MCP, CLI, and library
-    // calls honor this setting unless the caller passes a per-call budget control.
-    // With no setting or per-call control, MCP calls default to compact.
-    // preset: "compact" | "balanced" | "full"; numeric fields override the preset.
-    "outputBudget": {
-      "preset": "balanced",
-      "maxPostTextChars": 8000,
-      "maxPostMarkdownChars": 10000,
-      "maxThreadItems": 10,
-      "maxCommentItems": 10,
-      "maxOutboundLinks": 60,
-      "maxAssets": 15,
-      "maxBlocks": 20,
-      "maxBlockTextChars": 500,
-      "maxItemTextChars": 500
-    },
-    // Optional default named browser profile for authenticated smartfetch calls.
-    // Names are 1-64 characters: letters, numbers, dots, underscores, or hyphens;
-    // they must start with a letter or number and cannot end with a dot.
-    "browserProfile": ""
-  },
-
-  "network": {
-    // Allow localhost/private/reserved fetch targets (default: false)
-    // Applies to smartfetch, smartcrawl, and Korean public API helpers.
-    "allowPrivateHosts": false
-  }
-}
-```
-
-### Common setups
-
-#### Default — no config file needed
-
-Out of the box `smart-web` works with sensible defaults. Create a settings file only when you need to change something.
-
-#### Minimal: profile only
-
-```json
-{
-  "profile": "balanced"
-}
-```
-
-#### Private / local-first
-
-```json
-{
-  "profile": "private",
-  "search": {
-    "searxngBaseUrl": "http://localhost:8080"
-  }
-}
-```
-
-In `private` mode, relay-style provider requests are blocked, public no-key search fallbacks are disabled, and SearXNG bases must resolve to localhost or private addresses. Hosted API-key search providers such as Exa, Tavily, and Brave Search are intentionally not part of the default smartsearch runtime; use local/site-native search, DuckDuckGo/Brave HTML fallbacks, or self-hosted SearXNG instead.
-
-#### Disable Playwright auto-install
-
-```json
-{
-  "fetch": {
-    "autoInstallPlaywright": false
-  }
-}
-```
-
-The server will return an actionable `playwright_browsers_missing` or `playwright_browsers_outdated` error instead.
-
-#### Opt in to Jina Reader for weak generic article pages
-
-```json
-{
-  "fetch": {
-    "enableJinaReader": true
-  }
-}
-```
-
-This stays opt-in because it is a third-party relay path. When enabled, it only applies to weak generic/article direct fetches and prefers Jina JSON mode when that improves the page. `profile: "private"` hard-disables relay-style providers.
-
-Relay eligibility is provider-policy based. Generic and document-like providers can opt in, while browser/auth/social/commerce providers stay off unless their provider declares support. `pipeline.policy.relay_style_allowed` and `pipeline.policy.relay_style_used` tell hosts whether a relay path was allowed and whether it was used.
-
-#### Opt in to a local adaptive scraper command
-
-```json
-{
-  "fetch": {
-    "enableAdaptiveScraper": true,
-    "adaptiveScraperCommand": "scrapling",
-    "adaptiveScraperTimeoutMs": 20000
-  }
-}
-```
-
-This lane is for local, no-secret deployments that install Scrapling or a compatible CLI themselves. It runs only after direct retrieval still looks weak, blocked, or shell-like and before the built-in browser fallback. The attempt is recorded as `adaptive_scraper` in `pipeline.acquire.attempt_map`; if the command is missing, times out, returns empty text, or does not improve the active result, smartfetch keeps the existing result and continues through the normal fallback chain.
-
-#### Tune smartfetch output budgets
+MCP host configuration:
 
 ```json
 {
-  "fetch": {
-    "outputBudget": {
-      "maxPostTextChars": 16000,
-      "maxThreadItems": 25,
-      "maxOutboundLinks": 80
-    }
+  "mcpServers": {
+    "smart-web": { "command": "npx", "args": ["-y", "smart-web-mcp"] }
   }
 }
 ```
 
-Extraction runs before budgeting so providers can work from complete page content. The budget applies to returned `smartfetch` projections and records truncation under `budget.fields`.
+Use `--init-settings` to create the default settings file, or `--print-settings-example` to inspect it. The published npm package can lag source; verify the runtime you actually use.
 
-#### Force archive fallback off
-
-```json
-{
-  "fetch": {
-    "enableArchiveFallback": false
-  }
-}
-```
-
-#### Legal OA DOI recovery for paywalled paper pages
-
-```json
-{
-  "fetch": {
-    "enableUnpaywall": true,
-    "unpaywallEmail": "research@example.com",
-    "enableSemanticScholar": true,
-    "enableCoreDiscovery": true
-  }
-}
-```
-
-#### Enable undetected-chromedriver from a dedicated virtualenv
-
-```json
-{
-  "fetch": {
-    "enableUndetectedChromedriver": true,
-    "undetectedChromedriverPython": "/absolute/path/to/venv/bin/python"
-  }
-}
-```
-
-This helper is optional. `smart-web` still works without it, but when installed and reachable it can act as a second browser engine for stubborn pages where Playwright alone is not enough.
-
-#### Custom temp directory
-
-```json
-{
-  "runtime": {
-    "tempDir": "/absolute/path/to/smart-web-tmp"
-  }
-}
-```
-
-### Advanced overrides
-
-These keys live inside `settings.json` when you need to force one provider on or off:
-
-**Search**
-
-- `search.enableSearxng`
-- `search.enableDuckDuckGo`
-- `search.enableBraveHtml`
-
-**Fetch**
-
-- `fetch.enableFxTwitter`
-- `fetch.enableXOembed`
-- `fetch.autoInstallPlaywright`
-- `fetch.enableJinaReader`
-- `fetch.jinaReaderBaseUrl`
-- `fetch.enableAcademicFallback`
-- `fetch.enableOpenAlex`
-- `fetch.enableEuropePmc`
-- `fetch.enableBiorxivApi`
-- `fetch.enableUnpaywall`
-- `fetch.unpaywallEmail`
-- `fetch.enableSemanticScholar`
-- `fetch.semanticScholarApiKey`
-- `fetch.enableCoreDiscovery`
-- `fetch.coreApiKey`
-- `fetch.enableUndetectedChromedriver`
-- `fetch.undetectedChromedriverPython`
-- `fetch.enableRedditJson`
-- `fetch.enableYoutubeTranscript`
-- `fetch.enableArchiveFallback`
-- `fetch.enableWayback`
-- `fetch.enableArchiveMd`
-- `fetch.outputBudget.maxPostTextChars`
-- `fetch.outputBudget.maxPostMarkdownChars`
-- `fetch.outputBudget.maxThreadItems`
-- `fetch.outputBudget.maxCommentItems`
-- `fetch.outputBudget.maxOutboundLinks`
-- `fetch.outputBudget.maxAssets`
-- `fetch.outputBudget.maxBlocks`
-- `fetch.outputBudget.maxBlockTextChars`
-- `fetch.outputBudget.maxItemTextChars`
-- `fetch.outputBudget.maxEnvelopeArrayItems`
-- `fetch.outputBudget.maxAssessmentSignals`
-- `fetch.outputBudget.maxTotalEnvelopeEntries`
-- `fetch.outputBudget.maxTotalEnvelopeStringChars`
-
-**Compatibility**
-
-- `network.localOnly`: advanced override that forces local-only behavior regardless of profile
-
-## Provider benchmark
-
-Use the provider benchmark when deciding whether to build, wrap, or replace retrieval lanes:
-
-```bash
-npm run benchmark:providers
-npm run benchmark:providers:measure
-npm run benchmark:providers:compare
-npm run --silent benchmark:providers:json
-npm run benchmark:providers:json-file
-npm run benchmark:providers:report
-npm run benchmark:providers:gaps
-npm run benchmark:providers:live
-npm run benchmark:providers:live:jina
-npm run benchmark:providers:live:cp
-npm run benchmark:providers:live:browser
-npm run benchmark:providers:live:youtube
-npm run benchmark:providers:live:blocked
-npm run benchmark:providers:live:naver-blog
-npm run benchmark:providers:live:naver-map
-npm run benchmark:providers:live:naver-cafe
-npm run benchmark:providers:live:x-twitter
-npm run benchmark:providers:live:no-secret
-npm run benchmark:providers:live:no-secret:gaps
-npm run watchdog:no-secret:live
-npm run benchmark:providers:trend
-npm run benchmark:smartsearch
-npm run --silent benchmark:smartsearch:json
-npm run watchdog:no-secret
-npm run watchdog:partial-handoff
-```
-
-The default report is availability-only and safe for CI. `--run-measurements` (`npm run benchmark:providers:measure`) adds deterministic fixture records for available local/core and optional no-key lanes: success, useful character counts, projected/text character counts, useful-to-projected and useful-to-text yield ratios, latency, extractor/provider id, confidence, partial status, error class, lane label/family/category metadata, provider-family/category lane-status rollups, surface coverage, and recommendation sections grouped by lane, case, and surface.
-
-Use `--compare-external` (`npm run benchmark:providers:compare`) to add an `externalComparisons` section that compares current smart-web fixture evidence against optional no-secret/local candidates. The comparison includes the local Crawl4AI command lane, the public Jina Reader relay fixture, and available direct/Playwright baselines. If `crawl4ai` is not installed, the Crawl4AI row is `skipped` with `action: "install-local-prerequisite"` and `prerequisite.command: "crawl4ai"`; this is an expected setup signal, not a benchmark failure or required dependency.
-
-Reports carry the `provider-benchmark.v1` schema version plus non-fatal git and runtime metadata so archived JSON, Markdown, and NDJSON trend artifacts are parseable, traceable, and comparable as the harness evolves. Markdown, text, JSON, and NDJSON outputs include aggregate fixture yield averages, compact yield rollups by provider family/category/lane, and a best-current-evidence surface table that ranks measured/skipped candidates so review jobs can compare token efficiency and build-vs-wrap posture between core, browser, relay, and specialist paths without reprocessing the full measurement table.
-
-Measurement artifacts also include a live-readiness manifest listing each lane's env/command/settings prerequisites, a deterministic fixture smoke, a safe MCP live-smoke candidate, and surfaces that still lack live evidence. `--run-live-smokes` (`npm run benchmark:providers:live`) is an explicit gated mode that runs only no-secret, core-surface live checks for static articles, long docs/Wikipedia, Hacker News, and docs export; optional no-secret lanes remain gated until explicitly allowlisted: `--allow-jina-live-smoke` (`npm run benchmark:providers:live:jina`) exercises the public Jina Reader relay with a secret-stripped settings file, `--allow-competitive-programming-live-smoke` (`npm run benchmark:providers:live:cp`) adds a public BOJ/Codeforces representative smoke for the competitive-programming specialist surface, `--allow-browser-live-smoke` (`npm run benchmark:providers:live:browser`) runs a Playwright `--force-dynamic` smoke for the JS-rendered article surface only when the local browser lane is available, `--allow-youtube-live-smoke` (`npm run benchmark:providers:live:youtube`) adds the public YouTube seed-lane representative smoke, `--allow-blocked-authwall-live-smoke` (`npm run benchmark:providers:live:blocked`) adds a deterministic HTTP Basic Auth challenge smoke that treats honest blocked/error/handoff signals as positive calibration without private credentials, `--allow-naver-blog-live-smoke` (`npm run benchmark:providers:live:naver-blog`) adds a stable public Naver Blog representative smoke for Korean specialist-lane calibration, `--allow-naver-map-live-smoke` (`npm run benchmark:providers:live:naver-map`) adds a stable public Naver Map place smoke with browser-preflight diagnostics, `--allow-naver-cafe-live-smoke` (`npm run benchmark:providers:live:naver-cafe`) adds a stable public Naver Cafe shared-link smoke that treats useful no-credential partial output as honest calibration, and `--allow-x-twitter-live-smoke` (`npm run benchmark:providers:live:x-twitter`) adds a public X/Twitter post smoke that keeps no-secret social-provider drift visible without private cookies. `--allow-all-no-secret-live-smokes` (`npm run benchmark:providers:live:no-secret`) enables every no-secret gate in one run; hosted API-key search lanes are excluded from the benchmark surface by product policy.
-
-Live smoke artifacts record per-case `present`/`missing`/`skipped` evidence next to fixture evidence without printing secret values. When fixture rows exist but all are skipped by missing local commands, comparison and gap artifacts report fixture evidence as `skipped` with a `fixture-skipped-*` comparison instead of implying the fixture rows are absent. Browser-lane live smoke rows also include a secret-safe Playwright preflight diagnostic with executable path, launch timing, `domcontentloaded` navigation timing, HTTP status/final URL, and truncated failure reason so runtime/cache/network failures can be separated from smartfetch normalization failures.
-
-The same JSON, text, Markdown, and NDJSON outputs include a compact live-vs-fixture comparison rollup so replacement reviews can distinguish calibrated surfaces from fixture-only, skipped-live, live-disagreeing, and fixture-skipped-by-prerequisite surfaces. Secret-safe provider diagnostics list prerequisite names with satisfied/missing status and never print env values. Use `npm run benchmark:providers:json-file` or add `--json-file <path>` to write the full JSON artifact without relying on stdout capture. Use `npm run benchmark:providers:report` or add `--report-file <path>` with measurement mode to write a Markdown review artifact containing lane availability, diagnostics, surface/corpus coverage, measurement highlights, yield comparisons, per-surface evidence, live readiness, live-smoke diagnostics, and build-vs-wrap recommendations without changing stdout or JSON behavior. Use `npm run benchmark:providers:gaps` or add `--gap-file <path>` with measurement mode to write a compact `provider-benchmark-gaps.v1` JSON artifact containing only the ranked provider evidence gap queue, summary counts, git/runtime metadata, and safe smoke candidates. Use `npm run benchmark:providers:trend` or add `--trend-file <path>` with measurement mode to append compact NDJSON trend records with aggregate lane, prerequisite, measurement, recommendation, surface-evidence, and fixture-yield counts. Unknown `--flag` options fail fast so typoed benchmark jobs do not produce misleading artifacts. The corpus covers static articles, JS-rendered articles, blocked/authwall-like pages, long docs/Wikipedia pages, Naver Blog/Cafe/Map, Hacker News, YouTube, X/Twitter, BOJ/Codeforces, relay, and docs export surfaces. Optional local/no-secret lanes without commands remain explicit skips instead of silently failing; API-key provider candidates are excluded from the default corpus and reports by product policy.
-
-Provider benchmark reports also include `providerEvidenceGaps`: a ranked next-action queue built from the live-vs-fixture rollup. Each gap records the surface, priority score, selected and candidate lanes, follow-up action, unlock prerequisites, fixture/live useful chars, fixture skipped-case lanes/reasons, live-smoke failure error classes/reasons, and safe live smoke candidate so replacement reviews can pick the next provider experiment without scanning every measurement row or re-running a timed live calibration. Use `npm run benchmark:providers:live:no-secret:gaps` to refresh only the compact all-no-secret gap artifact at `reports/provider-benchmark-live-no-secret-gaps.json`. When fixture rows exist but were skipped by missing local commands, the live-vs-fixture and gap actions are `unlock-skipped-fixture-prerequisites` rather than fixture-creation actions.
-
-`npm run benchmark:smartsearch` is the deterministic quality gate for site-native smartsearch routing. It patches `fetch`, disables generic web-search fallbacks with a temporary settings file, and exercises representative `site:` queries for Reddit, GitHub repo discovery, GitHub repo docs, npm, Hacker News, Stack Exchange, Wikipedia, Velog, MDN, and crates.io without live network access. The human report is compact; `npm run --silent benchmark:smartsearch:json` emits `smartsearch-quality-benchmark.v1` JSON with expected engine, top URL, URL-substring checks, actual engine, top URL, notes, and pass/fail status per case.
-
-See `reports/README.md` for the curated report archive index. It identifies `provider-benchmark-live-no-secret.{json,md}` and `provider-benchmark-live-no-secret-gaps.json` as the current maintenance artifacts, and labels older cumulative live-smoke reports as historical calibration snapshots.
-
-`npm run watchdog:no-secret` is the deterministic current-surface gate for no-secret maintenance. It fails when `reports/provider-benchmark-live-no-secret-gaps.json` contains current gaps, when runtime/docs/examples/scripts reintroduce hosted API-key provider markers for Exa, Tavily, Firecrawl, Browserless, or Brave Search API, or when the partial/handoff watchdog no longer has a compact text ceiling. It intentionally ignores historical `CHANGELOG.md` entries and archived `reports/` output, and does not run live network smokes.
-
-`npm run watchdog:no-secret:live` is the live regression gate for no-secret maintenance. It first refreshes `reports/provider-benchmark-live-no-secret-gaps.json` with `npm run benchmark:providers:live:no-secret:gaps`, then runs `npm run watchdog:no-secret` so non-empty live provider gaps fail locally or in CI. The scheduled/manual GitHub Actions workflow `.github/workflows/no-secret-live-watchdog.yml` runs the same alias weekly with read-only permissions, no secrets, and a short-retention gap artifact.
-
-`npm run watchdog:partial-handoff` builds the package and runs a compact no-secret drift watchdog over representative partial/blocked/handoff `smartfetch` outputs (httpbin authwall, LinkedIn profile authwall, Naver Blog unavailable/deleted posts, Threads unavailable/join-login output, DCInside unavailable/deleted posts, Telegram missing/private posts, a non-existent Wikidocs article, Naver Cafe, Naver Map search, and BOJ unavailable/shutdown output). It exits non-zero when expected handoff cases are silently reclassified as non-partial, when partial/handoff `content[0].text` grows beyond the compact watchdog ceiling, or when compact text or structured links/assets leak chrome URLs, script globals, or style-shell tokens, and prints a `partial-handoff-watchdog.v1` JSON report without raw page dumps or credentials. For runner artifacts without JSON stdout, run `node scripts/partial-handoff-watchdog.mjs --report-file <path>` after `npm run build`; the command keeps stdout human-readable and writes the full JSON report to the requested path. Fixture cases can set `expectPartialHandoff: true` to make classifier drift fail even if the output would otherwise be skipped.
-
-## Quick verification
-
-Sanity-check the server with a few real calls:
-
-- `smartfetch` on a normal article URL
-- `smartfetch` on a Medium or other member-only article URL — confirm it returns either archive-backed content or an honest `reference_only` preview
-- `smartfetch` on a Naver Map, Kakao Map, or `naver.me` place-share URL
-- `smartfetch` on an arXiv abstract URL
-- `smartfetch` on a PubMed, PMC, or bioRxiv/medRxiv paper URL
-- `smartfetch` on a known product URL
-- `smartsearch` on a `site:` query
-- `smartcrawl` on a docs site or board you actually use
-
-## Reporting issues
-
-If `smart-web` returns reproducible incorrect or misleading output, open a GitHub issue with:
-
-- exact input and tool arguments
-- observed output
-- expected behavior
-- `smart-web-mcp` version
-
-Skip obvious transient network, auth, or rate-limit failures unless the classification itself looks wrong.
+## Tool routing
 
-## License
+- `smartfetch` first when the user already gave a URL or shortlink.
+- `smartsearch` when the user gave a topic, keywords, or a `site:` query.
+- `smartcrawl` when you already know the site and need multiple relevant pages.
+- `smartvertical` for Korean/local structured lookups.
+- `smartbrowser` for explicit approved-browser input/click/filter/wait/screenshot batches; see [action scope and limits](https://github.com/jojo-labs/smart-web/blob/main/docs/operator-actions.md).
 
-All rights reserved. See [LICENSE](LICENSE) for terms.
+Use compact output first; request a larger budget when needed. Partial output is honest evidence, **not task completion**. Never substitute a search snippet, loading screen, or unrelated listing for the requested source.
+The CLI `smart-web fetch <url> --format json` (or `--json`) prints structured output; `--budget full` controls bounded delivery, not source completeness. Invalid budget names fail before retrieval rather than silently using compact output.
+For CLI search, `--max-chars` caps rendered text; use `--json` when the complete projected structured result is needed.
 
-This software is licensed under a proprietary license that permits installation and use as a local MCP server but prohibits modification, redistribution, reverse engineering, or use in competing products.
+## Documentation
 
-## References
+- [Product contract and acceptance](https://github.com/jojo-labs/smart-web/blob/main/docs/operator-contract.md)
+- [Host setup](https://github.com/jojo-labs/smart-web/blob/main/docs/host-setup.md) · [Configuration](https://github.com/jojo-labs/smart-web/blob/main/docs/configuration.md)
+- [Architecture and code map](https://github.com/jojo-labs/smart-web/blob/main/docs/architecture.md) · [Verification and maintenance](https://github.com/jojo-labs/smart-web/blob/main/docs/benchmarks.md)
+- [All docs](https://github.com/jojo-labs/smart-web/blob/main/docs/README.md) · [Agent instructions](https://github.com/jojo-labs/smart-web/blob/main/AGENTS.md)
 
-[^1]: Model Context Protocol, "Tools" specification — MCP tools are a retrieval and integration surface, not a requirement to emulate full browser-automation flows inside one tool.
+Report reproducible problems in [GitHub Issues](https://github.com/jojo-labs/smart-web/issues), with sanitized input, expected/observed behavior, and runtime version. Never attach credentials or private browsing content.
 
-[^2]: Model Context Protocol Blog, "Tool Annotations as Risk Vocabulary: What Hints Can and Can't Do" (2026-03-16) — structured tool output and accurate hints improve host-side routing and escalation decisions.
+[Package](https://www.npmjs.com/package/smart-web-mcp) · MCP Registry: `io.github.jojo-labs/smart-web` · [License](https://github.com/jojo-labs/smart-web/blob/main/LICENSE) (proprietary; all rights reserved).