smart-web-mcp 0.7.6 → 0.7.7

package/CHANGELOG.md

@@ -6,6 +6,12 @@ The format is based on Keep a Changelog[^1] and this project uses SemVer.
 
 ## [Unreleased]
 
+## [0.7.7] - 2026-03-29
+
+### Changed
+
+- the README now leads with agent and MCP-host routing concerns instead of first-time human onboarding flow, which makes the package positioning and tool contract clearer for the way this repo is actually used
+
 ## [0.7.6] - 2026-03-29
 
 ### Added

package/README.md

@@ -2,74 +2,67 @@
 
 # smart-web
 
-One local MCP server for web search, page fetch, and site crawl.
+Local MCP for agentic web retrieval.
 
-If you are installing this for the first time, the short version is:
+`smart-web` bundles three tools in one stdio server:
 
-1. install the MCP with `npx -y smart-web-mcp`
-2. wire it into your host once
-3. use `smartfetch` for direct URLs, `smartsearch` for discovery, and `smartcrawl` for multi-page traversal
+- `smartfetch` for direct URLs
+- `smartsearch` for discovery
+- `smartcrawl` for same-site multi-page traversal
 
-`smart-web` is built for **content retrieval**, not form-filling automation. It tries to return clean, structured results first, then tells the host when a real browser-task runtime would likely do better.
+It is designed for **agents and MCP hosts**, not for manual browser automation. The job is to return structured retrieval output first, then signal when a browser-task runtime would be a better next step.
 
-`smart-web` ships three primary tools in one stdio server:
-
-- `smartsearch`: fallback web search with optional site-crawl handoff
-- `smartfetch`: browser-aware direct-URL fetch with target-specific normalization
-- `smartcrawl`: site exploration for docs, boards, pagination, scroll-heavy pages, and optional local Markdown export via `output_dir`
-
-All three tools now return a structured `assessment` block so hosts can tell when deterministic retrieval was strong enough versus when a browser-task runtime would likely do better.
-
-Migration note: if you previously called `smartdocscrawl`, switch to `smartcrawl` with the same crawl parameters plus `output_dir`.
-
-## Why people use it
-
-- one local MCP instead of separate search/fetch/crawl servers
-- better normalized outputs for common web surfaces instead of raw HTML dumps
-- explicit confidence/blocking/handoff signals so agents can make better decisions
-- local-first and privacy-first by default
-
-Current high-signal targets include:
-
-- communities: Reddit, DCInside, LinkedIn public posts, Algumon
-- reference/article pages: NamuWiki, Wikipedia, Naver Blog, Tistory, Velog
-- commerce pages: Coupang, Danawa, Aladin, AliExpress, Amazon
-- media/social: YouTube, X, Threads, Instagram, Telegram
-
-## What it is not
+- npm: `https://www.npmjs.com/package/smart-web-mcp`
+- MCP Registry: `io.github.rich-jojo/smart-web`
+- GitHub: `https://github.com/rich-jojo/smart-web`
 
-- not a full browser automation tool for clicking and form filling
-- not an enterprise anti-bot proxy network
-- not a cloud crawler that tries to out-scale Bright Data / Firecrawl / Apify
+## Highlights
 
-Instead, `smart-web` aims to be the strong **local deterministic retrieval layer** that you put in front of heavier browser-task or cloud scraping systems when those are actually needed.
+- one local MCP instead of separate search, fetch, and crawl servers
+- explicit routing: URL -> `smartfetch`, query -> `smartsearch`, known site -> `smartcrawl`
+- structured output with `assessment` and `pipeline` fields for host-side decision making
+- local-first defaults with privacy-aware search/fetch behavior
+- useful normalization for common public surfaces instead of raw HTML dumps
 
 ## Tool routing
 
 Use the tools like this:
 
 - `smartfetch` first when the user already gave a URL or shortlink
-- `smartsearch` when the user gave a topic, keywords, or a `site:` query and you need discovery
-- `smartcrawl` when you already know the site and need multiple relevant pages instead of one page, optionally with `output_dir` when you want a local Markdown mirror instead of only a ranked summary
+- `smartsearch` when the user gave a topic, keywords, or a `site:` query
+- `smartcrawl` when you already know the site and need multiple relevant pages
 
-`smartfetch` also works for commerce-style pages now: search/listing pages can be normalized into product results, and product detail pages can surface title, price, rating, and availability when the source site exposes them.
+| Situation | Tool |
+| --- | --- |
+| User already gave a URL or shortlink | `smartfetch` |
+| User gave a topic, keywords, or `site:` query | `smartsearch` |
+| You already know the site and need multiple pages | `smartcrawl` |
 
-That includes explicit support for Amazon, Coupang, Danawa, Aladin, and AliExpress commerce surfaces, plus direct extraction support for Algumon hot-deal pages and reference pages such as NamuWiki and Wikipedia. When a storefront exposes too little stable metadata, `smartfetch` should stay partial and tell the host that the page is not confidently verified yet.
+`smart-web` is a retrieval layer. It is not a general-purpose click/type/form-fill browser agent.[^1]
 
-For hostile or authwalled surfaces, `smartfetch` now also has a few targeted upgrades:
+## What the tools return
 
-- X status URLs such as `/i/status/...` prefer public relay expansion and return a much cleaner post summary by default
-- LinkedIn profile URLs return a short authwall-aware summary instead of dumping the whole signup wall
-- LinkedIn profile `smartcrawl` is now a best-effort public-post discovery path instead of crawling irrelevant LinkedIn navigation pages
-- an optional Python `undetected-chromedriver` helper is enabled by default when the helper is actually installed and reachable, and can be pointed at a virtualenv python for tougher browser fetches when Playwright alone is not enough
+All three tools are shaped for agent consumption.
 
-For LinkedIn profile URLs specifically, `smartcrawl` should currently be read as **best-effort recent public post discovery**, not full historical “all posts” pagination.
+Common high-signal fields include:
 
-This routing mirrors MCP tool-design guidance: clear human-readable descriptions, schema field descriptions, and explicit tool behavior hints make host/tool selection more reliable.[^1][^2]
+- `assessment`: confidence, block/auth hints, recommended handoff
+- `pipeline`: resolve/acquire/normalize/assess stages
+- `errors`: structured failure reasons
+- `post`, `thread`, `comments`: normalized content surfaces
+- `assets` and `download`: file-like outputs when present
 
-- npm: `https://www.npmjs.com/package/smart-web-mcp`
-- MCP Registry: `io.github.rich-jojo/smart-web`
-- GitHub: `https://github.com/rich-jojo/smart-web`
+This lets hosts decide whether deterministic retrieval was good enough or whether they should escalate to a browser-task runtime.[^2]
+
+## Supported high-signal surfaces
+
+- communities: Reddit, DCInside, LinkedIn public posts, Algumon
+- reference/article pages: NamuWiki, Wikipedia, Naver Blog, Tistory, Velog
+- media/social: YouTube, X, Threads, Instagram, Telegram
+- commerce pages: Amazon, Coupang, Danawa, Aladin, AliExpress
+- archives/reference wrappers: Wayback snapshots, archive.md lookup pages, arXiv abstract pages with direct paper links
+
+When a source stays blocked or under-specified, `smart-web` prefers partial but honest output over pretending to have verified content.
 
 ## Install
 
@@ -79,19 +72,17 @@ npx -y smart-web-mcp
 
 `npm` is the canonical package-manager path for this repo and package.
 
-## Quick start
-
-### 1) Add the MCP to your host
+## Host setup
 
-Claude Code:
+### Claude Code
 
 ```bash
 claude mcp add smart-web -- npx -y smart-web-mcp
 ```
 
-If you prefer Claude Code marketplace install/discovery, the repo also keeps a small Claude plugin wrapper. It is worth keeping even without custom hooks/skills because the plugin gives Claude-native marketplace distribution and a single optional `settings_file` user config instead of a long MCP env block.
+The repo also ships a small Claude plugin wrapper for marketplace-style distribution.
 
-OpenCode:
+### OpenCode
 
 ```json
 {
@@ -106,42 +97,27 @@ OpenCode:
 }
 ```
 
-### 2) Verify it with a few real calls
-
-- `smartfetch` on a normal article URL
-- `smartfetch` on a known Reddit, YouTube, or product URL
-- `smartsearch` on a `site:` query
-- `smartcrawl` on a docs site or board you actually use
-
-### 3) Start using the routing rule daily
-
-- direct URL already provided -> `smartfetch`
-- topic/keywords only -> `smartsearch`
-- same site, many pages needed -> `smartcrawl`
-
 ## Configuration
 
-`smart-web` now prefers one settings file instead of a long MCP `env` block.
-
-Default path:
+Default settings path:
 
 ```text
 ~/.config/smart-web/settings.json
 ```
 
-You can print a template with:
+Print a template:
 
 ```bash
 npx -y smart-web-mcp --print-settings-example
 ```
 
-Or initialize the default file with:
+Initialize the default file:
 
 ```bash
 npx -y smart-web-mcp --init-settings
 ```
 
-If you need a non-default path, set only one variable in the host config:
+Use a non-default path:
 
 ```json
 {
@@ -151,27 +127,34 @@ If you need a non-default path, set only one variable in the host config:
 }
 ```
 
-Environment variables are still supported as direct overrides, but the recommended steady-state setup is one `settings.json` file.[^3]
+Recommended steady state: keep most runtime configuration in one settings file instead of a long MCP `env` block.
+
+Most installations only need:
 
-Most users only need:
+- `profile: "balanced"`
+- `profile: "private"`
+- `search.searxngBaseUrl`
+- `search.exaApiKey`
+- `search.braveSearchApiKey`
 
-- `profile: "balanced"`: default behavior
-- `profile: "private"`: disables relay-style providers and public search helpers by default
-- `search.searxngBaseUrl`: self-hosted private fallback search
-- `search.exaApiKey` and `search.braveSearchApiKey`: optional paid search tiers
+## Quick verification
 
-Everything else lives in the same JSON file as advanced overrides rather than exploding the MCP config.
+Sanity-check the server with a few real calls:
 
-## Smoke test
+- `smartfetch` on a normal article URL
+- `smartfetch` on an arXiv abstract URL
+- `smartfetch` on a known product URL
+- `smartsearch` on a `site:` query
+- `smartcrawl` on a docs site or board you actually use
 
-- ask `smartsearch` for `site:velog.io react useeffect cleanup`
-- ask `smartfetch` to fetch a normal article URL
-- ask `smartfetch` to fetch `https://www.algumon.com/deal/rank?categoryId=1`
-- ask `smartfetch` to fetch `https://namu.wiki/w/%EC%95%8C%EA%B5%AC%EB%AA%AC`
-- ask `smartfetch` to fetch a known Coupang/Danawa/Aladin/AliExpress product or search URL
-- ask `smartfetch` with `download: true` on a direct PDF or media URL when you want the result shaped around the file target
-- ask `smartcrawl` to explore a docs site or forum you actually use
-- ask `smartcrawl` with `output_dir` to mirror a docs section into a local folder you can grep, index, or commit
+## Examples
+
+- [Claude Code](examples/claude.mcp.json)
+- [OpenCode](examples/opencode.json)
+- [OpenCode dev hot-reload](examples/opencode.dev.json)
+- [Settings template](examples/smart-web.settings.json)
+
+These examples use `/absolute/path/to/...` placeholders for local checkouts.
 
 ## Docs
 
@@ -185,15 +168,6 @@ Everything else lives in the same JSON file as advanced overrides rather than ex
 - [Contributing](CONTRIBUTING.md)
 - [CHANGELOG](CHANGELOG.md)
 
-## Examples
-
-- [Claude Code](examples/claude.mcp.json)
-- [OpenCode](examples/opencode.json)
-- [OpenCode dev hot-reload](examples/opencode.dev.json)
-- [Settings template](examples/smart-web.settings.json)
-
-These examples use `/absolute/path/to/...` placeholders for local checkouts.
-
 ## Development
 
 ```bash
@@ -203,18 +177,16 @@ npm run check
 
 `npm install` also installs local git hooks for fast TypeScript checks on commit and the full verification suite on push.
 
-### Dev MCP without restarting OpenCode
-
-If you run `smart-web` from a local repo checkout, you can use the dev entrypoint so normal implementation changes are picked up after a rebuild without restarting the MCP host:
+### Dev MCP without restarting the host
 
 ```bash
 npm run dev:watch
 smart-web-dev
 ```
 
-`smart-web-dev` hot-swaps the latest rebuilt `dist/*.js` modules on each tool call. That means existing tool implementations update after `tsc` rebuilds, but adding brand-new tools or changing tool schemas still needs a client reconnect because MCP tool registration happens at process start.
+`smart-web-dev` hot-swaps rebuilt `dist/*.js` modules on each tool call. New implementations update after rebuilds, but brand-new MCP tools or schema changes still require a client reconnect because MCP registration happens at process start.
 
-By default the dev/runtime staging snapshots now live under the platform cache root (for example `~/.cache/smart-web/tmp` on Linux) instead of inheriting an ambient home-directory temp root. Set `SMART_WEB_TMP_DIR` if you need an explicit override.
+By default, staging snapshots live under the platform cache root (for example `~/.cache/smart-web/tmp` on Linux). Set `SMART_WEB_TMP_DIR` if you need an explicit override.
 
 ## License
 
@@ -222,6 +194,5 @@ MIT
 
 ## References
 
-[^1]: Model Context Protocol, "Tools" specification — tool definitions may include a title, human-readable descriptions, JSON Schema input metadata, and tool annotations such as `readOnlyHint` and `openWorldHint`.
-[^2]: Model Context Protocol Blog, "Tool Annotations as Risk Vocabulary: What Hints Can and Can't Do" (2026-03-16) — `title` is useful UI metadata, while behavior annotations should be accurate but treated as hints.
-[^3]: VS Code MCP configuration reference documents `env` and `inputs` on the client side, which is why `smart-web` supports both env-based overrides and a single settings file while keeping the main runtime behavior centralized.
+[^1]: Model Context Protocol, "Tools" specification — MCP tools are a retrieval and integration surface, not a requirement to emulate full browser-automation flows inside one tool.
+[^2]: Model Context Protocol Blog, "Tool Annotations as Risk Vocabulary: What Hints Can and Can't Do" (2026-03-16) — structured tool output and accurate hints improve host-side routing and escalation decisions.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "smart-web-mcp",
   "mcpName": "io.github.rich-jojo/smart-web",
-  "version": "0.7.6",
+  "version": "0.7.7",
   "packageManager": "npm@10.8.2",
   "description": "Portable MCP server for web search, direct-URL fetch, site crawling, and docs export.",
   "homepage": "https://github.com/rich-jojo/smart-web",