smart-web-mcp 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 rich-jojo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,191 @@
+# smart-web
+
+`smart-web` is a portable MCP server that exposes two tools:
+
+- `smartsearch`: fallback web search with API-key tiers and no-key fallbacks
+- `smartfetch`: browser-aware page retrieval with escalation to Playwright when needed
+
+It is built for Claude Code, OpenCode, and any other MCP client that can run a local stdio server.
+
+## Why this exists
+
+Most AI coding setups still treat search and fetch as separate, inconsistent integrations.
+
+- Good search APIs are often paid
+- no-key fallbacks are usually weak or privacy-unclear
+- page fetchers often fail on JS-heavy or bot-protected sites
+- tool setups are usually client-specific instead of portable
+
+`smart-web` packages a practical default stack into one MCP server with two focused tools.
+
+## What you get
+
+- One MCP install, two tools
+- Official MCP SDK over stdio
+- Structured outputs plus readable text/markdown rendering
+- Search fallback chain: Exa -> Brave -> SearXNG -> DuckDuckGo HTML
+- Fetch fallback chain: impit -> stealth Playwright
+- Target-aware normalization for Reddit, X, and DCInside
+- Default blocking for localhost, private IP ranges, and reserved address space
+- Shared setup examples for Claude Code and OpenCode
+- `server.json` metadata for MCP Registry alignment
+
+## Why MCP is the right abstraction
+
+For your goal, yes: MCP is the best primary interface.
+
+- Claude Code already treats MCP as the standard way to add real tools
+- OpenCode can consume local MCP servers too, so one implementation works across both
+- plugins/marketplaces are packaging layers; MCP is the actual interoperability layer
+- if you later want Claude marketplace distribution, the plugin should wrap this MCP server, not replace it
+
+Short version:
+
+- `MCP` = the core product
+- `Claude plugin` = a distribution wrapper for Claude users
+- `Marketplace` = discovery/install channel
+
+## Install
+
+Canonical packaging is npm-first because MCP Registry publication and the lowest-friction Claude/OpenCode install path both expect a normal npm package.
+
+```bash
+npm install
+npm run build
+```
+
+Run locally:
+
+```bash
+node dist/index.js
+```
+
+Once the npm package is published, the intended install path is:
+
+```bash
+npx -y smart-web-mcp
+```
+
+## Quick start
+
+Claude Code:
+
+```bash
+claude mcp add smart-web -- node /absolute/path/to/smart-web/dist/index.js
+```
+
+OpenCode:
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "mcp": {
+    "smart-web": {
+      "type": "local",
+      "command": ["node", "/absolute/path/to/smart-web/dist/index.js"],
+      "enabled": true
+    }
+  }
+}
+```
+
+More complete examples live in `examples/claude.mcp.json` and `examples/opencode.json`.
+
+## Tool behavior
+
+### `smartsearch`
+
+- uses `EXA_API_KEY` when available
+- falls back to `BRAVE_SEARCH_API_KEY` when available
+- falls back to self-hosted `SEARXNG_BASE_URL` when available
+- falls back to DuckDuckGo HTML when no better option succeeds
+
+### `smartfetch`
+
+- starts with `impit` for lightweight fetch realism
+- escalates to stealth Playwright when content looks blocked or JS-rendered
+- returns normalized output for Reddit, X, and DCInside
+- can render `json`, `text`, or `markdown`
+- blocks localhost/private/reserved network targets unless explicitly overridden
+
+## Configuration
+
+Environment variables:
+
+- `EXA_API_KEY`: optional first-tier search provider
+- `BRAVE_SEARCH_API_KEY`: optional second-tier search provider
+- `SEARXNG_BASE_URL`: optional self-hosted private fallback before public no-key search
+- `SMARTFETCH_CHROME_CHANNEL`: optional Playwright browser channel override
+- `SMARTFETCH_CHROME_PATH`: optional Playwright executable path override
+- `SMARTFETCH_ENABLE_FXTWITTER`: opt in to FxTwitter relay enrichment for X posts
+- `SMARTFETCH_ENABLE_X_OEMBED`: enable or disable X oEmbed fallback
+- `SMARTFETCH_ENABLE_REDDIT_JSON`: enable or disable Reddit JSON enrichment
+- `SMART_WEB_ALLOW_PRIVATE_HOSTS`: opt in to fetching localhost/private network targets
+- `SMART_WEB_LOCAL_ONLY`: disable public no-key or relay-style fallbacks and keep traffic limited to explicitly chosen providers and direct targets
+
+Provider toggles:
+
+- `SMARTSEARCH_ENABLE_EXA`
+- `SMARTSEARCH_ENABLE_BRAVE`
+- `SMARTSEARCH_ENABLE_SEARXNG`
+- `SMARTSEARCH_ENABLE_DUCKDUCKGO`
+
+For local development, copy `.env.example` and export the values in your shell before starting your MCP client.
+
+## Security and privacy
+
+- For stdio MCP servers, secrets should be injected through environment variables, not committed config
+- Claude Code supports env expansion in `.mcp.json` with `${VAR}` syntax
+- OpenCode supports env expansion in `opencode.json` with `{env:VAR}` syntax
+- `smartsearch` sends only the query to the active backend
+- `smartfetch` sends the target URL to the destination site and, for some enrichments, may call additional public endpoints like Reddit JSON or X oEmbed
+- FxTwitter enrichment is opt-in via `SMARTFETCH_ENABLE_FXTWITTER`
+- provider enable/disable env vars let you turn individual backends on or off without code changes
+- Playwright may use browser-managed temporary state during execution; this project does not intentionally persist its own fetch history or secrets
+- localhost/private/reserved targets are blocked by default to reduce agent-side SSRF risk
+
+If query privacy matters, prefer self-hosted `SearXNG` and avoid public no-key fallbacks for sensitive searches.
+
+## Claude Code plugin and marketplace
+
+This repo now includes an optional Claude plugin scaffold in `plugins/claude-smart-web` and a marketplace manifest in `.claude-plugin/marketplace.json`.
+
+Recommendation:
+
+- use raw MCP config first for fastest adoption
+- add the plugin/marketplace path once the npm package is published
+
+That gives you the right layering:
+
+- MCP for cross-client compatibility
+- plugin for Claude-native install UX
+- marketplace for discovery and team rollout
+
+## Publish path
+
+This project is already structured for:
+
+1. npm publish of `smart-web-mcp`
+2. MCP Registry publish using `server.json`
+3. Claude marketplace/plugin distribution using the included plugin scaffold
+
+## Development
+
+```bash
+npm run check
+```
+
+CI runs the same checks on GitHub Actions, including metadata consistency and `npm pack --dry-run`.
+
+## Design notes
+
+- small shared library first, client wrappers second
+- one server with two tools is the best default UX
+- split packages remain possible later if demand appears
+
+## Roadmap
+
+- publish `smart-web-mcp` to npm
+- publish `smart-web` to the MCP Registry
+- improve site-specific normalizers and ranking heuristics
+- add more client wrappers only where they improve installation, not core behavior

package/dist/index.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/index.js

@@ -0,0 +1,62 @@
+#!/usr/bin/env node
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+import { runSmartfetch, renderSmartfetch } from "./smartfetch.js";
+import { runSmartsearch, renderSmartsearch } from "./smartsearch.js";
+const server = new McpServer({
+    name: "smart-web",
+    version: "0.1.0",
+});
+server.tool("smartsearch", "Search the web with fallback engines. Uses Exa if EXA_API_KEY is set, then Brave if BRAVE_SEARCH_API_KEY is set, then DuckDuckGo HTML as a no-key fallback.", {
+    query: z.string().min(1).describe("Web search query"),
+    numResults: z.number().int().min(1).max(20).optional().describe("Number of search results to return"),
+    type: z.enum(["auto", "fast", "deep"]).optional().describe("Search depth hint"),
+    contextMaxCharacters: z.number().int().min(1000).max(20000).optional().describe("Maximum snippet/context size hint"),
+}, async ({ query, numResults, type, contextMaxCharacters }) => {
+    const output = await runSmartsearch({
+        query,
+        ...(numResults === undefined ? {} : { numResults }),
+        ...(type === undefined ? {} : { searchType: type }),
+        ...(contextMaxCharacters === undefined ? {} : { contextMaxCharacters }),
+    });
+    return {
+        structuredContent: output,
+        content: [{ type: "text", text: renderSmartsearch(output) }],
+    };
+});
+server.tool("smartfetch", "Fetch a URL with browser-aware fallbacks. Uses impit first, then stealth Playwright when the page looks JS-rendered or blocked.", {
+    url: z.string().min(1).describe("Target URL to fetch"),
+    target: z.enum(["auto", "generic", "reddit_post", "x_post", "dcinside_post"]).optional().describe("Content type hint"),
+    timeout_ms: z.number().int().min(3000).max(120000).optional().describe("Fetch timeout in milliseconds"),
+    force_dynamic: z.boolean().optional().describe("Skip light fetches and go straight to Playwright"),
+    strict_schema: z.boolean().optional().describe("Return schema-safe fallback output when normalization breaks"),
+    format: z.enum(["json", "text", "markdown"]).optional().describe("Text rendering format for the human-readable content"),
+}, async ({ url, target, timeout_ms, force_dynamic, strict_schema, format }) => {
+    const output = await runSmartfetch({
+        url,
+        ...(target === undefined ? {} : { target }),
+        ...(timeout_ms === undefined ? {} : { timeoutMs: timeout_ms }),
+        ...(force_dynamic === undefined ? {} : { forceDynamic: force_dynamic }),
+        ...(strict_schema === undefined ? {} : { strictSchema: strict_schema }),
+    });
+    const requestedFormat = format || "json";
+    return {
+        structuredContent: output,
+        content: [
+            {
+                type: "text",
+                text: requestedFormat === "json" ? JSON.stringify(output, null, 2) : renderSmartfetch(output, requestedFormat),
+            },
+        ],
+    };
+});
+async function main() {
+    const transport = new StdioServerTransport();
+    await server.connect(transport);
+}
+main().catch((error) => {
+    console.error(error);
+    process.exit(1);
+});
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,SAAS,EAAE,MAAM,yCAAyC,CAAA;AACnE,OAAO,EAAE,oBAAoB,EAAE,MAAM,2CAA2C,CAAA;AAChF,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAA;AAEvB,OAAO,EAAE,aAAa,EAAE,gBAAgB,EAAE,MAAM,iBAAiB,CAAA;AACjE,OAAO,EAAE,cAAc,EAAE,iBAAiB,EAAE,MAAM,kBAAkB,CAAA;AAEpE,MAAM,MAAM,GAAG,IAAI,SAAS,CAAC;IAC3B,IAAI,EAAE,WAAW;IACjB,OAAO,EAAE,OAAO;CACjB,CAAC,CAAA;AAEF,MAAM,CAAC,IAAI,CACT,aAAa,EACb,6JAA6J,EAC7J;IACE,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,kBAAkB,CAAC;IACrD,UAAU,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,oCAAoC,CAAC;IACrG,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,mBAAmB,CAAC;IAC/E,oBAAoB,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,mCAAmC,CAAC;CACrH,EACD,KAAK,EAAE,EAAE,KAAK,EAAE,UAAU,EAAE,IAAI,EAAE,oBAAoB,EAAE,EAAE,EAAE;IAC1D,MAAM,MAAM,GAAG,MAAM,cAAc,CAAC;QAClC,KAAK;QACL,GAAG,CAAC,UAAU,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,CAAC;QACnD,GAAG,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,IAAI,EAAE,CAAC;QACnD,GAAG,CAAC,oBAAoB,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,oBAAoB,EAAE,CAAC;KACxE,CAAC,CAAA;IAEF,OAAO;QACL,iBAAiB,EAAE,MAAM;QACzB,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,iBAAiB,CAAC,MAAM,CAAC,EAAE,CAAC;KAC7D,CAAA;AACH,CAAC,CACF,CAAA;AAED,MAAM,CAAC,IAAI,CACT,YAAY,EACZ,iIAAiI,EACjI;IACE,GAAG,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,qBAAqB,CAAC;IACtD,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,SAAS,EAAE,aAAa,EAAE,QAAQ,EAAE,eAAe,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,mBAAmB,CAAC;IACtH,UAAU,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,+BAA+B,CAAC;IACvG,aAAa,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,kDAAkD,CAAC;IAClG,aAAa,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,8DAA8D,CAAC;IAC9G,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,sDAAsD,CAAC;CACzH,EACD,KAAK,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,UAAU,EAAE,aAAa,EAAE,aAAa,EAAE,MAAM,EAAE,EAAE,EAAE;IAC1E,MAAM,MAAM,GAAG,MAAM,aAAa,CAAC;QACjC,GAAG;QACH,GAAG,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC;QAC3C,GAAG,CAAC,UAAU,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,UAAU,EAAE,CAAC;QAC9D,GAAG,CAAC,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,aAAa,EAAE,CAAC;QACvE,GAAG,CAAC,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,aAAa,EAAE,CAAC;KACxE,CAAC,CAAA;IAEF,MAAM,eAAe,GAAG,MAAM,IAAI,MAAM,CAAA;IACxC,OAAO;QACL,iBAAiB,EAAE,MAAM;QACzB,OAAO,EAAE;YACP;gBACE,IAAI,EAAE,MAAM;gBACZ,IAAI,EAAE,eAAe,KAAK,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,gBAAgB,CAAC,MAAM,EAAE,eAAe,CAAC;aAC/G;SACF;KACF,CAAA;AACH,CAAC,CACF,CAAA;AAED,KAAK,UAAU,IAAI;IACjB,MAAM,SAAS,GAAG,IAAI,oBAAoB,EAAE,CAAA;IAC5C,MAAM,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,CAAA;AACjC,CAAC;AAED,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,KAAK,EAAE,EAAE;IACrB,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAA;IACpB,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA;AACjB,CAAC,CAAC,CAAA"}

package/dist/lib.d.ts

@@ -0,0 +1,3 @@
+export { runSmartfetch, renderSmartfetch } from "./smartfetch.js";
+export { runSmartsearch, renderSmartsearch } from "./smartsearch.js";
+export type { AdapterResult, ErrorCategory, SearchResponse, SearchResult, SmartfetchOutput, SmartfetchRenderFormat, Target, ToolError, } from "./shared.js";

package/dist/lib.js

@@ -0,0 +1,3 @@
+export { runSmartfetch, renderSmartfetch } from "./smartfetch.js";
+export { runSmartsearch, renderSmartsearch } from "./smartsearch.js";
+//# sourceMappingURL=lib.js.map

package/dist/lib.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"lib.js","sourceRoot":"","sources":["../src/lib.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,gBAAgB,EAAE,MAAM,iBAAiB,CAAA;AACjE,OAAO,EAAE,cAAc,EAAE,iBAAiB,EAAE,MAAM,kBAAkB,CAAA"}

package/dist/shared.d.ts

@@ -0,0 +1,105 @@
+export type Target = "auto" | "generic" | "reddit_post" | "x_post" | "dcinside_post";
+export type ResolvedTarget = Exclude<Target, "auto">;
+export type ErrorCategory = "timeout" | "block" | "parse_error" | "unavailable";
+export type ToolError = {
+    category: ErrorCategory;
+    code: string;
+    message: string;
+};
+export type AdapterResult = {
+    ok: boolean;
+    method: string;
+    content: string;
+    links: string[];
+    error?: ToolError;
+};
+export type SearchResult = {
+    title: string;
+    url: string;
+    snippet: string;
+    engine: string;
+};
+export type SearchResponse = {
+    engine: string;
+    query: string;
+    results: SearchResult[];
+    fallbacks_tried: string[];
+    notes: string[];
+};
+export type SmartfetchOutput = {
+    source: "smartfetch";
+    url: string;
+    target: ResolvedTarget;
+    retrieval_method: string[];
+    risk_level: "low" | "medium" | "high";
+    partial: boolean;
+    errors: ToolError[];
+    post: Record<string, unknown> | null;
+    thread: Record<string, unknown>[];
+    comments: Record<string, unknown>[];
+    outbound_links: string[];
+    evidence: {
+        impit_attempted: boolean;
+        impit_success: boolean;
+        playwright_attempted: boolean;
+        playwright_success: boolean;
+        generated_at: string;
+    };
+};
+export type SmartfetchRenderFormat = "text" | "markdown";
+export declare function asString(value: unknown): string;
+export declare function asNumber(value: unknown): number;
+export declare function decodeHtml(value: string): string;
+export declare function stripTags(value: string): string;
+export declare function trimUrlToken(value: string): string;
+export declare function dedupeUrls(values: string[]): string[];
+export declare function cleanLinks(values: string[]): string[];
+export declare function extractUrls(value: string): string[];
+export declare function extractAnchorHrefs(value: string): string[];
+export declare function uniqueResults(results: SearchResult[]): SearchResult[];
+export declare function resolveDuckDuckGoUrl(input: string): string;
+export declare function capture(value: string, regex: RegExp): string;
+export declare function extractTitleFromHtml(html: string): string;
+export declare function extractMetaDescription(html: string): string;
+export declare function inferTarget(url: string, target: Target): ResolvedTarget;
+export declare function envFlag(name: string): boolean;
+export declare function envEnabled(name: string, defaultValue: boolean): boolean;
+export declare function validateOutboundUrl(url: string, options?: {
+    allowPrivateHosts?: boolean;
+}): Promise<{
+    ok: false;
+    reason: string;
+    message: string;
+} | {
+    ok: true;
+    reason?: never;
+    message?: never;
+}>;
+export declare function isXUrl(url: string): boolean;
+export declare function isRedditUrl(url: string): boolean;
+export declare function isDcinsideUrl(url: string): boolean;
+export declare function needsDynamicCrawl(result: AdapterResult): boolean;
+export declare function fetchText(url: string, timeoutMs: number, init?: RequestInit): Promise<{
+    ok: boolean;
+    status: number;
+    text: string;
+    error?: never;
+} | {
+    ok: boolean;
+    status: number;
+    text: string;
+    error: string;
+}>;
+export declare function fetchJson(url: string, timeoutMs: number, init?: RequestInit): Promise<{
+    data: any;
+    ok: boolean;
+    status: number;
+    text: string;
+    error?: never;
+} | {
+    data: any;
+    ok: boolean;
+    status: number;
+    text: string;
+    error: string;
+}>;