crawlforge-mcp-server 5.6.10 → 5.8.0

package/CLAUDE.md

@@ -60,7 +60,7 @@ These guidelines are working if: fewer unnecessary changes in diffs, fewer rewri
 
 ## Project Overview
 
-CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 29 web scraping, crawling, and content processing tools (6 inline + 23 advanced).
+CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 30 web scraping, crawling, and content processing tools (6 inline + 24 advanced).
 
 **Current Version:** 5.6.6
 
@@ -162,18 +162,19 @@ Tools are organized in subdirectories by category:
 - `crawl/` - crawlDeep, mapSite
 - `extract/` - analyzeContent, extractContent, extractStructured, extractWithLlm, listOllamaModels, processDocument, summarizeContent
 - `research/` - deepResearch
+- `result/` - readResult (`read_result`: search/slice/lines/json_path over a stored over-threshold result by `result_handle`, v5.8.0)
 - `search/` - searchWeb (proxied through CrawlForge.dev API), serpRank (real Google organic rank via DataForSEO; adapter in `search/adapters/dataforseoSearch.js`), redditSearch (Reddit posts/comments/threads via Arctic Shift + PullPush community archives — reddit.com blocks scrapers)
 - `templates/` - ScrapeTemplateTool (10 pre-built site templates, v4.0.0)
 - `tracking/` - trackChanges
 - `llmstxt/` - generateLLMsTxt
 
-### Available MCP Tools (29 total)
+### Available MCP Tools (30 total)
 
 **Basic Tools (server.js inline, 6):**
 fetch_url, extract_text, extract_links, extract_metadata, extract_embedded_state, scrape_structured
 
-**Advanced Tools (23):**
-search_web, serp_rank, reddit_search, crawl_deep, map_site, extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models, batch_scrape, get_batch_results, scrape_with_actions, deep_research, track_changes, generate_llms_txt, stealth_mode, localization, scrape_template, scrape, agent
+**Advanced Tools (24):**
+search_web, serp_rank, reddit_search, crawl_deep, map_site, extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models, batch_scrape, get_batch_results, scrape_with_actions, deep_research, track_changes, generate_llms_txt, stealth_mode, localization, scrape_template, scrape, agent, read_result
 
 **serp_rank (DataForSEO):**
 - `serp_rank` — reports where a target domain ranks in Google's REAL organic results for a keyword (the SERP position Google Custom Search / `search_web` cannot give). Backed by the DataForSEO Google Organic SERP API (Live Advanced, `POST /v3/serp/google/organic/live/advanced`, HTTP Basic auth). Credentials via `DATAFORSEO_LOGIN` / `DATAFORSEO_PASSWORD`, billed to the user's own DataForSEO account (~US$0.002 per 10 results of `depth`, so $0.004 at the default `depth:20` and $0.02 at `depth:100`), separate from CrawlForge credits. When unconfigured it returns `{ configured:false }` and charges **0** credits; when configured, **Cost: 5**. Never fabricates a rank.

package/README.md

@@ -3,7 +3,7 @@
 </p>
 
 <p align="center">
-  <b>29 web scraping, crawling, deep-research &amp; autonomous-extraction tools for Claude, Cursor &amp; any MCP client.</b><br>
+  <b>30 web scraping, crawling, deep-research &amp; autonomous-extraction tools for Claude, Cursor &amp; any MCP client.</b><br>
   Clean Markdown &amp; structured JSON from any site. Get started with <b>1,000 free credits</b> — no credit card required.
 </p>
 
@@ -35,7 +35,7 @@
 
 ## 🎯 Why CrawlForge?
 
-- **29 MCP-native tools** — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous `agent`, a unified multi-format `scrape`, document processing, stealth browsing, and more, callable directly from your AI assistant.
+- **30 MCP-native tools** — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous `agent`, a unified multi-format `scrape`, document processing, stealth browsing, and more, callable directly from your AI assistant.
 - **Generous free tier** — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire.
 - **Local-LLM by default** — `extract_with_llm` runs against a local **Ollama** model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
 - **LLM-ready output** — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
@@ -47,7 +47,7 @@
 
 | | **CrawlForge MCP** | Firecrawl | Raw scraping API |
 |---|:---:|:---:|:---:|
-| Native MCP server | ✅ 29 tools | ✅ | ❌ |
+| Native MCP server | ✅ 30 tools | ✅ | ❌ |
 | Free tier | ✅ 1,000 credits, rollover | Limited | Varies |
 | Self-hosted / local LLM extraction (Ollama) | ✅ default, $0/token | ❌ | ❌ |
 | Autonomous agent (no URLs needed) | ✅ `agent` | ✅ | ❌ |
@@ -178,7 +178,8 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `scrape_template` | 1 | Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors |
 | `list_ollama_models` | 1 | List the Ollama models installed locally (helps you pick a `model` for `extract_with_llm`) |
 | `get_batch_results` | 1 | Retrieve paginated results for a `batch_scrape` job by `batchId` |
-| `scrape` | 2 | **Unified single-fetch, multi-format extraction.** Pass a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) plus `onlyMainContent`; one fetch serves every requested format with per-format partial-success warnings |
+| `read_result` | 1 | Search, slice, read lines or a JSON path from a result a tool returned with `truncated: true` and a `result_handle` (kept 1 hour on your own machine) — never fetch the page again |
+| `scrape` | 2 | **Unified single-fetch, multi-format extraction.** Pass a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema, plus `{type:"highlights",query}` and `{type:"question",question}` for only the matching sentences, table rows and code blocks, verbatim with offsets into the markdown: +1 credit once per call, `mode:"model"` +3) plus `onlyMainContent`; one fetch serves every requested format with per-format partial-success warnings |
 | `scrape_structured` | 2 | Extract structured data with CSS selectors |
 | `extract_embedded_state` | 2 | Read a page's embedded JavaScript state — `__NEXT_DATA__`, React Server Component payloads, Nuxt, Apollo, Redux, `<script type="application/json">` — with a `path` to scope the result. No LLM in the extraction path |
 | `extract_content` | 2 | Enhanced content extraction |
@@ -201,6 +202,8 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `agent` | 8 | **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM) |
 | `deep_research` | 10 | Multi-stage research with source verification |
 
+Ten tools (`scrape`, `fetch_url`, `extract_content`, `crawl_deep`, `batch_scrape`, `stealth_mode`, `scrape_with_actions`, `process_document`, `deep_research`, `extract_embedded_state`) accept `max_inline_chars` (default 40,000; env `CRAWLFORGE_MAX_INLINE_CHARS`): a result over it comes back as a `preview` plus a `result_handle` for `read_result`, with the full result kept for 1 hour under `~/.crawlforge/results/` on your own machine — nothing is uploaded.
+
 For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see [SKILL.md](docs/SKILL.md).
 
 <p align="right"><a href="#table-of-contents">↑ Back to top</a></p>
@@ -217,7 +220,7 @@ For the full canonical capabilities reference (all tools, CLI commands, stealth
 | **Business** ($399) | 250,000 / month | Large scale operations |
 
 **All plans include:**
-- Access to all 29 tools
+- Access to all 30 tools
 - Credits never expire; paid-plan credits roll over month to month
 - API access and webhook notifications
 

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "5.6.10",
+  "version": "5.8.0",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
-  "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 29 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
+  "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 30 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
   "bin": {
     "crawlforge": "src/cli/index.js",
@@ -116,7 +116,7 @@
     "cheerio": "^1.1.2",
     "commander": "^14.0.3",
     "compromise": "^14.14.4",
-    "crawlforge-extractors": "^1.6.5",
+    "crawlforge-extractors": "^1.8.0",
     "diff": "^9.0.0",
     "dotenv": "^17.2.1",
     "franc": "^6.2.0",

package/server.js

@@ -26,7 +26,7 @@ import { DeepResearchTool } from "./src/tools/research/deepResearch.js";
 import { TrackChangesTool } from "./src/tools/tracking/trackChanges/index.js";
 import { GenerateLLMsTxtTool } from "./src/tools/llmstxt/generateLLMsTxt.js";
 import { ScrapeTemplateTool } from "./src/tools/templates/ScrapeTemplateTool.js"; // D3.3
-import { UnifiedScrapeTool } from "./src/tools/scrape/unifiedScrape.js"; // D4 D1
+import { UnifiedScrapeTool, SCRAPE_INPUT_SHAPE } from "./src/tools/scrape/unifiedScrape.js"; // D4 D1
 import { AgentTool } from "./src/tools/agent/agent.js"; // D4 D2
 import { StealthBrowserManager } from "./src/core/StealthBrowserManager.js";
 import { LocalizationManager } from "./src/core/LocalizationManager.js";
@@ -55,6 +55,8 @@ import { extractLinksHandler } from "./src/tools/basic/extractLinks.js";
 import { extractMetadataHandler } from "./src/tools/basic/extractMetadata.js";
 import { scrapeStructuredHandler } from "./src/tools/basic/scrapeStructured.js";
 import { extractEmbeddedStateHandler } from "./src/tools/extract/extractEmbeddedState.js";
+import { READ_RESULT_INPUT_SHAPE, readResultHandler } from "./src/tools/result/readResult.js"; // Phase 2
+import { MAX_INLINE_CHARS_PARAM } from "./src/server/inlineThreshold.js"; // Phase 2
 // D1.1 Resources + D1.2 Prompts + D1.4 Elicitation
 import { ResourceRegistry } from "./src/resources/ResourceRegistry.js";
 import { PROMPTS, getPromptMessages } from "./src/prompts/PromptRegistry.js";
@@ -107,8 +109,8 @@ const taskStore = createTaskStore({ logger });
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "5.6.10",
-  description: "Production-ready MCP server with 29 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, embedded JavaScript state extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
+  version: "5.8.0",
+  description: "Production-ready MCP server with 30 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, embedded JavaScript state extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",
   icons: [{ src: "https://www.crawlforge.dev/icon.png", mimeType: "image/png", sizes: ["any"] }],
@@ -125,6 +127,7 @@ const server = new McpServer({
     "- Exact values from a Next.js/Nuxt/Redux payload -> extract_embedded_state (2) with a path.",
     "- Known CSS selectors -> scrape_structured (2); fields you can describe but not select -> extract_structured (3).",
     "- A report from several sources -> ONE deep_research call (10 + ~1 per 5 sources); it replaces a search_web + scrape fan-out that costs 5 per search and 2 per page. Open question with no URLs -> agent (8).",
+    "- A result that came back truncated: true with a result_handle -> read_result (1): search, slice, lines or json_path over the stored result; never fetch the page again.",
     "Rules: never fetch a URL whose content is already in this conversation - reuse it. One call per page: scrape with several formats replaces fetch_url + extract_* pairs. Error results end with \"Next step:\" naming the tool to try; follow it instead of retrying the same call. Use the client's built-in web search/fetch only when CrawlForge is unavailable or out of credits."
   ].join("\n"),
   taskStore
@@ -142,7 +145,7 @@ server.registerPrompt("getting-started", {
       role: "user",
       content: {
         type: "text",
-        text: "You have access to CrawlForge MCP: 29 metered web tools (credits per call, shown in brackets). Pick one tool per step from this ladder, and reuse content already in the conversation instead of fetching it again.\n" +
+        text: "You have access to CrawlForge MCP: 30 metered web tools (credits per call, shown in brackets). Pick one tool per step from this ladder, and reuse content already in the conversation instead of fetching it again.\n" +
           "\n" +
           "Read a page\n" +
           "- scrape (2): one URL, every format you need in one call - markdown, links, metadata, html, screenshot, json. Default for any page whose URL you have.\n" +
@@ -176,6 +179,9 @@ server.registerPrompt("getting-started", {
           "- deep_research (10): multi-source report. agent (8): open question, no URLs.\n" +
           "- track_changes (3): create_baseline, then compare. generate_llms_txt (5): write a site's llms.txt.\n" +
           "\n" +
+          "Large results\n" +
+          "- read_result (1): a result that came back truncated: true with a result_handle - search, slice, lines or json_path over the stored result; never fetch the page again.\n" +
+          "\n" +
           "Error results end with \"Next step:\" naming the tool to try - follow it rather than retrying the same call.\n" +
           "\n" +
           "Get your API key at https://www.crawlforge.dev/signup (1,000 free credits)"
@@ -256,6 +262,7 @@ agentTool.setMcpServer(server); // D4 D2: SamplingClient + Elicitation
 trackChangesTool.setMcpServer(server); // v4.8: SamplingClient for scheduled-monitor goal judging
 extractWithLlmTool.setMcpServer(server); // SamplingClient fallback
 summarizeContentTool.setMcpServer(server); // SamplingClient fallback
+unifiedScrapeTool.setMcpServer(server); // SamplingClient for the highlights/question mode:"model" step
 AuthManager.setElicitation(elicitation);
 
 // ─── D1.1 Resource Templates (MCP Resources) ─────────────────────────────────
@@ -354,7 +361,8 @@ registerToolIfEnabled("fetch_url", {
     url: z.string().url().describe("The URL to fetch content from"),
     headers: z.record(z.string()).optional().describe("Custom HTTP headers to include in the request"),
     timeout: z.number().min(1000).max(30000).optional().default(10000).describe("Request timeout in milliseconds (1000-30000)"),
-    ...COMPLIANCE_PARAMS
+    ...COMPLIANCE_PARAMS,
+    ...MAX_INLINE_CHARS_PARAM
   }
 }, withAuth("fetch_url", fetchUrlHandler));
 
@@ -401,7 +409,8 @@ registerToolIfEnabled("extract_embedded_state", {
   inputSchema: {
     url: z.string().url().describe("The URL to read embedded state from"),
     path: z.string().optional().describe("Return only this subtree instead of the whole payload. Dotted keys and array indexes, e.g. \"next_data.props.pageProps\" or \"next_f[0].f\" — not JSONPath (no wildcards, filters or recursion). State payloads are routinely over a megabyte; scope them."),
-    ...COMPLIANCE_PARAMS
+    ...COMPLIANCE_PARAMS,
+    ...MAX_INLINE_CHARS_PARAM
   }
 }, withAuth("extract_embedded_state", extractEmbeddedStateHandler));
 
@@ -574,7 +583,8 @@ if (toolFilter.isEnabled("crawl_deep")) {
           headers: z.record(z.string()).optional(),
           body: z.string().optional()
         }).optional()
-      }).optional().describe("Shared cookie-jar/session for login-then-crawl workflows")
+      }).optional().describe("Shared cookie-jar/session for login-then-crawl workflows"),
+      ...MAX_INLINE_CHARS_PARAM
     },
     outputSchema: OUTPUT_SCHEMAS.crawl_deep,
     execution: TASK_EXECUTION
@@ -636,7 +646,8 @@ registerToolIfEnabled("extract_content", {
   inputSchema: {
     url: z.string().url().describe("The URL to extract content from"),
     options: z.object({}).passthrough().optional().describe("Additional extraction options"),
-    ...COMPLIANCE_PARAMS
+    ...COMPLIANCE_PARAMS,
+    ...MAX_INLINE_CHARS_PARAM
   }
 }, withAuth("extract_content", async (params) => {
   try {
@@ -660,7 +671,8 @@ registerToolIfEnabled("process_document", {
     // C3: passthrough so granular options (maxPages, pageRange:{start,end},
     // extractText, outputFormat, etc.) reach the tool instead of being stripped.
     options: z.object({}).passthrough().optional().describe("Additional processing options (maxPages, pageRange:{start,end}, extractText, extractMetadata, outputFormat, ...)"),
-    ...COMPLIANCE_PARAMS
+    ...COMPLIANCE_PARAMS,
+    ...MAX_INLINE_CHARS_PARAM
   }
 }, withAuth("process_document", async (params) => {
   try {
@@ -825,7 +837,8 @@ if (toolFilter.isEnabled("batch_scrape")) {
         maxRetries: z.number().min(0).max(5).default(1),
         tags: z.array(z.string()).default([])
       }).optional().describe("Job management options for async processing"),
-      ...COMPLIANCE_PARAMS
+      ...COMPLIANCE_PARAMS,
+      ...MAX_INLINE_CHARS_PARAM
     },
     execution: TASK_EXECUTION
   }, makeTaskToolHandler({
@@ -864,6 +877,14 @@ registerToolIfEnabled("get_batch_results", {
   }
 }));
 
+// Tool: read_result — Phase 2: read a stored result by the handle a truncated result returned
+registerToolIfEnabled("read_result", {
+  description: "Use this to read a result that came back with truncated: true and a result_handle - the tool kept the whole result for 1 hour and returned a preview. operation:\"search\" finds a literal query with offsets and context, \"slice\" returns characters from an offset, \"lines\" pages by line, \"json_path\" reads one subtree of a JSON result (crawl_deep pages, batch results, a fetch_url JSON body). Not a fetching tool: never call the original tool again while the handle is valid, and not for a result that arrived whole. Cost: 1 credit. Example: read_result({handle: \"res_…\", operation: \"search\", query: \"pricing\"})",
+  annotations: { title: "Read Result", readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },
+  inputSchema: READ_RESULT_INPUT_SHAPE,
+  outputSchema: OUTPUT_SCHEMAS.read_result
+}, withAuth("read_result", readResultHandler));
+
 // Tool: scrape_with_actions
 registerToolIfEnabled("scrape_with_actions", {
   description: "Use this when you must interact with a page before scraping - login, click buttons, fill forms, scroll, or wait for dynamic content to load - for SPAs, login-gated content, or multi-step flows. Actions: wait, click, type, press, scroll, screenshot, executeJavaScript, select (dropdowns), hover, navigate. Set browserOptions.stealth:true to run the chain in the stealth browser. robots.txt is respected on every navigation. Screenshots from this tool are stored as crawlforge://screenshot/{actionId} resources. Not for pages that render without interaction (scrape) and not as the first attempt on a blocked site (stealth_mode operation:\"scrape\"). Cost: 5 credits. Example: scrape_with_actions({url: \"https://app.com/dashboard\", actions: [{type:\"click\",selector:\"#login\"},{type:\"type\",selector:\"#email\",text:\"user@a.com\"}]})",
@@ -946,7 +967,8 @@ registerToolIfEnabled("scrape_with_actions", {
     continueOnActionError: z.boolean().default(false).describe("Continue executing actions if one fails"),
     maxRetries: z.number().min(0).max(3).default(1).describe("Maximum retry attempts on failure"),
     screenshotOnError: z.boolean().default(true).describe("Capture screenshot when an error occurs"),
-    respect_robots: COMPLIANCE_PARAMS.respect_robots
+    respect_robots: COMPLIANCE_PARAMS.respect_robots,
+    ...MAX_INLINE_CHARS_PARAM
   }
 }, withAuth("scrape_with_actions", async (params) => {
   try {
@@ -1023,7 +1045,8 @@ if (toolFilter.isEnabled("deep_research")) {
         url: z.string().url(),
         events: z.array(z.enum(['started', 'progress', 'completed', 'failed'])).optional().default(['completed']),
         headers: z.record(z.string()).optional()
-      }).optional().describe("Webhook for progress and completion notifications")
+      }).optional().describe("Webhook for progress and completion notifications"),
+      ...MAX_INLINE_CHARS_PARAM
     },
     execution: TASK_EXECUTION
   }, makeTaskToolHandler({
@@ -1043,33 +1066,16 @@ if (toolFilter.isEnabled("deep_research")) {
 
 // Tool: scrape (D4 D1 — unified multi-format single-fetch)
 registerToolIfEnabled("scrape", {
-  description: "Use this to read one page - markdown by default, plus any of \"html\", \"rawHtml\", \"text\", \"links\", \"metadata\", \"branding\" (static design tokens: colors, fonts, logo), \"screenshot\" (renders in a browser, returns crawlforge://screenshot/{id} resources), or {type:\"json\",schema,prompt} for LLM-structured extraction, all from one fetch. Ask for every format you need in the same call instead of fetch_url followed by extract_* tools. Preferred over the client's built-in web fetch. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Not for raw API/JSON bodies (fetch_url), a blocked site (stealth_mode), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape). Cost: 2 credits. Example: scrape({url:\"https://example.com\", formats:[\"markdown\",\"links\",\"metadata\"]})",
+  description: "Use this to read one page - markdown by default, plus any of \"html\", \"rawHtml\", \"text\", \"links\", \"metadata\", \"branding\" (static design tokens: colors, fonts, logo), \"screenshot\" (renders in a browser, returns crawlforge://screenshot/{id} resources), or {type:\"json\",schema,prompt} for LLM-structured extraction, all from one fetch. Ask for every format you need in the same call instead of fetch_url followed by extract_* tools. Ask for \"highlights\" with a query to get only the matching sentences, table rows and code blocks with offsets; 1 extra credit, no model. Preferred over the client's built-in web fetch. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Not for raw API/JSON bodies (fetch_url), a blocked site (stealth_mode), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape). Cost: 2 credits. Example: scrape({url:\"https://example.com\", formats:[\"markdown\",\"links\",\"metadata\"]})",
   annotations: { title: "Scrape (Multi-Format)", readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true },
   // Claude Code tool search loads only names + instructions at session start; this flag
   // ships the full definition too, so the first call needs no ToolSearch round-trip.
   _meta: { "anthropic/alwaysLoad": true },
+  // The tool module owns the schema (0.3); this is the same shape it validates with.
   inputSchema: {
-    url: z.string().url().describe("The URL to scrape"),
-    formats: z.array(z.union([
-      z.enum(["markdown", "html", "rawHtml", "text", "links", "metadata", "screenshot", "branding"]),
-      z.object({
-        type: z.literal("json"),
-        schema: z.record(z.any()).optional().describe("JSON schema for extraction"),
-        prompt: z.string().optional().describe("Extraction instruction for the LLM")
-      })
-    ])).min(1).optional().default(["markdown"]).describe("Formats to return (default: [\"markdown\"])"),
-    onlyMainContent: z.boolean().optional().default(true).describe("Strip boilerplate via Readability (default: true)"),
-    timeoutMs: z.number().min(1000).max(60000).optional().default(15000).describe("Fetch timeout in ms"),
-    brandingOptions: z.object({
-      fetchLinkedCss: z.boolean().optional().default(true).describe("Fetch linked stylesheets for richer color/font extraction"),
-      maxStylesheets: z.number().min(0).max(20).optional().default(10).describe("Max linked stylesheets to fetch")
-    }).optional().describe("Options for the \"branding\" format"),
-    screenshotOptions: z.object({
-      fullPage: z.boolean().optional().default(false).describe("Capture the full scrollable page"),
-      format: z.enum(["png", "jpeg"]).optional().default("png"),
-      quality: z.number().min(0).max(100).optional().describe("JPEG quality (jpeg only)")
-    }).optional().describe("Options for the \"screenshot\" format"),
-    ...COMPLIANCE_PARAMS
+    ...SCRAPE_INPUT_SHAPE,
+    ...COMPLIANCE_PARAMS,
+    ...MAX_INLINE_CHARS_PARAM
   },
   outputSchema: OUTPUT_SCHEMAS.scrape
 }, withAuth("scrape", async (params) => {
@@ -1089,6 +1095,11 @@ registerToolIfEnabled("scrape", {
         return shot;
       });
     }
+    // A blocked verdict (0.1) is an error result: withAuth bills it at the
+    // error rate and appends the hint naming stealth_mode, and the SDK skips
+    // outputSchema validation on isError. The JSON body still carries the
+    // verdict (success:false, error, blocked) the way stealth_mode's does.
+    if (result.success === false) return { ...dualOutput(result), isError: true };
     return dualOutput(result);
   } catch (error) {
     return { content: [{ type: "text", text: `Scrape failed: ${error.message}` }], isError: true };
@@ -1381,7 +1392,8 @@ registerToolIfEnabled("stealth_mode", {
     formats: z.array(z.enum(["markdown", "html", "text", "links", "metadata", "screenshot"])).optional().default(["markdown"]).describe("Formats to return from operation:\"scrape\" (default: [\"markdown\"]). \"screenshot\" returns a crawlforge://screenshot/{id} resource URI."),
     wait_for: z.number().min(0).max(30000).optional().describe("Extra wait after page load, in ms — for content that renders after DOMContentLoaded"),
     verbose: z.boolean().optional().default(false).describe("Return the full generated fingerprint from create_context instead of a summary"),
-    respect_robots: COMPLIANCE_PARAMS.respect_robots
+    respect_robots: COMPLIANCE_PARAMS.respect_robots,
+    ...MAX_INLINE_CHARS_PARAM
   }
 }, withAuth("stealth_mode", async ({ operation, stealthConfig, contextId, urlToTest, url, formats, wait_for, verbose, engine, respect_robots }) => {
   try {
@@ -1730,10 +1742,10 @@ async function runServer() {
   console.error("Search enabled: true (via CrawlForge proxy)");
 
   const allTools = [
-    "fetch_url", "extract_text", "extract_links", "extract_metadata", "scrape_structured",
+    "fetch_url", "extract_text", "extract_links", "extract_metadata", "scrape_structured", "extract_embedded_state",
     "search_web", "serp_rank", "reddit_search", "crawl_deep", "map_site",
     "extract_content", "process_document", "summarize_content", "analyze_content",
-    "batch_scrape", "get_batch_results", "scrape_with_actions",
+    "batch_scrape", "get_batch_results", "read_result", "scrape_with_actions",
     "deep_research", "track_changes", "generate_llms_txt",
     "stealth_mode", "localization", "extract_structured", "extract_with_llm",
     "list_ollama_models", "scrape_template", // D3.3

package/src/core/AuthManager.js

@@ -5,12 +5,18 @@
 
 // Using native fetch (Node.js 18+)
 import fs from 'fs/promises';
+import { readFileSync } from 'fs';
 import path from 'path';
 import { randomUUID } from 'crypto';
 import { isCreatorModeVerified } from './creatorMode.js';
 import { resolveApiEndpoint } from './endpointGuard.js';
 import { logger } from '../utils/Logger.js';
 import { maskSecrets } from '../utils/secretMask.js';
+import { scrapeFormatSurcharge } from '../tools/scrape/formats.js';
+// Stamped on every usage report so support can tell which client version
+// made a call; it read '3.0.3' from 3.0.3 to 5.6.10 (the website stores it
+// per record from its Phase 1.3 on).
+const PACKAGE_VERSION = JSON.parse(readFileSync(new URL('../../package.json', import.meta.url), 'utf8')).version;
 // D1.4: Elicitation for low-credit warnings (lazy import to avoid circular dep)
 let _ElicitationHelper = null;
 function getElicitationHelper() {
@@ -368,7 +374,7 @@ class AuthManager {
       timestamp: new Date().toISOString(),
       requestId,
       idempotencyKey,
-      version: '3.0.3'
+      version: PACKAGE_VERSION
     };
 
     try {
@@ -493,7 +499,7 @@ class AuthManager {
             timestamp: entry.timestamp,
             requestId: entry.requestId,
             idempotencyKey,
-            version: '3.0.3'
+            version: PACKAGE_VERSION
           }),
           signal: AbortSignal.timeout(5000)
         });
@@ -554,6 +560,7 @@ class AuthManager {
       scrape_template: 1,
       list_ollama_models: 1,
       get_batch_results: 1, // retrieval of an already-paid batch job
+      read_result: 1, // reads a stored result an earlier call already paid for
 
       // 2 credits
       scrape_structured: 2,
@@ -618,6 +625,14 @@ class AuthManager {
       return costs.search_web;
     }
 
+    // The query-scoped formats (highlights, question) add 1 once per call,
+    // and mode:"model" adds 3 once; the base price in the table is what the
+    // backend sees. The rule lives with the format schemas.
+    if (tool === 'scrape') {
+      const { query, model } = scrapeFormatSurcharge(params?.formats);
+      return costs.scrape + query + model;
+    }
+
     return costs[tool] ?? 1;
   }
 
@@ -674,7 +689,7 @@ class AuthManager {
         break;
       case 'scrape': {
         projected = base;
-        note = 'Fixed cost per invocation. json format may incur external LLM cost (billed by your provider).';
+        note = 'Base 2; a highlights or question format adds 1 once per call, mode:"model" adds 3 once (charged at the extractive price when no LLM route exists). json format may incur external LLM cost (billed by your provider).';
         break;
       }
       case 'agent': {

package/src/core/ResultStore.js

@@ -0,0 +1,230 @@
+/**
+ * ResultStore — where a tool result lives once it is too large to return
+ * inline (Phase 2, result handles).
+ *
+ * Local and per-process: the store is on the machine the server runs on,
+ * which is where the result already was. Nothing leaves it. Entries are
+ * written under ~/.crawlforge/results/ (same root as SnapshotManager's
+ * snapshots), indexed in memory, evicted least-recently-used once the total
+ * passes maxBytes, and dropped after ttlMs whatever happens. A result that
+ * cannot be written to disk stays in memory instead — put never fails for a
+ * disk error.
+ *
+ * Handles are `res_<uuid>` (or a caller-supplied key such as a batch id).
+ * RESULT_HANDLE_PATTERN is the path-traversal guard: every method rejects a
+ * handle that does not match before touching the filesystem.
+ */
+
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { randomUUID } from 'node:crypto';
+
+export const RESULT_HANDLE_PATTERN = /^(res|batch)_[A-Za-z0-9_-]{1,80}$/;
+const KEY_PATTERN = /^[A-Za-z0-9_-]{4,80}$/;
+
+export class ResultStore {
+  /**
+   * @param {object} [options]
+   * @param {string} [options.baseDir] — directory for the result files (default ~/.crawlforge/results)
+   * @param {number} [options.maxBytes] — LRU budget in serialised bytes (default 200 MB)
+   * @param {number} [options.ttlMs] — how long an entry is readable (default 1 hour)
+   * @param {number} [options.sweepIntervalMs] — expiry sweep period (default 5 minutes)
+   * @param {object} [options.logger] — optional logger with warn()
+   */
+  constructor({
+    baseDir,
+    maxBytes = 200 * 1024 * 1024,
+    ttlMs = 60 * 60 * 1000,
+    sweepIntervalMs = 5 * 60 * 1000,
+    logger = null
+  } = {}) {
+    this.baseDir = baseDir || path.join(os.homedir(), '.crawlforge', 'results');
+    this.maxBytes = maxBytes;
+    this.ttlMs = ttlMs;
+    this.logger = logger;
+    // Insertion order is LRU order: get() re-inserts the entry it read.
+    this.index = new Map();
+    this.totalBytes = 0;
+
+    this._cleanupStaleFiles();
+
+    // .unref() so this timer never blocks process exit on its own — matches
+    // SnapshotManager's cleanupTimer.
+    this._sweepTimer = setInterval(() => this.sweep(), sweepIntervalMs);
+    if (typeof this._sweepTimer.unref === 'function') this._sweepTimer.unref();
+  }
+
+  /**
+   * Store a payload. Never throws for a disk error.
+   * @param {string} toolName
+   * @param {unknown} payload — any JSON-serialisable value
+   * @param {{ key?: string, meta?: object }} [options] — key: caller-supplied handle
+   * @returns {string} handle
+   */
+  put(toolName, payload, { key, meta = {} } = {}) {
+    let handle;
+    if (key !== undefined) {
+      if (typeof key !== 'string' || !KEY_PATTERN.test(key)) {
+        throw new Error(`ResultStore: invalid key "${key}"`);
+      }
+      if (!RESULT_HANDLE_PATTERN.test(key)) {
+        throw new Error(`ResultStore: key "${key}" must start with res_ or batch_ to be readable as a handle`);
+      }
+      handle = key;
+    } else {
+      handle = `res_${randomUUID()}`;
+    }
+
+    const json = JSON.stringify(payload);
+    const bytes = Buffer.byteLength(json);
+    const createdAt = Date.now();
+    const entry = { handle, toolName, bytes, createdAt, expiresAt: createdAt + this.ttlMs, meta };
+
+    // Re-put under the same key replaces the old entry (and its bytes).
+    if (this.index.has(handle)) this.delete(handle);
+
+    try {
+      fs.mkdirSync(this.baseDir, { recursive: true });
+      fs.writeFileSync(this._file(handle), json);
+    } catch (error) {
+      entry.inMemory = true;
+      entry.json = json;
+      this.logger?.warn?.(`ResultStore: could not write ${handle} to ${this.baseDir}, keeping it in memory: ${error.message}`);
+    }
+
+    this.index.set(handle, entry);
+    this.totalBytes += bytes;
+    this._evict(handle);
+    return handle;
+  }
+
+  /**
+   * @param {string} handle
+   * @returns {{ handle: string, toolName: string, payload: unknown, bytes: number, createdAt: number, expiresAt: number, meta: object } | null}
+   */
+  get(handle) {
+    const entry = this._live(handle);
+    if (!entry) return null;
+
+    let payload;
+    try {
+      const json = entry.inMemory ? entry.json : fs.readFileSync(this._file(handle), 'utf8');
+      payload = JSON.parse(json);
+    } catch {
+      // A missing or corrupt file is treated as gone, never thrown.
+      this.delete(handle);
+      return null;
+    }
+
+    // Refresh recency.
+    this.index.delete(handle);
+    this.index.set(handle, entry);
+
+    const { toolName, bytes, createdAt, expiresAt, meta } = entry;
+    return { handle, toolName, payload, bytes, createdAt, expiresAt, meta };
+  }
+
+  /** Live and unexpired, without a disk read. */
+  has(handle) {
+    return this._live(handle) !== null;
+  }
+
+  delete(handle) {
+    if (!RESULT_HANDLE_PATTERN.test(handle)) return false;
+    const entry = this.index.get(handle);
+    if (!entry) return false;
+    this.index.delete(handle);
+    this.totalBytes -= entry.bytes;
+    if (!entry.inMemory) {
+      try { fs.unlinkSync(this._file(handle)); } catch { /* already gone */ }
+    }
+    return true;
+  }
+
+  /** Index entries without payloads. */
+  list() {
+    return [...this.index.values()].map(({ handle, toolName, bytes, createdAt, expiresAt, meta, inMemory }) => (
+      { handle, toolName, bytes, createdAt, expiresAt, meta, inMemory: inMemory === true }
+    ));
+  }
+
+  /** Remove expired entries and their files. */
+  sweep() {
+    const now = Date.now();
+    for (const [handle, entry] of this.index) {
+      if (entry.expiresAt <= now) this.delete(handle);
+    }
+  }
+
+  close() {
+    if (this._sweepTimer) {
+      clearInterval(this._sweepTimer);
+      this._sweepTimer = null;
+    }
+  }
+
+  // ── Private ────────────────────────────────────────────────────────────────
+
+  _file(handle) {
+    return path.join(this.baseDir, `${handle}.json`);
+  }
+
+  /** The index entry when the handle is valid, known and unexpired; else null. */
+  _live(handle) {
+    if (typeof handle !== 'string' || !RESULT_HANDLE_PATTERN.test(handle)) return null;
+    const entry = this.index.get(handle);
+    if (!entry) return null;
+    if (entry.expiresAt <= Date.now()) {
+      this.delete(handle);
+      return null;
+    }
+    return entry;
+  }
+
+  /** Evict least-recently-used entries until the budget holds; never `keep`. */
+  _evict(keep) {
+    for (const handle of this.index.keys()) {
+      if (this.totalBytes <= this.maxBytes) return;
+      if (handle === keep) return;
+      this.delete(handle);
+    }
+  }
+
+  /**
+   * Best-effort removal of result files older than the TTL. Other server
+   * processes may share the directory, so a file younger than the TTL is
+   * never touched here even though this process did not write it.
+   */
+  _cleanupStaleFiles() {
+    try {
+      const cutoff = Date.now() - this.ttlMs;
+      for (const name of fs.readdirSync(this.baseDir)) {
+        if (!name.endsWith('.json') || !RESULT_HANDLE_PATTERN.test(name.slice(0, -5))) continue;
+        const file = path.join(this.baseDir, name);
+        try {
+          if (fs.statSync(file).mtimeMs < cutoff) fs.unlinkSync(file);
+        } catch { /* raced with another process */ }
+      }
+    } catch { /* no directory yet, or unreadable — nothing to clean */ }
+  }
+}
+
+// ── Process singleton ────────────────────────────────────────────────────────
+
+let singleton = null;
+
+/** The process-wide store; CRAWLFORGE_RESULTS_DIR overrides its directory. */
+export function getResultStore() {
+  if (!singleton) {
+    singleton = new ResultStore({ baseDir: process.env.CRAWLFORGE_RESULTS_DIR || undefined });
+  }
+  return singleton;
+}
+
+/** Test seam: replace (or with null, reset) the singleton. */
+export function setResultStoreForTests(store) {
+  singleton = store;
+}
+
+export default ResultStore;