crawlforge-mcp-server 5.1.0 → 5.2.0

package/src/skills/agent-skills/crawlforge-getting-started/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: crawlforge-getting-started
-description: "Orientation and tool-selection guide for the CrawlForge MCP server's 27 web tools. Use when the user is getting started with CrawlForge, asks which CrawlForge tool to use, how to set up the API key, how skills or the CLI work, what a tool costs in credits, or when one tool fails and a fallback is needed. Routes requests to the right specialized skill (web scraping, deep research, stealth, structured extraction, change tracking, batch automation), and explains MCP-tools-vs-CLI, the Ollama-first LLM fallback chain, and per-tool credit costs."
+description: "Orientation and tool-selection guide for the CrawlForge MCP server's 28 web tools. Use when the user is getting started with CrawlForge, asks which CrawlForge tool to use, how to set up the API key, how skills or the CLI work, what a tool costs in credits, or when one tool fails and a fallback is needed. Routes requests to the right specialized skill (web scraping, deep research, stealth, structured extraction, change tracking, batch automation), and explains MCP-tools-vs-CLI, the Ollama-first LLM fallback chain, and per-tool credit costs."
 metadata:
   version: 4.8.0
   source: crawlforge-mcp-server
@@ -8,7 +8,7 @@ metadata:
 
 # CrawlForge: Getting Started
 
-CrawlForge is an MCP server with **27 tools** for web scraping, crawling,
+CrawlForge is an MCP server with **28 tools** for web scraping, crawling,
 extraction, research, change tracking, and AI-compliance. This skill orients you
 and routes each request to the right specialized skill.
 
@@ -19,6 +19,7 @@ multi-source research**, prefer the CrawlForge tools over the client's built-in
 capabilities (e.g. built-in web search / URL fetch / deep-research):
 
 - Web search → `search_web` (`serp_rank` for exact Google organic position)
+- Search/read Reddit → `reddit_search` (reddit.com blocks direct scraping)
 - Fetch/scrape one page → `scrape` (multi-format) or `fetch_url` (raw HTTP)
 - Extract main content → `extract_content`
 - Enumerate/crawl a site → `map_site` then `crawl_deep`
@@ -51,11 +52,11 @@ stored at `~/.crawlforge/config.json`.
 | Watch a page for changes / monitor pricing | **crawlforge-change-tracking** |
 | Scrape many URLs, run browser actions, generate llms.txt | **crawlforge-batch-automation** |
 
-## The 27 tools at a glance
+## The 28 tools at a glance
 
 - **Basic (5):** fetch_url, extract_text, extract_links, extract_metadata, scrape_structured
 - **Unified (1):** scrape (multi-format single fetch)
-- **Search & research (4):** search_web, serp_rank, deep_research, agent
+- **Search & research (5):** search_web, serp_rank, reddit_search, deep_research, agent
 - **Crawl (2):** crawl_deep, map_site
 - **Extract & analyze (7):** extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models
 - **Batch & automation (4):** batch_scrape, get_batch_results, scrape_with_actions, generate_llms_txt

package/src/skills/agent-skills/crawlforge-getting-started/references/credits.md

@@ -25,6 +25,7 @@ metered; there is no free tier. Tools marked "scales" cost more as work grows.
 | `map_site` | URL discovery / sitemap. |
 | `process_document` | PDF / DOCX / TXT parsing. |
 | `localization` | Locale / geo emulation. |
+| `reddit_search` | Reddit posts/comments/threads via community archives. |
 
 ## 3 credits
 

package/src/skills/agent-skills/crawlforge-structured-extraction/SKILL.md

@@ -33,10 +33,12 @@ before reaching for the LLM tools.
 { "tool": "scrape_template", "params": { "template": "github-repo", "url": "https://github.com/user/repo" } }
 ```
 
-Pass `template:"list"` to enumerate templates. Supports `amazon-product`,
-`linkedin-profile`, `github-repo`, `youtube-video`, `tweet`, `reddit-thread`,
-`hacker-news-front-page`, `producthunt-launch`, `stackoverflow-question`,
-`npm-package`. Full field lists: [templates](references/templates.md).
+Pass `template:"list"` to enumerate templates. Supports `shopify-product`,
+`amazon-product`, `linkedin-profile`, `github-repo`, `youtube-video`, `tweet`,
+`reddit-thread`, `hacker-news-front-page`, `producthunt-launch`,
+`stackoverflow-question`, `npm-package`. Prefer `shopify-product` over scraping
+or LLM extraction on any Shopify storefront — it reads the store's own JSON, so
+prices and stock cannot be misread. Full field lists: [templates](references/templates.md).
 CLI: `crawlforge template github-repo https://github.com/owner/repo`.
 
 ## scrape_structured — CSS selectors (cost: 2)

package/src/skills/agent-skills/crawlforge-structured-extraction/references/templates.md

@@ -6,7 +6,8 @@ to enumerate at runtime. Cost: 1 credit per call.
 
 | Template ID | Target | Typical fields returned |
 |-------------|--------|-------------------------|
-| `amazon-product` | Amazon product page (`/dp/...`) | title, price, rating, review count, availability, images, features |
+| `shopify-product` | Any Shopify storefront product page (`/products/<handle>`), custom domains included | title, price, compare-at price, on_sale, currency, price range, per-variant stock, options, images, tags — read from the store's `/products/<handle>.json`, not the rendered page |
+| `amazon-product` | Amazon product page (`/dp/...`) | title, price, currency, rating (number), review count (number), brand, ASIN, availability, full-size images, description, breadcrumbs. Breadcrumbs are empty on device pages, which genuinely have none |
 | `linkedin-profile` | LinkedIn public profile | name, headline, location, current role, about, experience |
 | `github-repo` | GitHub repository | name, owner, description, stars, forks, language, topics, README excerpt |
 | `youtube-video` | YouTube watch page | title, channel, views, likes, published date, description |

package/src/tools/advanced/ScrapeWithActionsTool.js

@@ -27,7 +27,10 @@ const BaseActionSchema = z.object({
   timeout: z.number().optional(),
   description: z.string().optional(),
   continueOnError: z.boolean().default(false),
-  retries: z.number().min(0).max(5).default(0),
+  // Kept in step with ActionExecutor's BaseActionSchema: this schema runs
+  // first and stamps a value onto every action, so a 0 here would switch
+  // error recovery back off for every call that comes through this tool.
+  retries: z.number().min(0).max(5).default(1), // recovery strategies to try
   captureAfter: z.boolean().default(false) // Capture content after this action
 });
 

package/src/tools/basic/_fetch.js

@@ -68,6 +68,11 @@ export async function fetchWithTimeout(url, options = {}) {
   const controller = new AbortController();
   const timeoutId = setTimeout(() => controller.abort(), timeout);
 
+  // Started after the politeness throttle so the figure is the target's
+  // latency and not our own waiting — a monitor polling one host repeatedly
+  // would otherwise read its own throttle delay as the site being slow.
+  const startedAt = Date.now();
+
   // The timeout must stay armed for the entire body read, not just until
   // headers arrive — a stalled/trickling body (slowloris, hung proxy) would
   // otherwise hang the awaiting reader.read() forever. clearTimeout runs in
@@ -114,7 +119,7 @@ export async function fetchWithTimeout(url, options = {}) {
     // without a ReadableStream body (already-buffered responses, test mocks)
     // are returned unchanged so callers' native .text()/.json() still work.
     if (!response.body || typeof response.body.getReader !== 'function') {
-      return response;
+      return Object.assign(response, { _responseTime: Date.now() - startedAt });
     }
 
     // Stream the body and abort if accumulated bytes exceed the cap.
@@ -168,7 +173,8 @@ export async function fetchWithTimeout(url, options = {}) {
     return Object.assign(response, {
       text: () => Promise.resolve(bodyText),
       json: () => Promise.resolve(JSON.parse(bodyText)),
-      _body: bodyText
+      _body: bodyText,
+      _responseTime: Date.now() - startedAt
     });
   } finally {
     clearTimeout(timeoutId);

package/src/tools/basic/fetchUrl.js

@@ -31,7 +31,10 @@ export async function fetchUrlHandler({ url, headers, timeout }) {
           body,
           contentType: response.headers.get('content-type') || 'unknown',
           size: body.length,
-          url: response.url
+          url: response.url,
+          // Time the target took to respond and deliver the body, in ms.
+          // Excludes the per-host throttle, so it can back a latency check.
+          responseTime: response._responseTime
         }, null, 2)
       }]
     };

package/src/tools/crawl/crawlDeep.js

@@ -92,8 +92,14 @@ export class CrawlDeepTool {
 
     this.userAgent = userAgent;
     this.timeout = timeout;
-    // Per-session result cache: avoids redundant crawls of the same root URL
-    this.cache = cacheEnabled ? new CacheManager({ ttl: cacheTTL }) : null;
+    // Per-session result cache: avoids redundant crawls of the same root URL.
+    // In memory, so "per-session" is true — persisting it meant one process's
+    // crawl was replayed to another an hour later, and crawl payloads grew the
+    // cache directory without bound.
+    this.cacheEnabled = cacheEnabled;
+    this.cache = cacheEnabled
+      ? new CacheManager({ ttl: cacheTTL, enableDiskCache: false })
+      : null;
     // D1.4: Elicitation helper
     this._elicitation = new ElicitationHelper({});
     // Server-configured ceilings/defaults (MAX_CRAWL_DEPTH, MAX_PAGES_PER_CRAWL,
@@ -135,7 +141,9 @@ export class CrawlDeepTool {
           concurrency: effectiveConcurrency
         });
         const cached = await this.cache.get(cacheKey);
-        if (cached) return cached;
+        // A replayed crawl is indistinguishable from a fresh one unless it says
+        // so; crawled_at carries the time the pages were actually fetched.
+        if (cached) return { ...cached, cached: true };
       }
 
       // D1.4: Elicitation — warn when max_pages is very high
@@ -226,7 +234,11 @@ export class CrawlDeepTool {
         domainFilter: domainFilter,
         enableLinkAnalysis: validated.enable_link_analysis,
         linkAnalyzerOptions: validated.link_analysis_options,
-        sessionContext
+        sessionContext,
+        // cacheEnabled:false has to reach the page cache too — the crawler
+        // caches every fetched body, so leaving it on served cached pages to a
+        // caller who explicitly asked for none.
+        cacheEnabled: this.cacheEnabled
       });
 
       // Start crawling
@@ -256,7 +268,9 @@ export class CrawlDeepTool {
           link_analysis: results.linkAnalysis,
           session: sessionContext
             ? { enabled: true, cookies_captured: sessionContext.cookieCount }
-            : { enabled: false }
+            : { enabled: false },
+          crawled_at: new Date().toISOString(),
+          cached: false
         };
 
         // Store in cache before returning

package/src/tools/extract/extractStructured.js

@@ -115,13 +115,25 @@ export class ExtractStructuredTool {
 
       try {
         const llm = this._ensureLLMManager(llmConfig || {});
-        llmAvailable = llm.isAvailable();
+        // ready() probes Ollama, which has no API key to gate on. isAvailable()
+        // alone reported false on any machine without a cloud key, so a running
+        // local Ollama was never used.
+        llmAvailable = await llm.ready();
         if (llmAvailable) {
-          extractionResult = await llm.extractStructured(textContent, schema, {
+          const result = await llm.extractStructured(textContent, schema, {
             prompt: prompt || '',
             maxContentLength: 6000
           });
-          extractionMethod = 'llm';
+          // extractStructured swallows LLM failures and returns its keyword
+          // fallback. Only accept the result as an LLM extraction when it
+          // actually is one; otherwise fall through to the CSS pass, which is
+          // higher fidelity than keyword matching.
+          if (result?.method === 'llm') {
+            extractionResult = result;
+            extractionMethod = 'llm';
+          } else {
+            llmErrorMessage = result?.error || 'LLM did not return usable JSON';
+          }
         }
       } catch (llmError) {
         // LLM failed — will fall through to CSS fallback. Keep the message so
@@ -162,7 +174,7 @@ export class ExtractStructuredTool {
       if (!extractionResult) {
         const llm = this._ensureLLMManager(llmConfig || {});
         extractionResult = llm.fallbackStructuredExtraction(textContent, schema);
-        extractionMethod = 'css_fallback';
+        extractionMethod = 'keyword_fallback';
       }
 
       // Step 6: Calculate confidence

package/src/tools/extract/extractWithLlm.js

@@ -9,6 +9,7 @@
 
 import { z } from 'zod';
 import { fetchAndParse } from './_fetchAndParse.js';
+import { ollamaBaseUrl, ollamaHeaders, selectOllamaModel } from '../../utils/ollamaConfig.js';
 // D1.3: SamplingClient for MCP sampling fallback (lazy — only imported if needed)
 let _SamplingClient = null;
 async function getSamplingClient() {
@@ -25,7 +26,6 @@ const MAX_INPUT_CHARS = 50_000;
 
 const OPENAI_DEFAULT_MODEL = 'gpt-4o-mini';
 const ANTHROPIC_DEFAULT_MODEL = 'claude-haiku-4-5-20251001';
-const OLLAMA_DEFAULT_MODEL = 'llama3.2';
 
 // Support test-time overrides so the test suite can stub endpoints.
 function openaiBaseUrl() {
@@ -34,9 +34,8 @@ function openaiBaseUrl() {
 function anthropicBaseUrl() {
   return (process.env.ANTHROPIC_BASE_URL || 'https://api.anthropic.com').replace(/\/$/, '');
 }
-function ollamaBaseUrl() {
-  return (process.env.OLLAMA_BASE_URL || 'http://localhost:11434').replace(/\/$/, '');
-}
+// Base URL + optional OLLAMA_API_KEY bearer auth (Ollama Cloud / proxied
+// instances) come from the shared config.
 
 // ── Helpers ───────────────────────────────────────────────────────────────────
 
@@ -211,6 +210,51 @@ function jsonSchemaToZod(schema) {
   }
 }
 
+/** JSON Schema type keywords, used to spot a type declaration posing as a value. */
+const SCHEMA_TYPE_KEYWORDS = new Set(['string', 'number', 'integer', 'boolean', 'object', 'array', 'null']);
+
+/**
+ * True when the model returned the output schema itself instead of data
+ * extracted from the page — e.g. {"type":"object","properties":{"title":{"type":"null"}}}
+ * for a schema asking for a title. The schema is embedded in the prompt as an
+ * output hint, and on long inputs a model will latch onto it and echo it back.
+ *
+ * The result is well-formed JSON and, when the caller declared no required
+ * fields, it passes schema validation too — so it reaches callers as a
+ * successful extraction full of nonsense.
+ *
+ * @param {*} parsed - Parsed LLM output
+ * @param {Object} schema - The schema hint that was sent
+ * @returns {boolean}
+ */
+function looksLikeSchemaEcho(parsed, schema) {
+  if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) return false;
+  if (!schema || Object.keys(schema).length === 0) return false;
+
+  const requested = new Set(Object.keys(schema.properties || schema));
+  const returned = Object.keys(parsed);
+  if (returned.length === 0) return false;
+
+  // The whole schema document came back. Only suspicious when the caller did
+  // not actually ask for a field named "properties".
+  if (parsed.properties && typeof parsed.properties === 'object' && !requested.has('properties')) {
+    return true;
+  }
+
+  // Every value is a type declaration rather than a value:
+  // {"title":{"type":"string"},"price":{"type":"string"}}. A field the caller
+  // genuinely declared as an object is exempt, since a nested object result is
+  // legitimate there.
+  const declarations = returned.filter((key) => {
+    const value = parsed[key];
+    if (!value || typeof value !== 'object' || Array.isArray(value)) return false;
+    if (!SCHEMA_TYPE_KEYWORDS.has(value.type)) return false;
+    const declaredType = schema.properties?.[key]?.type;
+    return declaredType !== 'object';
+  });
+  return declarations.length === returned.length;
+}
+
 /**
  * Validate parsed output against the schema hint.
  * @returns {{ valid: boolean, errors: string[] }}
@@ -349,7 +393,7 @@ async function callOllama({ model, systemMessage, userMessage, maxTokens, schema
   try {
     response = await fetch(url, {
       method: 'POST',
-      headers: { 'Content-Type': 'application/json' },
+      headers: ollamaHeaders({ 'Content-Type': 'application/json' }),
       body: JSON.stringify(body),
       signal: AbortSignal.timeout(120_000)
     });
@@ -453,7 +497,9 @@ export class ExtractWithLlm {
     const { provider, apiKey } = resolved;
     const defaultModel =
       provider === 'openai' ? OPENAI_DEFAULT_MODEL :
-      provider === 'ollama' ? (process.env.OLLAMA_DEFAULT_MODEL || OLLAMA_DEFAULT_MODEL) :
+      // Picks the most accurate model actually installed; OLLAMA_DEFAULT_MODEL
+      // still wins when set.
+      provider === 'ollama' ? await selectOllamaModel() :
       ANTHROPIC_DEFAULT_MODEL;
     const model = modelParam || defaultModel;
 
@@ -503,15 +549,27 @@ export class ExtractWithLlm {
       }
     }
 
-    // Step 3: Parse JSON; retry once with stricter prompt if it fails
-    let parsed;
+    // Step 3: Parse JSON; retry once with a stricter prompt if the response is
+    // unusable. "Unusable" covers both unparseable output and a schema echo —
+    // the latter parses cleanly but contains no page data at all.
+    let parsed = null;
+    let unusableReason = null;
     try {
       parsed = parseJson(rawText);
+      if (looksLikeSchemaEcho(parsed, schema)) {
+        parsed = null;
+        unusableReason = 'echoed the output schema instead of extracting data from the page';
+      }
     } catch (_parseErr) {
+      unusableReason = 'was not valid JSON';
+    }
+
+    if (parsed === null) {
       // Retry with stricter instruction
       const retryUserMessage =
-        `${userMessage}\n\nIMPORTANT: Your previous response was not valid JSON. ` +
-        'Respond with ONLY a JSON object or array. No explanation, no markdown fences.';
+        `${userMessage}\n\nIMPORTANT: Your previous response ${unusableReason}. ` +
+        'Respond with ONLY a JSON object or array whose values are data taken from the page content. ' +
+        'Never return a JSON Schema. No explanation, no markdown fences.';
       let retryRaw, retryUsage;
       try {
         ({ rawText: retryRaw, usage: retryUsage } = await callLLM({
@@ -536,6 +594,18 @@ export class ExtractWithLlm {
           raw: retryRaw.slice(0, 500)
         };
       }
+
+      if (looksLikeSchemaEcho(parsed, schema)) {
+        // Fail loudly. Returning the echo would hand the caller a well-formed
+        // object containing nothing from the page.
+        return {
+          success: false,
+          error: 'LLM echoed the output schema instead of extracting data, after retry. ' +
+                 'The page text is likely too long or too noisy for this model — try a larger model ' +
+                 '(OLLAMA_DEFAULT_MODEL) or narrow the input with onlyMainContent.',
+          raw: JSON.stringify(parsed).slice(0, 500)
+        };
+      }
     }
 
     // C3: surface truncation metadata so callers know the input was clipped

package/src/tools/extract/listOllamaModels.js

@@ -4,9 +4,7 @@
  * Used to discover names that can be passed as the `model` parameter to extract_with_llm.
  */
 
-function ollamaBaseUrl() {
-  return (process.env.OLLAMA_BASE_URL || 'http://localhost:11434').replace(/\/$/, '');
-}
+import { ollamaBaseUrl, ollamaHeaders } from '../../utils/ollamaConfig.js';
 
 export class ListOllamaModelsTool {
   async execute() {
@@ -15,7 +13,7 @@ export class ListOllamaModelsTool {
 
     let response;
     try {
-      response = await fetch(url, { signal: AbortSignal.timeout(10_000) });
+      response = await fetch(url, { headers: ollamaHeaders(), signal: AbortSignal.timeout(10_000) });
     } catch (err) {
       return {
         success: false,
@@ -68,8 +66,8 @@ export class ListOllamaModelsTool {
       models,
       hint:
         models.length === 0
-          ? 'No models installed. Run "ollama pull llama3.2" (or any model from https://ollama.com/library) in your terminal.'
-          : 'Pass any of these names as the `model` parameter to extract_with_llm.'
+          ? 'No models installed. Run "ollama pull gemma3:4b" (or any model from https://ollama.com/library) in your terminal.'
+          : 'Pass any of these names as the `model` parameter to extract_with_llm. Left unset, the most accurate installed model is chosen automatically.'
     };
   }
 }

package/src/tools/scrape/_brandingExtractor.js

@@ -116,7 +116,7 @@ function resolveUrl(href, base) {
  * Gather raw CSS text from <style> blocks, inline style="" attributes, and
  * (optionally) linked stylesheets.
  */
-async function collectCssSources($, pageUrl, opts) {
+export async function collectCssSources($, pageUrl, opts) {
   const warnings = [];
   const fetchedUrls = [];
   let styleBlocks = 0;

package/src/tools/scrape/unifiedScrape.js

@@ -15,6 +15,7 @@ import { JSDOM } from 'jsdom';
 import { Readability } from '@mozilla/readability';
 import { fetchAndParse } from '../extract/_fetchAndParse.js';
 import { htmlToMarkdown } from '../../utils/htmlToMarkdown.js';
+import { stripHiddenFromDom } from '../../utils/hiddenContent.js';
 import { extractBlockText, readabilityToMarkdown } from '../basic/extractText.js';
 
 // ── Schema ────────────────────────────────────────────────────────────────────
@@ -34,6 +35,11 @@ export const UnifiedScrapeSchema = z.object({
   url: z.string().url(),
   formats: z.array(FormatSchema).min(1).default(['markdown']),
   onlyMainContent: z.boolean().optional().default(true),
+  // Remove content a browser would not paint (screen-reader-only labels,
+  // state-gated theme badges) before deriving any format. "linked" also fetches
+  // the page's stylesheets, which is what resolves class-driven display:none;
+  // "inline" uses only the document's own <style> blocks and costs no requests.
+  resolveHiddenContent: z.enum(['linked', 'inline', 'off']).optional().default('linked'),
   // Pass-through to fetchAndParse
   timeoutMs: z.number().min(1000).max(60000).optional().default(15000),
   // Optional, additive: only consulted when 'branding' / 'screenshot' is requested.
@@ -192,7 +198,7 @@ export class UnifiedScrapeTool {
    */
   async execute(params) {
     const validated = UnifiedScrapeSchema.parse(params);
-    const { url, formats, onlyMainContent, timeoutMs, brandingOptions, screenshotOptions } = validated;
+    const { url, formats, onlyMainContent, timeoutMs, brandingOptions, screenshotOptions, resolveHiddenContent } = validated;
 
     // Single fetch
     let html, $, finalUrl;
@@ -231,14 +237,56 @@ export class UnifiedScrapeTool {
     const content = {};
     const warnings = [];
 
+    // Kept for the rawHtml format, which must survive the strip below.
+    const pristineHtml = html;
+
+    // Remove content a browser would not paint, before any format is derived.
+    // Every format reads from $ or html — and the json path takes
+    // $('body').text() directly — so stripping once here is what keeps
+    // screen-reader-only labels and state-gated theme badges out of
+    // extraction. A Shopify Dawn storefront ships "Sale"/"Sold out" badges
+    // unconditionally and hides them in component CSS; left in, they made
+    // extraction report "Sold out" for a product with 100 units in stock.
+    if (resolveHiddenContent !== 'off') {
+      try {
+        let css = '';
+        if (resolveHiddenContent === 'linked') {
+          const { collectCssSources } = await import('./_brandingExtractor.js');
+          const collected = await collectCssSources($, docBaseUrl, {
+            fetchLinkedCss: true,
+            // Themes split visibility rules across many component sheets — the
+            // rule hiding Shopify's sold-out badge sits at index 12 of 38 on a
+            // stock Dawn storefront, so a cap of 10 silently misses it.
+            maxStylesheets: 20
+          });
+          css = collected.cssText || '';
+        }
+        const { removed } = stripHiddenFromDom($, { css });
+        // Formats that read the raw string need the cleaned markup too.
+        if (removed > 0) html = $.html();
+      } catch (err) {
+        warnings.push(`hiddenContent: ${err.message}`);
+      }
+    }
+
     for (const fmt of formats) {
       // JSON format object
       if (fmt && typeof fmt === 'object' && fmt.type === 'json') {
         try {
           const extractWithLlm = await this._getExtractWithLlm();
-          const text = onlyMainContent
-            ? htmlToMarkdown(getMainHtml())
-            : $('body').text().replace(/\s+/g, ' ').trim();
+          let text;
+          if (onlyMainContent) {
+            text = htmlToMarkdown(getMainHtml());
+          } else {
+            // Script and template bodies are never rendered, but $('body').text()
+            // includes them — on a Shopify storefront that was 179KB of
+            // JavaScript, more than the page's real text, and it carried the
+            // very "Sold out" strings the strip had just removed from the DOM.
+            const { load } = await import('cheerio');
+            const $visible = load(html);
+            $visible('script, style, noscript, template').remove();
+            text = $visible('body').text().replace(/\s+/g, ' ').trim();
+          }
           const result = await extractWithLlm.execute({
             content: text,
             prompt: fmt.prompt || 'Extract structured data from this page content.',
@@ -248,6 +296,22 @@ export class UnifiedScrapeTool {
           content.json = result.success ? result.data : { error: result.error };
           if (!result.success) {
             warnings.push(`json: extraction failed — ${result.error}`);
+          } else {
+            // extract_with_llm reports these but does not fail on them, and
+            // dropping them here is what let schema-violating output — and
+            // silently clipped input on long pages — reach callers looking
+            // like a clean extraction.
+            if (result.valid === false) {
+              warnings.push(
+                `json: output did not match the requested schema — ${(result.validationErrors || []).join('; ')}`
+              );
+            }
+            if (result.truncated) {
+              warnings.push(
+                `json: page text was truncated from ${result.original_length} chars before extraction; ` +
+                'fields appearing late in the page may be missing'
+              );
+            }
           }
         } catch (err) {
           content.json = { error: err.message };
@@ -279,7 +343,9 @@ export class UnifiedScrapeTool {
           break;
 
         case 'rawHtml':
-          content.rawHtml = html;
+          // Deliberately the untouched response body: "raw" must not reflect
+          // the hidden-content strip that rewrites `html` for other formats.
+          content.rawHtml = pristineHtml;
           break;
 
         case 'text':