crawlforge-mcp-server 5.5.9 → 5.6.1

package/CLAUDE.md

@@ -60,9 +60,9 @@ These guidelines are working if: fewer unnecessary changes in diffs, fewer rewri
 
 ## Project Overview
 
-CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 28 web scraping, crawling, and content processing tools (5 inline + 23 advanced).
+CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 29 web scraping, crawling, and content processing tools (6 inline + 23 advanced).
 
-**Current Version:** 5.3.0
+**Current Version:** 5.6.0
 
 ## Development Commands
 
@@ -75,11 +75,6 @@ npm run setup
 # Or provide API key via environment:
 export CRAWLFORGE_API_KEY="your_api_key_here"
 
-# Creator Mode (for package maintainer only)
-# Set your creator secret in .env file:
-# CRAWLFORGE_CREATOR_SECRET=your-secret-uuid
-# This enables unlimited access for development/testing
-
 # Run the server (production)
 npm start
 
@@ -172,13 +167,13 @@ Tools are organized in subdirectories by category:
 - `tracking/` - trackChanges
 - `llmstxt/` - generateLLMsTxt
 
-### Available MCP Tools (28 total)
+### Available MCP Tools (29 total)
 
-**Basic Tools (server.js inline, 5):**
-fetch_url, extract_text, extract_links, extract_metadata, scrape_structured
+**Basic Tools (server.js inline, 6):**
+fetch_url, extract_text, extract_links, extract_metadata, extract_embedded_state, scrape_structured
 
 **Advanced Tools (23):**
-search_web, serp_rank, reddit_search, crawl_deep, map_site, extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models, batch_scrape, scrape_with_actions, deep_research, track_changes, generate_llms_txt, stealth_mode, localization, scrape_template, scrape, agent
+search_web, serp_rank, reddit_search, crawl_deep, map_site, extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models, batch_scrape, get_batch_results, scrape_with_actions, deep_research, track_changes, generate_llms_txt, stealth_mode, localization, scrape_template, scrape, agent
 
 **serp_rank (DataForSEO):**
 - `serp_rank` — reports where a target domain ranks in Google's REAL organic results for a keyword (the SERP position Google Custom Search / `search_web` cannot give). Backed by the DataForSEO Google Organic SERP API (Live Advanced, `POST /v3/serp/google/organic/live/advanced`, HTTP Basic auth). Credentials via `DATAFORSEO_LOGIN` / `DATAFORSEO_PASSWORD`, billed to the user's own DataForSEO account (~US$0.002 per 10 results of `depth`, so $0.004 at the default `depth:20` and $0.02 at `depth:100`), separate from CrawlForge credits. When unconfigured it returns `{ configured:false }` and charges **0** credits; when configured, **Cost: 5**. Never fabricates a rank.

package/README.md

@@ -192,7 +192,7 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `summarize_content` | 4 | Generate intelligent summaries |
 | `crawl_deep` | 4 | Deep crawl entire websites |
 | `search_web` | 5 | Search the web using Google Search API |
-| `reddit_search` | 5 | Search Reddit posts/comments or read a full thread — reddit.com blocks direct scraping, so this queries the Arctic Shift + PullPush community archives (free, no Reddit credentials). A Reddit-wide search spends a web search to discover posts, so it is priced with `search_web` |
+| `reddit_search` | 5 | Search Reddit posts/comments or read a full thread — reddit.com blocks direct scraping, so this reads the Arctic Shift community archive (free, no Reddit credentials). A Reddit-wide search spends a web search to discover posts, so it is priced with `search_web` |
 | `serp_rank` | 5 | Check where a domain ranks in Google's **real organic SERP** for a keyword (the position `search_web` can't give). Powered by DataForSEO (`DATAFORSEO_LOGIN`/`DATAFORSEO_PASSWORD`, billed to your own DataForSEO account). Returns `{ configured:false }` and charges **0** credits until configured |
 | `batch_scrape` | 5 | Process multiple URLs simultaneously |
 | `scrape_with_actions` | 5 | Browser automation chains |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "5.5.9",
+  "version": "5.6.1",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 29 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
@@ -114,7 +114,7 @@
     "cheerio": "^1.1.2",
     "commander": "^14.0.3",
     "compromise": "^14.14.4",
-    "crawlforge-extractors": "^1.5.3",
+    "crawlforge-extractors": "^1.6.1",
     "diff": "^9.0.0",
     "dotenv": "^17.2.1",
     "franc": "^6.2.0",

package/server.js

@@ -105,7 +105,7 @@ const taskStore = createTaskStore({ logger });
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "5.5.9",
+  version: "5.6.1",
   description: "Production-ready MCP server with 29 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, embedded JavaScript state extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",
@@ -1574,7 +1574,7 @@ registerToolIfEnabled("localization", {
         break;
       case 'generate_timezone_spoof':
         result = {
-          timezoneScript: await localizationManager.generateTimezoneSpoof(params.countryCode),
+          timezoneScript: await localizationManager.generateTimezoneSpoof(params.countryCode, params.timezone),
           countryCode: params.countryCode || localizationManager.getCurrentSettings().countryCode
         };
         break;

package/src/core/AgentOrchestrator.js

@@ -203,7 +203,13 @@ export class AgentOrchestrator {
       const { text } = await this._getSamplingClient().complete(planPrompt, { maxTokens: 200 });
       const lines = text.split('\n')
         .map(l => l.replace(/^[-*\d.)\s]+/, '').trim())
+        // A model that fences its list ("```\nq1\nq2\n```") or quotes each
+        // query leaks the fence and quote characters into the queries; the
+        // R14 Julia run spent two of its three searches on the literal "```".
+        .map(l => l.replace(/^```\w*\s*|\s*```$/g, '').replace(/^[`"']+|[`"']+$/g, '').trim())
         .filter(Boolean)
+        // A line with no letter or digit left is punctuation, not a query.
+        .filter(l => /[\p{L}\p{N}]/u.test(l))
         // Drop preamble/garbage lines ("Here are 3 concise web search queries:", …)
         // so they never become search query #1 and poison the URL queue.
         .filter(l =>

package/src/core/LLMsTxtAnalyzer.js

@@ -1,5 +1,6 @@
 import { z } from 'zod';
 import { load } from 'cheerio';
+import { readBody } from 'crawlforge-extractors';
 import { MapSiteTool } from '../tools/crawl/mapSite.js';
 import { CrawlDeepTool } from '../tools/crawl/crawlDeep.js';
 import { normalizeUrl, getBaseUrl } from '../utils/urlNormalizer.js';
@@ -240,7 +241,7 @@ export class LLMsTxtAnalyzer {
       // Look for API documentation references
       const mainPageResponse = await this.fetchWithTimeout(baseUrl);
       if (mainPageResponse.ok) {
-        const html = await mainPageResponse.text();
+        const html = await readBody(mainPageResponse);
         const $ = load(html);
         
         // Find API documentation links
@@ -490,7 +491,7 @@ export class LLMsTxtAnalyzer {
       const robotsUrl = `${baseUrl}/robots.txt`;
       const response = await this.fetchWithTimeout(robotsUrl);
       if (response.ok) {
-        return await response.text();
+        return await readBody(response);
       }
     } catch {
       // No robots.txt found
@@ -677,7 +678,9 @@ export class LLMsTxtAnalyzer {
       }
 
       const contentType = response.headers.get('content-type') || '';
-      const html = await response.text();
+      // Charset-aware read; .text() decodes UTF-8 only and mangled
+      // ISO-8859-1 page titles in the generated llms.txt.
+      const html = await readBody(response);
       const $ = load(html);
 
       // Check for forms

package/src/core/LocalizationManager.js

@@ -428,9 +428,13 @@ export class LocalizationManager extends EventEmitter {
    * @param {string} countryCode - Target country code
    * @returns {string} - JavaScript injection code
    */
-  async generateTimezoneSpoof(countryCode = null) {
+  async generateTimezoneSpoof(countryCode = null, timezone = null) {
     const targetCountry = countryCode || this.currentSettings.countryCode;
-    const config = await this.getLocalizationConfig(targetCountry);
+    let config = await this.getLocalizationConfig(targetCountry);
+    // An explicit IANA timezone wins over the country default: the tool
+    // accepted `timezone` and silently answered with the configured country's
+    // zone (Asia/Tokyo asked, Europe/Berlin returned — R14).
+    if (timezone) config = { ...config, timezone };
     
     const timezoneOffset = this.getTimezoneOffset(config.timezone);
     

package/src/core/StealthBrowserManager.js

@@ -616,6 +616,14 @@ export class StealthBrowserManager {
 
     // Select browser based on distribution and OS compatibility
     let availableBrowsers = { ...this.browserDistribution };
+    // The engine decides the pool. Chromium exposes navigator.vendor
+    // "Google Inc.", window.chrome and Chrome's PDF viewer plugins, so a
+    // Firefox or Safari User-Agent on it contradicts what the page can see —
+    // sannysoft printed a Firefox UA beside "Chrome: present" (R14). Camoufox
+    // presents its own Firefox identity and never draws from this pool.
+    if (config.engine !== 'camoufox') {
+      availableBrowsers = { chrome: 1 };
+    }
     if (selectedOS === 'linux' && availableBrowsers.safari) {
       delete availableBrowsers.safari;
       // Redistribute safari's weight
@@ -1005,7 +1013,9 @@ export class StealthBrowserManager {
       processor: selectedProcessor.name,
       architecture: 'x86_64',
       memory: Math.floor(Math.random() * 24) + 8, // 8-32 GB
-      deviceMemory: Math.pow(2, Math.floor(Math.random() * 3) + 3), // 8, 16, or 32 GB
+      // navigator.deviceMemory is capped at 8 in every Chromium build, so 16
+      // or 32 is a value no real browser reports (sannysoft CHR_MEMORY: FAIL).
+      deviceMemory: Math.random() < 0.7 ? 8 : 4,
       platform: this.selectRealisticPlatform(selectedOS)
     };
   }
@@ -1215,29 +1225,64 @@ export class StealthBrowserManager {
         })
       });
 
-      // Override plugin array
-      Object.defineProperty(navigator, 'plugins', {
-        get: function() {
-          const plugins = [
-            {
-              0: {
-                type: "application/x-google-chrome-pdf",
-                suffixes: "pdf",
-                description: "Portable Document Format"
-              },
-              description: "Portable Document Format",
-              filename: "internal-pdf-viewer",
-              length: 1,
-              name: "Chrome PDF Plugin"
-            }
-          ];
-          plugins.item = function(index) { return this[index] || null; };
-          plugins.namedItem = function(name) { 
-            return this.find(plugin => plugin.name === name) || null; 
-          };
-          return plugins;
-        }
-      });
+      // Plugin list. New headless Chromium already reports the same five PDF
+      // viewer entries as a real PluginArray, and replacing them with a plain
+      // Array fails the "plugins is of type PluginArray" check every scanner
+      // runs. Only an empty list — the legacy headless tell — is replaced.
+      if (navigator.plugins.length === 0 && typeof PluginArray !== 'undefined') {
+        // Built on the real prototypes so `instanceof PluginArray`, the
+        // toStringTag and the DOM-style accessors all hold — a plain Array here
+        // failed bot.sannysoft.com's "plugins is of type PluginArray" check.
+        // The five entries are what every desktop Chrome reports.
+        const mimeSpecs = [
+          { type: 'application/pdf', suffixes: 'pdf', description: 'Portable Document Format' },
+          { type: 'text/pdf', suffixes: 'pdf', description: 'Portable Document Format' }
+        ];
+        const makePlugin = (name) => {
+          const plugin = Object.create(Plugin.prototype);
+          const mimeTypes = mimeSpecs.map((spec) => {
+            const mimeType = Object.create(MimeType.prototype);
+            Object.defineProperties(mimeType, {
+              type: { value: spec.type },
+              suffixes: { value: spec.suffixes },
+              description: { value: spec.description },
+              enabledPlugin: { get: () => plugin }
+            });
+            return mimeType;
+          });
+          Object.defineProperties(plugin, {
+            name: { value: name },
+            filename: { value: 'internal-pdf-viewer' },
+            description: { value: 'Portable Document Format' },
+            length: { value: mimeTypes.length },
+            item: { value: (index) => mimeTypes[index] || null },
+            namedItem: { value: (type) => mimeTypes.find((m) => m.type === type) || null },
+            [Symbol.iterator]: { value: function* () { yield* mimeTypes; } }
+          });
+          mimeTypes.forEach((mimeType, index) => {
+            Object.defineProperty(plugin, index, { value: mimeType, enumerable: true });
+            Object.defineProperty(plugin, mimeType.type, { value: mimeType });
+          });
+          return plugin;
+        };
+        const plugins = [
+          'PDF Viewer', 'Chrome PDF Viewer', 'Chromium PDF Viewer',
+          'Microsoft Edge PDF Viewer', 'WebKit built-in PDF'
+        ].map(makePlugin);
+        const pluginArray = Object.create(PluginArray.prototype);
+        Object.defineProperties(pluginArray, {
+          length: { value: plugins.length },
+          item: { value: (index) => plugins[index] || null },
+          namedItem: { value: (name) => plugins.find((p) => p.name === name) || null },
+          refresh: { value: () => {} },
+          [Symbol.iterator]: { value: function* () { yield* plugins; } }
+        });
+        plugins.forEach((plugin, index) => {
+          Object.defineProperty(pluginArray, index, { value: plugin, enumerable: true });
+          Object.defineProperty(pluginArray, plugin.name, { value: plugin });
+        });
+        Object.defineProperty(navigator, 'plugins', { get: () => pluginArray });
+      }
 
       // Override languages with the fingerprint's own locale — a hardcoded
       // en-US here contradicts navigator.language and Accept-Language whenever
@@ -1367,14 +1412,23 @@ export class StealthBrowserManager {
         HTMLCanvasElement.prototype.getContext = function(contextType, contextAttributes) {
           const ctx = getContext.call(this, contextType, contextAttributes);
           
-          if (contextType === 'webgl' || contextType === 'experimental-webgl') {
+          // Headless Chromium without a GPU path answers getContext('webgl')
+          // with null; wrapping it threw a TypeError where a real browser
+          // returns null, which is itself a tell.
+          if (ctx && (contextType === 'webgl' || contextType === 'experimental-webgl')) {
             const originalGetParameter = ctx.getParameter;
+            // Real Chrome answers VENDOR/RENDERER with "WebKit"/"WebKit WebGL";
+            // the GPU identity lives behind WEBGL_debug_renderer_info as the
+            // UNMASKED_VENDOR_WEBGL (37445) / UNMASKED_RENDERER_WEBGL (37446)
+            // parameters. Those are what a scanner reads, so those are spoofed.
+            const UNMASKED_VENDOR_WEBGL = 37445;
+            const UNMASKED_RENDERER_WEBGL = 37446;
             ctx.getParameter = function(parameter) {
               // Spoof specific WebGL parameters
-              if (parameter === ctx.RENDERER) {
+              if (parameter === UNMASKED_RENDERER_WEBGL) {
                 return webglConfig.renderer;
               }
-              if (parameter === ctx.VENDOR) {
+              if (parameter === UNMASKED_VENDOR_WEBGL) {
                 return webglConfig.vendor;
               }
               if (parameter === ctx.VERSION) {
@@ -1387,13 +1441,10 @@ export class StealthBrowserManager {
               return originalGetParameter.call(this, parameter);
             };
             
-            const originalGetExtension = ctx.getExtension;
-            ctx.getExtension = function(name) {
-              if (webglConfig.extensions.includes(name)) {
-                return originalGetExtension.call(this, name) || {};
-              }
-              return null;
-            };
+            // getExtension is left alone: returning null for an extension the
+            // fingerprint's list did not name made WEBGL_debug_renderer_info
+            // vanish, and a page that reads it threw a TypeError where a real
+            // browser reports a GPU (sannysoft "WebGL Vendor: Error", R14).
           }
           
           return ctx;

package/src/core/crawlers/BFSCrawler.js

@@ -1,4 +1,5 @@
 import { load } from 'cheerio';
+import { readBody } from 'crawlforge-extractors';
 import { QueueManager } from '../queue/QueueManager.js';
 import { CacheManager } from '../cache/CacheManager.js';
 import { RateLimiter } from '../../utils/rateLimiter.js';
@@ -366,7 +367,9 @@ export class BFSCrawler {
         throw new Error(`Non-HTML content type: ${contentType}`);
       }
 
-      const html = await response.text();
+      // readBody decodes with the body's real charset — response.text() is
+      // UTF-8-only and mangled ISO-8859-1 pages ("portugu�s" on lua.org).
+      const html = await readBody(response);
       return this.parsePage(html, url);
     } catch (error) {
       clearTimeout(timeoutId);

package/src/core/llm/LLMManager.js

@@ -1,4 +1,5 @@
 import { OpenAIProvider } from './OpenAIProvider.js';
+import { extractionFormat } from '../../utils/extractionFormat.js';
 import { AnthropicProvider } from './AnthropicProvider.js';
 import { OllamaProvider } from './OllamaProvider.js';
 import { Logger } from '../../utils/Logger.js';
@@ -919,7 +920,7 @@ Return the list and nothing else.`;
     const schemaFields = Object.keys(schema.properties || {}).length;
     const scaledTokens = Math.min(2000, Math.max(maxTokens, schemaFields * 100 + 500));
 
-    const systemPrompt = `You are a structured data extraction expert. Extract data from the provided content and return ONLY valid JSON that conforms to the given JSON Schema. Do not include any explanation or markdown — only the raw JSON object.`;
+    const systemPrompt = `You are a structured data extraction expert. Extract data from the provided content and return ONLY valid JSON that conforms to the given JSON Schema. Use null for any field the content does not state — never guess, infer, or fill a value from memory. Do not include any explanation or markdown — only the raw JSON object.`;
 
     const schemaStr = JSON.stringify(schema, null, 2);
     const guidance = userPrompt ? `\n\nExtraction guidance: ${userPrompt}` : '';
@@ -937,9 +938,11 @@ Extract the data and return valid JSON:`;
         systemPrompt,
         maxTokens: scaledTokens,
         temperature: 0.1,
-        // Constrain the output to a parseable object. Small local models
-        // otherwise wrap the JSON in prose and the parse below throws.
-        format: 'json'
+        // Constrain the output to the caller's shape with every field
+        // nullable, so a model shown content that does not state a field can
+        // answer null instead of being decoded into an invented string. Small
+        // local models otherwise wrap the JSON in prose and the parse throws.
+        format: extractionFormat(schema)
       });
 
       // Strip markdown code fences if present
@@ -972,7 +975,9 @@ Extract the data and return valid JSON:`;
     const required = schema.required || [];
 
     for (const field of required) {
-      if (!(field in data)) {
+      // The decoder is told to answer null for a field the content never
+      // states, so a null here is "not filled in", the same as absent.
+      if (!(field in data) || data[field] === null) {
         errors.push(`Missing required field: ${field}`);
       }
     }
@@ -980,6 +985,10 @@ Extract the data and return valid JSON:`;
     for (const [key, fieldSchema] of Object.entries(properties)) {
       if (key in data) {
         const value = data[key];
+        // A null in a field the schema does not require is the honest answer
+        // for content that never states it, not a type error (typeof null is
+        // 'object', which used to read as "expected number, got object").
+        if (value === null) continue;
         const expectedType = fieldSchema.type;
         if (expectedType) {
           const actualType = Array.isArray(value) ? 'array' : typeof value;

package/src/skills/agent-skills/crawlforge-deep-research/SKILL.md

@@ -53,8 +53,8 @@ Returns titles, URLs, snippets. Supports `lang`, `site` (domain filter),
 
 ## reddit_search (cost: 5)
 
-reddit.com 403-blocks direct scraping, so this queries the Arctic Shift and
-PullPush community archives instead (free, no Reddit credentials).
+reddit.com 403-blocks direct scraping, so this reads the Arctic Shift community
+archive instead (free, no Reddit credentials).
 
 ```json
 {
@@ -65,8 +65,10 @@ PullPush community archives instead (free, no Reddit credentials).
 
 Modes: `posts` (default), `comments`, and `thread` (a post plus its nested
 comment tree via `link_id`). Scope with `subreddit`/`author` for near-real-time
-results (Arctic Shift); an unscoped `query` searches all of Reddit via PullPush
-(best-effort — it rate-limits aggressively). `after`/`before` accept ISO dates,
+results (Arctic Shift); an unscoped `query` searches all of Reddit via web
+discovery — a site-restricted web search finds posts, then they are read from the
+archive by ID (max 10 per call, ordered by relevance, and `after`/`before` do not
+apply on that path). `after`/`before` accept ISO dates,
 epoch seconds, or offsets like `"7d"`. Scores of posts under ~36h old read 0/1
 until the archive backfills. Space calls out rather than firing them in
 parallel — the archives shed load under concurrent bursts.

package/src/skills/agent-skills/crawlforge-structured-extraction/references/templates.md

@@ -12,7 +12,7 @@ to enumerate at runtime. Cost: 1 credit per call.
 | `youtube-video` | YouTube watch page | title, channel, views, likes, published date, description |
 | `reddit-thread` | Reddit post (read from the Arctic Shift archive, not reddit.com, which blocks plain fetchers) | id, title, subreddit, author, score, upvote_ratio, num_comments, posted, body, link_url, flair, removed. Pass `id` to `reddit_search` with `mode:"thread"` for the comment tree |
 | `hacker-news-front-page` | HN front page | ranked stories: title, URL, points, author, comment count |
-| `producthunt-launch` | Product Hunt launch | name, tagline, upvotes, maker, description |
+| `producthunt-launch` | Product Hunt product page (`/products/...`; old `/posts/...` links redirect there) | name, tagline, description, topics, website, followers, reviews_count, reviews_rating. The /products layout has no product-level vote count — followers and reviews are its engagement numbers |
 | `stackoverflow-question` | Stack Overflow question (read from the keyless Stack Exchange API, not the page) | title, body, votes, views, tags, author, asked, answered, accepted_answer_id, answer_count, top 5 answers (accepted first: votes, author, posted, body) |
 | `npm-package` | npm package page | name, version, description, weekly downloads, license, repo link |
 

package/src/tools/advanced/batchScrape/worker.js

@@ -5,6 +5,7 @@
  */
 
 import { load } from 'cheerio';
+import { readBody } from 'crawlforge-extractors';
 import { config as appConfig } from '../../../constants/config.js';
 import { ssrfGuard, isSsrfError } from '../../../utils/ssrfGuard.js';
 import { noteRetryAfter } from '../../../utils/hostRateLimiter.js';
@@ -36,7 +37,10 @@ export async function fetchUrl(url, options = {}) {
     if (response.status === 429 || response.status === 503) {
       noteRetryAfter(url, response.headers?.get?.('retry-after'));
     }
-    const html = await readBodyCapped(response, maxBodySize);
+    // crawlforge-extractors' readBody: same size cap, but decoded with the
+    // body's real charset — the local reader this replaces decoded everything
+    // as UTF-8, which turned lua.org's ISO-8859-1 "português" into "portugu�s".
+    const html = await readBody(response, { maxBytes: maxBodySize });
     return { response, html, warnings: gate.warnings };
   } catch (error) {
     if (isSsrfError(error)) throw new Error(error.cause?.message || error.message);
@@ -50,47 +54,6 @@ export async function fetchUrl(url, options = {}) {
   }
 }
 
-/**
- * Read a Response body as text, enforcing maxBodySize (Content-Length
- * pre-check, then a running byte count while streaming). Falls back to plain
- * .text() for responses without a streamable body (e.g. test mocks).
- */
-async function readBodyCapped(response, maxBodySize) {
-  const contentLengthHeader = response.headers?.get?.('content-length') ?? null;
-  if (contentLengthHeader !== null) {
-    const declared = parseInt(contentLengthHeader, 10);
-    if (!isNaN(declared) && declared > maxBodySize) {
-      throw new Error(`Response body too large: Content-Length ${declared} exceeds limit of ${maxBodySize} bytes`);
-    }
-  }
-
-  if (!response.body || typeof response.body.getReader !== 'function') {
-    return response.text();
-  }
-
-  const reader = response.body.getReader();
-  const chunks = [];
-  let totalBytes = 0;
-  while (true) {
-    const { done, value } = await reader.read();
-    if (done) break;
-    totalBytes += value.byteLength;
-    if (totalBytes > maxBodySize) {
-      reader.cancel();
-      throw new Error(`Response body too large: exceeded limit of ${maxBodySize} bytes`);
-    }
-    chunks.push(value);
-  }
-
-  const merged = new Uint8Array(totalBytes);
-  let offset = 0;
-  for (const chunk of chunks) {
-    merged.set(chunk, offset);
-    offset += chunk.byteLength;
-  }
-  return new TextDecoder().decode(merged);
-}
-
 /**
  * Scrape a single URL and return a result object.
  */

package/src/tools/extract/extractStructured.js

@@ -54,6 +54,37 @@ function mainContentText($, html, url) {
   return [title, flattenBodyText(load(mainHtml))].filter(Boolean).join('\n');
 }
 
+/** Characters of page text the model is shown; about 6k tokens. */
+const SHOWN_TEXT_BUDGET = 24000;
+
+/**
+ * The text the model reads: main content first, then the whole page when both
+ * fit the budget.
+ *
+ * Main content alone is what keeps the Cloudflare blog's headline from being
+ * "Skip to content", but Readability also drops the page chrome, and that is
+ * where racket-lang.org announces "Racket version 9.3 is available". A model
+ * that never sees the answer fills a required field with the nearest heading,
+ * which the provenance guard cannot fault because it carries no digits, and
+ * the full-text retry below never fires because nothing was nulled. Showing
+ * the page text after the main content puts the banner in view while the
+ * article still leads. A page too long for both gets main content alone, up
+ * to the budget — the old 6000-character cap cut a 20k-character article to
+ * its first third.
+ *
+ * @param {import('cheerio').CheerioAPI} $
+ * @param {string} html
+ * @param {string} url
+ * @param {string} textContent - whole-page text from fetchAndParse
+ * @returns {string}
+ */
+function shownText($, html, url, textContent) {
+  const main = mainContentText($, html, url);
+  if (!main) return textContent;
+  if (main.length + textContent.length > SHOWN_TEXT_BUDGET) return main;
+  return `${main}\n\n${textContent}`;
+}
+
 /**
  * Missing, null, blank string or empty array — what "the extraction did not
  * fill this field in" looks like across every extraction method.
@@ -147,6 +178,9 @@ export class ExtractStructuredTool {
         tool: 'extract_structured'
       });
 
+      // What the model reads — see shownText().
+      const shown = shownText($, html, url, textContent);
+
       // Step 3: Try LLM extraction first
       let extractionResult = null;
       let extractionMethod = 'llm';
@@ -160,9 +194,9 @@ export class ExtractStructuredTool {
         // local Ollama was never used.
         llmAvailable = await llm.ready();
         if (llmAvailable) {
-          const result = await llm.extractStructured(mainContentText($, html, url) || textContent, schema, {
+          const result = await llm.extractStructured(shown, schema, {
             prompt: prompt || '',
-            maxContentLength: 6000
+            maxContentLength: SHOWN_TEXT_BUDGET
           });
           // extractStructured swallows LLM failures and returns its keyword
           // fallback. Only accept the result as an LLM extraction when it
@@ -246,13 +280,12 @@ export class ExtractStructuredTool {
         // So when a REQUIRED field is nulled, retry once on the full page text,
         // which is exactly what the guard will accept. Only a strictly better
         // result is kept, so a retry can never make the answer worse.
-        const shownText = mainContentText($, html, url) || textContent;
-        if (guarded.missingRequired.length > 0 && shownText !== textContent && textContent) {
+        if (guarded.missingRequired.length > 0 && shown !== textContent && textContent) {
           try {
             const llm = this._ensureLLMManager(llmConfig || {});
             const retry = await llm.extractStructured(textContent, schema, {
               prompt: prompt || '',
-              maxContentLength: 6000
+              maxContentLength: SHOWN_TEXT_BUDGET
             });
             if (retry?.method === 'llm') {
               const retryGuarded = applyGuard(retry);

package/src/tools/extract/extractWithLlm.js

@@ -11,6 +11,7 @@ import { z } from 'zod';
 import { fetchAndParse } from './_fetchAndParse.js';
 import { ollamaBaseUrl, ollamaHeaders, selectOllamaModel } from '../../utils/ollamaConfig.js';
 import { verifyNumericProvenance } from '../../utils/provenance.js';
+import { extractionFormat } from '../../utils/extractionFormat.js';
 import { fenceUntrusted } from '../../utils/untrustedContent.js';
 // D1.3: SamplingClient for MCP sampling fallback (lazy — only imported if needed)
 let _SamplingClient = null;
@@ -164,16 +165,12 @@ function extractBalancedJson(str) {
  * as an object schema.
  */
 function buildInputSchema(schema) {
-  if (schema && (schema.type === 'object' || schema.properties)) {
-    return { additionalProperties: true, ...schema, type: 'object' };
-  }
-  // Flat hint map → object schema with string-typed properties for any
-  // non-object hint values (Anthropic requires a valid JSON Schema).
-  const properties = {};
-  for (const [key, val] of Object.entries(schema || {})) {
-    properties[key] = (val && typeof val === 'object') ? val : { type: 'string' };
-  }
-  return { type: 'object', properties, additionalProperties: true };
+  // Every field nullable, no `required`: the decoder must be allowed to say
+  // "not stated" — a required string it cannot leave null becomes an
+  // invention (R14: `scrape` json on racket-lang.org answered "Racket 5.1.0"
+  // for a page that says 9.3). The caller's schema still drives validation.
+  const format = extractionFormat(schema);
+  return format === 'json' ? { type: 'object', properties: {}, additionalProperties: true } : format;
 }
 
 /**
@@ -189,7 +186,7 @@ function jsonSchemaToZod(schema) {
   if (!isJsonSchema) {
     const shape = {};
     for (const [key, val] of Object.entries(schema)) {
-      shape[key] = jsonSchemaToZod(typeof val === 'string' ? { type: val } : val).optional();
+      shape[key] = jsonSchemaToZod(typeof val === 'string' ? { type: val } : val).nullable().optional();
     }
     return z.object(shape).passthrough();
   }
@@ -206,7 +203,11 @@ function jsonSchemaToZod(schema) {
       const required = Array.isArray(schema.required) ? schema.required : [];
       for (const [key, val] of Object.entries(schema.properties || {})) {
         const field = jsonSchemaToZod(val);
-        shape[key] = required.includes(key) ? field : field.optional();
+        // The model is told to answer null for a field the content never
+        // states, so null is the honest answer for a field the schema does
+        // not require — not a type violation. A required field stays strict:
+        // null there is exactly what the caller needs to hear about.
+        shape[key] = required.includes(key) ? field : field.nullable().optional();
       }
       return z.object(shape).passthrough();
     }
@@ -561,7 +562,8 @@ export class ExtractWithLlm {
     }
 
     const systemMessage =
-      'You extract structured data from web content per the user\'s instructions. Return JSON only.';
+      'You extract structured data from web content per the user\'s instructions. Return JSON only. ' +
+      'Use null for any field the content does not state — never guess, infer, or fill a value from memory.';
 
     const { userMessage, truncated: inputTruncated, original_length } = buildUserMessage(prompt, text, schema);
 

package/src/tools/llmstxt/generateLLMsTxt.js

@@ -1,5 +1,6 @@
 import { z } from 'zod';
 import { load } from 'cheerio';
+import { readBody } from 'crawlforge-extractors';
 import { LLMsTxtAnalyzer } from '../../core/LLMsTxtAnalyzer.js';
 import { Logger } from '../../utils/Logger.js';
 import { getBaseUrl } from '../../utils/urlNormalizer.js';
@@ -382,7 +383,8 @@ export class GenerateLLMsTxtTool {
         headers: { ...gate.headers }
       });
       if (!response.ok) return null;
-      return this.extractHomePageMetadata(await response.text());
+      // Charset-aware read; .text() decodes UTF-8 only.
+      return this.extractHomePageMetadata(await readBody(response));
     } catch {
       return null;
     } finally {

package/src/tools/scrape/unifiedScrape.js

@@ -281,19 +281,21 @@ export class UnifiedScrapeTool {
       if (fmt && typeof fmt === 'object' && fmt.type === 'json') {
         try {
           const extractWithLlm = await this._getExtractWithLlm();
-          let text;
-          if (onlyMainContent) {
-            text = htmlToMarkdown(getMainHtml());
-          } else {
-            // Script and template bodies are never rendered, but $('body').text()
-            // includes them — on a Shopify storefront that was 179KB of
-            // JavaScript, more than the page's real text, and it carried the
-            // very "Sold out" strings the strip had just removed from the DOM.
-            const { load } = await import('cheerio');
-            const $visible = load(html);
-            $visible('script, style, noscript, template').remove();
-            text = $visible('body').text().replace(/\s+/g, ' ').trim();
-          }
+          // Script and template bodies are never rendered, but $('body').text()
+          // includes them — on a Shopify storefront that was 179KB of
+          // JavaScript, more than the page's real text, and it carried the
+          // very "Sold out" strings the strip had just removed from the DOM.
+          const { load } = await import('cheerio');
+          const $visible = load(html);
+          $visible('script, style, noscript, template').remove();
+          const pageText = $visible('body').text().replace(/\s+/g, ' ').trim();
+          // Main content first, then the whole page: Readability drops the page
+          // chrome, and racket-lang.org states its version there — shown the
+          // main content alone, the model answered "Racket" for a version the
+          // page gives as 9.3 (R14). The article still leads.
+          const text = onlyMainContent
+            ? `${htmlToMarkdown(getMainHtml())}\n\n${pageText}`
+            : pageText;
           const result = await extractWithLlm.execute({
             content: text,
             prompt: fmt.prompt || 'Extract structured data from this page content.',

package/src/tools/search/adapters/redditOfficialApi.js

@@ -133,7 +133,7 @@ export class RedditOfficialApiAdapter {
     // The official API cannot honor an arbitrary date range (only coarse `t`
     // buckets). Reject so `auto` mode falls back to the archives, which can.
     if (v.after || v.before) {
-      throw new Error('official Reddit API cannot filter by date range — unset after/before, or use source:"arctic_shift"/"pullpush"');
+      throw new Error('official Reddit API cannot filter by date range — unset after/before, or use source:"arctic_shift" (scoped searches only — date filters need an archive)');
     }
     const sr = stripNamePrefix(subreddit);
     const au = stripNamePrefix(author);

package/src/tools/search/redditSearch.js

@@ -166,12 +166,12 @@ export class RedditSearchTool {
         throw new Error('source:"reddit_api" needs Reddit app credentials — set REDDIT_CLIENT_ID and REDDIT_CLIENT_SECRET (create a "script" app at https://www.reddit.com/prefs/apps)');
       }
       if (v.mode === 'comments') {
-        throw new Error('the official Reddit API has no comment full-text search — use comments mode with source:"arctic_shift"/"pullpush", or read a whole thread with mode:"thread"');
+        throw new Error('the official Reddit API has no comment full-text search — use comments mode with source:"arctic_shift" (scoped) or "web_discovery" (unscoped), or read a whole thread with mode:"thread"');
       }
       order = ['reddit_api'];
     } else if (v.source === 'arctic_shift') {
       if (!arcticPossible) {
-        throw new Error('Arctic Shift cannot keyword-search across all of Reddit — add a subreddit or author scope, or use source:"pullpush"');
+        throw new Error('Arctic Shift cannot keyword-search across all of Reddit — add a subreddit or author scope, or drop source so auto routes the query to web_discovery');
       }
       order = ['arctic_shift'];
     } else if (v.source === 'pullpush') {
@@ -364,12 +364,15 @@ export class RedditSearchTool {
     try {
       data = await this.#get(`${this.arcticBaseUrl}${path}`, query);
     } catch (error) {
-      // Arctic Shift sheds a scoped comment keyword search over its whole
-      // history with the same 422 "Timeout. Maybe slow down a bit" it uses for
-      // throttling (observed live 2026-08-30: r/webdev body=react failed
-      // unbounded and at 7d, answered at 3d). When the caller set no window,
-      // narrow it step by step rather than hand the caller the 422.
-      const narrowable = v.mode === 'comments' && Boolean(v.query) && !v.after;
+      // Arctic Shift sheds a scoped keyword search over its whole history
+      // with the same 422 "Timeout. Maybe slow down a bit" it uses for
+      // throttling (observed live 2026-08-30: r/webdev comments body=react
+      // failed unbounded and at 7d, answered at 3d; 2026-09-01: an
+      // r/selfhosted posts search failed unbounded and answered at 7d). When
+      // the caller set no window, narrow it step by step rather than hand the
+      // caller the 422. Thread mode returned above, so both remaining modes
+      // (posts, comments) narrow.
+      const narrowable = Boolean(v.query) && !v.after;
       if (!error.retryable || !narrowable) throw error;
       let lastError = error;
       for (const window of ['7d', '3d', '1d']) {
@@ -386,7 +389,7 @@ export class RedditSearchTool {
       if (!data) {
         throw new Error(
           `${lastError.message} — Arctic Shift timed out at every window (full history, 7d, 3d, 1d), ` +
-          'so its comment search is under load right now. Retry later, pass a narrower after/before, or scope to one post with link_id.'
+          'so its search is under load right now. Retry later, pass a narrower after/before, or scope to one post with link_id.'
         );
       }
       notes.push(`Arctic Shift timed out searching the full history; results are limited to the last ${windowApplied}. Pass after/before to choose the window.`);

package/src/tools/tracking/trackChanges/differ.js

@@ -3,6 +3,7 @@
  * URL content fetching and history/stat helper functions.
  */
 
+import { readBody } from 'crawlforge-extractors';
 import { safeFetch } from '../../../utils/ssrfGuard.js';
 import { preflightFetch } from '../../../utils/robotsGate.js';
 import { noteRetryAfter } from '../../../utils/hostRateLimiter.js';
@@ -81,7 +82,10 @@ export async function fetchContent(url, options = {}) {
       throw new Error(`HTTP ${response.status}: ${response.statusText}`);
     }
 
-    const content = await response.text();
+    // Charset-aware read; .text() decodes UTF-8 only, and mojibake here
+    // would both corrupt the reported diff and fake changes on a page whose
+    // bytes never moved.
+    const content = await readBody(response);
 
     return {
       content,
@@ -96,7 +100,11 @@ export async function fetchContent(url, options = {}) {
       }
     };
   } catch (error) {
-    throw new Error(`Failed to fetch content: ${error.message}`);
+    // undici reports every network failure as bare "fetch failed" and puts
+    // the real reason (ENOTFOUND, ECONNREFUSED, timeout) in error.cause —
+    // without it, a dead host and a typoed domain read identically.
+    const cause = error.cause?.message ? ` (${error.cause.message})` : '';
+    throw new Error(`Failed to fetch content: ${error.message}${cause}`);
   }
 }
 

package/src/utils/extractionFormat.js

@@ -0,0 +1,52 @@
+/**
+ * The constrained-decoding shape for an LLM extraction.
+ *
+ * A caller's JSON Schema says which fields are REQUIRED of the result. Handed
+ * to the model verbatim as the output format, that word means something else:
+ * the decoder cannot emit null for a required string, so a model shown content
+ * that never states the field is forced to invent one. R14 caught it twice on
+ * the same day — racket-lang.org's version came back as the page title,
+ * rust-lang.org's as the word "stable" — while the same model on the same
+ * pages answered correctly the moment the fact was in front of it.
+ *
+ * So the format the model decodes against makes every property nullable and
+ * drops `required`. The caller's schema still drives validation and the
+ * provenance retry: a required field the model answered null is what those
+ * exist to catch, and a null is honest where a guess is not.
+ *
+ * @param {object} schema - caller's JSON Schema (or a flat hint map)
+ * @returns {object|'json'} an object schema for the decoder, or 'json' when the
+ *   input carries no properties to constrain
+ */
+export function extractionFormat(schema) {
+  if (!schema || typeof schema !== 'object') return 'json';
+  const source = schema.properties && typeof schema.properties === 'object'
+    ? schema
+    : { type: 'object', properties: flatHintProperties(schema) };
+  if (!source.properties || Object.keys(source.properties).length === 0) return 'json';
+
+  const properties = {};
+  for (const [key, prop] of Object.entries(source.properties)) {
+    properties[key] = nullable(prop);
+  }
+  const { required: _required, ...rest } = source;
+  return { additionalProperties: true, ...rest, type: 'object', properties };
+}
+
+/** A flat `{ field: 'string' }` hint map as JSON-Schema properties. */
+function flatHintProperties(hints) {
+  const properties = {};
+  for (const [key, val] of Object.entries(hints)) {
+    properties[key] = (val && typeof val === 'object') ? val : { type: 'string' };
+  }
+  return properties;
+}
+
+/** The property with `null` admitted to its type, when it names a single type. */
+function nullable(prop) {
+  if (!prop || typeof prop !== 'object' || Array.isArray(prop)) return prop;
+  if (typeof prop.type !== 'string' || prop.type === 'null' || prop.type === 'object') return prop;
+  return { ...prop, type: [prop.type, 'null'] };
+}
+
+export default extractionFormat;

package/src/utils/hiddenContent.js

@@ -112,6 +112,61 @@ function specificity(selector) {
   return ids * 10000 + classes * 100 + elements;
 }
 
+/**
+ * Iterate a stylesheet's top-level rules as {selector, body}, skipping any
+ * block nested inside a rule's body.
+ *
+ * The flat regex this replaces — ([^{}]+)\{([^{}]*)\} — had no concept of
+ * native CSS nesting, so it read a nested rule as a top-level one under its
+ * inner selector. legalblogs.wolterskluwer.com ships
+ *   .cg-splitpane-aside-container.cg-is-closed .cg-splitpane-aside-header-cstm{
+ *     ... h6{display:none;} a{display:none;} }
+ * — links hidden only in a closed sidebar pane — which the stripper read as a
+ * bare a{display:none} and deleted 56 of an article's 68 anchors. Nested rules
+ * are skipped rather than resolved: expanding them into full descendant
+ * selectors needs a real CSS parser, and skipping errs on keeping content.
+ * A rule left unclosed at end of input (stylesheets are size-capped upstream,
+ * so truncation mid-rule is real) is dropped for the same reason.
+ */
+function* topLevelRules(css) {
+  const n = css.length;
+  let i = 0;
+  let selStart = 0;
+  while (i < n) {
+    const ch = css[i];
+    if (ch === '{') {
+      const selector = css.slice(selStart, i);
+      let depth = 1;
+      let j = i + 1;
+      let segStart = j;
+      let body = '';
+      while (j < n && depth > 0) {
+        const c = css[j];
+        if (c === '{') {
+          if (depth === 1) {
+            // The text since the last ';' is the nested rule's selector, not a
+            // declaration of this rule — keep only the completed declarations.
+            const seg = css.slice(segStart, j);
+            body += seg.slice(0, seg.lastIndexOf(';') + 1);
+          }
+          depth++;
+        } else if (c === '}') {
+          depth--;
+          if (depth === 1) segStart = j + 1;
+          else if (depth === 0) body += css.slice(segStart, j);
+        }
+        j++;
+      }
+      if (depth === 0) yield { selector, body };
+      i = j;
+      selStart = j;
+    } else {
+      if (ch === '}') selStart = i + 1; // stray close — resync
+      i++;
+    }
+  }
+}
+
 /**
  * Parse CSS into rules that hide content and rules that re-show it, each
  * carrying the specificity and source order a browser would use to resolve
@@ -125,13 +180,9 @@ export function collectVisibilitySelectors(css) {
   if (!css) return { hide, show };
 
   const flat = stripConditionalBlocks(css);
-  const rule = /([^{}]+)\{([^{}]*)\}/g;
-  let m;
   let order = 0;
 
-  while ((m = rule.exec(flat)) !== null) {
-    const selectorList = m[1];
-    const body = m[2];
+  for (const { selector: selectorList, body } of topLevelRules(flat)) {
     order++;
 
     const display = /(?:^|[;{\s])display\s*:\s*([a-z-]+)/i.exec(body);

package/src/utils/languageDetection.js

@@ -214,12 +214,17 @@ export function detectLanguage(text, options = {}) {
 
     const confidence = Math.min(1, 0.5 + (text.length / 500) * 0.5);
 
+    // francAll scores are similarity relative to the winner (whose own score
+    // is 1), while `confidence` above is length-based — two different scales.
+    // Printed raw, a runner-up could read 0.82 against a winner shown at 0.8.
+    // Scaling by the winner's confidence keeps the ranking and the invariant
+    // that no alternative outranks the pick.
     const alternative = francAll(text, { minLength, whitelist: Object.keys(LANGUAGE_NAMES) })
       .slice(1, 4)
       .map(([code, score]) => ({
         code,
         name: LANGUAGE_NAMES[code] || code,
-        confidence: Math.round(score * 100) / 100
+        confidence: Math.round(score * confidence * 100) / 100
       }));
 
     return {

package/src/utils/provenance.js

@@ -228,6 +228,28 @@ function literalReadings(value) {
   return [...spellings];
 }
 
+/** A version with two or more dot-separated numeric segments, inside prose. */
+const EMBEDDED_VERSION = /(?<![\w.])v?\d+(?:\.\d+){2,}(?![\w.])/gi;
+
+/**
+ * The spellings of every version-shaped token inside a prose string, or null
+ * when the value is not prose or carries none. "Racket 5.1.0" on a page that
+ * says 9.3 is the case that found this: the whole value is prose, so
+ * `literalReadings` left it alone, and the fabricated version rode through.
+ *
+ * @param {*} value
+ * @returns {string[][]|null} one spelling list per token
+ */
+function embeddedVersionReadings(value) {
+  if (typeof value !== 'string' || !/\s/.test(value.trim())) return null;
+  const tokens = value.match(EMBEDDED_VERSION);
+  if (!tokens) return null;
+  return tokens.map((token) => {
+    const lower = token.toLowerCase();
+    return [...new Set([lower, lower.replace(VERSION_PREFIX, '')])];
+  });
+}
+
 /**
  * Replace every numeric value that is not present in the source with null.
  *
@@ -286,8 +308,22 @@ export function verifyNumericProvenance(data, source) {
     // Not a number, but possibly a version/date/SKU-shaped literal, which is
     // the other thing a model invents when the page does not say.
     const literals = literalReadings(node);
-    if (literals === null) return node;
-    if (inSource(literals)) {
+    if (literals !== null) {
+      if (inSource(literals)) {
+        verified++;
+        return node;
+      }
+      unverified.push({ path: path || '(root)', value: node, reason: 'not_found_in_source' });
+      return null;
+    }
+
+    // Prose that carries a version-shaped token ("Racket 5.1.0"): the words
+    // may be the model's own re-wording, but the token has exactly one
+    // spelling and is checked like a bare one. Decimals ("1.5 million") and
+    // bare integers stay prose — a model may legitimately re-word those.
+    const embedded = embeddedVersionReadings(node);
+    if (embedded === null) return node;
+    if (embedded.every((spellings) => inSource(spellings))) {
       verified++;
       return node;
     }

package/src/utils/robotsGate.js

@@ -135,6 +135,29 @@ export async function robotsPreflight(url, options = {}) {
  *   warnings: string[], overridden: boolean }>}
  * @throws {BlockedHostError|RobotsDisallowedError}
  */
+/**
+ * The headers an HTTP page fetch carries: identity, the Web Bot Auth signature
+ * when one is configured, and an EMPTY Accept-Language.
+ *
+ * Node's fetch adds `Accept-Language: *` whenever the caller sets none. Amazon
+ * answers this identity with a captcha interstitial when ANY Accept-Language
+ * rides along — `*`, en-US, de-DE, a full browser list — and serves the page
+ * when the header is absent (R14, bisected header by header with curl, then
+ * reproduced with Node's fetch). fetch offers no way to leave the header out,
+ * but an empty value is sent as-is instead of `*`, and Amazon treats empty as
+ * absent. The identity stays honest; no language preference is claimed.
+ * @param {string} [userAgent]
+ * @param {Record<string,string>} [signature]
+ * @returns {Record<string,string>}
+ */
+export function outboundHeaders(userAgent, signature = {}) {
+  return {
+    ...identityHeaders({ userAgent }),
+    'Accept-Language': '',
+    ...signature
+  };
+}
+
 export async function preflightFetch(url, options = {}) {
   const decision = await robotsPreflight(url, options);
   if (!decision.allowed) {
@@ -153,7 +176,7 @@ export async function preflightFetch(url, options = {}) {
   const signature = signRequestHeaders(url) || {};
 
   return {
-    headers: { ...identityHeaders({ userAgent: decision.userAgent }), ...signature },
+    headers: outboundHeaders(decision.userAgent, signature),
     userAgent: decision.userAgent,
     warnings: decision.warnings,
     overridden: decision.overridden