@aborruso/ckan-mcp-server 0.4.130 → 0.4.132

package/LOG.md

@@ -1,7 +1,37 @@
 # LOG
 
+## 2026-10-27
+
+### v0.4.132
+
+Ships #565: worker telemetry logs returned search results. Tests 601 passed. Smoke 16/17: the MQA case times out because data.europa.eu `/api/mqa/cache/datasets/{id}` (and its successor `/resources/{id}`) hang for every dataset id tried, while `/global`, `/countries`, `/catalogues/dati-gov-it` answer in 0.2 s; released anyway on the user's decision. The official spec (`openapi.yaml` 6.0.1-SNAPSHOT) marks `/datasets/{id}` deprecated, alias of `/resources/{id}`: switching is a separate change.
+
+### Worker telemetry: returned search results
+
+`ckan_package_search` and `ckan_find_relevant_datasets` now record the datasets they return (ids and names in rank order, max 20), the total count (zero included) and the start offset. The worker adds them to its log line (`result_count`, `result_start`, `result_ids`, `result_names`), and the archiver carries them, plus `fq` and `sort`, into `data/worker_events_flat.jsonl`. Goal: query → result pairs for search fine-tuning. No session id exists (stateless worker): a later `package_show` on an id in `result_ids` links to the search. Side channel keyed by the per-request `McpServer` (`src/utils/search-telemetry.ts`), so concurrent requests cannot mix. Checked with `wrangler dev` on dati.gov.it, zero-result case included.
+
+## 2026-10-26
+
+### Update Worker Telemetry: DuckDB pinned to v2.0.0
+
+The workflow had failed twice a day since 2026-10-21: DuckDB v2.0.0 (released that day) renamed the CLI asset from `duckdb_cli-linux-amd64.zip` to `duckdb-cli-linux-amd64.tar.gz`, so `releases/latest/download/...zip` returned 404. Now downloads the v2.0.0 tarball explicitly (pinned, so a future rename does not break it again). Checked locally: `scripts/worker_daily_stats.sh` with v2.0.0 and v1.5.6 in UTC gives output identical to the committed `data/worker_daily_calls.jsonl`. Note: v2.0.0 reads `...Z` timestamps as `TIMESTAMPTZ`, so `::TIMESTAMP` follows the machine time zone; on the UTC runner nothing changes, on a non-UTC machine the daily counts shift.
+
+## 2026-10-22
+
+### v0.4.131
+
+Ships #563: Anubis challenge retry with an honest User-Agent (#7). Tests 595 passed, smoke 17/17. Live on npm (with provenance; the registry took ~5 min to serve it after the CI publish), the MCP Registry (`isLatest`), the Worker (`/health` 0.4.131) and the GitHub release (DXT + skill attached).
+
+### Anubis-fronted portals: retry once with an honest User-Agent (#7)
+
+Reported by @umlwbin: our Chrome-impersonating headers (needed by dati.anticorruzione.it's WAF) trip Anubis anti-bot gates, which answer with an HTML proof-of-work page that axios cannot solve. Anubis challenges any UA containing `Mozilla`, including `Mozilla/5.0 (compatible; ...)`, so no fixed UA serves both portals. `makeCkanRequest` now detects the challenge page (`anubis_challenge` marker, `isAnubisChallenge`) and retries once with `CKAN-MCP-Server/1.0` and no `Sec-CH-UA*` headers, on both the axios and fetch/Workers paths. The retry acquires its own rate-limit token (Greptile review). +6 tests (fail without the fix). Tests 595 passed; e2e `package_search` OK on canwin-datahub.ad.umanitoba.ca/data (48,947), dati.anticorruzione.it (70), dati.gov.it.
+
 ## 2026-10-19
 
+### Advisory GHSA-w3gp-3hrh-v465 published
+
+Reported by @olevasseur, accepted and published as Medium (CVSS 6.5): affected `>= 0.4.114, <= 0.4.129`, patched `0.4.130`. CVE requested, not yet assigned. v0.4.130 is live on npm (with provenance), the MCP Registry and the Worker. `SECURITY.md` table updated.
+
 ### v0.4.130
 
 Ships #561: linear shrink in `truncateJson` (GHSA-w3gp-3hrh-v465). Tests 589 passed, smoke 17/17.

package/dist/index.js

@@ -932,6 +932,11 @@ function auditLog(serverUrl, action, params, cacheHit) {
   } catch {
   }
 }
+var BROWSER_USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36";
+var HONEST_USER_AGENT = "CKAN-MCP-Server/1.0";
+function isAnubisChallenge(body) {
+  return typeof body === "string" && body.includes("anubis_challenge");
+}
 async function makeCkanRequest(serverUrl, action, params = {}, opts = {}) {
   const isNode = typeof process !== "undefined" && !!process.versions?.node;
   validateServerUrl(serverUrl);
@@ -963,95 +968,106 @@ async function makeCkanRequest(serverUrl, action, params = {}, opts = {}) {
   _lastCacheHit = false;
   const rateLimitConfig = getRateLimitConfig();
   const rateLimitEnabled = rateLimitConfig.enabled && opts.rateLimit !== false;
-  if (rateLimitEnabled) {
-    const hostname = new URL(resolvedServerUrl).hostname;
-    await getRateLimiter().acquire(hostname);
-  }
+  const acquireRateToken = async () => {
+    if (rateLimitEnabled) {
+      const hostname = new URL(resolvedServerUrl).hostname;
+      await getRateLimiter().acquire(hostname);
+    }
+  };
+  await acquireRateToken();
   try {
-    let decodedData;
-    if (isNode) {
-      const safeAgents = await getSafeAgents();
-      const response = await axios.get(url, {
-        params,
-        timeout: 3e4,
-        responseType: "arraybuffer",
-        maxRedirects: 5,
-        // Re-validate every redirect hop: Node skips the agent's `lookup` for IP
-        // literals, so a public host 302-ing to e.g. 169.254.169.254 would otherwise
-        // be followed. Also enforces CKAN_ALLOWED_DOMAINS on redirect targets.
-        beforeRedirect: (redirectOptions) => {
-          validateServerUrl(redirectOptions.href);
-        },
-        // Never route through HTTP_PROXY/HTTPS_PROXY: a proxy would connect to the
-        // target itself, bypassing the SSRF-safe lookup pinned in the agents below.
-        proxy: false,
-        maxContentLength: MAX_RESPONSE_BYTES,
-        maxBodyLength: MAX_RESPONSE_BYTES,
-        ...safeAgents ? { httpAgent: safeAgents.httpAgent, httpsAgent: safeAgents.httpsAgent } : {},
-        headers: {
-          Accept: "application/json, text/plain, */*",
-          "Accept-Language": "en-US,en;q=0.9,it;q=0.8",
-          "Accept-Encoding": "gzip, deflate, br",
-          Connection: "keep-alive",
-          "Sec-CH-UA": '"Chromium";v="120", "Not?A_Brand";v="24", "Google Chrome";v="120"',
-          "Sec-CH-UA-Mobile": "?0",
-          "Sec-CH-UA-Platform": '"Linux"',
-          "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
-        }
-      });
-      decodedData = await decodePossiblyCompressed(
-        response.data,
-        response.headers
-      );
-    } else {
-      const searchParams = new URLSearchParams();
-      for (const [key, value] of Object.entries(params)) {
-        if (value === void 0 || value === null) {
-          continue;
-        }
-        searchParams.set(key, String(value));
-      }
-      const fetchUrl = searchParams.toString() ? `${url}?${searchParams.toString()}` : url;
-      const controller = new AbortController();
-      const timeoutId = setTimeout(() => controller.abort(), 3e4);
-      let response;
-      try {
-        response = await safeFetch(fetchUrl, {
-          method: "GET",
-          signal: controller.signal,
+    const fetchDecoded = async (honest) => {
+      if (isNode) {
+        const safeAgents = await getSafeAgents();
+        const response = await axios.get(url, {
+          params,
+          timeout: 3e4,
+          responseType: "arraybuffer",
+          maxRedirects: 5,
+          // Re-validate every redirect hop: Node skips the agent's `lookup` for IP
+          // literals, so a public host 302-ing to e.g. 169.254.169.254 would otherwise
+          // be followed. Also enforces CKAN_ALLOWED_DOMAINS on redirect targets.
+          beforeRedirect: (redirectOptions) => {
+            validateServerUrl(redirectOptions.href);
+          },
+          // Never route through HTTP_PROXY/HTTPS_PROXY: a proxy would connect to the
+          // target itself, bypassing the SSRF-safe lookup pinned in the agents below.
+          proxy: false,
+          maxContentLength: MAX_RESPONSE_BYTES,
+          maxBodyLength: MAX_RESPONSE_BYTES,
+          ...safeAgents ? { httpAgent: safeAgents.httpAgent, httpsAgent: safeAgents.httpsAgent } : {},
           headers: {
             Accept: "application/json, text/plain, */*",
-            "Accept-Encoding": "identity",
-            "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
+            "Accept-Language": "en-US,en;q=0.9,it;q=0.8",
+            "Accept-Encoding": "gzip, deflate, br",
+            Connection: "keep-alive",
+            ...honest ? {} : {
+              "Sec-CH-UA": '"Chromium";v="120", "Not?A_Brand";v="24", "Google Chrome";v="120"',
+              "Sec-CH-UA-Mobile": "?0",
+              "Sec-CH-UA-Platform": '"Linux"'
+            },
+            "User-Agent": honest ? HONEST_USER_AGENT : BROWSER_USER_AGENT
           }
-        }, { maxHops: 5 });
-      } catch (error) {
-        if (controller.signal.aborted) {
-          throw new CkanTransportError(`Request timeout connecting to ${serverUrl}`, serverUrl);
+        });
+        return decodePossiblyCompressed(
+          response.data,
+          response.headers
+        );
+      } else {
+        const searchParams = new URLSearchParams();
+        for (const [key, value] of Object.entries(params)) {
+          if (value === void 0 || value === null) {
+            continue;
+          }
+          searchParams.set(key, String(value));
+        }
+        const fetchUrl = searchParams.toString() ? `${url}?${searchParams.toString()}` : url;
+        const controller = new AbortController();
+        const timeoutId = setTimeout(() => controller.abort(), 3e4);
+        let response;
+        try {
+          response = await safeFetch(fetchUrl, {
+            method: "GET",
+            signal: controller.signal,
+            headers: {
+              Accept: "application/json, text/plain, */*",
+              "Accept-Encoding": "identity",
+              "User-Agent": honest ? HONEST_USER_AGENT : BROWSER_USER_AGENT
+            }
+          }, { maxHops: 5 });
+        } catch (error) {
+          if (controller.signal.aborted) {
+            throw new CkanTransportError(`Request timeout connecting to ${serverUrl}`, serverUrl);
+          }
+          if (error instanceof TypeError) {
+            throw new CkanTransportError(`Network error: ${error.message}`, serverUrl);
+          }
+          throw error;
+        } finally {
+          clearTimeout(timeoutId);
         }
-        if (error instanceof TypeError) {
-          throw new CkanTransportError(`Network error: ${error.message}`, serverUrl);
+        if (!response.ok) {
+          throw new CkanApiError(`CKAN API error (${response.status}): ${response.statusText}`, response.status, action, serverUrl);
         }
-        throw error;
-      } finally {
-        clearTimeout(timeoutId);
-      }
-      if (!response.ok) {
-        throw new CkanApiError(`CKAN API error (${response.status}): ${response.statusText}`, response.status, action, serverUrl);
-      }
-      const declaredLength = Number(response.headers.get("content-length"));
-      if (Number.isFinite(declaredLength) && declaredLength > MAX_RESPONSE_BYTES) {
-        throw new CkanApiError(`Response too large (${declaredLength} bytes)`, void 0, action, serverUrl);
-      }
-      const buffer = await response.arrayBuffer();
-      if (buffer.byteLength > MAX_RESPONSE_BYTES) {
-        throw new CkanApiError(`Response too large (${buffer.byteLength} bytes)`, void 0, action, serverUrl);
+        const declaredLength = Number(response.headers.get("content-length"));
+        if (Number.isFinite(declaredLength) && declaredLength > MAX_RESPONSE_BYTES) {
+          throw new CkanApiError(`Response too large (${declaredLength} bytes)`, void 0, action, serverUrl);
+        }
+        const buffer = await response.arrayBuffer();
+        if (buffer.byteLength > MAX_RESPONSE_BYTES) {
+          throw new CkanApiError(`Response too large (${buffer.byteLength} bytes)`, void 0, action, serverUrl);
+        }
+        const headers = {};
+        response.headers.forEach((headerValue, headerKey) => {
+          headers[headerKey] = headerValue;
+        });
+        return decodePossiblyCompressed(buffer, headers);
       }
-      const headers = {};
-      response.headers.forEach((headerValue, headerKey) => {
-        headers[headerKey] = headerValue;
-      });
-      decodedData = await decodePossiblyCompressed(buffer, headers);
+    };
+    let decodedData = await fetchDecoded(false);
+    if (isAnubisChallenge(decodedData)) {
+      await acquireRateToken();
+      decodedData = await fetchDecoded(true);
     }
     if (decodedData && decodedData.success === true) {
       const result = decodedData.result;
@@ -1106,6 +1122,23 @@ async function makeCkanRequest(serverUrl, action, params = {}, opts = {}) {
   }
 }
 
+// src/utils/search-telemetry.ts
+var SEARCH_TELEMETRY_MAX_RESULTS = 20;
+var MAX_FIELD_LENGTH = 100;
+var lastResults = /* @__PURE__ */ new WeakMap();
+function recordSearchResults(server, count, start, datasets, replyText) {
+  const shown = datasets.filter(
+    (d) => d.id && replyText.includes(d.id) || d.name && replyText.includes(d.name)
+  );
+  const kept = shown.slice(0, SEARCH_TELEMETRY_MAX_RESULTS);
+  lastResults.set(server, {
+    result_count: count,
+    result_start: start,
+    result_ids: kept.map((d) => String(d.id ?? "").slice(0, MAX_FIELD_LENGTH)),
+    result_names: kept.map((d) => String(d.name || d.id || "").slice(0, MAX_FIELD_LENGTH))
+  });
+}
+
 // src/utils/formatting.ts
 function truncateText(text, limit = CHARACTER_LIMIT) {
   if (text.length <= limit) {
@@ -2232,8 +2265,10 @@ Typical workflow: ckan_status_show (check locale) \u2192 ckan_package_search (qu
             params.server_url,
             effectiveQuery !== params.q ? effectiveQuery : void 0
           );
+          const text2 = truncateJson(compact);
+          recordSearchResults(server, result.count ?? 0, effectiveStart, result.results ?? [], text2);
           return {
-            content: [{ type: "text", text: truncateJson(compact) }]
+            content: [{ type: "text", text: text2 }]
           };
         }
         let hvdNote = "";
@@ -2363,8 +2398,10 @@ Note: showing top ${sorted.length} only. Use \`response_format: json\` for full
 `;
           }
         }
+        const text = truncateText(addDemoFooter(markdown));
+        recordSearchResults(server, result.count ?? 0, effectiveStart, result.results ?? [], text);
         return {
-          content: [{ type: "text", text: truncateText(addDemoFooter(markdown)) }]
+          content: [{ type: "text", text }]
         };
       } catch (error) {
         return {
@@ -2492,7 +2529,9 @@ Typical workflow: ckan_find_relevant_datasets \u2192 ckan_package_show (inspect
           results: top
         };
         if (params.response_format === "json" /* JSON */) {
-          return jsonToolResult(payload);
+          const jsonResult = jsonToolResult(payload);
+          recordSearchResults(server, payload.total_results, 0, top, jsonResult.content[0].text);
+          return jsonResult;
         }
         let markdown = `# Relevant CKAN Datasets
 
@@ -2568,8 +2607,10 @@ Typical workflow: ckan_find_relevant_datasets \u2192 ckan_package_show (inspect
 `;
           });
         }
+        const text = truncateText(addDemoFooter(markdown));
+        recordSearchResults(server, payload.total_results, 0, top, text);
         return {
-          content: [{ type: "text", text: truncateText(addDemoFooter(markdown)) }]
+          content: [{ type: "text", text }]
         };
       } catch (error) {
         return {
@@ -6101,7 +6142,7 @@ var registerAllPrompts = (server) => {
 function createServer() {
   return new McpServer({
     name: "ckan-mcp-server",
-    version: "0.4.130"
+    version: "0.4.132"
   });
 }
 function registerAll(server) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aborruso/ckan-mcp-server",
-  "version": "0.4.130",
+  "version": "0.4.132",
   "mcpName": "io.github.aborruso/ckan-mcp-server",
   "description": "MCP server for interacting with CKAN open data portals",
   "repository": {