@aborruso/ckan-mcp-server 0.4.131 → 0.4.133

package/LOG.md

@@ -1,10 +1,45 @@
 # LOG
 
+## 2026-10-28
+
+### v0.4.133
+
+Ships #568. For callers: `ckan_find_portals` works again on the hosted Worker (it failed on every call since at least 2026-09-05); `ckan_status_show` on a site that is not a CKAN API now says so and points at `ckan_find_portals`, instead of telling the caller to call `ckan_status_show`; `open.toronto.ca` works as a server URL in every tool (routed to its CKAN host). Tests 616 passed. Smoke 16/17: same MQA outage as v0.4.132 (per-dataset `/api/mqa/cache/*/{id}` still not answering, `/global` 0.2 s), tagged with `SKIP_RELEASE_GATE=1` on the user's decision.
+
+### Telemetry errors: find_portals fixed on Workers, error messages logged, non-CKAN hints
+
+From the error analysis of `worker_events_flat.jsonl` (app errors are logged only since 2026-09-05: 754 of 3,752 calls, 537 of them invalid input from scanners).
+
+- `ckan_find_portals` failed on every Worker call (96/96 since 2026-09-05): it was the only tool calling axios directly, and axios's fetch adapter breaks on Workers ("The 'cache' field on 'RequestInitializerDict' is not implemented"). Now uses `safeFetch`, like MQA; SSRF checks on redirects come with it. Checked with `wrangler dev`: 945 portals, about 5 s (the datashades.info list is 22 MB).
+- The worker log line of a failed call now has `error_message` (first 200 chars, one line) and `error_kind` (`validation`, `not_allowed`, `timeout`, `upstream_http`, `unreachable`, `other`, `src/utils/error-telemetry.ts`); the archiver carries both into the flat file. The find_portals failure stayed hidden for weeks because the log only said `error`.
+- `ckan_status_show` on a site that is not the CKAN API (47% failure on real URLs: open.toronto.ca, open.ottawa.ca, data.europa.eu/data, ...) told the caller to call `ckan_status_show`. It now says the site may not run CKAN or keep the API on another host, and points at `ckan_find_portals`. `open.toronto.ca` is routed to its CKAN host `ckan0.cf.opendata.inter.prod-toronto.ca` via `portals.json`.
+- Tests 615 passed (+14, each new test fails without its fix), Python tests 5.
+
+## 2026-10-27
+
+### Archiver: empty result lists for zero-result searches
+
+Manual archiver run on a scratchpad copy (7 live events): search fields reach `worker_events_flat.jsonl`, but a zero-result search came in with `result_ids`/`result_names` null, since the Observability API drops empty arrays from `source`. `flatten()` now restores `[]` whenever `result_count` is present, so "0 results" stays distinct from "not a search". Rechecked on a fresh copy: `zzqxkwvnonesiste` → `result_count` 0, lists `[]`. Tests in `tests/python/test_worker_telemetry_archiver.py` (stdlib `unittest`: zero count with omitted lists, non-empty lists, non-search event, field set); the zero-count case fails on the previous `flatten()`. They run in `update-telemetry.yml` before each archiver run (Greptile review).
+
+### v0.4.132
+
+Ships #565: worker telemetry logs returned search results. Tests 601 passed. Smoke 16/17: the MQA case times out because data.europa.eu `/api/mqa/cache/datasets/{id}` (and its successor `/resources/{id}`) hang for every dataset id tried, while `/global`, `/countries`, `/catalogues/dati-gov-it` answer in 0.2 s; released anyway on the user's decision. Live on npm (provenance), the MCP Registry (`isLatest`), the Worker (`/health` 0.4.132) and the GitHub release (DXT + skill attached). Observability API checked read-only: `result_ids` arrives as a list and `flatten()` maps it. The official spec (`openapi.yaml` 6.0.1-SNAPSHOT) marks `/datasets/{id}` deprecated, alias of `/resources/{id}`: switching is a separate change.
+
+### Worker telemetry: returned search results
+
+`ckan_package_search` and `ckan_find_relevant_datasets` now record the datasets they return (ids and names in rank order, max 20), the total count (zero included) and the start offset. The worker adds them to its log line (`result_count`, `result_start`, `result_ids`, `result_names`), and the archiver carries them, plus `fq` and `sort`, into `data/worker_events_flat.jsonl`. Goal: query → result pairs for search fine-tuning. No session id exists (stateless worker): a later `package_show` on an id in `result_ids` links to the search. Side channel keyed by the per-request `McpServer` (`src/utils/search-telemetry.ts`), so concurrent requests cannot mix. Checked with `wrangler dev` on dati.gov.it, zero-result case included.
+
+## 2026-10-26
+
+### Update Worker Telemetry: DuckDB pinned to v2.0.0
+
+The workflow had failed twice a day since 2026-10-21: DuckDB v2.0.0 (released that day) renamed the CLI asset from `duckdb_cli-linux-amd64.zip` to `duckdb-cli-linux-amd64.tar.gz`, so `releases/latest/download/...zip` returned 404. Now downloads the v2.0.0 tarball explicitly (pinned, so a future rename does not break it again). Checked locally: `scripts/worker_daily_stats.sh` with v2.0.0 and v1.5.6 in UTC gives output identical to the committed `data/worker_daily_calls.jsonl`. Note: v2.0.0 reads `...Z` timestamps as `TIMESTAMPTZ`, so `::TIMESTAMP` follows the machine time zone; on the UTC runner nothing changes, on a non-UTC machine the daily counts shift.
+
 ## 2026-10-22
 
 ### v0.4.131
 
-Ships #563: Anubis challenge retry with an honest User-Agent (#7). Tests 595 passed, smoke 17/17.
+Ships #563: Anubis challenge retry with an honest User-Agent (#7). Tests 595 passed, smoke 17/17. Live on npm (with provenance; the registry took ~5 min to serve it after the CI publish), the MCP Registry (`isLatest`), the Worker (`/health` 0.4.131) and the GitHub release (DXT + skill attached).
 
 ### Anubis-fronted portals: retry once with an honest User-Agent (#7)
 

package/dist/index.js

@@ -96,6 +96,16 @@ var portals_default = {
         "http://open.canada.ca"
       ]
     },
+    {
+      id: "open-toronto",
+      name: "open.toronto.ca",
+      api_url: "https://ckan0.cf.opendata.inter.prod-toronto.ca",
+      api_url_aliases: [
+        "https://open.toronto.ca",
+        "http://open.toronto.ca"
+      ],
+      dataset_view_url: "https://open.toronto.ca/dataset/{name}/"
+    },
     {
       id: "data-gov-au",
       name: "data.gov.au",
@@ -1122,6 +1132,23 @@ async function makeCkanRequest(serverUrl, action, params = {}, opts = {}) {
   }
 }
 
+// src/utils/search-telemetry.ts
+var SEARCH_TELEMETRY_MAX_RESULTS = 20;
+var MAX_FIELD_LENGTH = 100;
+var lastResults = /* @__PURE__ */ new WeakMap();
+function recordSearchResults(server, count, start, datasets, replyText) {
+  const shown = datasets.filter(
+    (d) => d.id && replyText.includes(d.id) || d.name && replyText.includes(d.name)
+  );
+  const kept = shown.slice(0, SEARCH_TELEMETRY_MAX_RESULTS);
+  lastResults.set(server, {
+    result_count: count,
+    result_start: start,
+    result_ids: kept.map((d) => String(d.id ?? "").slice(0, MAX_FIELD_LENGTH)),
+    result_names: kept.map((d) => String(d.name || d.id || "").slice(0, MAX_FIELD_LENGTH))
+  });
+}
+
 // src/utils/formatting.ts
 function truncateText(text, limit = CHARACTER_LIMIT) {
   if (text.length <= limit) {
@@ -2248,8 +2275,10 @@ Typical workflow: ckan_status_show (check locale) \u2192 ckan_package_search (qu
             params.server_url,
             effectiveQuery !== params.q ? effectiveQuery : void 0
           );
+          const text2 = truncateJson(compact);
+          recordSearchResults(server, result.count ?? 0, effectiveStart, result.results ?? [], text2);
           return {
-            content: [{ type: "text", text: truncateJson(compact) }]
+            content: [{ type: "text", text: text2 }]
           };
         }
         let hvdNote = "";
@@ -2379,8 +2408,10 @@ Note: showing top ${sorted.length} only. Use \`response_format: json\` for full
 `;
           }
         }
+        const text = truncateText(addDemoFooter(markdown));
+        recordSearchResults(server, result.count ?? 0, effectiveStart, result.results ?? [], text);
         return {
-          content: [{ type: "text", text: truncateText(addDemoFooter(markdown)) }]
+          content: [{ type: "text", text }]
         };
       } catch (error) {
         return {
@@ -2508,7 +2539,9 @@ Typical workflow: ckan_find_relevant_datasets \u2192 ckan_package_show (inspect
           results: top
         };
         if (params.response_format === "json" /* JSON */) {
-          return jsonToolResult(payload);
+          const jsonResult = jsonToolResult(payload);
+          recordSearchResults(server, payload.total_results, 0, top, jsonResult.content[0].text);
+          return jsonResult;
         }
         let markdown = `# Relevant CKAN Datasets
 
@@ -2584,8 +2617,10 @@ Typical workflow: ckan_find_relevant_datasets \u2192 ckan_package_show (inspect
 `;
           });
         }
+        const text = truncateText(addDemoFooter(markdown));
+        recordSearchResults(server, payload.total_results, 0, top, text);
         return {
-          content: [{ type: "text", text: truncateText(addDemoFooter(markdown)) }]
+          content: [{ type: "text", text }]
         };
       } catch (error) {
         return {
@@ -3623,14 +3658,15 @@ Typical workflow: ckan_status_show (verify server is up) \u2192 ckan_package_sea
           structuredContent: cappedStructured(result)
         };
       } catch (error) {
-        const migrated = error instanceof CkanApiError && error.serverUrl && getPortalMigration(error.serverUrl);
-        const detail = formatCkanError(error, "ckan_status_show");
+        const migrated = (error instanceof CkanApiError || error instanceof CkanTransportError) && error.serverUrl && getPortalMigration(error.serverUrl);
+        if (migrated) {
+          return { content: [{ type: "text", text: formatCkanError(error, "ckan_status_show") }], isError: true };
+        }
+        const detail = error instanceof CkanApiError || error instanceof CkanTransportError ? `${error.message}
+\u2192 No CKAN API answered at this address. The site may be up but not run CKAN, or keep its CKAN API on another host. Use \`ckan_find_portals\` to look up the portal's CKAN API URL; retry later only if this address usually works.` : formatCkanError(error, "ckan_status_show");
         return {
-          content: [{
-            type: "text",
-            text: migrated ? detail : `Server appears to be offline or not a valid CKAN instance:
-${detail}`
-          }],
+          content: [{ type: "text", text: `Server appears to be offline or not a valid CKAN instance:
+${detail}` }],
           isError: true
         };
       }
@@ -5382,14 +5418,17 @@ ${error instanceof Error ? error.message : String(error)}`;
 
 // src/tools/portal-discovery.ts
 import { z as z11 } from "zod";
-import axios2 from "axios";
 var DATASHADES_URL = "https://datashades.info/api/portal/list";
 async function fetchPortals() {
-  const resp = await axios2.get(DATASHADES_URL, {
-    timeout: 15e3,
-    headers: { "User-Agent": "CKAN-MCP-Server/1.0" }
-  });
-  return resp.data.portals;
+  const resp = await safeFetch(DATASHADES_URL, {
+    headers: { "User-Agent": "CKAN-MCP-Server/1.0" },
+    signal: AbortSignal.timeout(15e3)
+  }, { httpsOnly: true });
+  if (!resp.ok) {
+    throw new Error(`datashades.info returned ${resp.status} ${resp.statusText}`);
+  }
+  const data = await resp.json();
+  return data.portals;
 }
 function deduplicateByHostname(portals) {
   const seen = /* @__PURE__ */ new Map();
@@ -6117,7 +6156,7 @@ var registerAllPrompts = (server) => {
 function createServer() {
   return new McpServer({
     name: "ckan-mcp-server",
-    version: "0.4.131"
+    version: "0.4.133"
   });
 }
 function registerAll(server) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aborruso/ckan-mcp-server",
-  "version": "0.4.131",
+  "version": "0.4.133",
   "mcpName": "io.github.aborruso/ckan-mcp-server",
   "description": "MCP server for interacting with CKAN open data portals",
   "repository": {