@aborruso/ckan-mcp-server 0.4.120 → 0.4.122

package/LOG.md

@@ -2,6 +2,103 @@
 
 ## 2026-09-05
 
+### v0.4.122 - Solr parser fix
+
+Ships #534: the `text:(...)` wrapper is reserved for the queries dismax cannot serve, the escaping preserves unary operators and balanced grouping, and the parser probe measures two terms taken from the catalog on every portal. `force_text_field` is gone from `portals.json`.
+
+### Solr: wrap only boolean queries, and measure every portal instead of storing a verdict
+
+The usage review surfaced an LLM client reformulating the same request against
+`www.dati.gov.it/opendata` for three hours on 29 July. The data was there from the first
+attempt: `bonifica siti contaminati Piemonte` returns 22 datasets on that portal, the
+Piedmont contaminated-sites registry first. Our `text:(...)` rewrite was returning 5.
+
+`package_search` hands a colon-free query to Solr's dismax parser with `q.op=AND`,
+`mm='2<-1 5<80%'` and `qf='name^4 title^4 tags^2 groups^2 text'` (`ckan/lib/search/query.py`).
+dismax has no boolean syntax, so `A OR B` collapses into `A AND B`: on dati.gov.it
+`aria OR Milano` returns 59, exactly what `aria AND Milano` returns. A colon takes the query
+off dismax, which is what the wrapper exploits — the wrapped form returns 3421. The same
+switch is why it hurts everything else: it drops the `qf` boosts that rank titles and tags
+first, searches the catch-all `text` field alone, and ANDs every term instead of applying
+`mm`. This is CKAN's own default, not a per-portal defect, which is why the same pattern
+showed up on Milano, Toscana, Sicilia, Ucraina and open.canada.ca.
+
+Measured on dati.gov.it: `defibrillatori Comune di Lecce` 678 -> 1, `qualità aria Milano`
+650 -> 51, `musei roma arte opere catalogo` 9 -> 0 — the second most repeated query of the
+semester, answered with an empty page while nine datasets existed. On 40 real queries from
+telemetry the separation is clean: with a boolean operator the wrapper helped 8 times and
+hurt none; without one it helped none and hurt 10, six of them down to zero.
+
+So the wrapper now applies only to queries carrying a boolean operator. The shape that an
+LLM client generates from a user's request — a run of keywords, no operators — is left to
+the portal's own parser, boosts and `mm` included.
+
+`probePortalParser` was rewritten and now runs for every portal, configured ones included;
+`force_text_field` is gone from `portals.json` (8 entries) so there is one source of truth.
+The old probe asked `data OR dati`, two words common enough to saturate: on Milano `data`
+alone and `data OR dati` both return 2564, so it read the portal as healthy while
+`aria OR acqua` returned 0 against 54 and 33 for the single terms. It now picks two terms
+from the catalog itself — single-word tag facets between 0.5% and 30%, falling back to
+frequent title words — and compares `A`, `B`, `A OR B`, `text:(A OR B)`. An OR returning
+fewer hits than either operand is not being honoured, and the wrapper is the answer only if
+the wrapped form returns more. `data.stadt-zuerich.ch`, where the `text` field returns 0 for
+every query, is correctly left alone.
+
+Cost: a plain query pays nothing, since nothing but a boolean query can be wrapped. The
+probe costs 5 extra `rows=0` calls the first time a boolean query reaches a portal in a
+session, then nothing.
+
+Verified e2e against live portals: dati.gov.it 5 -> 22 and 0 -> 9 on the two queries from
+the telemetry, Milano `aria OR acqua` 0 -> 87, Zurich unchanged at 10 and 172. 516 tests
+pass, 18 added.
+
+### v0.4.121 - clearer 404 on datastore_search_sql
+
+Patch release for the fix in #532: a 404 on `datastore_search_sql` now says the portal does not expose the SQL endpoint, instead of sending the caller to `ckan_package_show` for a resource_id that was already valid. Also ships the telemetry pipeline fixes and the DEPLOYMENT.md/CLAUDE.md realignment from earlier today.
+
+### DEPLOYMENT.md and CLAUDE.md realigned with the actual deployment
+
+Six things the deployment guide got wrong, all verified against the live system:
+
+- tool count stuck at 7; `/health` reports 20 tools, 7 resources, 6 prompts on v0.4.120
+- bundle "~400KB" in two places, while the doc's own sample output said 541 KiB; rebuilt: 533 KB
+- the free tier presented as the configuration in use. Workers Observability keeps 7 days of telemetry on this account (measured: events at 7 days back, none at 8), which is the paid retention — the free-tier request and CPU limits are not the ones this deployment operates under
+- `demo.ckan.org` in three curl examples, which CLAUDE.md forbids for tests
+- the release workflow told you to commit and push straight to `main`, in both files. Now branch, PR, squash-merge, then tag on the merged commit
+- a stale `Co-Authored-By` line in the sample commit
+
+Also corrected the troubleshooting entry calling CPU-limit errors "rare": 22 in the first four days of September against 11 in all of August. Left flagged as unexplained.
+
+Left alone on purpose: archived OpenSpec proposals, historical LOG entries, and `demo.ckan.org` where it appears as an example for end users (README, EXAMPLES, SKILL) rather than as a test target.
+
+### 404 on datastore_search_sql: hint sent callers round a loop
+
+A 404 on `datastore_search_sql` fell into the generic `datastore_search` branch of
+`formatCkanError`, which answers "get a valid resource_id first: call `ckan_package_show`".
+On a portal that does not expose SQL the resource_id is fine and `ckan_package_show`
+hands back the same one, so a client following the hint retries into the same 404.
+
+SQL access is optional in CKAN and widely disabled: `datastore_search_sql?sql=SELECT 1`
+returns 404 on Toronto's portal and 200 on dati.comune.messina.it, while
+`datastore_search` answers 200 on both. A bad table name comes back as a 400, not a 404,
+so a 404 on that action means the endpoint itself is missing. The hint now says so and
+points at `ckan_datastore_search`.
+
+Verified e2e: Toronto returns the new hint; Messina counts 202.906 rows in a single SQL
+call. 498 tests pass, one added.
+
+### Telemetry pipeline: cross-worker contamination, missing outcome, unattributable errors
+
+A usage review of `worker_events_flat.jsonl` surfaced three defects in the measurement itself, all confirmed against the live Observability API:
+
+- The query filtered only on `$metadata.type = cf-worker`, with no `service` filter, so events from `opensdmx-mcp` (another Worker on the same account) were archived here: 7 out of 31 events in a 48h probe, 589 `sdmx_*` records in the May-July history. Added `$metadata.service = ckan-mcp-server` (env override `CF_SCRIPT_NAME`); the probe then returned 24/24 ckan events. The existing records cannot be cleaned: retention is 3 days.
+- `$workers.outcome` disappeared from the API on 2026-07-12 (the object now carries `spanId`/`traceId` instead). Not a regression of ours: the archiver's only refactor is 7e78b32, March. The replacement was already there and unused: `src/worker.ts` logs its own `status` per call, plus `cache_hit`, `duration_ms` and `limit`, and the archiver was throwing all four away. `resolve_outcome()` now reads `$workers.outcome`, then `source.status`, then `$metadata.level`, and writes `unknown` rather than assuming `ok`. On a 47-event live sample it resolved every record (40 ok, 7 error, no unknown), CPU-limit crashes included. July-September stays null and cannot be backfilled: retention is 3 days.
+- `worker_daily_stats.sh` had been writing `ok:0, errors:0` on every row since that date. It now derives the error flag from `error IS NOT NULL OR outcome IN (...)`, which stays valid across the schema change. March figures are unchanged; August and September are populated again (2026-09-03: 240 calls, 220 ok, 20 errors).
+
+Flat schema gains `script_version`, `request_id`, `trigger`, `cache_hit`, `duration_ms`, `limit`. Without the first, a CPU-limit event cannot be tied to a release, and those events carry no `tool` or `server` either; they now do carry the version, so September's 22 CPU errors in 4 days (against 11 in the whole of August) become attributable from here on, not retroactively. `cache_hit` and `limit` answer the two questions this review could only guess at: cache fragmentation from URL spelling, and page size on the long datastore loops. Older records are padded to the same field set on rewrite.
+
+Usage, excluding March (evaluation traffic: 2.953 calls, 242 portals) and the sdmx events: ~470-1.450 calls a month, median 13,5 a day, very spiky. `ckan_datastore_search` leads (1.142 since June), and on 2026-08-24 a single client generated 392 calls with 6 distinct queries paging Toronto's datastore. Callers spell the same portal in three or four ways (`dati.gov.it/opendata`, `www.` variant, trailing slash; `open.canada.ca` in three forms), which fragments the cache key.
+
 ### v0.4.120 - SSRF hardening release + dependency audit
 
 Releases PR #531 (env-proxy refusal, multicast/reserved/site-local ranges, see 2026-09-04 entry). Also:

package/dist/index.js

@@ -33,9 +33,6 @@ var portals_default = {
         "https://dati.gov.it",
         "https://www.dati.gov.it"
       ],
-      search: {
-        force_text_field: true
-      },
       hvd: {
         category_field: "hvd_category"
       },
@@ -52,10 +49,7 @@ var portals_default = {
       api_url: "https://dati.arpae.it",
       api_url_aliases: [
         "http://dati.arpae.it"
-      ],
-      search: {
-        force_text_field: true
-      }
+      ]
     },
     {
       id: "anac-opendata",
@@ -77,9 +71,6 @@ var portals_default = {
         "http://www.data.gov.uk"
       ],
       api_path: "/api/action",
-      search: {
-        force_text_field: true
-      },
       dataset_view_url: "https://www.data.gov.uk/dataset/{id}/{name}",
       organization_view_url: "https://www.data.gov.uk/organization/{name}"
     },
@@ -89,10 +80,7 @@ var portals_default = {
       api_url: "https://catalog.data.gov",
       api_url_aliases: [
         "http://catalog.data.gov"
-      ],
-      search: {
-        force_text_field: true
-      }
+      ]
     },
     {
       id: "open-canada",
@@ -102,10 +90,7 @@ var portals_default = {
         "http://open.canada.ca/data",
         "https://open.canada.ca",
         "http://open.canada.ca"
-      ],
-      search: {
-        force_text_field: true
-      }
+      ]
     },
     {
       id: "data-gov-au",
@@ -147,10 +132,7 @@ var portals_default = {
       api_url: "https://data.gov.ua",
       api_url_aliases: [
         "http://data.gov.ua"
-      ],
-      search: {
-        force_text_field: false
-      }
+      ]
     },
     {
       id: "dati-regione-sicilia",
@@ -158,10 +140,7 @@ var portals_default = {
       api_url: "https://dati.regione.sicilia.it",
       api_url_aliases: [
         "http://dati.regione.sicilia.it"
-      ],
-      search: {
-        force_text_field: false
-      }
+      ]
     },
     {
       id: "bdap-rgs-mef",
@@ -190,10 +169,7 @@ var portals_default = {
   ],
   defaults: {
     dataset_view_url: "{server_url}/dataset/{name}",
-    organization_view_url: "{server_url}/organization/{name}",
-    search: {
-      force_text_field: false
-    }
+    organization_view_url: "{server_url}/organization/{name}"
   }
 };
 
@@ -224,10 +200,6 @@ function getPortalSearchConfig(serverUrl) {
     force_text_field: portal?.search?.force_text_field ?? defaults.force_text_field ?? false
   };
 }
-function isPortalSearchExplicitlyConfigured(serverUrl) {
-  const portal = getPortalConfig(serverUrl);
-  return portal?.search?.force_text_field !== void 0;
-}
 function normalizePortalUrl(serverUrl) {
   return normalizeUrl(serverUrl);
 }
@@ -496,7 +468,9 @@ function formatCkanError(error, _toolName) {
   const { status, action, message } = error;
   let hint = "";
   if (status === 404) {
-    if (action.startsWith("datastore_search")) {
+    if (action === "datastore_search_sql") {
+      hint = "\u2192 This portal does not expose the SQL endpoint (optional in CKAN, often disabled). Use `ckan_datastore_search` with `filters`/`sort` instead; the resource_id is probably fine.";
+    } else if (action.startsWith("datastore_search")) {
       hint = "\u2192 Get a valid resource_id first: call `ckan_package_show` on a dataset, then pick a resource where `datastore_active` is true.";
     } else if (action === "package_show") {
       hint = "\u2192 Use `ckan_package_search` to find a valid dataset name or ID.";
@@ -1220,12 +1194,51 @@ function getOrganizationViewUrl(serverUrl, org) {
 // src/utils/search.ts
 var DEFAULT_SEARCH_QUERY = "*:*";
 var FIELD_QUERY_PATTERN = /\b[a-zA-Z_][\w-]*:/;
+var EXPLICIT_BOOL_PATTERN = /\b(AND|OR|NOT)\b|[+\-!]/;
+var BOOL_KEYWORD_PATTERN = /\b(AND|OR|NOT)\b/;
+var INTRAWORD_PUNCT_PATTERN = /\S[+\-!]\S/;
 var SOLR_SPECIAL_CHARS = /[+\-!(){}[\]^~*?:\\/|&]/g;
 function isFieldedQuery(query) {
   return FIELD_QUERY_PATTERN.test(query);
 }
-function escapeSolrQuery(query) {
-  return query.replace(SOLR_SPECIAL_CHARS, "\\$&");
+function hasExplicitBooleanOperator(query) {
+  const trimmed = query.trim();
+  if (BOOL_KEYWORD_PATTERN.test(trimmed)) return true;
+  return INTRAWORD_PUNCT_PATTERN.test(trimmed);
+}
+function mayNeedTextWrapping(query) {
+  const trimmed = query.trim();
+  return trimmed !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmed) && hasExplicitBooleanOperator(trimmed);
+}
+function isEscaped(query, offset) {
+  let backslashes = 0;
+  for (let i = offset - 1; i >= 0 && query[i] === "\\"; i--) backslashes++;
+  return backslashes % 2 === 1;
+}
+function hasBalancedParens(query) {
+  let depth = 0;
+  for (let i = 0; i < query.length; i++) {
+    if (isEscaped(query, i)) continue;
+    if (query[i] === "(") depth++;
+    else if (query[i] === ")" && --depth < 0) return false;
+  }
+  return depth === 0;
+}
+function escapeForTextWrapping(query) {
+  const keepGroups = hasBalancedParens(query);
+  return query.replace(
+    SOLR_SPECIAL_CHARS,
+    (char, offset, whole) => {
+      if (keepGroups && (char === "(" || char === ")") && !isEscaped(whole, offset)) {
+        return char;
+      }
+      const isUnary = char === "+" || char === "-" || char === "!";
+      const previous = offset === 0 ? "" : whole[offset - 1];
+      const atTermStart = offset === 0 || /\s/.test(previous) || keepGroups && previous === "(";
+      const followedByTerm = offset + 1 < whole.length && !/\s/.test(whole[offset + 1]);
+      return isUnary && atTermStart && followedByTerm ? char : `\\${char}`;
+    }
+  );
 }
 function convertDateMathForUnsupportedFields(query) {
   const now = /* @__PURE__ */ new Date();
@@ -1252,7 +1265,6 @@ function convertDateMathForUnsupportedFields(query) {
     return `${field}:[${startIso} TO ${nowIso}]`;
   });
 }
-var EXPLICIT_BOOL_PATTERN = /\b(AND|OR|NOT)\b|[+\-!]/;
 function isPlainMultiTermQuery(query) {
   const trimmed = query.trim();
   if (trimmed === "*:*" || trimmed === "") return false;
@@ -1281,27 +1293,69 @@ function resolveSearchQuery(serverUrl, query, parserOverride) {
     forceTextField = false;
   } else if (portalForce) {
     const trimmedQuery = query.trim();
-    forceTextField = trimmedQuery !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmedQuery);
+    forceTextField = trimmedQuery !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmedQuery) && hasExplicitBooleanOperator(trimmedQuery);
   }
-  let effectiveQuery = forceTextField ? `text:(${escapeSolrQuery(query)})` : query;
+  let effectiveQuery = forceTextField ? `text:(${escapeForTextWrapping(query)})` : query;
   effectiveQuery = convertDateMathForUnsupportedFields(effectiveQuery);
   return { effectiveQuery, forcedTextField: forceTextField };
 }
 
 // src/tools/package.ts
 var _portalParserCache = /* @__PURE__ */ new Map();
+var LITERAL_TERM = /^[\p{L}\p{N}_]+$/u;
+async function pickProbeTerms(serverUrl) {
+  const facetRes = await makeCkanRequest(serverUrl, "package_search", {
+    q: "*:*",
+    rows: 0,
+    "facet.field": '["tags"]',
+    "facet.limit": 100
+  }).catch(() => null);
+  const total = facetRes?.count ?? 0;
+  if (!total) return null;
+  const items = facetRes?.search_facets?.tags?.items ?? [];
+  const usable = items.filter((i) => typeof i.name === "string" && LITERAL_TERM.test(i.name)).filter((i) => (i.count ?? 0) >= total * 5e-3 && (i.count ?? 0) <= total * 0.3).sort((a, b) => (b.count ?? 0) - (a.count ?? 0));
+  if (usable.length >= 2) return [usable[0].name, usable[1].name];
+  const sampleRes = await makeCkanRequest(serverUrl, "package_search", {
+    q: "*:*",
+    rows: 25
+  }).catch(() => null);
+  const titles = (sampleRes?.results ?? []).map((r) => r?.title ?? "");
+  const words = titles.map((t) => new Set(t.toLowerCase().split(/[^\p{L}\p{N}_]+/u).filter((w) => w.length > 4)));
+  const freq = /* @__PURE__ */ new Map();
+  for (const set of words) for (const w of set) freq.set(w, (freq.get(w) ?? 0) + 1);
+  const ranked = [...freq.entries()].filter(([w]) => LITERAL_TERM.test(w)).sort((a, b) => b[1] - a[1]);
+  const first = ranked[0]?.[0];
+  if (!first) return null;
+  const independent = ranked.slice(1).find(([w]) => words.some((set) => set.has(w) && !set.has(first)));
+  const second = independent?.[0] ?? ranked[1]?.[0];
+  return second ? [first, second] : null;
+}
 async function probePortalParser(serverUrl) {
   const key = serverUrl.replace(/\/$/, "").toLowerCase();
   if (_portalParserCache.has(key)) return _portalParserCache.get(key);
-  const probe = "data OR dati";
-  const [defaultRes, textRes] = await Promise.allSettled([
-    makeCkanRequest(serverUrl, "package_search", { q: probe, rows: 0 }),
-    makeCkanRequest(serverUrl, "package_search", { q: `text:(${probe})`, rows: 0 })
-  ]);
-  const defaultCount = defaultRes.status === "fulfilled" ? defaultRes.value.count ?? 0 : 0;
-  const textCount = textRes.status === "fulfilled" ? textRes.value.count ?? 0 : 0;
-  const needsText = textCount > 0 && (defaultCount === 0 || textCount > defaultCount * 2);
-  _portalParserCache.set(key, needsText);
+  const terms = await pickProbeTerms(serverUrl).catch(() => null);
+  if (!terms) return false;
+  let needsText = false;
+  let measured = false;
+  {
+    const [a, b] = terms;
+    const count = async (q) => {
+      const res = await makeCkanRequest(serverUrl, "package_search", { q, rows: 0 }).catch(() => null);
+      return typeof res?.count === "number" ? res.count : null;
+    };
+    const [ca, cb, cOr, cText] = await Promise.all([
+      count(a),
+      count(b),
+      count(`${a} OR ${b}`),
+      count(`text:(${a} OR ${b})`)
+    ]);
+    if (ca !== null && cb !== null && cOr !== null && cText !== null) {
+      const booleanIgnored = cOr < Math.max(ca, cb);
+      needsText = booleanIgnored && cText > cOr;
+      measured = true;
+    }
+  }
+  if (measured) _portalParserCache.set(key, needsText);
   return needsText;
 }
 var DEFAULT_RELEVANCE_WEIGHTS = {
@@ -1932,7 +1986,7 @@ Typical workflow: ckan_status_show (check locale) \u2192 ckan_package_search (qu
           if (!effectiveSort) effectiveSort = "issued desc, metadata_created desc";
         }
         let parserOverride = params.query_parser;
-        if (!parserOverride && !isPortalSearchExplicitlyConfigured(params.server_url)) {
+        if (!parserOverride && mayNeedTextWrapping(query)) {
           const needsText = await probePortalParser(params.server_url);
           if (needsText) parserOverride = "text";
         }
@@ -1965,7 +2019,10 @@ Typical workflow: ckan_status_show (check locale) \u2192 ckan_package_search (qu
           const { effectiveQuery: strippedEffective } = resolveSearchQuery(
             params.server_url,
             strippedQuery,
-            params.query_parser
+            // parserOverride, not params.query_parser: the probe's verdict must
+            // survive the retry, or an accented boolean query is re-sent to the
+            // parser that ignores booleans.
+            parserOverride
           );
           const fallbackResult = await makeCkanRequest(
             params.server_url,
@@ -2090,7 +2147,7 @@ Note: showing top ${sorted.length} only. Use \`response_format: json\` for full
 `;
           if (isPlainMultiTermQuery(params.q)) {
             markdown += `
-> **Tip**: Multi-term queries use AND by default (all terms must match). Try OR to broaden the search:
+> **Tip**: With several terms the portal requires most of them to match. Spelling out OR broadens the search:
 `;
             markdown += `> \`q: "${buildOrQuery(params.q)}"\`
 `;
@@ -5800,7 +5857,7 @@ var registerAllPrompts = (server) => {
 function createServer() {
   return new McpServer({
     name: "ckan-mcp-server",
-    version: "0.4.120"
+    version: "0.4.122"
   });
 }
 function registerAll(server) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aborruso/ckan-mcp-server",
-  "version": "0.4.120",
+  "version": "0.4.122",
   "mcpName": "io.github.aborruso/ckan-mcp-server",
   "description": "MCP server for interacting with CKAN open data portals",
   "repository": {