@aborruso/ckan-mcp-server 0.4.121 → 0.4.122

package/LOG.md

@@ -2,6 +2,56 @@
 
 ## 2026-09-05
 
+### v0.4.122 - Solr parser fix
+
+Ships #534: the `text:(...)` wrapper is reserved for the queries dismax cannot serve, the escaping preserves unary operators and balanced grouping, and the parser probe measures two terms taken from the catalog on every portal. `force_text_field` is gone from `portals.json`.
+
+### Solr: wrap only boolean queries, and measure every portal instead of storing a verdict
+
+The usage review surfaced an LLM client reformulating the same request against
+`www.dati.gov.it/opendata` for three hours on 29 July. The data was there from the first
+attempt: `bonifica siti contaminati Piemonte` returns 22 datasets on that portal, the
+Piedmont contaminated-sites registry first. Our `text:(...)` rewrite was returning 5.
+
+`package_search` hands a colon-free query to Solr's dismax parser with `q.op=AND`,
+`mm='2<-1 5<80%'` and `qf='name^4 title^4 tags^2 groups^2 text'` (`ckan/lib/search/query.py`).
+dismax has no boolean syntax, so `A OR B` collapses into `A AND B`: on dati.gov.it
+`aria OR Milano` returns 59, exactly what `aria AND Milano` returns. A colon takes the query
+off dismax, which is what the wrapper exploits — the wrapped form returns 3421. The same
+switch is why it hurts everything else: it drops the `qf` boosts that rank titles and tags
+first, searches the catch-all `text` field alone, and ANDs every term instead of applying
+`mm`. This is CKAN's own default, not a per-portal defect, which is why the same pattern
+showed up on Milano, Toscana, Sicilia, Ucraina and open.canada.ca.
+
+Measured on dati.gov.it: `defibrillatori Comune di Lecce` 678 -> 1, `qualità aria Milano`
+650 -> 51, `musei roma arte opere catalogo` 9 -> 0 — the second most repeated query of the
+semester, answered with an empty page while nine datasets existed. On 40 real queries from
+telemetry the separation is clean: with a boolean operator the wrapper helped 8 times and
+hurt none; without one it helped none and hurt 10, six of them down to zero.
+
+So the wrapper now applies only to queries carrying a boolean operator. The shape that an
+LLM client generates from a user's request — a run of keywords, no operators — is left to
+the portal's own parser, boosts and `mm` included.
+
+`probePortalParser` was rewritten and now runs for every portal, configured ones included;
+`force_text_field` is gone from `portals.json` (8 entries) so there is one source of truth.
+The old probe asked `data OR dati`, two words common enough to saturate: on Milano `data`
+alone and `data OR dati` both return 2564, so it read the portal as healthy while
+`aria OR acqua` returned 0 against 54 and 33 for the single terms. It now picks two terms
+from the catalog itself — single-word tag facets between 0.5% and 30%, falling back to
+frequent title words — and compares `A`, `B`, `A OR B`, `text:(A OR B)`. An OR returning
+fewer hits than either operand is not being honoured, and the wrapper is the answer only if
+the wrapped form returns more. `data.stadt-zuerich.ch`, where the `text` field returns 0 for
+every query, is correctly left alone.
+
+Cost: a plain query pays nothing, since nothing but a boolean query can be wrapped. The
+probe costs 5 extra `rows=0` calls the first time a boolean query reaches a portal in a
+session, then nothing.
+
+Verified e2e against live portals: dati.gov.it 5 -> 22 and 0 -> 9 on the two queries from
+the telemetry, Milano `aria OR acqua` 0 -> 87, Zurich unchanged at 10 and 172. 516 tests
+pass, 18 added.
+
 ### v0.4.121 - clearer 404 on datastore_search_sql
 
 Patch release for the fix in #532: a 404 on `datastore_search_sql` now says the portal does not expose the SQL endpoint, instead of sending the caller to `ckan_package_show` for a resource_id that was already valid. Also ships the telemetry pipeline fixes and the DEPLOYMENT.md/CLAUDE.md realignment from earlier today.

package/dist/index.js

@@ -33,9 +33,6 @@ var portals_default = {
         "https://dati.gov.it",
         "https://www.dati.gov.it"
       ],
-      search: {
-        force_text_field: true
-      },
       hvd: {
         category_field: "hvd_category"
       },
@@ -52,10 +49,7 @@ var portals_default = {
       api_url: "https://dati.arpae.it",
       api_url_aliases: [
         "http://dati.arpae.it"
-      ],
-      search: {
-        force_text_field: true
-      }
+      ]
     },
     {
       id: "anac-opendata",
@@ -77,9 +71,6 @@ var portals_default = {
         "http://www.data.gov.uk"
       ],
       api_path: "/api/action",
-      search: {
-        force_text_field: true
-      },
       dataset_view_url: "https://www.data.gov.uk/dataset/{id}/{name}",
       organization_view_url: "https://www.data.gov.uk/organization/{name}"
     },
@@ -89,10 +80,7 @@ var portals_default = {
       api_url: "https://catalog.data.gov",
       api_url_aliases: [
         "http://catalog.data.gov"
-      ],
-      search: {
-        force_text_field: true
-      }
+      ]
     },
     {
       id: "open-canada",
@@ -102,10 +90,7 @@ var portals_default = {
         "http://open.canada.ca/data",
         "https://open.canada.ca",
         "http://open.canada.ca"
-      ],
-      search: {
-        force_text_field: true
-      }
+      ]
     },
     {
       id: "data-gov-au",
@@ -147,10 +132,7 @@ var portals_default = {
       api_url: "https://data.gov.ua",
       api_url_aliases: [
         "http://data.gov.ua"
-      ],
-      search: {
-        force_text_field: false
-      }
+      ]
     },
     {
       id: "dati-regione-sicilia",
@@ -158,10 +140,7 @@ var portals_default = {
       api_url: "https://dati.regione.sicilia.it",
       api_url_aliases: [
         "http://dati.regione.sicilia.it"
-      ],
-      search: {
-        force_text_field: false
-      }
+      ]
     },
     {
       id: "bdap-rgs-mef",
@@ -190,10 +169,7 @@ var portals_default = {
   ],
   defaults: {
     dataset_view_url: "{server_url}/dataset/{name}",
-    organization_view_url: "{server_url}/organization/{name}",
-    search: {
-      force_text_field: false
-    }
+    organization_view_url: "{server_url}/organization/{name}"
   }
 };
 
@@ -224,10 +200,6 @@ function getPortalSearchConfig(serverUrl) {
     force_text_field: portal?.search?.force_text_field ?? defaults.force_text_field ?? false
   };
 }
-function isPortalSearchExplicitlyConfigured(serverUrl) {
-  const portal = getPortalConfig(serverUrl);
-  return portal?.search?.force_text_field !== void 0;
-}
 function normalizePortalUrl(serverUrl) {
   return normalizeUrl(serverUrl);
 }
@@ -1222,12 +1194,51 @@ function getOrganizationViewUrl(serverUrl, org) {
 // src/utils/search.ts
 var DEFAULT_SEARCH_QUERY = "*:*";
 var FIELD_QUERY_PATTERN = /\b[a-zA-Z_][\w-]*:/;
+var EXPLICIT_BOOL_PATTERN = /\b(AND|OR|NOT)\b|[+\-!]/;
+var BOOL_KEYWORD_PATTERN = /\b(AND|OR|NOT)\b/;
+var INTRAWORD_PUNCT_PATTERN = /\S[+\-!]\S/;
 var SOLR_SPECIAL_CHARS = /[+\-!(){}[\]^~*?:\\/|&]/g;
 function isFieldedQuery(query) {
   return FIELD_QUERY_PATTERN.test(query);
 }
-function escapeSolrQuery(query) {
-  return query.replace(SOLR_SPECIAL_CHARS, "\\$&");
+function hasExplicitBooleanOperator(query) {
+  const trimmed = query.trim();
+  if (BOOL_KEYWORD_PATTERN.test(trimmed)) return true;
+  return INTRAWORD_PUNCT_PATTERN.test(trimmed);
+}
+function mayNeedTextWrapping(query) {
+  const trimmed = query.trim();
+  return trimmed !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmed) && hasExplicitBooleanOperator(trimmed);
+}
+function isEscaped(query, offset) {
+  let backslashes = 0;
+  for (let i = offset - 1; i >= 0 && query[i] === "\\"; i--) backslashes++;
+  return backslashes % 2 === 1;
+}
+function hasBalancedParens(query) {
+  let depth = 0;
+  for (let i = 0; i < query.length; i++) {
+    if (isEscaped(query, i)) continue;
+    if (query[i] === "(") depth++;
+    else if (query[i] === ")" && --depth < 0) return false;
+  }
+  return depth === 0;
+}
+function escapeForTextWrapping(query) {
+  const keepGroups = hasBalancedParens(query);
+  return query.replace(
+    SOLR_SPECIAL_CHARS,
+    (char, offset, whole) => {
+      if (keepGroups && (char === "(" || char === ")") && !isEscaped(whole, offset)) {
+        return char;
+      }
+      const isUnary = char === "+" || char === "-" || char === "!";
+      const previous = offset === 0 ? "" : whole[offset - 1];
+      const atTermStart = offset === 0 || /\s/.test(previous) || keepGroups && previous === "(";
+      const followedByTerm = offset + 1 < whole.length && !/\s/.test(whole[offset + 1]);
+      return isUnary && atTermStart && followedByTerm ? char : `\\${char}`;
+    }
+  );
 }
 function convertDateMathForUnsupportedFields(query) {
   const now = /* @__PURE__ */ new Date();
@@ -1254,7 +1265,6 @@ function convertDateMathForUnsupportedFields(query) {
     return `${field}:[${startIso} TO ${nowIso}]`;
   });
 }
-var EXPLICIT_BOOL_PATTERN = /\b(AND|OR|NOT)\b|[+\-!]/;
 function isPlainMultiTermQuery(query) {
   const trimmed = query.trim();
   if (trimmed === "*:*" || trimmed === "") return false;
@@ -1283,27 +1293,69 @@ function resolveSearchQuery(serverUrl, query, parserOverride) {
     forceTextField = false;
   } else if (portalForce) {
     const trimmedQuery = query.trim();
-    forceTextField = trimmedQuery !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmedQuery);
+    forceTextField = trimmedQuery !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmedQuery) && hasExplicitBooleanOperator(trimmedQuery);
   }
-  let effectiveQuery = forceTextField ? `text:(${escapeSolrQuery(query)})` : query;
+  let effectiveQuery = forceTextField ? `text:(${escapeForTextWrapping(query)})` : query;
   effectiveQuery = convertDateMathForUnsupportedFields(effectiveQuery);
   return { effectiveQuery, forcedTextField: forceTextField };
 }
 
 // src/tools/package.ts
 var _portalParserCache = /* @__PURE__ */ new Map();
+var LITERAL_TERM = /^[\p{L}\p{N}_]+$/u;
+async function pickProbeTerms(serverUrl) {
+  const facetRes = await makeCkanRequest(serverUrl, "package_search", {
+    q: "*:*",
+    rows: 0,
+    "facet.field": '["tags"]',
+    "facet.limit": 100
+  }).catch(() => null);
+  const total = facetRes?.count ?? 0;
+  if (!total) return null;
+  const items = facetRes?.search_facets?.tags?.items ?? [];
+  const usable = items.filter((i) => typeof i.name === "string" && LITERAL_TERM.test(i.name)).filter((i) => (i.count ?? 0) >= total * 5e-3 && (i.count ?? 0) <= total * 0.3).sort((a, b) => (b.count ?? 0) - (a.count ?? 0));
+  if (usable.length >= 2) return [usable[0].name, usable[1].name];
+  const sampleRes = await makeCkanRequest(serverUrl, "package_search", {
+    q: "*:*",
+    rows: 25
+  }).catch(() => null);
+  const titles = (sampleRes?.results ?? []).map((r) => r?.title ?? "");
+  const words = titles.map((t) => new Set(t.toLowerCase().split(/[^\p{L}\p{N}_]+/u).filter((w) => w.length > 4)));
+  const freq = /* @__PURE__ */ new Map();
+  for (const set of words) for (const w of set) freq.set(w, (freq.get(w) ?? 0) + 1);
+  const ranked = [...freq.entries()].filter(([w]) => LITERAL_TERM.test(w)).sort((a, b) => b[1] - a[1]);
+  const first = ranked[0]?.[0];
+  if (!first) return null;
+  const independent = ranked.slice(1).find(([w]) => words.some((set) => set.has(w) && !set.has(first)));
+  const second = independent?.[0] ?? ranked[1]?.[0];
+  return second ? [first, second] : null;
+}
 async function probePortalParser(serverUrl) {
   const key = serverUrl.replace(/\/$/, "").toLowerCase();
   if (_portalParserCache.has(key)) return _portalParserCache.get(key);
-  const probe = "data OR dati";
-  const [defaultRes, textRes] = await Promise.allSettled([
-    makeCkanRequest(serverUrl, "package_search", { q: probe, rows: 0 }),
-    makeCkanRequest(serverUrl, "package_search", { q: `text:(${probe})`, rows: 0 })
-  ]);
-  const defaultCount = defaultRes.status === "fulfilled" ? defaultRes.value.count ?? 0 : 0;
-  const textCount = textRes.status === "fulfilled" ? textRes.value.count ?? 0 : 0;
-  const needsText = textCount > 0 && (defaultCount === 0 || textCount > defaultCount * 2);
-  _portalParserCache.set(key, needsText);
+  const terms = await pickProbeTerms(serverUrl).catch(() => null);
+  if (!terms) return false;
+  let needsText = false;
+  let measured = false;
+  {
+    const [a, b] = terms;
+    const count = async (q) => {
+      const res = await makeCkanRequest(serverUrl, "package_search", { q, rows: 0 }).catch(() => null);
+      return typeof res?.count === "number" ? res.count : null;
+    };
+    const [ca, cb, cOr, cText] = await Promise.all([
+      count(a),
+      count(b),
+      count(`${a} OR ${b}`),
+      count(`text:(${a} OR ${b})`)
+    ]);
+    if (ca !== null && cb !== null && cOr !== null && cText !== null) {
+      const booleanIgnored = cOr < Math.max(ca, cb);
+      needsText = booleanIgnored && cText > cOr;
+      measured = true;
+    }
+  }
+  if (measured) _portalParserCache.set(key, needsText);
   return needsText;
 }
 var DEFAULT_RELEVANCE_WEIGHTS = {
@@ -1934,7 +1986,7 @@ Typical workflow: ckan_status_show (check locale) \u2192 ckan_package_search (qu
           if (!effectiveSort) effectiveSort = "issued desc, metadata_created desc";
         }
         let parserOverride = params.query_parser;
-        if (!parserOverride && !isPortalSearchExplicitlyConfigured(params.server_url)) {
+        if (!parserOverride && mayNeedTextWrapping(query)) {
           const needsText = await probePortalParser(params.server_url);
           if (needsText) parserOverride = "text";
         }
@@ -1967,7 +2019,10 @@ Typical workflow: ckan_status_show (check locale) \u2192 ckan_package_search (qu
           const { effectiveQuery: strippedEffective } = resolveSearchQuery(
             params.server_url,
             strippedQuery,
-            params.query_parser
+            // parserOverride, not params.query_parser: the probe's verdict must
+            // survive the retry, or an accented boolean query is re-sent to the
+            // parser that ignores booleans.
+            parserOverride
           );
           const fallbackResult = await makeCkanRequest(
             params.server_url,
@@ -2092,7 +2147,7 @@ Note: showing top ${sorted.length} only. Use \`response_format: json\` for full
 `;
           if (isPlainMultiTermQuery(params.q)) {
             markdown += `
-> **Tip**: Multi-term queries use AND by default (all terms must match). Try OR to broaden the search:
+> **Tip**: With several terms the portal requires most of them to match. Spelling out OR broadens the search:
 `;
             markdown += `> \`q: "${buildOrQuery(params.q)}"\`
 `;
@@ -5802,7 +5857,7 @@ var registerAllPrompts = (server) => {
 function createServer() {
   return new McpServer({
     name: "ckan-mcp-server",
-    version: "0.4.121"
+    version: "0.4.122"
   });
 }
 function registerAll(server) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aborruso/ckan-mcp-server",
-  "version": "0.4.121",
+  "version": "0.4.122",
   "mcpName": "io.github.aborruso/ckan-mcp-server",
   "description": "MCP server for interacting with CKAN open data portals",
   "repository": {