@aborruso/ckan-mcp-server 0.4.122 → 0.4.123

package/LOG.md

@@ -2,6 +2,63 @@
 
 ## 2026-09-05
 
+### v0.4.123 - relevance scoring and shared parser probe
+
+Ships #536: field scoring by share of matched terms, Italian stopwords with acronyms preserved, Unicode-aware term matching, a wider candidate window, the parser probe shared with `ckan_find_relevant_datasets`, and accent-safe filters in `ckan_organization_search` and `ckan_tag_list`. `openspec/specs/ckan-search/spec.md` rewritten around the query-building path.
+
+### Relevance scoring: three defects that only became visible once search worked
+
+Testing v0.4.122 through an MCP client, not curl, showed `ckan_find_relevant_datasets`
+answering its own documented example badly. `defibrillatori Comune di Lecce` on
+dati.gov.it used to return the catalog's only Lecce dataset — because the search
+returned exactly one result. With recall restored it returns 679, and the top three
+became Trento, Martina Franca and Desio while the Lecce dataset fell out of the window.
+
+The wrapping fix did not cause this; it removed the cover. Three defects, all older:
+
+- `scoreTextField` awarded the **whole** field weight when **any** query term matched.
+  "Comune di Martina Franca" and "Comune di Lecce" both scored a full holder match, so
+  the right dataset could not outrank the wrong ones. It now scores the share of terms
+  the field carries.
+- the stopword list was English-only, so `di` counted as a term: "Provincia Autonoma
+  **di** Trento" earned a full holder match on a query asking for Lecce. Italian
+  stopwords added.
+- `\b` is ASCII-only in JavaScript, so a term ending in an accented letter never found
+  its word boundary: `mobilità` scored 0 against "mobilità urbana", `qualità` against
+  "qualità dell'aria". On a catalog that is mostly not in English this silently sank
+  every accented query. Replaced with Unicode lookarounds.
+
+Looking for more of the same family turned up three more, all cases of a local filter
+running over a truncated or wrongly-normalised set:
+
+- `ckan_find_relevant_datasets` never called the parser probe. `portals.json` used to
+  cover it; removing `force_text_field` left it sending boolean queries to the parser that
+  ignores them. On dati.comune.milano.it `aria OR acqua` returned 0 there against 87 from
+  `ckan_package_search`. Same probe now applies to both.
+- `ckan_organization_search` builds a Solr wildcard, which bypasses the analysis chain, so
+  the pattern must be pre-normalised the way CKAN builds a name slug. It lowercased but did
+  not fold accents: `città` returned 0 while `citta` matched 135 datasets.
+- `ckan_tag_list` applied `tag_query` after faceting, with `facet.limit` set to the
+  caller's `limit`. On dati.gov.it 53 tags contain "citta" and none is in the top 100, so
+  the filter answered "no tags" while they existed. The facet is now widened when a filter
+  is given.
+
+`openspec/specs/ckan-search/spec.md` described the parser as a property of
+`ckan_package_search` alone, which is what let the second caller go unnoticed — and after
+yesterday it was also wrong, still describing the per-portal default that was removed.
+Rewritten as a property of the query-building path, naming every tool that shares it.
+
+Also raised the candidate window to at least 50: the local ranking only sees what Solr
+returns first, and `limit: 3` shrank it to 15 — enough when a search returned a handful
+of results, not enough now.
+
+`defibrillatori Comune di Lecce` puts the Lecce dataset first again. 532 tests, 5 added.
+
+How it was missed: yesterday's verification checked result **counts** through
+`ckan_package_search`, never the ranked output of `ckan_find_relevant_datasets` — the
+third most used tool in the telemetry, and the second caller of `resolveSearchQuery`.
+Counting results proves recall, not usefulness.
+
 ### v0.4.122 - Solr parser fix
 
 Ships #534: the `text:(...)` wrapper is reserved for the queries dismax cannot serve, the escaping preserves unary operators and balanced grouping, and the parser probe measures two terms taken from the catalog on every portal. `force_text_field` is gone from `portals.json`.

package/dist/index.js

@@ -1367,6 +1367,41 @@ var DEFAULT_RELEVANCE_WEIGHTS = {
   publisher: 2
 };
 var QUERY_STOPWORDS = /* @__PURE__ */ new Set([
+  // Italian: without these, `defibrillatori Comune di Lecce` scored a full holder
+  // match against "Provincia Autonoma di Trento" on the strength of "di" alone.
+  "di",
+  "del",
+  "dello",
+  "della",
+  "dei",
+  "degli",
+  "delle",
+  "il",
+  "lo",
+  "la",
+  "i",
+  "gli",
+  "le",
+  "un",
+  "uno",
+  "una",
+  "e",
+  "ed",
+  "per",
+  "con",
+  "su",
+  "da",
+  "dal",
+  "dalla",
+  "nel",
+  "nella",
+  "al",
+  "alla",
+  "che",
+  "non",
+  "come",
+  "dove",
+  "sono",
   "a",
   "an",
   "the",
@@ -1410,18 +1445,32 @@ var QUERY_STOPWORDS = /* @__PURE__ */ new Set([
   "those"
 ]);
 var extractQueryTerms = (query) => {
-  const matches = query.toLowerCase().match(/[\p{L}\p{N}]+/gu) ?? [];
-  const terms = matches.filter((term) => term.length > 1 && !QUERY_STOPWORDS.has(term));
+  const raw = query.normalize("NFC").match(/[\p{L}\p{N}]+/gu) ?? [];
+  const terms = raw.filter((token) => {
+    const term = token.toLowerCase();
+    if (term.length <= 1) return false;
+    if (!QUERY_STOPWORDS.has(term)) return true;
+    return token.length > 1 && token === token.toUpperCase() && token !== term;
+  }).map((token) => token.toLowerCase());
   return Array.from(new Set(terms));
 };
 var escapeRegExp = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+var termPattern = (term) => new RegExp(`(?<![\\p{L}\\p{N}])${escapeRegExp(term.normalize("NFC"))}(?![\\p{L}\\p{N}])`, "iu");
+var normalizeForMatch = (text) => text.normalize("NFC").toLowerCase().replace(/_/g, " ");
 var textMatchesTerms = (text, terms) => {
   if (!text || terms.length === 0) return false;
-  const normalized = text.toLowerCase().replace(/_/g, " ");
-  return terms.some((term) => new RegExp(`\\b${escapeRegExp(term)}\\b`, "i").test(normalized));
+  const normalized = normalizeForMatch(text);
+  return terms.some((term) => termPattern(term).test(normalized));
+};
+var countMatchingTerms = (text, terms) => {
+  if (!text || terms.length === 0) return 0;
+  const normalized = normalizeForMatch(text);
+  return terms.filter((term) => termPattern(term).test(normalized)).length;
 };
 var scoreTextField = (text, terms, weight) => {
-  return textMatchesTerms(text, terms) ? weight : 0;
+  const matched = countMatchingTerms(text, terms);
+  if (matched === 0) return 0;
+  return Math.round(weight * matched / terms.length * 10) / 10;
 };
 var readDcatExtra = (dataset, key) => {
   const extras = Array.isArray(dataset.extras) ? dataset.extras : [];
@@ -2242,11 +2291,16 @@ Typical workflow: ckan_find_relevant_datasets \u2192 ckan_package_show (inspect
           ...DEFAULT_RELEVANCE_WEIGHTS,
           ...params.weights ?? {}
         };
-        const rows = Math.min(Math.max(params.limit * 5, params.limit), 100);
+        const rows = Math.min(Math.max(params.limit * 5, 50), 100);
+        let parserOverride = params.query_parser;
+        if (!parserOverride && mayNeedTextWrapping(params.query)) {
+          const needsText = await probePortalParser(params.server_url);
+          if (needsText) parserOverride = "text";
+        }
         const { effectiveQuery } = resolveSearchQuery(
           params.server_url,
           params.query,
-          params.query_parser
+          parserOverride
         );
         const searchResult = await makeCkanRequest(
           params.server_url,
@@ -2987,7 +3041,7 @@ Typical workflow: ckan_organization_search \u2192 ckan_organization_show (get de
     },
     async (params) => {
       try {
-        const query = `organization:*${params.pattern.toLowerCase()}*`;
+        const query = `organization:*${stripAccents(params.pattern.toLowerCase())}*`;
         const result = await makeCkanRequest(
           params.server_url,
           "package_search",
@@ -3491,7 +3545,13 @@ Typical workflow: ckan_tag_list \u2192 ckan_package_search with fq="tags:tag_nam
           q: params.q,
           rows: 0,
           "facet.field": JSON.stringify(["tags"]),
-          "facet.limit": params.limit
+          // The filter runs on whatever the facet returned, so with a tag_query the
+          // facet has to be wider than the caller's limit: on dati.gov.it 53 tags
+          // contain "citta" and none is in the top 100, so asking for 100 answered
+          // "no tags" while they existed. CKAN rejects Solr's own `facet.contains`
+          // ("Invalid search parameters"), so the filtering cannot move server-side;
+          // this window is the compromise, and WIDE_FACET below is the escape hatch.
+          "facet.limit": params.tag_query ? Math.max(params.limit * 20, 500) : params.limit
         };
         if (params.fq) apiParams.fq = params.fq;
         const result = await makeCkanRequest(
@@ -3501,8 +3561,18 @@ Typical workflow: ckan_tag_list \u2192 ckan_package_search with fq="tags:tag_nam
         );
         let tags = normalizeTagFacets(result);
         if (params.tag_query) {
-          const needle = params.tag_query.toLowerCase();
-          tags = tags.filter((tag) => tag.name.toLowerCase().includes(needle));
+          const needle = stripAccents(params.tag_query.toLowerCase());
+          const matching = (candidates) => candidates.filter((tag) => stripAccents(tag.name.toLowerCase()).includes(needle));
+          tags = matching(tags);
+          if (tags.length < params.limit) {
+            const wide = await makeCkanRequest(
+              params.server_url,
+              "package_search",
+              { ...apiParams, "facet.limit": -1 }
+            );
+            tags = matching(normalizeTagFacets(wide));
+          }
+          tags = tags.sort((a, b) => b.count - a.count).slice(0, params.limit);
         }
         tags = tags.sort((a, b) => b.count - a.count || a.name.localeCompare(b.name)).slice(0, params.limit);
         if (params.response_format === "json" /* JSON */) {
@@ -5857,7 +5927,7 @@ var registerAllPrompts = (server) => {
 function createServer() {
   return new McpServer({
     name: "ckan-mcp-server",
-    version: "0.4.122"
+    version: "0.4.123"
   });
 }
 function registerAll(server) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aborruso/ckan-mcp-server",
-  "version": "0.4.122",
+  "version": "0.4.123",
   "mcpName": "io.github.aborruso/ckan-mcp-server",
   "description": "MCP server for interacting with CKAN open data portals",
   "repository": {