@aborruso/ckan-mcp-server 0.4.121 → 0.4.123
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LOG.md +107 -0
- package/dist/index.js +188 -63
- package/package.json +1 -1
package/LOG.md
CHANGED
|
@@ -2,6 +2,113 @@
|
|
|
2
2
|
|
|
3
3
|
## 2026-09-05
|
|
4
4
|
|
|
5
|
+
### v0.4.123 - relevance scoring and shared parser probe
|
|
6
|
+
|
|
7
|
+
Ships #536: field scoring by share of matched terms, Italian stopwords with acronyms preserved, Unicode-aware term matching, a wider candidate window, the parser probe shared with `ckan_find_relevant_datasets`, and accent-safe filters in `ckan_organization_search` and `ckan_tag_list`. `openspec/specs/ckan-search/spec.md` rewritten around the query-building path.
|
|
8
|
+
|
|
9
|
+
### Relevance scoring: three defects that only became visible once search worked
|
|
10
|
+
|
|
11
|
+
Testing v0.4.122 through an MCP client, not curl, showed `ckan_find_relevant_datasets`
|
|
12
|
+
answering its own documented example badly. `defibrillatori Comune di Lecce` on
|
|
13
|
+
dati.gov.it used to return the catalog's only Lecce dataset — because the search
|
|
14
|
+
returned exactly one result. With recall restored it returns 679, and the top three
|
|
15
|
+
became Trento, Martina Franca and Desio while the Lecce dataset fell out of the window.
|
|
16
|
+
|
|
17
|
+
The wrapping fix did not cause this; it removed the cover. Three defects, all older:
|
|
18
|
+
|
|
19
|
+
- `scoreTextField` awarded the **whole** field weight when **any** query term matched.
|
|
20
|
+
"Comune di Martina Franca" and "Comune di Lecce" both scored a full holder match, so
|
|
21
|
+
the right dataset could not outrank the wrong ones. It now scores the share of terms
|
|
22
|
+
the field carries.
|
|
23
|
+
- the stopword list was English-only, so `di` counted as a term: "Provincia Autonoma
|
|
24
|
+
**di** Trento" earned a full holder match on a query asking for Lecce. Italian
|
|
25
|
+
stopwords added.
|
|
26
|
+
- `\b` is ASCII-only in JavaScript, so a term ending in an accented letter never found
|
|
27
|
+
its word boundary: `mobilità` scored 0 against "mobilità urbana", `qualità` against
|
|
28
|
+
"qualità dell'aria". On a catalog that is mostly not in English this silently sank
|
|
29
|
+
every accented query. Replaced with Unicode lookarounds.
|
|
30
|
+
|
|
31
|
+
Looking for more of the same family turned up three more, all cases of a local filter
|
|
32
|
+
running over a truncated or wrongly-normalised set:
|
|
33
|
+
|
|
34
|
+
- `ckan_find_relevant_datasets` never called the parser probe. `portals.json` used to
|
|
35
|
+
cover it; removing `force_text_field` left it sending boolean queries to the parser that
|
|
36
|
+
ignores them. On dati.comune.milano.it `aria OR acqua` returned 0 there against 87 from
|
|
37
|
+
`ckan_package_search`. Same probe now applies to both.
|
|
38
|
+
- `ckan_organization_search` builds a Solr wildcard, which bypasses the analysis chain, so
|
|
39
|
+
the pattern must be pre-normalised the way CKAN builds a name slug. It lowercased but did
|
|
40
|
+
not fold accents: `città` returned 0 while `citta` matched 135 datasets.
|
|
41
|
+
- `ckan_tag_list` applied `tag_query` after faceting, with `facet.limit` set to the
|
|
42
|
+
caller's `limit`. On dati.gov.it 53 tags contain "citta" and none is in the top 100, so
|
|
43
|
+
the filter answered "no tags" while they existed. The facet is now widened when a filter
|
|
44
|
+
is given.
|
|
45
|
+
|
|
46
|
+
`openspec/specs/ckan-search/spec.md` described the parser as a property of
|
|
47
|
+
`ckan_package_search` alone, which is what let the second caller go unnoticed — and after
|
|
48
|
+
yesterday it was also wrong, still describing the per-portal default that was removed.
|
|
49
|
+
Rewritten as a property of the query-building path, naming every tool that shares it.
|
|
50
|
+
|
|
51
|
+
Also raised the candidate window to at least 50: the local ranking only sees what Solr
|
|
52
|
+
returns first, and `limit: 3` shrank it to 15 — enough when a search returned a handful
|
|
53
|
+
of results, not enough now.
|
|
54
|
+
|
|
55
|
+
`defibrillatori Comune di Lecce` puts the Lecce dataset first again. 532 tests, 5 added.
|
|
56
|
+
|
|
57
|
+
How it was missed: yesterday's verification checked result **counts** through
|
|
58
|
+
`ckan_package_search`, never the ranked output of `ckan_find_relevant_datasets` — the
|
|
59
|
+
third most used tool in the telemetry, and the second caller of `resolveSearchQuery`.
|
|
60
|
+
Counting results proves recall, not usefulness.
|
|
61
|
+
|
|
62
|
+
### v0.4.122 - Solr parser fix
|
|
63
|
+
|
|
64
|
+
Ships #534: the `text:(...)` wrapper is reserved for the queries dismax cannot serve, the escaping preserves unary operators and balanced grouping, and the parser probe measures two terms taken from the catalog on every portal. `force_text_field` is gone from `portals.json`.
|
|
65
|
+
|
|
66
|
+
### Solr: wrap only boolean queries, and measure every portal instead of storing a verdict
|
|
67
|
+
|
|
68
|
+
The usage review surfaced an LLM client reformulating the same request against
|
|
69
|
+
`www.dati.gov.it/opendata` for three hours on 29 July. The data was there from the first
|
|
70
|
+
attempt: `bonifica siti contaminati Piemonte` returns 22 datasets on that portal, the
|
|
71
|
+
Piedmont contaminated-sites registry first. Our `text:(...)` rewrite was returning 5.
|
|
72
|
+
|
|
73
|
+
`package_search` hands a colon-free query to Solr's dismax parser with `q.op=AND`,
|
|
74
|
+
`mm='2<-1 5<80%'` and `qf='name^4 title^4 tags^2 groups^2 text'` (`ckan/lib/search/query.py`).
|
|
75
|
+
dismax has no boolean syntax, so `A OR B` collapses into `A AND B`: on dati.gov.it
|
|
76
|
+
`aria OR Milano` returns 59, exactly what `aria AND Milano` returns. A colon takes the query
|
|
77
|
+
off dismax, which is what the wrapper exploits — the wrapped form returns 3421. The same
|
|
78
|
+
switch is why it hurts everything else: it drops the `qf` boosts that rank titles and tags
|
|
79
|
+
first, searches the catch-all `text` field alone, and ANDs every term instead of applying
|
|
80
|
+
`mm`. This is CKAN's own default, not a per-portal defect, which is why the same pattern
|
|
81
|
+
showed up on Milano, Toscana, Sicilia, Ucraina and open.canada.ca.
|
|
82
|
+
|
|
83
|
+
Measured on dati.gov.it: `defibrillatori Comune di Lecce` 678 -> 1, `qualità aria Milano`
|
|
84
|
+
650 -> 51, `musei roma arte opere catalogo` 9 -> 0 — the second most repeated query of the
|
|
85
|
+
semester, answered with an empty page while nine datasets existed. On 40 real queries from
|
|
86
|
+
telemetry the separation is clean: with a boolean operator the wrapper helped 8 times and
|
|
87
|
+
hurt none; without one it helped none and hurt 10, six of them down to zero.
|
|
88
|
+
|
|
89
|
+
So the wrapper now applies only to queries carrying a boolean operator. The shape that an
|
|
90
|
+
LLM client generates from a user's request — a run of keywords, no operators — is left to
|
|
91
|
+
the portal's own parser, boosts and `mm` included.
|
|
92
|
+
|
|
93
|
+
`probePortalParser` was rewritten and now runs for every portal, configured ones included;
|
|
94
|
+
`force_text_field` is gone from `portals.json` (8 entries) so there is one source of truth.
|
|
95
|
+
The old probe asked `data OR dati`, two words common enough to saturate: on Milano `data`
|
|
96
|
+
alone and `data OR dati` both return 2564, so it read the portal as healthy while
|
|
97
|
+
`aria OR acqua` returned 0 against 54 and 33 for the single terms. It now picks two terms
|
|
98
|
+
from the catalog itself — single-word tag facets between 0.5% and 30%, falling back to
|
|
99
|
+
frequent title words — and compares `A`, `B`, `A OR B`, `text:(A OR B)`. An OR returning
|
|
100
|
+
fewer hits than either operand is not being honoured, and the wrapper is the answer only if
|
|
101
|
+
the wrapped form returns more. `data.stadt-zuerich.ch`, where the `text` field returns 0 for
|
|
102
|
+
every query, is correctly left alone.
|
|
103
|
+
|
|
104
|
+
Cost: a plain query pays nothing, since nothing but a boolean query can be wrapped. The
|
|
105
|
+
probe costs 5 extra `rows=0` calls the first time a boolean query reaches a portal in a
|
|
106
|
+
session, then nothing.
|
|
107
|
+
|
|
108
|
+
Verified e2e against live portals: dati.gov.it 5 -> 22 and 0 -> 9 on the two queries from
|
|
109
|
+
the telemetry, Milano `aria OR acqua` 0 -> 87, Zurich unchanged at 10 and 172. 516 tests
|
|
110
|
+
pass, 18 added.
|
|
111
|
+
|
|
5
112
|
### v0.4.121 - clearer 404 on datastore_search_sql
|
|
6
113
|
|
|
7
114
|
Patch release for the fix in #532: a 404 on `datastore_search_sql` now says the portal does not expose the SQL endpoint, instead of sending the caller to `ckan_package_show` for a resource_id that was already valid. Also ships the telemetry pipeline fixes and the DEPLOYMENT.md/CLAUDE.md realignment from earlier today.
|
package/dist/index.js
CHANGED
|
@@ -33,9 +33,6 @@ var portals_default = {
|
|
|
33
33
|
"https://dati.gov.it",
|
|
34
34
|
"https://www.dati.gov.it"
|
|
35
35
|
],
|
|
36
|
-
search: {
|
|
37
|
-
force_text_field: true
|
|
38
|
-
},
|
|
39
36
|
hvd: {
|
|
40
37
|
category_field: "hvd_category"
|
|
41
38
|
},
|
|
@@ -52,10 +49,7 @@ var portals_default = {
|
|
|
52
49
|
api_url: "https://dati.arpae.it",
|
|
53
50
|
api_url_aliases: [
|
|
54
51
|
"http://dati.arpae.it"
|
|
55
|
-
]
|
|
56
|
-
search: {
|
|
57
|
-
force_text_field: true
|
|
58
|
-
}
|
|
52
|
+
]
|
|
59
53
|
},
|
|
60
54
|
{
|
|
61
55
|
id: "anac-opendata",
|
|
@@ -77,9 +71,6 @@ var portals_default = {
|
|
|
77
71
|
"http://www.data.gov.uk"
|
|
78
72
|
],
|
|
79
73
|
api_path: "/api/action",
|
|
80
|
-
search: {
|
|
81
|
-
force_text_field: true
|
|
82
|
-
},
|
|
83
74
|
dataset_view_url: "https://www.data.gov.uk/dataset/{id}/{name}",
|
|
84
75
|
organization_view_url: "https://www.data.gov.uk/organization/{name}"
|
|
85
76
|
},
|
|
@@ -89,10 +80,7 @@ var portals_default = {
|
|
|
89
80
|
api_url: "https://catalog.data.gov",
|
|
90
81
|
api_url_aliases: [
|
|
91
82
|
"http://catalog.data.gov"
|
|
92
|
-
]
|
|
93
|
-
search: {
|
|
94
|
-
force_text_field: true
|
|
95
|
-
}
|
|
83
|
+
]
|
|
96
84
|
},
|
|
97
85
|
{
|
|
98
86
|
id: "open-canada",
|
|
@@ -102,10 +90,7 @@ var portals_default = {
|
|
|
102
90
|
"http://open.canada.ca/data",
|
|
103
91
|
"https://open.canada.ca",
|
|
104
92
|
"http://open.canada.ca"
|
|
105
|
-
]
|
|
106
|
-
search: {
|
|
107
|
-
force_text_field: true
|
|
108
|
-
}
|
|
93
|
+
]
|
|
109
94
|
},
|
|
110
95
|
{
|
|
111
96
|
id: "data-gov-au",
|
|
@@ -147,10 +132,7 @@ var portals_default = {
|
|
|
147
132
|
api_url: "https://data.gov.ua",
|
|
148
133
|
api_url_aliases: [
|
|
149
134
|
"http://data.gov.ua"
|
|
150
|
-
]
|
|
151
|
-
search: {
|
|
152
|
-
force_text_field: false
|
|
153
|
-
}
|
|
135
|
+
]
|
|
154
136
|
},
|
|
155
137
|
{
|
|
156
138
|
id: "dati-regione-sicilia",
|
|
@@ -158,10 +140,7 @@ var portals_default = {
|
|
|
158
140
|
api_url: "https://dati.regione.sicilia.it",
|
|
159
141
|
api_url_aliases: [
|
|
160
142
|
"http://dati.regione.sicilia.it"
|
|
161
|
-
]
|
|
162
|
-
search: {
|
|
163
|
-
force_text_field: false
|
|
164
|
-
}
|
|
143
|
+
]
|
|
165
144
|
},
|
|
166
145
|
{
|
|
167
146
|
id: "bdap-rgs-mef",
|
|
@@ -190,10 +169,7 @@ var portals_default = {
|
|
|
190
169
|
],
|
|
191
170
|
defaults: {
|
|
192
171
|
dataset_view_url: "{server_url}/dataset/{name}",
|
|
193
|
-
organization_view_url: "{server_url}/organization/{name}"
|
|
194
|
-
search: {
|
|
195
|
-
force_text_field: false
|
|
196
|
-
}
|
|
172
|
+
organization_view_url: "{server_url}/organization/{name}"
|
|
197
173
|
}
|
|
198
174
|
};
|
|
199
175
|
|
|
@@ -224,10 +200,6 @@ function getPortalSearchConfig(serverUrl) {
|
|
|
224
200
|
force_text_field: portal?.search?.force_text_field ?? defaults.force_text_field ?? false
|
|
225
201
|
};
|
|
226
202
|
}
|
|
227
|
-
function isPortalSearchExplicitlyConfigured(serverUrl) {
|
|
228
|
-
const portal = getPortalConfig(serverUrl);
|
|
229
|
-
return portal?.search?.force_text_field !== void 0;
|
|
230
|
-
}
|
|
231
203
|
function normalizePortalUrl(serverUrl) {
|
|
232
204
|
return normalizeUrl(serverUrl);
|
|
233
205
|
}
|
|
@@ -1222,12 +1194,51 @@ function getOrganizationViewUrl(serverUrl, org) {
|
|
|
1222
1194
|
// src/utils/search.ts
|
|
1223
1195
|
var DEFAULT_SEARCH_QUERY = "*:*";
|
|
1224
1196
|
var FIELD_QUERY_PATTERN = /\b[a-zA-Z_][\w-]*:/;
|
|
1197
|
+
var EXPLICIT_BOOL_PATTERN = /\b(AND|OR|NOT)\b|[+\-!]/;
|
|
1198
|
+
var BOOL_KEYWORD_PATTERN = /\b(AND|OR|NOT)\b/;
|
|
1199
|
+
var INTRAWORD_PUNCT_PATTERN = /\S[+\-!]\S/;
|
|
1225
1200
|
var SOLR_SPECIAL_CHARS = /[+\-!(){}[\]^~*?:\\/|&]/g;
|
|
1226
1201
|
function isFieldedQuery(query) {
|
|
1227
1202
|
return FIELD_QUERY_PATTERN.test(query);
|
|
1228
1203
|
}
|
|
1229
|
-
function
|
|
1230
|
-
|
|
1204
|
+
function hasExplicitBooleanOperator(query) {
|
|
1205
|
+
const trimmed = query.trim();
|
|
1206
|
+
if (BOOL_KEYWORD_PATTERN.test(trimmed)) return true;
|
|
1207
|
+
return INTRAWORD_PUNCT_PATTERN.test(trimmed);
|
|
1208
|
+
}
|
|
1209
|
+
function mayNeedTextWrapping(query) {
|
|
1210
|
+
const trimmed = query.trim();
|
|
1211
|
+
return trimmed !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmed) && hasExplicitBooleanOperator(trimmed);
|
|
1212
|
+
}
|
|
1213
|
+
function isEscaped(query, offset) {
|
|
1214
|
+
let backslashes = 0;
|
|
1215
|
+
for (let i = offset - 1; i >= 0 && query[i] === "\\"; i--) backslashes++;
|
|
1216
|
+
return backslashes % 2 === 1;
|
|
1217
|
+
}
|
|
1218
|
+
function hasBalancedParens(query) {
|
|
1219
|
+
let depth = 0;
|
|
1220
|
+
for (let i = 0; i < query.length; i++) {
|
|
1221
|
+
if (isEscaped(query, i)) continue;
|
|
1222
|
+
if (query[i] === "(") depth++;
|
|
1223
|
+
else if (query[i] === ")" && --depth < 0) return false;
|
|
1224
|
+
}
|
|
1225
|
+
return depth === 0;
|
|
1226
|
+
}
|
|
1227
|
+
function escapeForTextWrapping(query) {
|
|
1228
|
+
const keepGroups = hasBalancedParens(query);
|
|
1229
|
+
return query.replace(
|
|
1230
|
+
SOLR_SPECIAL_CHARS,
|
|
1231
|
+
(char, offset, whole) => {
|
|
1232
|
+
if (keepGroups && (char === "(" || char === ")") && !isEscaped(whole, offset)) {
|
|
1233
|
+
return char;
|
|
1234
|
+
}
|
|
1235
|
+
const isUnary = char === "+" || char === "-" || char === "!";
|
|
1236
|
+
const previous = offset === 0 ? "" : whole[offset - 1];
|
|
1237
|
+
const atTermStart = offset === 0 || /\s/.test(previous) || keepGroups && previous === "(";
|
|
1238
|
+
const followedByTerm = offset + 1 < whole.length && !/\s/.test(whole[offset + 1]);
|
|
1239
|
+
return isUnary && atTermStart && followedByTerm ? char : `\\${char}`;
|
|
1240
|
+
}
|
|
1241
|
+
);
|
|
1231
1242
|
}
|
|
1232
1243
|
function convertDateMathForUnsupportedFields(query) {
|
|
1233
1244
|
const now = /* @__PURE__ */ new Date();
|
|
@@ -1254,7 +1265,6 @@ function convertDateMathForUnsupportedFields(query) {
|
|
|
1254
1265
|
return `${field}:[${startIso} TO ${nowIso}]`;
|
|
1255
1266
|
});
|
|
1256
1267
|
}
|
|
1257
|
-
var EXPLICIT_BOOL_PATTERN = /\b(AND|OR|NOT)\b|[+\-!]/;
|
|
1258
1268
|
function isPlainMultiTermQuery(query) {
|
|
1259
1269
|
const trimmed = query.trim();
|
|
1260
1270
|
if (trimmed === "*:*" || trimmed === "") return false;
|
|
@@ -1283,27 +1293,69 @@ function resolveSearchQuery(serverUrl, query, parserOverride) {
|
|
|
1283
1293
|
forceTextField = false;
|
|
1284
1294
|
} else if (portalForce) {
|
|
1285
1295
|
const trimmedQuery = query.trim();
|
|
1286
|
-
forceTextField = trimmedQuery !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmedQuery);
|
|
1296
|
+
forceTextField = trimmedQuery !== DEFAULT_SEARCH_QUERY && !isFieldedQuery(trimmedQuery) && hasExplicitBooleanOperator(trimmedQuery);
|
|
1287
1297
|
}
|
|
1288
|
-
let effectiveQuery = forceTextField ? `text:(${
|
|
1298
|
+
let effectiveQuery = forceTextField ? `text:(${escapeForTextWrapping(query)})` : query;
|
|
1289
1299
|
effectiveQuery = convertDateMathForUnsupportedFields(effectiveQuery);
|
|
1290
1300
|
return { effectiveQuery, forcedTextField: forceTextField };
|
|
1291
1301
|
}
|
|
1292
1302
|
|
|
1293
1303
|
// src/tools/package.ts
|
|
1294
1304
|
var _portalParserCache = /* @__PURE__ */ new Map();
|
|
1305
|
+
var LITERAL_TERM = /^[\p{L}\p{N}_]+$/u;
|
|
1306
|
+
async function pickProbeTerms(serverUrl) {
|
|
1307
|
+
const facetRes = await makeCkanRequest(serverUrl, "package_search", {
|
|
1308
|
+
q: "*:*",
|
|
1309
|
+
rows: 0,
|
|
1310
|
+
"facet.field": '["tags"]',
|
|
1311
|
+
"facet.limit": 100
|
|
1312
|
+
}).catch(() => null);
|
|
1313
|
+
const total = facetRes?.count ?? 0;
|
|
1314
|
+
if (!total) return null;
|
|
1315
|
+
const items = facetRes?.search_facets?.tags?.items ?? [];
|
|
1316
|
+
const usable = items.filter((i) => typeof i.name === "string" && LITERAL_TERM.test(i.name)).filter((i) => (i.count ?? 0) >= total * 5e-3 && (i.count ?? 0) <= total * 0.3).sort((a, b) => (b.count ?? 0) - (a.count ?? 0));
|
|
1317
|
+
if (usable.length >= 2) return [usable[0].name, usable[1].name];
|
|
1318
|
+
const sampleRes = await makeCkanRequest(serverUrl, "package_search", {
|
|
1319
|
+
q: "*:*",
|
|
1320
|
+
rows: 25
|
|
1321
|
+
}).catch(() => null);
|
|
1322
|
+
const titles = (sampleRes?.results ?? []).map((r) => r?.title ?? "");
|
|
1323
|
+
const words = titles.map((t) => new Set(t.toLowerCase().split(/[^\p{L}\p{N}_]+/u).filter((w) => w.length > 4)));
|
|
1324
|
+
const freq = /* @__PURE__ */ new Map();
|
|
1325
|
+
for (const set of words) for (const w of set) freq.set(w, (freq.get(w) ?? 0) + 1);
|
|
1326
|
+
const ranked = [...freq.entries()].filter(([w]) => LITERAL_TERM.test(w)).sort((a, b) => b[1] - a[1]);
|
|
1327
|
+
const first = ranked[0]?.[0];
|
|
1328
|
+
if (!first) return null;
|
|
1329
|
+
const independent = ranked.slice(1).find(([w]) => words.some((set) => set.has(w) && !set.has(first)));
|
|
1330
|
+
const second = independent?.[0] ?? ranked[1]?.[0];
|
|
1331
|
+
return second ? [first, second] : null;
|
|
1332
|
+
}
|
|
1295
1333
|
async function probePortalParser(serverUrl) {
|
|
1296
1334
|
const key = serverUrl.replace(/\/$/, "").toLowerCase();
|
|
1297
1335
|
if (_portalParserCache.has(key)) return _portalParserCache.get(key);
|
|
1298
|
-
const
|
|
1299
|
-
|
|
1300
|
-
|
|
1301
|
-
|
|
1302
|
-
|
|
1303
|
-
|
|
1304
|
-
|
|
1305
|
-
|
|
1306
|
-
|
|
1336
|
+
const terms = await pickProbeTerms(serverUrl).catch(() => null);
|
|
1337
|
+
if (!terms) return false;
|
|
1338
|
+
let needsText = false;
|
|
1339
|
+
let measured = false;
|
|
1340
|
+
{
|
|
1341
|
+
const [a, b] = terms;
|
|
1342
|
+
const count = async (q) => {
|
|
1343
|
+
const res = await makeCkanRequest(serverUrl, "package_search", { q, rows: 0 }).catch(() => null);
|
|
1344
|
+
return typeof res?.count === "number" ? res.count : null;
|
|
1345
|
+
};
|
|
1346
|
+
const [ca, cb, cOr, cText] = await Promise.all([
|
|
1347
|
+
count(a),
|
|
1348
|
+
count(b),
|
|
1349
|
+
count(`${a} OR ${b}`),
|
|
1350
|
+
count(`text:(${a} OR ${b})`)
|
|
1351
|
+
]);
|
|
1352
|
+
if (ca !== null && cb !== null && cOr !== null && cText !== null) {
|
|
1353
|
+
const booleanIgnored = cOr < Math.max(ca, cb);
|
|
1354
|
+
needsText = booleanIgnored && cText > cOr;
|
|
1355
|
+
measured = true;
|
|
1356
|
+
}
|
|
1357
|
+
}
|
|
1358
|
+
if (measured) _portalParserCache.set(key, needsText);
|
|
1307
1359
|
return needsText;
|
|
1308
1360
|
}
|
|
1309
1361
|
var DEFAULT_RELEVANCE_WEIGHTS = {
|
|
@@ -1315,6 +1367,41 @@ var DEFAULT_RELEVANCE_WEIGHTS = {
|
|
|
1315
1367
|
publisher: 2
|
|
1316
1368
|
};
|
|
1317
1369
|
var QUERY_STOPWORDS = /* @__PURE__ */ new Set([
|
|
1370
|
+
// Italian: without these, `defibrillatori Comune di Lecce` scored a full holder
|
|
1371
|
+
// match against "Provincia Autonoma di Trento" on the strength of "di" alone.
|
|
1372
|
+
"di",
|
|
1373
|
+
"del",
|
|
1374
|
+
"dello",
|
|
1375
|
+
"della",
|
|
1376
|
+
"dei",
|
|
1377
|
+
"degli",
|
|
1378
|
+
"delle",
|
|
1379
|
+
"il",
|
|
1380
|
+
"lo",
|
|
1381
|
+
"la",
|
|
1382
|
+
"i",
|
|
1383
|
+
"gli",
|
|
1384
|
+
"le",
|
|
1385
|
+
"un",
|
|
1386
|
+
"uno",
|
|
1387
|
+
"una",
|
|
1388
|
+
"e",
|
|
1389
|
+
"ed",
|
|
1390
|
+
"per",
|
|
1391
|
+
"con",
|
|
1392
|
+
"su",
|
|
1393
|
+
"da",
|
|
1394
|
+
"dal",
|
|
1395
|
+
"dalla",
|
|
1396
|
+
"nel",
|
|
1397
|
+
"nella",
|
|
1398
|
+
"al",
|
|
1399
|
+
"alla",
|
|
1400
|
+
"che",
|
|
1401
|
+
"non",
|
|
1402
|
+
"come",
|
|
1403
|
+
"dove",
|
|
1404
|
+
"sono",
|
|
1318
1405
|
"a",
|
|
1319
1406
|
"an",
|
|
1320
1407
|
"the",
|
|
@@ -1358,18 +1445,32 @@ var QUERY_STOPWORDS = /* @__PURE__ */ new Set([
|
|
|
1358
1445
|
"those"
|
|
1359
1446
|
]);
|
|
1360
1447
|
var extractQueryTerms = (query) => {
|
|
1361
|
-
const
|
|
1362
|
-
const terms =
|
|
1448
|
+
const raw = query.normalize("NFC").match(/[\p{L}\p{N}]+/gu) ?? [];
|
|
1449
|
+
const terms = raw.filter((token) => {
|
|
1450
|
+
const term = token.toLowerCase();
|
|
1451
|
+
if (term.length <= 1) return false;
|
|
1452
|
+
if (!QUERY_STOPWORDS.has(term)) return true;
|
|
1453
|
+
return token.length > 1 && token === token.toUpperCase() && token !== term;
|
|
1454
|
+
}).map((token) => token.toLowerCase());
|
|
1363
1455
|
return Array.from(new Set(terms));
|
|
1364
1456
|
};
|
|
1365
1457
|
var escapeRegExp = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
1458
|
+
var termPattern = (term) => new RegExp(`(?<![\\p{L}\\p{N}])${escapeRegExp(term.normalize("NFC"))}(?![\\p{L}\\p{N}])`, "iu");
|
|
1459
|
+
var normalizeForMatch = (text) => text.normalize("NFC").toLowerCase().replace(/_/g, " ");
|
|
1366
1460
|
var textMatchesTerms = (text, terms) => {
|
|
1367
1461
|
if (!text || terms.length === 0) return false;
|
|
1368
|
-
const normalized = text
|
|
1369
|
-
return terms.some((term) =>
|
|
1462
|
+
const normalized = normalizeForMatch(text);
|
|
1463
|
+
return terms.some((term) => termPattern(term).test(normalized));
|
|
1464
|
+
};
|
|
1465
|
+
var countMatchingTerms = (text, terms) => {
|
|
1466
|
+
if (!text || terms.length === 0) return 0;
|
|
1467
|
+
const normalized = normalizeForMatch(text);
|
|
1468
|
+
return terms.filter((term) => termPattern(term).test(normalized)).length;
|
|
1370
1469
|
};
|
|
1371
1470
|
var scoreTextField = (text, terms, weight) => {
|
|
1372
|
-
|
|
1471
|
+
const matched = countMatchingTerms(text, terms);
|
|
1472
|
+
if (matched === 0) return 0;
|
|
1473
|
+
return Math.round(weight * matched / terms.length * 10) / 10;
|
|
1373
1474
|
};
|
|
1374
1475
|
var readDcatExtra = (dataset, key) => {
|
|
1375
1476
|
const extras = Array.isArray(dataset.extras) ? dataset.extras : [];
|
|
@@ -1934,7 +2035,7 @@ Typical workflow: ckan_status_show (check locale) \u2192 ckan_package_search (qu
|
|
|
1934
2035
|
if (!effectiveSort) effectiveSort = "issued desc, metadata_created desc";
|
|
1935
2036
|
}
|
|
1936
2037
|
let parserOverride = params.query_parser;
|
|
1937
|
-
if (!parserOverride &&
|
|
2038
|
+
if (!parserOverride && mayNeedTextWrapping(query)) {
|
|
1938
2039
|
const needsText = await probePortalParser(params.server_url);
|
|
1939
2040
|
if (needsText) parserOverride = "text";
|
|
1940
2041
|
}
|
|
@@ -1967,7 +2068,10 @@ Typical workflow: ckan_status_show (check locale) \u2192 ckan_package_search (qu
|
|
|
1967
2068
|
const { effectiveQuery: strippedEffective } = resolveSearchQuery(
|
|
1968
2069
|
params.server_url,
|
|
1969
2070
|
strippedQuery,
|
|
1970
|
-
params.query_parser
|
|
2071
|
+
// parserOverride, not params.query_parser: the probe's verdict must
|
|
2072
|
+
// survive the retry, or an accented boolean query is re-sent to the
|
|
2073
|
+
// parser that ignores booleans.
|
|
2074
|
+
parserOverride
|
|
1971
2075
|
);
|
|
1972
2076
|
const fallbackResult = await makeCkanRequest(
|
|
1973
2077
|
params.server_url,
|
|
@@ -2092,7 +2196,7 @@ Note: showing top ${sorted.length} only. Use \`response_format: json\` for full
|
|
|
2092
2196
|
`;
|
|
2093
2197
|
if (isPlainMultiTermQuery(params.q)) {
|
|
2094
2198
|
markdown += `
|
|
2095
|
-
> **Tip**:
|
|
2199
|
+
> **Tip**: With several terms the portal requires most of them to match. Spelling out OR broadens the search:
|
|
2096
2200
|
`;
|
|
2097
2201
|
markdown += `> \`q: "${buildOrQuery(params.q)}"\`
|
|
2098
2202
|
`;
|
|
@@ -2187,11 +2291,16 @@ Typical workflow: ckan_find_relevant_datasets \u2192 ckan_package_show (inspect
|
|
|
2187
2291
|
...DEFAULT_RELEVANCE_WEIGHTS,
|
|
2188
2292
|
...params.weights ?? {}
|
|
2189
2293
|
};
|
|
2190
|
-
const rows = Math.min(Math.max(params.limit * 5,
|
|
2294
|
+
const rows = Math.min(Math.max(params.limit * 5, 50), 100);
|
|
2295
|
+
let parserOverride = params.query_parser;
|
|
2296
|
+
if (!parserOverride && mayNeedTextWrapping(params.query)) {
|
|
2297
|
+
const needsText = await probePortalParser(params.server_url);
|
|
2298
|
+
if (needsText) parserOverride = "text";
|
|
2299
|
+
}
|
|
2191
2300
|
const { effectiveQuery } = resolveSearchQuery(
|
|
2192
2301
|
params.server_url,
|
|
2193
2302
|
params.query,
|
|
2194
|
-
|
|
2303
|
+
parserOverride
|
|
2195
2304
|
);
|
|
2196
2305
|
const searchResult = await makeCkanRequest(
|
|
2197
2306
|
params.server_url,
|
|
@@ -2932,7 +3041,7 @@ Typical workflow: ckan_organization_search \u2192 ckan_organization_show (get de
|
|
|
2932
3041
|
},
|
|
2933
3042
|
async (params) => {
|
|
2934
3043
|
try {
|
|
2935
|
-
const query = `organization:*${params.pattern.toLowerCase()}*`;
|
|
3044
|
+
const query = `organization:*${stripAccents(params.pattern.toLowerCase())}*`;
|
|
2936
3045
|
const result = await makeCkanRequest(
|
|
2937
3046
|
params.server_url,
|
|
2938
3047
|
"package_search",
|
|
@@ -3436,7 +3545,13 @@ Typical workflow: ckan_tag_list \u2192 ckan_package_search with fq="tags:tag_nam
|
|
|
3436
3545
|
q: params.q,
|
|
3437
3546
|
rows: 0,
|
|
3438
3547
|
"facet.field": JSON.stringify(["tags"]),
|
|
3439
|
-
|
|
3548
|
+
// The filter runs on whatever the facet returned, so with a tag_query the
|
|
3549
|
+
// facet has to be wider than the caller's limit: on dati.gov.it 53 tags
|
|
3550
|
+
// contain "citta" and none is in the top 100, so asking for 100 answered
|
|
3551
|
+
// "no tags" while they existed. CKAN rejects Solr's own `facet.contains`
|
|
3552
|
+
// ("Invalid search parameters"), so the filtering cannot move server-side;
|
|
3553
|
+
// this window is the compromise, and WIDE_FACET below is the escape hatch.
|
|
3554
|
+
"facet.limit": params.tag_query ? Math.max(params.limit * 20, 500) : params.limit
|
|
3440
3555
|
};
|
|
3441
3556
|
if (params.fq) apiParams.fq = params.fq;
|
|
3442
3557
|
const result = await makeCkanRequest(
|
|
@@ -3446,8 +3561,18 @@ Typical workflow: ckan_tag_list \u2192 ckan_package_search with fq="tags:tag_nam
|
|
|
3446
3561
|
);
|
|
3447
3562
|
let tags = normalizeTagFacets(result);
|
|
3448
3563
|
if (params.tag_query) {
|
|
3449
|
-
const needle = params.tag_query.toLowerCase();
|
|
3450
|
-
|
|
3564
|
+
const needle = stripAccents(params.tag_query.toLowerCase());
|
|
3565
|
+
const matching = (candidates) => candidates.filter((tag) => stripAccents(tag.name.toLowerCase()).includes(needle));
|
|
3566
|
+
tags = matching(tags);
|
|
3567
|
+
if (tags.length < params.limit) {
|
|
3568
|
+
const wide = await makeCkanRequest(
|
|
3569
|
+
params.server_url,
|
|
3570
|
+
"package_search",
|
|
3571
|
+
{ ...apiParams, "facet.limit": -1 }
|
|
3572
|
+
);
|
|
3573
|
+
tags = matching(normalizeTagFacets(wide));
|
|
3574
|
+
}
|
|
3575
|
+
tags = tags.sort((a, b) => b.count - a.count).slice(0, params.limit);
|
|
3451
3576
|
}
|
|
3452
3577
|
tags = tags.sort((a, b) => b.count - a.count || a.name.localeCompare(b.name)).slice(0, params.limit);
|
|
3453
3578
|
if (params.response_format === "json" /* JSON */) {
|
|
@@ -5802,7 +5927,7 @@ var registerAllPrompts = (server) => {
|
|
|
5802
5927
|
function createServer() {
|
|
5803
5928
|
return new McpServer({
|
|
5804
5929
|
name: "ckan-mcp-server",
|
|
5805
|
-
version: "0.4.
|
|
5930
|
+
version: "0.4.123"
|
|
5806
5931
|
});
|
|
5807
5932
|
}
|
|
5808
5933
|
function registerAll(server) {
|