@maschinenlesbar.org/ckan-cli 0.7.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. package/README.md +56 -0
  2. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/client.d.ts +25 -1
  3. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/client.js +64 -23
  4. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/engine.d.ts +26 -4
  5. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/engine.js +152 -45
  6. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/errors.d.ts +52 -6
  7. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/errors.js +155 -17
  8. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/http.d.ts +2 -1
  9. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/http.js +33 -6
  10. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/index.d.ts +2 -1
  11. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/index.js +2 -1
  12. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/operations.d.ts +441 -0
  13. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/operations.js +1211 -0
  14. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/portals.d.ts +2 -0
  15. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/portals.js +2 -2
  16. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/validate.d.ts +8 -4
  17. package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/validate.js +21 -4
  18. package/dist/node_modules/@maschinenlesbar.org/ckan-log/dist/src/index.d.ts +1 -0
  19. package/dist/node_modules/@maschinenlesbar.org/ckan-log/dist/src/index.js +1 -0
  20. package/dist/node_modules/@maschinenlesbar.org/ckan-log/dist/src/log.d.ts +80 -0
  21. package/dist/node_modules/@maschinenlesbar.org/ckan-log/dist/src/log.js +124 -0
  22. package/dist/node_modules/@maschinenlesbar.org/ckan-log/package.json +25 -0
  23. package/dist/src/cli/index.js +3 -1
  24. package/dist/src/cli/io.d.ts +17 -0
  25. package/dist/src/cli/io.js +14 -0
  26. package/dist/src/cli/limits.d.ts +19 -0
  27. package/dist/src/cli/limits.js +53 -0
  28. package/dist/src/cli/program.js +3 -0
  29. package/dist/src/cli/run.d.ts +0 -7
  30. package/dist/src/cli/run.js +17 -16
  31. package/dist/src/cli/shared.d.ts +4 -0
  32. package/dist/src/cli/shared.js +16 -1
  33. package/node_modules/commander/Readme.md +6 -32
  34. package/node_modules/commander/index.js +1 -1
  35. package/node_modules/commander/lib/command.js +22 -9
  36. package/node_modules/commander/lib/help.js +6 -2
  37. package/node_modules/commander/lib/option.js +1 -1
  38. package/node_modules/commander/package.json +2 -1
  39. package/node_modules/commander/typings/index.d.ts +6 -3
  40. package/package.json +2 -2
package/README.md CHANGED
@@ -34,6 +34,59 @@ npm i -g @maschinenlesbar.org/ckan-cli
34
34
 
35
35
  This installs the **`ckan`** command. Requires **Node.js 22.12+**.
36
36
 
37
+ ### Docker
38
+
39
+ From 0.9.0 every release is also a container image for linux/amd64 and linux/arm64 on the
40
+ GitHub Container Registry, made from the same tarball as the npm package. It needs no Node.js and no
41
+ install:
42
+
43
+ ```bash
44
+ docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
45
+ ghcr.io/maschinenlesbar-org/ckan-cli:latest --portal berlin search Radverkehr --rows 5
46
+ ```
47
+
48
+ The image's entrypoint is the program, so the command follows the image name directly;
49
+ `… ckan-cli:latest ckan search Radverkehr` works too. Pass `CKAN_BASE_URL` with `-e` if you use it.
50
+
51
+ In a container with a memory limit (`--memory`) the default `--max-response-bytes` is an eighth
52
+ of that limit (at most the usual 100 MiB), so an oversized answer is refused with a clear
53
+ error instead of the container being killed for memory. A search with many rows of large
54
+ records can be several megabytes; most answers are small.
55
+ The CLI needs at least 12 processes and threads (`--pids-limit 12`): below that it refuses with
56
+ an error, and with 4 or fewer Node.js itself hangs before the CLI can check.
57
+
58
+ `latest` is the newest release. For scripts, name a release and pin its digest, which the
59
+ [release notes](https://github.com/maschinenlesbar-org/ckan-cli/releases) give:
60
+ `…/ckan-cli:0.9.0@sha256:<digest>`. Each image carries signed build provenance;
61
+ check it with
62
+
63
+ ```bash
64
+ gh attestation verify oci://ghcr.io/maschinenlesbar-org/ckan-cli:latest \
65
+ --repo maschinenlesbar-org/ckan-cli
66
+ ```
67
+
68
+ ### MCP server
69
+
70
+ From 0.9.0 the CKAN portals are also an [MCP](https://modelcontextprotocol.io/) server for AI
71
+ agents, `@maschinenlesbar.org/ckan-mcp` (command `ckan-mcp`, image
72
+ `ghcr.io/maschinenlesbar-org/ckan-mcp`): six read-only tools, to see which portals of the
73
+ built-in list have data on a topic (GovData apart, since it harvests the others), find datasets on
74
+ one portal with friendly filters (an organization or group by name, a file format in all its
75
+ spellings, a date range) and their licences, count a portal by organization, group, format,
76
+ licence, tag or type and per year (the spellings folded), search the Transparenzportal Hamburg's
77
+ documents by type, topic and date, read one record in full, and list a portal's organizations or
78
+ groups, on stdio or Streamable HTTP. The server's portal is Hamburg, or the one `--base-url`
79
+ names; per call a tool takes another portal of the list by its id, never a URL. Setup for Claude
80
+ Code, Claude Desktop and VS Code:
81
+ [MCP.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/MCP.md).
82
+
83
+ ```bash
84
+ npm i -g @maschinenlesbar.org/ckan-mcp
85
+ claude mcp add ckan -- ckan-mcp
86
+ ```
87
+
88
+ Every result gives each record's licence and the portal it came from, whose terms apply too.
89
+
37
90
  ## Quickstart
38
91
 
39
92
  The two examples from Hamburg's API page, as commands. A full-text search:
@@ -335,6 +388,9 @@ Transparenzportal's contracts, Gutachten and Senate papers
335
388
 
336
389
  Real runs of each skill are in [EXAMPLE.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/EXAMPLE.md).
337
390
 
391
+ The same use cases are the tools of the MCP server, for any MCP client: see
392
+ [MCP.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/MCP.md).
393
+
338
394
  ## Data license
339
395
 
340
396
  This CLI is a **client**. It accesses data it does not own or redistribute, and
@@ -1,5 +1,5 @@
1
1
  import { type EngineOptions } from "./engine.js";
2
- import type { QueryParams } from "./query.js";
2
+ import { type QueryParams } from "./query.js";
3
3
  import type { Group, GroupListParams, JsonValue, License, ListParams, Organization, Package, PackageSearchParams, PackageSearchResult, Resource, Status, TagListParams } from "./types.js";
4
4
  /**
5
5
  * Most pages one `all_fields` list call requests (10,000 entries at CKAN's page size
@@ -14,6 +14,19 @@ export declare const MAX_ALL_FIELDS_PAGES = 400;
14
14
  * second at most instead of thousands.
15
15
  */
16
16
  export declare const ALL_FIELDS_PAGE_DELAY_MS = 100;
17
+ /**
18
+ * The `package_search` parameters `packageSearch(params)` sends, checked first (CkanValidationError
19
+ * before any request: an unknown key, a refused text value, a string where a list belongs, a NaN or
20
+ * negative number would otherwise go out, or be dropped). See `packageSearch` for how each one is
21
+ * sent.
22
+ */
23
+ export declare function packageSearchQuery(params?: PackageSearchParams): QueryParams;
24
+ /**
25
+ * A request as `action(name, params)` sends it, under the Action API's base
26
+ * (`<site>/api/3/action/`): the action's name and its query string (`package_show?id=x`), the
27
+ * parameters checked and normalised as they go out. What an answer's source names.
28
+ */
29
+ export declare function actionRequest(name: string, params?: QueryParams): string;
17
30
  /**
18
31
  * The site root of a CKAN instance, from what a user is likely to paste: portals
19
32
  * document their API as `<site>/api/3/action`, so that suffix (or `/api/3`) is
@@ -25,6 +38,17 @@ export declare function siteRoot(baseUrl: string): string;
25
38
  export declare class CkanClient {
26
39
  #private;
27
40
  constructor(options?: EngineOptions);
41
+ /**
42
+ * The portal this client talks to as one identity (`portalKey`: host without `www.` and path,
43
+ * lower case; never a user name or password): to tell whether a portal of the list is this one.
44
+ */
45
+ get siteKey(): string;
46
+ /**
47
+ * A client for another CKAN portal with this one's options (transport, timeouts, retries, user
48
+ * agent, hooks): another portal of a known list, through the same transport (and so the same
49
+ * request limits). The base URL is checked as the constructor checks it.
50
+ */
51
+ withBaseUrl(baseUrl: string): CkanClient;
28
52
  /** Call any CKAN action by name and return its unwrapped `result`. */
29
53
  action<T = JsonValue>(name: string, params?: QueryParams): Promise<T>;
30
54
  /**
@@ -2,6 +2,7 @@
2
2
  // Action API (`<site>/api/3/action`).
3
3
  import { bigIntegerKeysOption, DEFAULT_BASE_URL, RequestEngine, sanitizeServerText, validateBaseUrl } from "./engine.js";
4
4
  import { CkanActionError, CkanParseError, CkanValidationError, cutForMessage, describeCkanError, quoteValue, redactUrl } from "./errors.js";
5
+ import { buildQueryString } from "./query.js";
5
6
  import { assertValid, countProblem, facetLimitProblem, fieldProblem, idProblem, listProblem, normalizeText, normalizeToken, queryValueProblem, sortProblem, textProblem, tokenProblem, } from "./validate.js";
6
7
  const ACTION = "/api/3/action";
7
8
  /**
@@ -153,6 +154,48 @@ function assertCount(name, value) {
153
154
  if (value !== undefined)
154
155
  assertValid(name, value, countProblem);
155
156
  }
157
+ /**
158
+ * The `package_search` parameters `packageSearch(params)` sends, checked first (CkanValidationError
159
+ * before any request: an unknown key, a refused text value, a string where a list belongs, a NaN or
160
+ * negative number would otherwise go out, or be dropped). See `packageSearch` for how each one is
161
+ * sent.
162
+ */
163
+ export function packageSearchQuery(params = {}) {
164
+ assertParams(params);
165
+ assertKeys("packageSearch", "package_search", params, SEARCH_KEYS);
166
+ assertOptional("q", params.q, textProblem);
167
+ assertList("fq", params.fq, textProblem);
168
+ assertOptional("sort", params.sort, sortProblem);
169
+ assertList("facet_field", params.facet_field, fieldProblem);
170
+ assertCount("rows", params.rows);
171
+ assertCount("start", params.start);
172
+ if (params.facet_limit !== undefined)
173
+ assertValid("facet_limit", params.facet_limit, facetLimitProblem);
174
+ const fq = (params.fq ?? []).map(normalizeText);
175
+ const facetFields = (params.facet_field ?? []).map(normalizeToken);
176
+ return {
177
+ q: params.q === undefined ? undefined : normalizeText(params.q),
178
+ fq: fq.length === 1 ? fq[0] : undefined,
179
+ fq_list: fq.length > 1 ? fq : undefined,
180
+ rows: params.rows,
181
+ start: params.start,
182
+ sort: params.sort === undefined ? undefined : normalizeToken(params.sort),
183
+ "facet.field": facetFields.length > 0 ? JSON.stringify(facetFields) : undefined,
184
+ "facet.limit": params.facet_limit,
185
+ };
186
+ }
187
+ /**
188
+ * A request as `action(name, params)` sends it, under the Action API's base
189
+ * (`<site>/api/3/action/`): the action's name and its query string (`package_show?id=x`), the
190
+ * parameters checked and normalised as they go out. What an answer's source names.
191
+ */
192
+ export function actionRequest(name, params = {}) {
193
+ if (typeof name !== "string" || !ACTION_NAME.test(name)) {
194
+ throw new CkanValidationError(`Invalid CKAN action name: ${quoteValue(typeof name === "string" ? name : String(name))}`);
195
+ }
196
+ const qs = buildQueryString(prune(params));
197
+ return qs === "" ? name : `${name}?${qs}`;
198
+ }
156
199
  /**
157
200
  * The site root of a CKAN instance, from what a user is likely to paste: portals
158
201
  * document their API as `<site>/api/3/action`, so that suffix (or `/api/3`) is
@@ -173,7 +216,10 @@ export class CkanClient {
173
216
  #site;
174
217
  /** The pause between `all_fields` pages (EngineOptions.sleep, for tests). */
175
218
  #sleep;
219
+ /** The options it was made with, for `withBaseUrl`. */
220
+ #options;
176
221
  constructor(options = {}) {
222
+ this.#options = options;
177
223
  this.#site = siteRoot(options.baseUrl ?? DEFAULT_BASE_URL);
178
224
  // The engine checks every option, `sleep` included, before the client uses it. Solr's
179
225
  // `_version_` (≈1.88e18, past 2^53) is always declared a big integer: a portal that sends it
@@ -185,6 +231,22 @@ export class CkanClient {
185
231
  });
186
232
  this.#sleep = options.sleep ?? realSleep;
187
233
  }
234
+ /**
235
+ * The portal this client talks to as one identity (`portalKey`: host without `www.` and path,
236
+ * lower case; never a user name or password): to tell whether a portal of the list is this one.
237
+ */
238
+ get siteKey() {
239
+ const url = new URL(this.#site);
240
+ return `${url.host.toLowerCase().replace(/^www\./, "")}${url.pathname.replace(/\/+$/, "").toLowerCase()}`;
241
+ }
242
+ /**
243
+ * A client for another CKAN portal with this one's options (transport, timeouts, retries, user
244
+ * agent, hooks): another portal of a known list, through the same transport (and so the same
245
+ * request limits). The base URL is checked as the constructor checks it.
246
+ */
247
+ withBaseUrl(baseUrl) {
248
+ return new CkanClient({ ...this.#options, baseUrl });
249
+ }
188
250
  /** Call any CKAN action by name and return its unwrapped `result`. */
189
251
  async action(name, params = {}) {
190
252
  return (await this.call(name, params)).result;
@@ -208,7 +270,7 @@ export class CkanClient {
208
270
  // A success:false envelope can come with HTTP 200, so it never passes the
209
271
  // engine's error-detail sanitising: strip terminal controls here too.
210
272
  const type = env.error?.__type;
211
- throw new CkanActionError(`${request}: CKAN action "${cutForMessage(name)}" failed: ${cutForMessage(sanitizeServerText(this.#engine.scrub(describeCkanError(env.error))))}`, { action: name, errorType: typeof type === "string" ? cutForMessage(sanitizeServerText(type)) : undefined, url });
273
+ throw new CkanActionError(`${request}: CKAN action "${cutForMessage(name)}" failed: ${cutForMessage(sanitizeServerText(this.#engine.scrub(describeCkanError(env.error))))}`, { action: name, errorType: typeof type === "string" ? cutForMessage(sanitizeServerText(this.#engine.scrub(type))) : undefined, url });
212
274
  }
213
275
  // `{"success": true}` without a result would print nothing useful (and the CLI
214
276
  // would crash rendering `undefined`); CKAN always sends one, `null` included.
@@ -250,28 +312,7 @@ export class CkanClient {
250
312
  * a Solr field name) as tokens (`normalizeToken`).
251
313
  */
252
314
  async packageSearch(params = {}) {
253
- assertParams(params);
254
- assertKeys("packageSearch", "package_search", params, SEARCH_KEYS);
255
- assertOptional("q", params.q, textProblem);
256
- assertList("fq", params.fq, textProblem);
257
- assertOptional("sort", params.sort, sortProblem);
258
- assertList("facet_field", params.facet_field, fieldProblem);
259
- assertCount("rows", params.rows);
260
- assertCount("start", params.start);
261
- if (params.facet_limit !== undefined)
262
- assertValid("facet_limit", params.facet_limit, facetLimitProblem);
263
- const fq = (params.fq ?? []).map(normalizeText);
264
- const facetFields = (params.facet_field ?? []).map(normalizeToken);
265
- return this.typed("package_search", {
266
- q: params.q === undefined ? undefined : normalizeText(params.q),
267
- fq: fq.length === 1 ? fq[0] : undefined,
268
- fq_list: fq.length > 1 ? fq : undefined,
269
- rows: params.rows,
270
- start: params.start,
271
- sort: params.sort === undefined ? undefined : normalizeToken(params.sort),
272
- "facet.field": facetFields.length > 0 ? JSON.stringify(facetFields) : undefined,
273
- "facet.limit": params.facet_limit,
274
- }, isSearchResult, "an object with a numeric count and a results array");
315
+ return this.typed("package_search", packageSearchQuery(params), isSearchResult, "an object with a numeric count and a results array");
275
316
  }
276
317
  /** A single dataset by id or name. */
277
318
  packageShow(id) {
@@ -131,19 +131,25 @@ export declare function isBidiControl(code: number): boolean;
131
131
  * `detail`, a CKAN `success:false` error, the echoed Content-Type — safe to print
132
132
  * into an error message on stderr:
133
133
  *
134
- * - C0 and C1 controls and DEL are dropped. A JSON error body can encode an escape
134
+ * - Whole escape sequences (CSI, OSC and other control strings) are dropped first,
135
+ * then any remaining C0 and C1 controls and DEL. A JSON error body can encode an escape
135
136
  * (U+001B) that JSON.parse turns into a real control byte; printed raw, a hostile
136
137
  * or MITM'd endpoint could drive ANSI/OSC sequences into the terminal (display
137
138
  * spoofing, title changes, OSC 52 clipboard writes).
138
- * - Bidi formatting characters (isBidiControl) are dropped, so server text cannot
139
- * reorder the visible message.
139
+ * - Unicode format characters (General Category Cf) are dropped: the bidi controls
140
+ * (isBidiControl), so server text cannot reorder the visible message, and the
141
+ * invisible ones — zero-width space/joiners, word joiner, soft hyphen, BOM — so
142
+ * two messages that look identical are identical.
140
143
  * - Every run of whitespace — newlines, tabs, U+2028/U+2029 included — becomes one
141
144
  * space and the ends are trimmed, so the text stays on one line and a server
142
145
  * cannot forge an `Error:` line of its own.
146
+ * - Only the first MAX_SERVER_TEXT (16 384) units are read, and every step is a linear
147
+ * regex replace: a hostile error body costs neither seconds nor the heap. A caller that
148
+ * scrubs credentials does so first, so a cut never splits one.
143
149
  *
144
150
  * The CLI's JSON output is escaped separately (`escapeControlChars` in
145
151
  * cli/shared.ts): `JSON.stringify` alone leaves DEL, C1 and bidi characters raw.
146
- * Written as a char-code filter so no raw control byte appears in this source.
152
+ * Written with escapes so no raw control byte appears in this source.
147
153
  */
148
154
  export declare function sanitizeServerText(text: string): string;
149
155
  /**
@@ -200,6 +206,15 @@ export declare class RequestEngine {
200
206
  * messages built from server text.
201
207
  */
202
208
  scrub(text: string): string;
209
+ /**
210
+ * An error body, already scrubbed as sent, as `CkanApiError.body` keeps it: when it is JSON
211
+ * with an escape in it, checked decoded too. An escape (`\"`, `\u00e4`) hides an echoed
212
+ * credential from the scrub of the text as sent, while a caller who parses the body sees it. Such
213
+ * a body is kept as the decoded JSON written again (`JSON.stringify`, one spelling per character)
214
+ * with the credential scrubbed in that spelling. A body that doesn't parse is text nobody
215
+ * decodes, and stays as sent; one too deeply nested to write again is left out.
216
+ */
217
+ private scrubBody;
203
218
  /**
204
219
  * A failure as the `cause` of the error the engine raises: the original when its
205
220
  * text carries no credentials, otherwise a copy with them scrubbed (message, `code`
@@ -223,6 +238,13 @@ export declare class RequestEngine {
223
238
  * becomes a rejection.
224
239
  */
225
240
  private callTransport;
241
+ /**
242
+ * The URL a redirect leads to as messages and errors show it: the Location with the base URL's
243
+ * credentials scrubbed as sent, before resolving percent-encodes what is left of them in a new
244
+ * spelling (`'` as `%27` in a query), then resolved against the URL shown so far, its userinfo
245
+ * left out. A Location the scrub made unparseable (a credential in its host) is shown as scrubbed.
246
+ */
247
+ private shownTarget;
226
248
  /** Perform a request with Accept negotiation and transient-error retries. */
227
249
  request(method: string, path: string, options?: {
228
250
  query?: QueryParams;