@maschinenlesbar.org/ckan-cli 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +56 -0
- package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/client.d.ts +25 -1
- package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/client.js +63 -22
- package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/index.d.ts +2 -1
- package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/index.js +2 -1
- package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/operations.d.ts +441 -0
- package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/operations.js +1211 -0
- package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/portals.d.ts +2 -0
- package/dist/node_modules/@maschinenlesbar.org/ckan-client/dist/src/portals.js +2 -2
- package/dist/node_modules/@maschinenlesbar.org/ckan-log/dist/src/index.d.ts +1 -0
- package/dist/node_modules/@maschinenlesbar.org/ckan-log/dist/src/index.js +1 -0
- package/dist/node_modules/@maschinenlesbar.org/ckan-log/dist/src/log.d.ts +80 -0
- package/dist/node_modules/@maschinenlesbar.org/ckan-log/dist/src/log.js +124 -0
- package/dist/node_modules/@maschinenlesbar.org/ckan-log/package.json +25 -0
- package/dist/src/cli/index.js +3 -1
- package/dist/src/cli/io.d.ts +17 -0
- package/dist/src/cli/io.js +14 -0
- package/dist/src/cli/limits.d.ts +19 -0
- package/dist/src/cli/limits.js +53 -0
- package/dist/src/cli/program.js +3 -0
- package/dist/src/cli/run.js +9 -1
- package/dist/src/cli/shared.d.ts +4 -0
- package/dist/src/cli/shared.js +16 -1
- package/node_modules/commander/Readme.md +6 -32
- package/node_modules/commander/index.js +1 -1
- package/node_modules/commander/lib/command.js +22 -9
- package/node_modules/commander/lib/help.js +6 -2
- package/node_modules/commander/lib/option.js +1 -1
- package/node_modules/commander/package.json +2 -1
- package/node_modules/commander/typings/index.d.ts +6 -3
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -34,6 +34,59 @@ npm i -g @maschinenlesbar.org/ckan-cli
|
|
|
34
34
|
|
|
35
35
|
This installs the **`ckan`** command. Requires **Node.js 22.12+**.
|
|
36
36
|
|
|
37
|
+
### Docker
|
|
38
|
+
|
|
39
|
+
From 0.9.0 every release is also a container image for linux/amd64 and linux/arm64 on the
|
|
40
|
+
GitHub Container Registry, made from the same tarball as the npm package. It needs no Node.js and no
|
|
41
|
+
install:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
|
|
45
|
+
ghcr.io/maschinenlesbar-org/ckan-cli:latest --portal berlin search Radverkehr --rows 5
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The image's entrypoint is the program, so the command follows the image name directly;
|
|
49
|
+
`… ckan-cli:latest ckan search Radverkehr` works too. Pass `CKAN_BASE_URL` with `-e` if you use it.
|
|
50
|
+
|
|
51
|
+
In a container with a memory limit (`--memory`) the default `--max-response-bytes` is an eighth
|
|
52
|
+
of that limit (at most the usual 100 MiB), so an oversized answer is refused with a clear
|
|
53
|
+
error instead of the container being killed for memory. A search with many rows of large
|
|
54
|
+
records can be several megabytes; most answers are small.
|
|
55
|
+
The CLI needs at least 12 processes and threads (`--pids-limit 12`): below that it refuses with
|
|
56
|
+
an error, and with 4 or fewer Node.js itself hangs before the CLI can check.
|
|
57
|
+
|
|
58
|
+
`latest` is the newest release. For scripts, name a release and pin its digest, which the
|
|
59
|
+
[release notes](https://github.com/maschinenlesbar-org/ckan-cli/releases) give:
|
|
60
|
+
`…/ckan-cli:0.9.0@sha256:<digest>`. Each image carries signed build provenance;
|
|
61
|
+
check it with
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
gh attestation verify oci://ghcr.io/maschinenlesbar-org/ckan-cli:latest \
|
|
65
|
+
--repo maschinenlesbar-org/ckan-cli
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
### MCP server
|
|
69
|
+
|
|
70
|
+
From 0.9.0 the CKAN portals are also an [MCP](https://modelcontextprotocol.io/) server for AI
|
|
71
|
+
agents, `@maschinenlesbar.org/ckan-mcp` (command `ckan-mcp`, image
|
|
72
|
+
`ghcr.io/maschinenlesbar-org/ckan-mcp`): six read-only tools, to see which portals of the
|
|
73
|
+
built-in list have data on a topic (GovData apart, since it harvests the others), find datasets on
|
|
74
|
+
one portal with friendly filters (an organization or group by name, a file format in all its
|
|
75
|
+
spellings, a date range) and their licences, count a portal by organization, group, format,
|
|
76
|
+
licence, tag or type and per year (the spellings folded), search the Transparenzportal Hamburg's
|
|
77
|
+
documents by type, topic and date, read one record in full, and list a portal's organizations or
|
|
78
|
+
groups, on stdio or Streamable HTTP. The server's portal is Hamburg, or the one `--base-url`
|
|
79
|
+
names; per call a tool takes another portal of the list by its id, never a URL. Setup for Claude
|
|
80
|
+
Code, Claude Desktop and VS Code:
|
|
81
|
+
[MCP.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/MCP.md).
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
npm i -g @maschinenlesbar.org/ckan-mcp
|
|
85
|
+
claude mcp add ckan -- ckan-mcp
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Every result gives each record's licence and the portal it came from, whose terms apply too.
|
|
89
|
+
|
|
37
90
|
## Quickstart
|
|
38
91
|
|
|
39
92
|
The two examples from Hamburg's API page, as commands. A full-text search:
|
|
@@ -335,6 +388,9 @@ Transparenzportal's contracts, Gutachten and Senate papers
|
|
|
335
388
|
|
|
336
389
|
Real runs of each skill are in [EXAMPLE.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/EXAMPLE.md).
|
|
337
390
|
|
|
391
|
+
The same use cases are the tools of the MCP server, for any MCP client: see
|
|
392
|
+
[MCP.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/MCP.md).
|
|
393
|
+
|
|
338
394
|
## Data license
|
|
339
395
|
|
|
340
396
|
This CLI is a **client**. It accesses data it does not own or redistribute, and
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { type EngineOptions } from "./engine.js";
|
|
2
|
-
import type
|
|
2
|
+
import { type QueryParams } from "./query.js";
|
|
3
3
|
import type { Group, GroupListParams, JsonValue, License, ListParams, Organization, Package, PackageSearchParams, PackageSearchResult, Resource, Status, TagListParams } from "./types.js";
|
|
4
4
|
/**
|
|
5
5
|
* Most pages one `all_fields` list call requests (10,000 entries at CKAN's page size
|
|
@@ -14,6 +14,19 @@ export declare const MAX_ALL_FIELDS_PAGES = 400;
|
|
|
14
14
|
* second at most instead of thousands.
|
|
15
15
|
*/
|
|
16
16
|
export declare const ALL_FIELDS_PAGE_DELAY_MS = 100;
|
|
17
|
+
/**
|
|
18
|
+
* The `package_search` parameters `packageSearch(params)` sends, checked first (CkanValidationError
|
|
19
|
+
* before any request: an unknown key, a refused text value, a string where a list belongs, a NaN or
|
|
20
|
+
* negative number would otherwise go out, or be dropped). See `packageSearch` for how each one is
|
|
21
|
+
* sent.
|
|
22
|
+
*/
|
|
23
|
+
export declare function packageSearchQuery(params?: PackageSearchParams): QueryParams;
|
|
24
|
+
/**
|
|
25
|
+
* A request as `action(name, params)` sends it, under the Action API's base
|
|
26
|
+
* (`<site>/api/3/action/`): the action's name and its query string (`package_show?id=x`), the
|
|
27
|
+
* parameters checked and normalised as they go out. What an answer's source names.
|
|
28
|
+
*/
|
|
29
|
+
export declare function actionRequest(name: string, params?: QueryParams): string;
|
|
17
30
|
/**
|
|
18
31
|
* The site root of a CKAN instance, from what a user is likely to paste: portals
|
|
19
32
|
* document their API as `<site>/api/3/action`, so that suffix (or `/api/3`) is
|
|
@@ -25,6 +38,17 @@ export declare function siteRoot(baseUrl: string): string;
|
|
|
25
38
|
export declare class CkanClient {
|
|
26
39
|
#private;
|
|
27
40
|
constructor(options?: EngineOptions);
|
|
41
|
+
/**
|
|
42
|
+
* The portal this client talks to as one identity (`portalKey`: host without `www.` and path,
|
|
43
|
+
* lower case; never a user name or password): to tell whether a portal of the list is this one.
|
|
44
|
+
*/
|
|
45
|
+
get siteKey(): string;
|
|
46
|
+
/**
|
|
47
|
+
* A client for another CKAN portal with this one's options (transport, timeouts, retries, user
|
|
48
|
+
* agent, hooks): another portal of a known list, through the same transport (and so the same
|
|
49
|
+
* request limits). The base URL is checked as the constructor checks it.
|
|
50
|
+
*/
|
|
51
|
+
withBaseUrl(baseUrl: string): CkanClient;
|
|
28
52
|
/** Call any CKAN action by name and return its unwrapped `result`. */
|
|
29
53
|
action<T = JsonValue>(name: string, params?: QueryParams): Promise<T>;
|
|
30
54
|
/**
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
// Action API (`<site>/api/3/action`).
|
|
3
3
|
import { bigIntegerKeysOption, DEFAULT_BASE_URL, RequestEngine, sanitizeServerText, validateBaseUrl } from "./engine.js";
|
|
4
4
|
import { CkanActionError, CkanParseError, CkanValidationError, cutForMessage, describeCkanError, quoteValue, redactUrl } from "./errors.js";
|
|
5
|
+
import { buildQueryString } from "./query.js";
|
|
5
6
|
import { assertValid, countProblem, facetLimitProblem, fieldProblem, idProblem, listProblem, normalizeText, normalizeToken, queryValueProblem, sortProblem, textProblem, tokenProblem, } from "./validate.js";
|
|
6
7
|
const ACTION = "/api/3/action";
|
|
7
8
|
/**
|
|
@@ -153,6 +154,48 @@ function assertCount(name, value) {
|
|
|
153
154
|
if (value !== undefined)
|
|
154
155
|
assertValid(name, value, countProblem);
|
|
155
156
|
}
|
|
157
|
+
/**
|
|
158
|
+
* The `package_search` parameters `packageSearch(params)` sends, checked first (CkanValidationError
|
|
159
|
+
* before any request: an unknown key, a refused text value, a string where a list belongs, a NaN or
|
|
160
|
+
* negative number would otherwise go out, or be dropped). See `packageSearch` for how each one is
|
|
161
|
+
* sent.
|
|
162
|
+
*/
|
|
163
|
+
export function packageSearchQuery(params = {}) {
|
|
164
|
+
assertParams(params);
|
|
165
|
+
assertKeys("packageSearch", "package_search", params, SEARCH_KEYS);
|
|
166
|
+
assertOptional("q", params.q, textProblem);
|
|
167
|
+
assertList("fq", params.fq, textProblem);
|
|
168
|
+
assertOptional("sort", params.sort, sortProblem);
|
|
169
|
+
assertList("facet_field", params.facet_field, fieldProblem);
|
|
170
|
+
assertCount("rows", params.rows);
|
|
171
|
+
assertCount("start", params.start);
|
|
172
|
+
if (params.facet_limit !== undefined)
|
|
173
|
+
assertValid("facet_limit", params.facet_limit, facetLimitProblem);
|
|
174
|
+
const fq = (params.fq ?? []).map(normalizeText);
|
|
175
|
+
const facetFields = (params.facet_field ?? []).map(normalizeToken);
|
|
176
|
+
return {
|
|
177
|
+
q: params.q === undefined ? undefined : normalizeText(params.q),
|
|
178
|
+
fq: fq.length === 1 ? fq[0] : undefined,
|
|
179
|
+
fq_list: fq.length > 1 ? fq : undefined,
|
|
180
|
+
rows: params.rows,
|
|
181
|
+
start: params.start,
|
|
182
|
+
sort: params.sort === undefined ? undefined : normalizeToken(params.sort),
|
|
183
|
+
"facet.field": facetFields.length > 0 ? JSON.stringify(facetFields) : undefined,
|
|
184
|
+
"facet.limit": params.facet_limit,
|
|
185
|
+
};
|
|
186
|
+
}
|
|
187
|
+
/**
|
|
188
|
+
* A request as `action(name, params)` sends it, under the Action API's base
|
|
189
|
+
* (`<site>/api/3/action/`): the action's name and its query string (`package_show?id=x`), the
|
|
190
|
+
* parameters checked and normalised as they go out. What an answer's source names.
|
|
191
|
+
*/
|
|
192
|
+
export function actionRequest(name, params = {}) {
|
|
193
|
+
if (typeof name !== "string" || !ACTION_NAME.test(name)) {
|
|
194
|
+
throw new CkanValidationError(`Invalid CKAN action name: ${quoteValue(typeof name === "string" ? name : String(name))}`);
|
|
195
|
+
}
|
|
196
|
+
const qs = buildQueryString(prune(params));
|
|
197
|
+
return qs === "" ? name : `${name}?${qs}`;
|
|
198
|
+
}
|
|
156
199
|
/**
|
|
157
200
|
* The site root of a CKAN instance, from what a user is likely to paste: portals
|
|
158
201
|
* document their API as `<site>/api/3/action`, so that suffix (or `/api/3`) is
|
|
@@ -173,7 +216,10 @@ export class CkanClient {
|
|
|
173
216
|
#site;
|
|
174
217
|
/** The pause between `all_fields` pages (EngineOptions.sleep, for tests). */
|
|
175
218
|
#sleep;
|
|
219
|
+
/** The options it was made with, for `withBaseUrl`. */
|
|
220
|
+
#options;
|
|
176
221
|
constructor(options = {}) {
|
|
222
|
+
this.#options = options;
|
|
177
223
|
this.#site = siteRoot(options.baseUrl ?? DEFAULT_BASE_URL);
|
|
178
224
|
// The engine checks every option, `sleep` included, before the client uses it. Solr's
|
|
179
225
|
// `_version_` (≈1.88e18, past 2^53) is always declared a big integer: a portal that sends it
|
|
@@ -185,6 +231,22 @@ export class CkanClient {
|
|
|
185
231
|
});
|
|
186
232
|
this.#sleep = options.sleep ?? realSleep;
|
|
187
233
|
}
|
|
234
|
+
/**
|
|
235
|
+
* The portal this client talks to as one identity (`portalKey`: host without `www.` and path,
|
|
236
|
+
* lower case; never a user name or password): to tell whether a portal of the list is this one.
|
|
237
|
+
*/
|
|
238
|
+
get siteKey() {
|
|
239
|
+
const url = new URL(this.#site);
|
|
240
|
+
return `${url.host.toLowerCase().replace(/^www\./, "")}${url.pathname.replace(/\/+$/, "").toLowerCase()}`;
|
|
241
|
+
}
|
|
242
|
+
/**
|
|
243
|
+
* A client for another CKAN portal with this one's options (transport, timeouts, retries, user
|
|
244
|
+
* agent, hooks): another portal of a known list, through the same transport (and so the same
|
|
245
|
+
* request limits). The base URL is checked as the constructor checks it.
|
|
246
|
+
*/
|
|
247
|
+
withBaseUrl(baseUrl) {
|
|
248
|
+
return new CkanClient({ ...this.#options, baseUrl });
|
|
249
|
+
}
|
|
188
250
|
/** Call any CKAN action by name and return its unwrapped `result`. */
|
|
189
251
|
async action(name, params = {}) {
|
|
190
252
|
return (await this.call(name, params)).result;
|
|
@@ -250,28 +312,7 @@ export class CkanClient {
|
|
|
250
312
|
* a Solr field name) as tokens (`normalizeToken`).
|
|
251
313
|
*/
|
|
252
314
|
async packageSearch(params = {}) {
|
|
253
|
-
|
|
254
|
-
assertKeys("packageSearch", "package_search", params, SEARCH_KEYS);
|
|
255
|
-
assertOptional("q", params.q, textProblem);
|
|
256
|
-
assertList("fq", params.fq, textProblem);
|
|
257
|
-
assertOptional("sort", params.sort, sortProblem);
|
|
258
|
-
assertList("facet_field", params.facet_field, fieldProblem);
|
|
259
|
-
assertCount("rows", params.rows);
|
|
260
|
-
assertCount("start", params.start);
|
|
261
|
-
if (params.facet_limit !== undefined)
|
|
262
|
-
assertValid("facet_limit", params.facet_limit, facetLimitProblem);
|
|
263
|
-
const fq = (params.fq ?? []).map(normalizeText);
|
|
264
|
-
const facetFields = (params.facet_field ?? []).map(normalizeToken);
|
|
265
|
-
return this.typed("package_search", {
|
|
266
|
-
q: params.q === undefined ? undefined : normalizeText(params.q),
|
|
267
|
-
fq: fq.length === 1 ? fq[0] : undefined,
|
|
268
|
-
fq_list: fq.length > 1 ? fq : undefined,
|
|
269
|
-
rows: params.rows,
|
|
270
|
-
start: params.start,
|
|
271
|
-
sort: params.sort === undefined ? undefined : normalizeToken(params.sort),
|
|
272
|
-
"facet.field": facetFields.length > 0 ? JSON.stringify(facetFields) : undefined,
|
|
273
|
-
"facet.limit": params.facet_limit,
|
|
274
|
-
}, isSearchResult, "an object with a numeric count and a results array");
|
|
315
|
+
return this.typed("package_search", packageSearchQuery(params), isSearchResult, "an object with a numeric count and a results array");
|
|
275
316
|
}
|
|
276
317
|
/** A single dataset by id or name. */
|
|
277
318
|
packageShow(id) {
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
export { ALL_FIELDS_PAGE_DELAY_MS, CkanClient, MAX_ALL_FIELDS_PAGES, siteRoot } from "./client.js";
|
|
1
|
+
export { actionRequest, ALL_FIELDS_PAGE_DELAY_MS, CkanClient, MAX_ALL_FIELDS_PAGES, packageSearchQuery, siteRoot } from "./client.js";
|
|
2
2
|
export { RequestEngine, DEFAULT_BASE_URL, assertHeaderValue, MAX_REDIRECTS, MAX_RETRIES, MAX_RETRY_AFTER_MS, parseRetryAfter, validateBaseUrl, cleartextProblem, } from "./engine.js";
|
|
3
3
|
export type { EngineOptions, RawResponse, RetryEvent } from "./engine.js";
|
|
4
4
|
export { MAX_TIMEOUT_MS, nodeHttpTransport } from "./http.js";
|
|
@@ -12,3 +12,4 @@ export { PORTALS } from "./portals-list.js";
|
|
|
12
12
|
export { DEFAULT_CHECK_CONCURRENCY, checkPortal, checkPortalUrls, checkPortals, findPortal, mapLimit, portalKey, withCheck, } from "./portals.js";
|
|
13
13
|
export type { CheckPortalsOptions, PortalCheck } from "./portals.js";
|
|
14
14
|
export * from "./types.js";
|
|
15
|
+
export * from "./operations.js";
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// Public entry point for the API client library.
|
|
2
|
-
export { ALL_FIELDS_PAGE_DELAY_MS, CkanClient, MAX_ALL_FIELDS_PAGES, siteRoot } from "./client.js";
|
|
2
|
+
export { actionRequest, ALL_FIELDS_PAGE_DELAY_MS, CkanClient, MAX_ALL_FIELDS_PAGES, packageSearchQuery, siteRoot } from "./client.js";
|
|
3
3
|
export { RequestEngine, DEFAULT_BASE_URL, assertHeaderValue, MAX_REDIRECTS, MAX_RETRIES, MAX_RETRY_AFTER_MS, parseRetryAfter, validateBaseUrl, cleartextProblem, } from "./engine.js";
|
|
4
4
|
export { MAX_TIMEOUT_MS, nodeHttpTransport } from "./http.js";
|
|
5
5
|
export { buildQueryString } from "./query.js";
|
|
@@ -8,3 +8,4 @@ export { assertValid, fieldProblem, idProblem, listProblem, normalizeText, norma
|
|
|
8
8
|
export { PORTALS } from "./portals-list.js";
|
|
9
9
|
export { DEFAULT_CHECK_CONCURRENCY, checkPortal, checkPortalUrls, checkPortals, findPortal, mapLimit, portalKey, withCheck, } from "./portals.js";
|
|
10
10
|
export * from "./types.js";
|
|
11
|
+
export * from "./operations.js";
|