@maschinenlesbar.org/ckan-cli 0.1.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/DATA_LICENSE.md +78 -0
- package/README.md +55 -12
- package/dist/src/cli/commands/catalogue.js +12 -12
- package/dist/src/cli/commands/portal.js +12 -3
- package/dist/src/cli/index.js +8 -3
- package/dist/src/cli/io.d.ts +35 -5
- package/dist/src/cli/io.js +65 -9
- package/dist/src/cli/log.d.ts +95 -0
- package/dist/src/cli/log.js +159 -0
- package/dist/src/cli/program.d.ts +7 -0
- package/dist/src/cli/program.js +88 -11
- package/dist/src/cli/run.d.ts +53 -1
- package/dist/src/cli/run.js +239 -8
- package/dist/src/cli/shared.d.ts +17 -1
- package/dist/src/cli/shared.js +33 -3
- package/dist/src/client/client.d.ts +23 -3
- package/dist/src/client/client.js +124 -22
- package/dist/src/client/engine.d.ts +62 -9
- package/dist/src/client/engine.js +350 -47
- package/dist/src/client/errors.d.ts +81 -1
- package/dist/src/client/errors.js +171 -5
- package/dist/src/client/http.d.ts +23 -0
- package/dist/src/client/http.js +14 -1
- package/dist/src/client/index.d.ts +3 -3
- package/dist/src/client/index.js +3 -3
- package/dist/src/client/portals.d.ts +1 -1
- package/dist/src/client/portals.js +39 -4
- package/dist/src/client/validate.d.ts +13 -1
- package/dist/src/client/validate.js +49 -3
- package/package.json +6 -5
package/DATA_LICENSE.md
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# Data license
|
|
2
|
+
|
|
3
|
+
> **This tool does not include, host, or redistribute any data.**
|
|
4
|
+
> `ckan-cli` is a *client*. It only accesses records served live by whichever
|
|
5
|
+
> CKAN portal you point it at. Those records are governed by **that portal's**
|
|
6
|
+
> terms, and often by a licence set per dataset. The license of this CLI's own
|
|
7
|
+
> source code is a separate matter — see [LICENSING.md](LICENSING.md).
|
|
8
|
+
|
|
9
|
+
> [!IMPORTANT]
|
|
10
|
+
> **The terms depend on the portal and usually on the record.** A generic CKAN
|
|
11
|
+
> client cannot know them in advance. Every CKAN record carries its own
|
|
12
|
+
> `license_id` / `license_title` / `license_url`; check them before reusing a
|
|
13
|
+
> record's contents. `ckan licenses` lists the licences a portal offers, and
|
|
14
|
+
> `ckan search --rows 0 --facet license_id` shows how often each is used.
|
|
15
|
+
|
|
16
|
+
## Default portal: Transparenzportal Hamburg
|
|
17
|
+
|
|
18
|
+
| | |
|
|
19
|
+
|---|---|
|
|
20
|
+
| **Data provider** | Freie und Hansestadt Hamburg; each record is the responsibility of its publishing body |
|
|
21
|
+
| **API / source** | `https://suche.transparenz.hamburg.de/api/3/action` (CKAN Action API) · portal: https://transparenz.hamburg.de/ |
|
|
22
|
+
| **Legal basis** | Hamburgisches Transparenzgesetz (HmbTG), § 10 Abs. 3: *"Die Nutzung, Weiterverwendung und Verbreitung der Informationen ist frei, sofern höherrangiges Recht oder spezialgesetzliche Regelungen nichts anderes bestimmen."* |
|
|
23
|
+
| **Portal statement** | *"Die Dokumente und Daten aus dem Transparenzportal Hamburg dürfen frei verwendet werden."* Records are published under two licences that allow free reuse, *"ggf. unter Nennung des Namens"*: Datenlizenz Deutschland – Namensnennung 2.0 and – Zero 2.0. |
|
|
24
|
+
| **Licence** | **Per record**, in `license_id`. As of 2026-09-19: `dl-de-by-2.0` (242,632 records), `cc-zero` (2,203), `dl-zero-de-2.0` / `dl-de-zero-2.0` (262), `odbl` (59), `geoNutz-20130319` (48). |
|
|
25
|
+
| **Attribution** | Per record, in the `terms_of_use` extra (`attribution_text`), e.g. *"Namensnennung: Freie und Hansestadt Hamburg, Bezirksversammlung Wandsbek"*. The wording names the publishing body, so it varies. |
|
|
26
|
+
| **Metadata** | Not licensed separately by the portal. |
|
|
27
|
+
| **Automated access** | Invited: the API page describes it as the way to fetch *"Inhalte und Metadaten aus dem Transparenzportal automatisiert"*. No rate limits or API terms are published; HmbTG § 10 Abs. 4 makes access *"kostenlos und anonym"*. |
|
|
28
|
+
|
|
29
|
+
### Watch for
|
|
30
|
+
|
|
31
|
+
- **`dl-de-by-2.0`** (the vast majority) requires attribution. Use the record's
|
|
32
|
+
own `attribution_text`, and link the licence
|
|
33
|
+
(https://www.govdata.de/dl-de/by-2-0).
|
|
34
|
+
- **`odbl`** records are share-alike: derived databases must stay under the ODbL.
|
|
35
|
+
- **`geoNutz-20130319`** is Hamburg's older geodata usage licence, not a
|
|
36
|
+
standard open licence. Read its terms before reusing those records.
|
|
37
|
+
- **Third-party rights.** HmbTG § 10 Abs. 3 S. 3 only obliges authorities to
|
|
38
|
+
*try* to obtain reuse rights for third-party works (reports, studies), and
|
|
39
|
+
§ 8 Abs. 1 lets intellectual-property protection override publication. A
|
|
40
|
+
document written by a third party can carry rights the record's licence does
|
|
41
|
+
not grant.
|
|
42
|
+
- **Personal data** is redacted before publication (HmbTG § 4), but some names
|
|
43
|
+
are published on purpose (contract partners, authors of reports, subsidy
|
|
44
|
+
recipients). Data-protection law still applies to what you do with them.
|
|
45
|
+
- **The website itself** (layout, graphics, editorial pages) is copyright
|
|
46
|
+
protected per the Impressum; that concerns transparenz.hamburg.de, not the
|
|
47
|
+
register records served by the API.
|
|
48
|
+
|
|
49
|
+
### Attribution
|
|
50
|
+
|
|
51
|
+
```
|
|
52
|
+
Source: Transparenzportal Hamburg (https://transparenz.hamburg.de/),
|
|
53
|
+
<attribution_text of the record>, licensed under <license_title of the record>
|
|
54
|
+
(<license_url of the record>).
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Other portals
|
|
58
|
+
|
|
59
|
+
With `--portal`, `--base-url` or `CKAN_BASE_URL` you are subject to that portal's
|
|
60
|
+
terms. Being in the built-in list (`ckan portals`) says only that a portal runs a
|
|
61
|
+
working CKAN, nothing about its terms.
|
|
62
|
+
Look them up on the portal itself; do not assume they match Hamburg's. For
|
|
63
|
+
GovData, see the `DATA_LICENSE.md` of
|
|
64
|
+
[govdata-cli](https://github.com/maschinenlesbar-org/govdata-cli).
|
|
65
|
+
|
|
66
|
+
## Sources
|
|
67
|
+
|
|
68
|
+
- https://transparenz.hamburg.de/verwendung-der-daten-aus-dem-transparenzportal-796520 — free use, the two licences
|
|
69
|
+
- https://transparenz.hamburg.de/api-796358 — API notes, automated access
|
|
70
|
+
- https://transparenz.hamburg.de/gesetzestext-des-hmbtg-796514 — HmbTG §§ 4, 7, 8, 10
|
|
71
|
+
- https://transparenz.hamburg.de/impressum-796468 — responsibility per publishing body, website copyright
|
|
72
|
+
- https://www.govdata.de/dl-de/by-2-0 — Datenlizenz Deutschland – Namensnennung – 2.0
|
|
73
|
+
- Record counts: `ckan search --rows 0 --facet license_id` against the live API, 2026-09-19
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
*Good-faith summary compiled 2026-09-19; not legal advice. The portal's terms
|
|
78
|
+
and each record's licence are authoritative and can change.*
|
package/README.md
CHANGED
|
@@ -24,7 +24,7 @@ search, inspect, filter and pipe straight into [`jq`](https://jqlang.github.io/j
|
|
|
24
24
|
`status`, `portals`, and a generic `action` escape hatch for everything else.
|
|
25
25
|
|
|
26
26
|
> Want to use this as a TypeScript library or understand how it's built?
|
|
27
|
-
> See **[DEVELOPING.md](DEVELOPING.md)**.
|
|
27
|
+
> See **[DEVELOPING.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/DEVELOPING.md)**.
|
|
28
28
|
|
|
29
29
|
## Install
|
|
30
30
|
|
|
@@ -32,7 +32,7 @@ search, inspect, filter and pipe straight into [`jq`](https://jqlang.github.io/j
|
|
|
32
32
|
npm i -g @maschinenlesbar.org/ckan-cli
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
This installs the **`ckan`** command. Requires **Node.js
|
|
35
|
+
This installs the **`ckan`** command. Requires **Node.js 22.12+**.
|
|
36
36
|
|
|
37
37
|
## Quickstart
|
|
38
38
|
|
|
@@ -79,10 +79,16 @@ sub-path keeps it (`https://open.canada.ca/data`). `ckan status` is a good first
|
|
|
79
79
|
call on an unfamiliar portal: it shows the CKAN version and the installed
|
|
80
80
|
extensions.
|
|
81
81
|
|
|
82
|
+
A portal behind a login takes its user name and password in the URL
|
|
83
|
+
(`https://user:password@host`). They go out as HTTP Basic auth to that site only:
|
|
84
|
+
a redirect to another host, port or scheme drops them, and no message, help text
|
|
85
|
+
or error shows them.
|
|
86
|
+
|
|
82
87
|
### Known portals
|
|
83
88
|
|
|
84
89
|
`ckan portals` lists the CKAN portals in Germany this version knows, with the
|
|
85
90
|
result of their last check; `ckan portals --check` checks them all live now.
|
|
91
|
+
Neither uses `CKAN_BASE_URL`, so both work even when the variable is broken.
|
|
86
92
|
The list is built into the CLI, so it needs no outside service:
|
|
87
93
|
|
|
88
94
|
| `--portal` | Portal | Datasets |
|
|
@@ -108,7 +114,7 @@ The list is built into the CLI, so it needs no outside service:
|
|
|
108
114
|
|
|
109
115
|
Counts as of the last check, 2026-09-19. The list is refreshed from Wikidata,
|
|
110
116
|
the CKAN project's instance registry and GovData's harvest sources, keeping only
|
|
111
|
-
portals that pass a live check (see [DEVELOPING.md](DEVELOPING.md#the-portal-list)).
|
|
117
|
+
portals that pass a live check (see [DEVELOPING.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/DEVELOPING.md#the-portal-list)).
|
|
112
118
|
Many portals those lists name have left CKAN (Köln, Düsseldorf, Bonn, …).
|
|
113
119
|
|
|
114
120
|
## Commands
|
|
@@ -152,7 +158,9 @@ action <name> [--param key=value …] call any CKAN action (generic)
|
|
|
152
158
|
> **Blank values are refused.** `ckan search ""`, `--fq ""`, `tags --query ""` or
|
|
153
159
|
> `action … --param q=` exit 1 before any request, instead of silently running an unfiltered search.
|
|
154
160
|
> So does a forgotten value (`--fq --rows 5`): a value of `--fq`, `--facet`,
|
|
155
|
-
> `--sort` or `tags --query` cannot start with `--`.
|
|
161
|
+
> `--sort` or `tags --query` cannot start with `--`. And so does a single-value
|
|
162
|
+
> option given twice (`--rows 5 --rows 50`, a second `--base-url`): only `--fq`,
|
|
163
|
+
> `--facet` and `--param` repeat.
|
|
156
164
|
|
|
157
165
|
### Paging and list flags
|
|
158
166
|
|
|
@@ -163,7 +171,9 @@ action <name> [--param key=value …] call any CKAN action (generic)
|
|
|
163
171
|
|
|
164
172
|
> **`--all-fields` lists are complete.** CKAN caps them at 25 entries
|
|
165
173
|
> (`ckan.group_and_organization_list_all_fields_max`) and drops the rest
|
|
166
|
-
> silently, even for a larger `--limit`; `ckan` pages through them for you
|
|
174
|
+
> silently, even for a larger `--limit`; `ckan` pages through them for you,
|
|
175
|
+
> 0.1 s apart. A server whose pages never end is stopped after 400 pages
|
|
176
|
+
> (10,000 entries, far more than any portal lists) with an error (exit 1).
|
|
167
177
|
| `tags` | `--query <substring>` |
|
|
168
178
|
|
|
169
179
|
### `action`
|
|
@@ -187,7 +197,7 @@ before any request is sent.
|
|
|
187
197
|
matches nothing, and a full word such as `verwaltungsvorschriften` over-matches.
|
|
188
198
|
List the values that work, with their counts:
|
|
189
199
|
`ckan --compact search --rows 0 --facet extras_registerobject_type --facet-limit -1`.
|
|
190
|
-
The [Glossary](GLOSSARY.md) has the details.
|
|
200
|
+
The [Glossary](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/GLOSSARY.md) has the details.
|
|
191
201
|
- **Several filters work.** Hamburg's API page says only one filter can be
|
|
192
202
|
used; with `fq_list` (repeat `--fq`) they combine:
|
|
193
203
|
`ckan search --fq extras_registerobject_type:verwaltungsvorschrift --fq organization:workflows`.
|
|
@@ -206,6 +216,9 @@ Found by running the same commands against many portals:
|
|
|
206
216
|
- **The default sort differs** (ZBW sorts by `metadata_created desc`).
|
|
207
217
|
- **Field types vary**: licence ids are URIs on GovData, and some portals send
|
|
208
218
|
booleans as the strings `"True"`/`"False"`.
|
|
219
|
+
- **A purely negative query matches nothing**: `ckan search -- -elbe` answers
|
|
220
|
+
`count: 0` on Hamburg rather than "everything but elbe". Negate a filter instead
|
|
221
|
+
(`--fq -organization:allris`), next to a positive query or none.
|
|
209
222
|
- **Solr syntax errors** come back as HTTP 409; `ckan` prints Solr's own reason,
|
|
210
223
|
e.g. `Search Error: Cannot parse 'title:(': Encountered "<EOF>" …`.
|
|
211
224
|
- **A site that is not (or no longer) a CKAN** answers with a 404, an HTML page or
|
|
@@ -218,6 +231,25 @@ Every command prints the **unwrapped `result`** as pretty JSON to stdout.
|
|
|
218
231
|
Errors go to stderr, so piping stdout into `jq` stays clean. `--compact`
|
|
219
232
|
prints one line. Every global option works before or after the command.
|
|
220
233
|
|
|
234
|
+
Each line on stderr is a **log record**: a timestamp (UTC), a level (`ERROR`, `WARN`,
|
|
235
|
+
`INFO`) and a topic, the program and the area it comes from (`ckan.cli` for usage
|
|
236
|
+
errors, `ckan.api` for the portal's answers, a malformed one included, `ckan.http` for the connection,
|
|
237
|
+
`ckan.output` when stdout can't be written). By default
|
|
238
|
+
it is written log4j style; `--log-format jsonl` writes one JSON object per line instead.
|
|
239
|
+
A record is always one line: a line break, a control character or a bidi control in a
|
|
240
|
+
message (a server's text, a value you typed) is written as an escape (`\n`, `\u001b`,
|
|
241
|
+
`\u202e`), so it can neither split a record nor forge another one, nor steer the terminal;
|
|
242
|
+
a message longer than 4000 characters is cut and ends in `… (N more characters)`:
|
|
243
|
+
|
|
244
|
+
```text
|
|
245
|
+
2026-10-09T14:03:12.481Z WARN [ckan.http] requests to ckan.example are sent unencrypted (http:, not https:)
|
|
246
|
+
2026-10-09T14:03:12.902Z ERROR [ckan.api] HTTP 404 for GET http://ckan.example/api/3/action/status_show: Not Found Error: Not found
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
```bash
|
|
250
|
+
ckan --log-format jsonl package no-such-id 2>log.jsonl # {"ts":"…","level":"ERROR","topic":"ckan.api","msg":"HTTP 404 …"}
|
|
251
|
+
```
|
|
252
|
+
|
|
221
253
|
```bash
|
|
222
254
|
# How many records a portal has
|
|
223
255
|
ckan search --rows 0 | jq '.count'
|
|
@@ -234,10 +266,13 @@ ckan package bezirk-wandsbek-drucksache-22-3573-2 | jq '.resources[] | {format,
|
|
|
234
266
|
| Code | Meaning |
|
|
235
267
|
| --- | --- |
|
|
236
268
|
| `0` | success (also `--help`, `help [command]`, `--version` and a bare `ckan`) |
|
|
237
|
-
| `4` | not found (HTTP `404`): an unknown id on `package`/`resource`/`organization`/`group`, or, on any command, no CKAN at this URL (a wrong `--base-url`, a portal that left CKAN) |
|
|
238
|
-
| `1` | any other error — bad usage, validation error (`409`), unknown action (`400`), CKAN `success:false`, network failure |
|
|
269
|
+
| `4` | not found (HTTP `404`, or CKAN's `Not Found Error` sent with HTTP `200`): an unknown id on `package`/`resource`/`organization`/`group`, or, on any command, no CKAN at this URL (a wrong `--base-url`, a portal that left CKAN) |
|
|
270
|
+
| `1` | any other error — bad usage (an unknown command, also after `help`), validation error (`409`), unknown action (`400`), any other CKAN `success:false`, network failure |
|
|
239
271
|
|
|
240
|
-
|
|
272
|
+
A reader that stops early (`| head`) ends `ckan` quietly with exit 0; a failed
|
|
273
|
+
run keeps its exit code even when nothing reads stderr.
|
|
274
|
+
|
|
275
|
+
CKAN's error text is shown on stderr (in an `ERROR` record of `ckan.api`), e.g.
|
|
241
276
|
`Validation Error: rows: Invalid integer` or
|
|
242
277
|
`Bad request - Action name not known: …`.
|
|
243
278
|
|
|
@@ -254,16 +289,24 @@ CKAN's error text is shown on stderr, e.g.
|
|
|
254
289
|
| `-V, --version` | Print the version number |
|
|
255
290
|
| `-h, --help` | Show help for the program or a command |
|
|
256
291
|
| `--compact` | Print JSON on a single line instead of pretty-printed |
|
|
292
|
+
| `--log-format <format>` | How errors, warnings and notes are written to stderr: `text` (default; log4j style, `2026-10-09T14:03:12.481Z WARN [ckan.http] …`) or `jsonl` (one JSON object per line: `ts`, `level`, `topic`, `msg`). stdout is not affected |
|
|
257
293
|
| `--base-url <url>` | CKAN site URL (env `CKAN_BASE_URL`; default `https://suche.transparenz.hamburg.de`) |
|
|
258
294
|
| `--portal <id>` | a known portal by id (`ckan portals`); cannot be combined with `--base-url` |
|
|
259
295
|
| `--timeout <ms>` | Per-request timeout (default `30000`; at most `2147483647`) |
|
|
260
296
|
| `--user-agent <ua>` | `User-Agent` header value |
|
|
261
|
-
| `--max-retries <n>` | Retries for transient `429`/`503` responses (`0`–`10`, default `2`). Each retry waits the server's `Retry-After`
|
|
297
|
+
| `--max-retries <n>` | Retries for transient `429`/`503` responses (`0`–`10`, default `2`). Each retry backs off linearly (0.2 s, 0.4 s, …) or waits the server's `Retry-After` when that is longer, up to 30 s; a longer one is not retried, and the error names the wait. A reset or other network error is not retried |
|
|
262
298
|
| `--max-response-bytes <n>` | Cap response body size in bytes (`0` = unlimited; default 100 MiB) |
|
|
263
299
|
|
|
300
|
+
A base URL on plain `http:` (from `--base-url` or `CKAN_BASE_URL`) to a host other than
|
|
301
|
+
loopback (`localhost`, `127.0.0.0/8`, `::1`) works, but the CLI writes one `WARN` record of
|
|
302
|
+
`ckan.http` to stderr before the first request, e.g.
|
|
303
|
+
`… WARN [ckan.http] requests to ckan.example are sent unencrypted (http:, not https:)`, or
|
|
304
|
+
`… WARN [ckan.http] the base URL's credentials are sent unencrypted to ckan.example (http:, not https:)`
|
|
305
|
+
when it carries a `user:password@` (never printed). stdout and the exit code are unchanged.
|
|
306
|
+
|
|
264
307
|
## Claude Code skills
|
|
265
308
|
|
|
266
|
-
Three [Agent Skills](SKILLS.md) teach Claude Code to use `ckan` for real questions: find
|
|
309
|
+
Three [Agent Skills](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/SKILLS.md) teach Claude Code to use `ckan` for real questions: find
|
|
267
310
|
which portals have data on a topic and rank the hits (**ckan-dataset-finder**), build
|
|
268
311
|
publisher/format/licence statistics (**ckan-catalogue-stats**), and search the Hamburg
|
|
269
312
|
Transparenzportal's contracts, Gutachten and Senate papers
|
|
@@ -274,7 +317,7 @@ Transparenzportal's contracts, Gutachten and Senate papers
|
|
|
274
317
|
/plugin install ckan@maschinenlesbar
|
|
275
318
|
```
|
|
276
319
|
|
|
277
|
-
Real runs of each skill are in [EXAMPLE.md](EXAMPLE.md).
|
|
320
|
+
Real runs of each skill are in [EXAMPLE.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/EXAMPLE.md).
|
|
278
321
|
|
|
279
322
|
## Data license
|
|
280
323
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
// and tags.
|
|
3
3
|
import { InvalidArgumentError } from "commander";
|
|
4
4
|
import { facetLimitProblem } from "../../client/validate.js";
|
|
5
|
-
import { action, collectNonEmpty, parseBoundedInt, parseIntArg, parseNonEmpty, parseOptionValue, renderJson, } from "../shared.js";
|
|
5
|
+
import { action, collectNonEmpty, once, parseBoundedInt, parseIntArg, parseNonEmpty, parseOptionValue, renderJson, } from "../shared.js";
|
|
6
6
|
/**
|
|
7
7
|
* commander value-parser for a list --limit: 1 or more. CKAN reads `limit=0` as
|
|
8
8
|
* "no limit" and would send the whole list; leave --limit out for that.
|
|
@@ -24,12 +24,12 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
24
24
|
.command("search")
|
|
25
25
|
.description("Search datasets (Solr query syntax)")
|
|
26
26
|
.argument("[query]", "full-text query, e.g. elbe or title:haushalt", parseNonEmpty)
|
|
27
|
-
.option("--rows <n>", "max results (servers cap this, usually at 1000)", parseIntArg)
|
|
28
|
-
.option("--start <n>", "offset for paging", parseIntArg)
|
|
29
|
-
.option("--sort <expr>", 'e.g. "metadata_modified desc"', parseOptionValue)
|
|
27
|
+
.option("--rows <n>", "max results (servers cap this, usually at 1000)", once(parseIntArg))
|
|
28
|
+
.option("--start <n>", "offset for paging", once(parseIntArg))
|
|
29
|
+
.option("--sort <expr>", 'e.g. "metadata_modified desc"', once(parseOptionValue))
|
|
30
30
|
.option("--fq <filter>", "filter query, e.g. organization:allris (repeatable; all must match)", collectNonEmpty)
|
|
31
31
|
.option("--facet <field>", "count values of a field, e.g. res_format (repeatable)", collectNonEmpty)
|
|
32
|
-
.option("--facet-limit <n>", "max values per facet (default 50; -1 = all)", parseFacetLimit)
|
|
32
|
+
.option("--facet-limit <n>", "max values per facet (default 50; -1 = all)", once(parseFacetLimit))
|
|
33
33
|
.action(action(deps, async ({ client, global, opts }, [query]) => {
|
|
34
34
|
renderJson(deps, global, await client.packageSearch({
|
|
35
35
|
q: query,
|
|
@@ -72,8 +72,8 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
72
72
|
program
|
|
73
73
|
.command("packages")
|
|
74
74
|
.description("List dataset names")
|
|
75
|
-
.option("--limit <n>", "max names (1 or more; omit for all)", parseLimit)
|
|
76
|
-
.option("--offset <n>", "offset for paging", parseIntArg)
|
|
75
|
+
.option("--limit <n>", "max names (1 or more; omit for all)", once(parseLimit))
|
|
76
|
+
.option("--offset <n>", "offset for paging", once(parseIntArg))
|
|
77
77
|
.action(action(deps, async ({ client, global, opts }) => {
|
|
78
78
|
renderJson(deps, global, await client.packageList({
|
|
79
79
|
limit: opts["limit"],
|
|
@@ -84,8 +84,8 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
84
84
|
.command("organizations")
|
|
85
85
|
.description("List organizations (data publishers)")
|
|
86
86
|
.option("--all-fields", "return full objects instead of names")
|
|
87
|
-
.option("--limit <n>", "max entries (1 or more; omit for all)", parseLimit)
|
|
88
|
-
.option("--offset <n>", "offset for paging", parseIntArg)
|
|
87
|
+
.option("--limit <n>", "max entries (1 or more; omit for all)", once(parseLimit))
|
|
88
|
+
.option("--offset <n>", "offset for paging", once(parseIntArg))
|
|
89
89
|
.action(action(deps, async ({ client, global, opts }) => {
|
|
90
90
|
renderJson(deps, global, await client.organizationList({
|
|
91
91
|
all_fields: opts["allFields"],
|
|
@@ -97,8 +97,8 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
97
97
|
.command("groups")
|
|
98
98
|
.description("List groups (themes/categories)")
|
|
99
99
|
.option("--all-fields", "return full objects instead of names")
|
|
100
|
-
.option("--limit <n>", "max entries (1 or more; omit for all)", parseLimit)
|
|
101
|
-
.option("--offset <n>", "offset for paging", parseIntArg)
|
|
100
|
+
.option("--limit <n>", "max entries (1 or more; omit for all)", once(parseLimit))
|
|
101
|
+
.option("--offset <n>", "offset for paging", once(parseIntArg))
|
|
102
102
|
.action(action(deps, async ({ client, global, opts }) => {
|
|
103
103
|
renderJson(deps, global, await client.groupList({
|
|
104
104
|
all_fields: opts["allFields"],
|
|
@@ -109,7 +109,7 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
109
109
|
program
|
|
110
110
|
.command("tags")
|
|
111
111
|
.description("List tags")
|
|
112
|
-
.option("--query <substring>", "only tags containing this substring", parseOptionValue)
|
|
112
|
+
.option("--query <substring>", "only tags containing this substring", once(parseOptionValue))
|
|
113
113
|
.action(action(deps, async ({ client, global, opts }) => {
|
|
114
114
|
renderJson(deps, global, await client.tagList({ query: opts["query"] }));
|
|
115
115
|
}));
|
|
@@ -4,6 +4,15 @@ import { action, renderJson, toEngineOptions } from "../shared.js";
|
|
|
4
4
|
import { checkPortals } from "../../client/portals.js";
|
|
5
5
|
import { PORTALS } from "../../client/portals-list.js";
|
|
6
6
|
import { isBlank } from "../../client/validate.js";
|
|
7
|
+
import { cutForMessage, redactUrl } from "../../client/errors.js";
|
|
8
|
+
/**
|
|
9
|
+
* A typed value as a message quotes it: cut at MAX_MESSAGE_VALUE_LENGTH, after its
|
|
10
|
+
* credentials are redacted (a cut could otherwise leave part of a password without the
|
|
11
|
+
* "@" the argv redaction keys on).
|
|
12
|
+
*/
|
|
13
|
+
function shown(value) {
|
|
14
|
+
return cutForMessage(redactUrl(value));
|
|
15
|
+
}
|
|
7
16
|
/** commander accumulator for repeatable `key=value` pairs into a record. */
|
|
8
17
|
function collectKeyValue(value, previous = {}) {
|
|
9
18
|
const eq = value.indexOf("=");
|
|
@@ -13,18 +22,18 @@ function collectKeyValue(value, previous = {}) {
|
|
|
13
22
|
const key = value.slice(0, Math.max(eq, 0));
|
|
14
23
|
// A blank key (`=x`, ` =x`) names no parameter: the library refuses it too.
|
|
15
24
|
if (eq < 0 || isBlank(key)) {
|
|
16
|
-
throw new InvalidArgumentError(`Invalid --param "${value}". Expected key=value.`);
|
|
25
|
+
throw new InvalidArgumentError(`Invalid --param "${shown(value)}". Expected key=value.`);
|
|
17
26
|
}
|
|
18
27
|
// Reject a duplicated key rather than silently overwriting the earlier value.
|
|
19
28
|
if (Object.prototype.hasOwnProperty.call(previous, key)) {
|
|
20
|
-
throw new InvalidArgumentError(`Duplicate --param key "${key}".`);
|
|
29
|
+
throw new InvalidArgumentError(`Duplicate --param key "${shown(key)}".`);
|
|
21
30
|
}
|
|
22
31
|
// A blank value is refused by the library as well (CKAN reads an empty
|
|
23
32
|
// parameter as unset, so `--param q=` would run the action unfiltered); here it
|
|
24
33
|
// is a usage error that names the flag.
|
|
25
34
|
const paramValue = value.slice(eq + 1);
|
|
26
35
|
if (isBlank(paramValue)) {
|
|
27
|
-
throw new InvalidArgumentError(`Invalid --param "${value}". The value must not be blank.`);
|
|
36
|
+
throw new InvalidArgumentError(`Invalid --param "${shown(value)}". The value must not be blank.`);
|
|
28
37
|
}
|
|
29
38
|
// A computed key in a literal is an own data property, even for `__proto__`.
|
|
30
39
|
return { ...previous, [key]: paramValue };
|
package/dist/src/cli/index.js
CHANGED
|
@@ -1,7 +1,12 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
// Binary entry point. Thin shim around run(); all logic lives in run.ts/program.ts.
|
|
3
3
|
import { handleOutputErrors } from "./io.js";
|
|
4
|
-
import {
|
|
5
|
-
|
|
6
|
-
const
|
|
4
|
+
import { installWarningLog } from "./log.js";
|
|
5
|
+
import { processLogger, run } from "./run.js";
|
|
6
|
+
const argv = process.argv.slice(2);
|
|
7
|
+
// What happens outside run() is logged too, in the format argv asks for.
|
|
8
|
+
const log = processLogger(argv);
|
|
9
|
+
installWarningLog(process, log);
|
|
10
|
+
handleOutputErrors(process, undefined, log);
|
|
11
|
+
const exitCode = await run(argv);
|
|
7
12
|
process.exitCode = exitCode;
|
package/dist/src/cli/io.d.ts
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import type { CkanClient } from "../client/client.js";
|
|
2
2
|
import type { EngineOptions } from "../client/engine.js";
|
|
3
|
+
import { type Logger } from "./log.js";
|
|
3
4
|
export interface CliIO {
|
|
4
5
|
out(text: string): void;
|
|
5
6
|
err(text: string): void;
|
|
@@ -10,7 +11,16 @@ export interface CliDeps {
|
|
|
10
11
|
env: Record<string, string | undefined>;
|
|
11
12
|
/** Build a client from the resolved global options (injectable for tests). */
|
|
12
13
|
createClient(options: EngineOptions): CkanClient;
|
|
14
|
+
/**
|
|
15
|
+
* Where diagnostics go: one record per line on stderr, in the `--log-format`
|
|
16
|
+
* (`log.ts`). `run()` sets it from argv; deps without it log text through `io.err`.
|
|
17
|
+
*/
|
|
18
|
+
log?: Logger;
|
|
19
|
+
/** The clock the log's timestamps come from. Unset, the real one. */
|
|
20
|
+
now?: () => Date;
|
|
13
21
|
}
|
|
22
|
+
/** The deps' logger, or one that writes text records through `io.err`. */
|
|
23
|
+
export declare function logOf(deps: CliDeps): Logger;
|
|
14
24
|
/** The two process streams, as far as `handleOutputErrors` needs them. */
|
|
15
25
|
export interface OutputStreams {
|
|
16
26
|
stdout: Pick<NodeJS.WriteStream, "on">;
|
|
@@ -22,10 +32,30 @@ export interface OutputStreams {
|
|
|
22
32
|
*
|
|
23
33
|
* A reader that stops early — `| head`, `| jq` exiting on the first match, a closed
|
|
24
34
|
* pager — closes the pipe while the CLI is still writing, and the next write fails
|
|
25
|
-
* with EPIPE
|
|
26
|
-
*
|
|
27
|
-
*
|
|
28
|
-
*
|
|
35
|
+
* with EPIPE (ENOTCONN when stdout is a socket whose peer has gone, as when a Node
|
|
36
|
+
* parent spawns the CLI with piped stdio on macOS). That is ordinary use, so the
|
|
37
|
+
* process exits 0 at once, quietly. Any
|
|
38
|
+
* other stdout error is an ERROR record of `ckan.output` (`Could not write to stdout:
|
|
39
|
+
* <message>`, through `log`, in the run's format) and exits 1. On stderr an EPIPE is
|
|
40
|
+
* ignored, so a failed run keeps its exit code; any other stderr error exits 1 silently
|
|
41
|
+
* (there is nowhere left to report it).
|
|
42
|
+
* The bin shim installs this once, before `run()`, with a logger for the format argv
|
|
43
|
+
* asks for (`processLogger`).
|
|
29
44
|
*/
|
|
30
|
-
export declare function handleOutputErrors(streams?: OutputStreams, exit?: (code: number) => void): void;
|
|
45
|
+
export declare function handleOutputErrors(streams?: OutputStreams, exit?: (code: number) => void, log?: Pick<Logger, "error">): void;
|
|
46
|
+
/** What `stderrAfterStdout` needs of stdout: its backlog, and the events that end one. */
|
|
47
|
+
export interface StdoutBacklog {
|
|
48
|
+
readonly writableLength: number;
|
|
49
|
+
on(event: "drain" | "close" | "error", listener: () => void): unknown;
|
|
50
|
+
}
|
|
51
|
+
/**
|
|
52
|
+
* A stderr writer that waits for stdout. With both streams on one pipe (`2>&1 |`) and a
|
|
53
|
+
* slow reader, stdout's data is still queued in the process while a record is written
|
|
54
|
+
* to stderr at once, so the record landed inside the JSON (at 64 KiB). Here a text is
|
|
55
|
+
* held while `stdout.writableLength > 0` and written, in order, once the backlog is
|
|
56
|
+
* gone: on stdout's `drain`, `close` or `error`, or at the latest when a short poll sees
|
|
57
|
+
* it empty (`drain` only follows a write that returned false). The poll keeps the
|
|
58
|
+
* process alive until then, so a held record is never lost at exit.
|
|
59
|
+
*/
|
|
60
|
+
export declare function stderrAfterStdout(stdout: StdoutBacklog, write: (text: string) => void): (text: string) => void;
|
|
31
61
|
export declare const defaultIO: CliIO;
|
package/dist/src/cli/io.js
CHANGED
|
@@ -1,29 +1,85 @@
|
|
|
1
1
|
// I/O seam for the CLI. Everything the CLI writes goes through a CliIO object so
|
|
2
2
|
// tests can capture output instead of hitting the real stdout/stderr, and the
|
|
3
3
|
// environment is injected so tests never depend on the real process.env.
|
|
4
|
+
import { createLogger } from "./log.js";
|
|
5
|
+
/** The deps' logger, or one that writes text records through `io.err`. */
|
|
6
|
+
export function logOf(deps) {
|
|
7
|
+
return deps.log ?? createLogger({ format: "text", write: (line) => deps.io.err(line), ...(deps.now === undefined ? {} : { now: deps.now }) });
|
|
8
|
+
}
|
|
4
9
|
/**
|
|
5
10
|
* Handle write errors on stdout/stderr, which Node otherwise reports as an
|
|
6
11
|
* unhandled 'error' event: a raw stack trace and exit 1.
|
|
7
12
|
*
|
|
8
13
|
* A reader that stops early — `| head`, `| jq` exiting on the first match, a closed
|
|
9
14
|
* pager — closes the pipe while the CLI is still writing, and the next write fails
|
|
10
|
-
* with EPIPE
|
|
11
|
-
*
|
|
12
|
-
*
|
|
13
|
-
*
|
|
15
|
+
* with EPIPE (ENOTCONN when stdout is a socket whose peer has gone, as when a Node
|
|
16
|
+
* parent spawns the CLI with piped stdio on macOS). That is ordinary use, so the
|
|
17
|
+
* process exits 0 at once, quietly. Any
|
|
18
|
+
* other stdout error is an ERROR record of `ckan.output` (`Could not write to stdout:
|
|
19
|
+
* <message>`, through `log`, in the run's format) and exits 1. On stderr an EPIPE is
|
|
20
|
+
* ignored, so a failed run keeps its exit code; any other stderr error exits 1 silently
|
|
21
|
+
* (there is nowhere left to report it).
|
|
22
|
+
* The bin shim installs this once, before `run()`, with a logger for the format argv
|
|
23
|
+
* asks for (`processLogger`).
|
|
14
24
|
*/
|
|
15
|
-
export function handleOutputErrors(streams = process, exit = (code) => process.exit(code)) {
|
|
25
|
+
export function handleOutputErrors(streams = process, exit = (code) => process.exit(code), log = createLogger({ format: "text", write: (line) => process.stderr.write(line + "\n") })) {
|
|
16
26
|
streams.stdout.on("error", (err) => {
|
|
17
|
-
if (err
|
|
27
|
+
if (readerGone(err))
|
|
18
28
|
return exit(0);
|
|
19
|
-
|
|
29
|
+
log.error("output", `Could not write to stdout: ${err.message}`);
|
|
20
30
|
exit(1);
|
|
21
31
|
});
|
|
32
|
+
// stderr's reader going away doesn't make a failed run a success: ignore EPIPE there
|
|
33
|
+
// and let the run's own exit code stand (`2>&1 | true` used to turn a usage error into 0).
|
|
22
34
|
streams.stderr.on("error", (err) => {
|
|
23
|
-
|
|
35
|
+
if (!readerGone(err))
|
|
36
|
+
exit(1);
|
|
24
37
|
});
|
|
25
38
|
}
|
|
39
|
+
/** True for the write errors that mean the reader has gone: EPIPE, or ENOTCONN on a socket. */
|
|
40
|
+
function readerGone(err) {
|
|
41
|
+
return err.code === "EPIPE" || err.code === "ENOTCONN";
|
|
42
|
+
}
|
|
43
|
+
/** How often a held record checks whether stdout's backlog is gone (ms). */
|
|
44
|
+
const BACKLOG_POLL_MS = 10;
|
|
45
|
+
/**
|
|
46
|
+
* A stderr writer that waits for stdout. With both streams on one pipe (`2>&1 |`) and a
|
|
47
|
+
* slow reader, stdout's data is still queued in the process while a record is written
|
|
48
|
+
* to stderr at once, so the record landed inside the JSON (at 64 KiB). Here a text is
|
|
49
|
+
* held while `stdout.writableLength > 0` and written, in order, once the backlog is
|
|
50
|
+
* gone: on stdout's `drain`, `close` or `error`, or at the latest when a short poll sees
|
|
51
|
+
* it empty (`drain` only follows a write that returned false). The poll keeps the
|
|
52
|
+
* process alive until then, so a held record is never lost at exit.
|
|
53
|
+
*/
|
|
54
|
+
export function stderrAfterStdout(stdout, write) {
|
|
55
|
+
const held = [];
|
|
56
|
+
let poll;
|
|
57
|
+
const flush = () => {
|
|
58
|
+
if (poll !== undefined)
|
|
59
|
+
clearInterval(poll);
|
|
60
|
+
poll = undefined;
|
|
61
|
+
for (const text of held.splice(0))
|
|
62
|
+
write(text);
|
|
63
|
+
};
|
|
64
|
+
let listening = false;
|
|
65
|
+
return (text) => {
|
|
66
|
+
if (held.length === 0 && stdout.writableLength === 0)
|
|
67
|
+
return write(text);
|
|
68
|
+
held.push(text);
|
|
69
|
+
if (!listening) {
|
|
70
|
+
listening = true;
|
|
71
|
+
for (const event of ["drain", "close", "error"])
|
|
72
|
+
stdout.on(event, flush);
|
|
73
|
+
}
|
|
74
|
+
poll ??= setInterval(() => {
|
|
75
|
+
if (stdout.writableLength === 0)
|
|
76
|
+
flush();
|
|
77
|
+
}, BACKLOG_POLL_MS);
|
|
78
|
+
};
|
|
79
|
+
}
|
|
80
|
+
const stderrLine = stderrAfterStdout(process.stdout, (text) => process.stderr.write(text + "\n"));
|
|
26
81
|
export const defaultIO = {
|
|
27
82
|
out: (text) => process.stdout.write(text + "\n"),
|
|
28
|
-
|
|
83
|
+
// A record never lands inside the data when both streams share a pipe.
|
|
84
|
+
err: stderrLine,
|
|
29
85
|
};
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
/** The log formats `--log-format` takes. */
|
|
2
|
+
export declare const LOG_FORMATS: readonly ["text", "jsonl"];
|
|
3
|
+
export type LogFormat = (typeof LOG_FORMATS)[number];
|
|
4
|
+
/** The format without `--log-format`. */
|
|
5
|
+
export declare const DEFAULT_LOG_FORMAT: LogFormat;
|
|
6
|
+
export type LogLevel = "ERROR" | "WARN" | "INFO";
|
|
7
|
+
/** The program's name in every topic: `ckan.<area>`. */
|
|
8
|
+
export declare const LOG_PROGRAM = "ckan";
|
|
9
|
+
export interface LogRecord {
|
|
10
|
+
/** ISO 8601, UTC, milliseconds. */
|
|
11
|
+
ts: string;
|
|
12
|
+
level: LogLevel;
|
|
13
|
+
/** `<program>.<area>`, like `ckan.http`. */
|
|
14
|
+
topic: string;
|
|
15
|
+
msg: string;
|
|
16
|
+
}
|
|
17
|
+
/**
|
|
18
|
+
* `text` with every character that could split a record, forge a second one or steer
|
|
19
|
+
* the terminal written as an escape: CR as `\r`, LF as `\n`, any other C0 control but
|
|
20
|
+
* TAB, DEL and C1 as `\u00XX`, U+2028, U+2029 and the bidi controls (U+061C, U+200E,
|
|
21
|
+
* U+200F, U+202A–U+202E, U+2066–U+2069) as `\uXXXX`. Backslashes stay as they are. On
|
|
22
|
+
* the output of `JSON.stringify` (no raw C0 left) every escape it adds is valid JSON,
|
|
23
|
+
* so the same helper serves both formats. Checked by char code, so the source stays
|
|
24
|
+
* free of those characters.
|
|
25
|
+
*/
|
|
26
|
+
export declare function escapeForRecord(text: string): string;
|
|
27
|
+
/**
|
|
28
|
+
* The longest message (in characters) a record carries. A longer one is cut at a
|
|
29
|
+
* code-point boundary and ends in `… (N more characters)`: one huge server text or typed
|
|
30
|
+
* value cannot flood stderr or a log store with a line of megabytes. Own messages that
|
|
31
|
+
* quote server or user data are bounded at their source too (`cutForMessage`); this
|
|
32
|
+
* is the backstop for every path.
|
|
33
|
+
*/
|
|
34
|
+
export declare const MAX_RECORD_MESSAGE = 4000;
|
|
35
|
+
/**
|
|
36
|
+
* One record as one line, whatever the message holds: `escapeForRecord` runs over the
|
|
37
|
+
* message (text) or over the whole JSON object (jsonl), so no text that reaches a
|
|
38
|
+
* record can split it, forge another one, or reach the terminal as a control sequence.
|
|
39
|
+
*/
|
|
40
|
+
export declare function formatLogRecord(record: LogRecord, format: LogFormat): string;
|
|
41
|
+
export interface Logger {
|
|
42
|
+
/**
|
|
43
|
+
* The format records are written in. Mutable for one reason: it is read from argv
|
|
44
|
+
* before commander parses (for commander's own errors), then set again from what
|
|
45
|
+
* commander parsed, before the action runs (`run()`'s preAction hook).
|
|
46
|
+
*/
|
|
47
|
+
format: LogFormat;
|
|
48
|
+
error(area: string, msg: string): void;
|
|
49
|
+
warn(area: string, msg: string): void;
|
|
50
|
+
info(area: string, msg: string): void;
|
|
51
|
+
}
|
|
52
|
+
/**
|
|
53
|
+
* A logger that writes each record, formatted, to `write` (stderr). `redact` replaces the
|
|
54
|
+
* secrets of the run in the message, before it is cut and escaped: the frame (time,
|
|
55
|
+
* level, topic) is never touched, so a secret equal to a year or a topic cannot corrupt
|
|
56
|
+
* it, and a secret holding DEL, C1 or bidi characters is found in its raw form.
|
|
57
|
+
*/
|
|
58
|
+
export declare function createLogger(options: {
|
|
59
|
+
format: LogFormat;
|
|
60
|
+
write: (line: string) => void;
|
|
61
|
+
now?: () => Date;
|
|
62
|
+
redact?: (text: string) => string;
|
|
63
|
+
}): Logger;
|
|
64
|
+
/** Why `value` is not a log format, or undefined. */
|
|
65
|
+
export declare function logFormatProblem(value: string): string | undefined;
|
|
66
|
+
/**
|
|
67
|
+
* The `--log-format` in `argv`, read before commander parses it: commander's own
|
|
68
|
+
* usage errors are logged too, and they happen while parsing. A missing or unknown
|
|
69
|
+
* value gives the default here; commander then reports an unknown one. The first
|
|
70
|
+
* `--log-format` counts: commander keeps it and rejects a second one (`once`), so that
|
|
71
|
+
* error is logged in the first one's format. This scan is only for the records of a
|
|
72
|
+
* parse error: once commander has parsed argv, its value is the format (the program's
|
|
73
|
+
* first preAction hook), so `--user-agent --log-format=jsonl` (a User-Agent) logs text.
|
|
74
|
+
* `valueOptions` names the program's options that take a value (`--user-agent`): the
|
|
75
|
+
* token after one is its value, never an option, as commander reads it, so `--user-agent
|
|
76
|
+
* --log-format jsonl` agrees with commander in a parse error too. A subcommand's value
|
|
77
|
+
* option (`--rows`) does not count: commander takes the program's own options out of
|
|
78
|
+
* argv first, so `search --rows --log-format jsonl` is jsonl and a `--rows` without its
|
|
79
|
+
* value.
|
|
80
|
+
*/
|
|
81
|
+
export declare function logFormatFromArgv(argv: readonly string[], valueOptions?: ReadonlySet<string>): LogFormat;
|
|
82
|
+
/** What `installWarningLog` needs of the process. */
|
|
83
|
+
export interface WarningSource {
|
|
84
|
+
removeAllListeners(event: "warning"): unknown;
|
|
85
|
+
on(event: "warning", listener: (warning: Error) => void): unknown;
|
|
86
|
+
}
|
|
87
|
+
/**
|
|
88
|
+
* Node's own process warnings (`(node:PID) Warning: …`, e.g. for
|
|
89
|
+
* `NODE_TLS_REJECT_UNAUTHORIZED=0`) as WARN records of `ckan.cli`:
|
|
90
|
+
* `(node) <name>: <message>`, in the log's format and with its redaction. Node's default
|
|
91
|
+
* listener, which prints the plain line, is removed (checked on Node 22, 24 and 26: that
|
|
92
|
+
* silences it; `--no-warnings` is not needed). The bin shim installs it once, before
|
|
93
|
+
* `run()`; a warning Node emits while it starts up, before any code runs, stays Node's.
|
|
94
|
+
*/
|
|
95
|
+
export declare function installWarningLog(source: WarningSource, log: Pick<Logger, "warn">): void;
|