@maschinenlesbar.org/ckan-cli 0.1.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,78 @@
1
+ # Data license
2
+
3
+ > **This tool does not include, host, or redistribute any data.**
4
+ > `ckan-cli` is a *client*. It only accesses records served live by whichever
5
+ > CKAN portal you point it at. Those records are governed by **that portal's**
6
+ > terms, and often by a licence set per dataset. The license of this CLI's own
7
+ > source code is a separate matter — see [LICENSING.md](LICENSING.md).
8
+
9
+ > [!IMPORTANT]
10
+ > **The terms depend on the portal and usually on the record.** A generic CKAN
11
+ > client cannot know them in advance. Every CKAN record carries its own
12
+ > `license_id` / `license_title` / `license_url`; check them before reusing a
13
+ > record's contents. `ckan licenses` lists the licences a portal offers, and
14
+ > `ckan search --rows 0 --facet license_id` shows how often each is used.
15
+
16
+ ## Default portal: Transparenzportal Hamburg
17
+
18
+ | | |
19
+ |---|---|
20
+ | **Data provider** | Freie und Hansestadt Hamburg; each record is the responsibility of its publishing body |
21
+ | **API / source** | `https://suche.transparenz.hamburg.de/api/3/action` (CKAN Action API) · portal: https://transparenz.hamburg.de/ |
22
+ | **Legal basis** | Hamburgisches Transparenzgesetz (HmbTG), § 10 Abs. 3: *"Die Nutzung, Weiterverwendung und Verbreitung der Informationen ist frei, sofern höherrangiges Recht oder spezialgesetzliche Regelungen nichts anderes bestimmen."* |
23
+ | **Portal statement** | *"Die Dokumente und Daten aus dem Transparenzportal Hamburg dürfen frei verwendet werden."* Records are published under two licences that allow free reuse, *"ggf. unter Nennung des Namens"*: Datenlizenz Deutschland – Namensnennung 2.0 and – Zero 2.0. |
24
+ | **Licence** | **Per record**, in `license_id`. As of 2026-09-19: `dl-de-by-2.0` (242,632 records), `cc-zero` (2,203), `dl-zero-de-2.0` / `dl-de-zero-2.0` (262), `odbl` (59), `geoNutz-20130319` (48). |
25
+ | **Attribution** | Per record, in the `terms_of_use` extra (`attribution_text`), e.g. *"Namensnennung: Freie und Hansestadt Hamburg, Bezirksversammlung Wandsbek"*. The wording names the publishing body, so it varies. |
26
+ | **Metadata** | Not licensed separately by the portal. |
27
+ | **Automated access** | Invited: the API page describes it as the way to fetch *"Inhalte und Metadaten aus dem Transparenzportal automatisiert"*. No rate limits or API terms are published; HmbTG § 10 Abs. 4 makes access *"kostenlos und anonym"*. |
28
+
29
+ ### Watch for
30
+
31
+ - **`dl-de-by-2.0`** (the vast majority) requires attribution. Use the record's
32
+ own `attribution_text`, and link the licence
33
+ (https://www.govdata.de/dl-de/by-2-0).
34
+ - **`odbl`** records are share-alike: derived databases must stay under the ODbL.
35
+ - **`geoNutz-20130319`** is Hamburg's older geodata usage licence, not a
36
+ standard open licence. Read its terms before reusing those records.
37
+ - **Third-party rights.** HmbTG § 10 Abs. 3 S. 3 only obliges authorities to
38
+ *try* to obtain reuse rights for third-party works (reports, studies), and
39
+ § 8 Abs. 1 lets intellectual-property protection override publication. A
40
+ document written by a third party can carry rights the record's licence does
41
+ not grant.
42
+ - **Personal data** is redacted before publication (HmbTG § 4), but some names
43
+ are published on purpose (contract partners, authors of reports, subsidy
44
+ recipients). Data-protection law still applies to what you do with them.
45
+ - **The website itself** (layout, graphics, editorial pages) is copyright
46
+ protected per the Impressum; that concerns transparenz.hamburg.de, not the
47
+ register records served by the API.
48
+
49
+ ### Attribution
50
+
51
+ ```
52
+ Source: Transparenzportal Hamburg (https://transparenz.hamburg.de/),
53
+ <attribution_text of the record>, licensed under <license_title of the record>
54
+ (<license_url of the record>).
55
+ ```
56
+
57
+ ## Other portals
58
+
59
+ With `--portal`, `--base-url` or `CKAN_BASE_URL` you are subject to that portal's
60
+ terms. Being in the built-in list (`ckan portals`) says only that a portal runs a
61
+ working CKAN, nothing about its terms.
62
+ Look them up on the portal itself; do not assume they match Hamburg's. For
63
+ GovData, see the `DATA_LICENSE.md` of
64
+ [govdata-cli](https://github.com/maschinenlesbar-org/govdata-cli).
65
+
66
+ ## Sources
67
+
68
+ - https://transparenz.hamburg.de/verwendung-der-daten-aus-dem-transparenzportal-796520 — free use, the two licences
69
+ - https://transparenz.hamburg.de/api-796358 — API notes, automated access
70
+ - https://transparenz.hamburg.de/gesetzestext-des-hmbtg-796514 — HmbTG §§ 4, 7, 8, 10
71
+ - https://transparenz.hamburg.de/impressum-796468 — responsibility per publishing body, website copyright
72
+ - https://www.govdata.de/dl-de/by-2-0 — Datenlizenz Deutschland – Namensnennung – 2.0
73
+ - Record counts: `ckan search --rows 0 --facet license_id` against the live API, 2026-09-19
74
+
75
+ ---
76
+
77
+ *Good-faith summary compiled 2026-09-19; not legal advice. The portal's terms
78
+ and each record's licence are authoritative and can change.*
package/README.md CHANGED
@@ -24,7 +24,7 @@ search, inspect, filter and pipe straight into [`jq`](https://jqlang.github.io/j
24
24
  `status`, `portals`, and a generic `action` escape hatch for everything else.
25
25
 
26
26
  > Want to use this as a TypeScript library or understand how it's built?
27
- > See **[DEVELOPING.md](DEVELOPING.md)**.
27
+ > See **[DEVELOPING.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/DEVELOPING.md)**.
28
28
 
29
29
  ## Install
30
30
 
@@ -32,7 +32,7 @@ search, inspect, filter and pipe straight into [`jq`](https://jqlang.github.io/j
32
32
  npm i -g @maschinenlesbar.org/ckan-cli
33
33
  ```
34
34
 
35
- This installs the **`ckan`** command. Requires **Node.js 20+**.
35
+ This installs the **`ckan`** command. Requires **Node.js 22.12+**.
36
36
 
37
37
  ## Quickstart
38
38
 
@@ -79,10 +79,16 @@ sub-path keeps it (`https://open.canada.ca/data`). `ckan status` is a good first
79
79
  call on an unfamiliar portal: it shows the CKAN version and the installed
80
80
  extensions.
81
81
 
82
+ A portal behind a login takes its user name and password in the URL
83
+ (`https://user:password@host`). They go out as HTTP Basic auth to that site only:
84
+ a redirect to another host, port or scheme drops them, and no message, help text
85
+ or error shows them.
86
+
82
87
  ### Known portals
83
88
 
84
89
  `ckan portals` lists the CKAN portals in Germany this version knows, with the
85
90
  result of their last check; `ckan portals --check` checks them all live now.
91
+ Neither uses `CKAN_BASE_URL`, so both work even when the variable is broken.
86
92
  The list is built into the CLI, so it needs no outside service:
87
93
 
88
94
  | `--portal` | Portal | Datasets |
@@ -108,7 +114,7 @@ The list is built into the CLI, so it needs no outside service:
108
114
 
109
115
  Counts as of the last check, 2026-09-19. The list is refreshed from Wikidata,
110
116
  the CKAN project's instance registry and GovData's harvest sources, keeping only
111
- portals that pass a live check (see [DEVELOPING.md](DEVELOPING.md#the-portal-list)).
117
+ portals that pass a live check (see [DEVELOPING.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/DEVELOPING.md#the-portal-list)).
112
118
  Many portals those lists name have left CKAN (Köln, Düsseldorf, Bonn, …).
113
119
 
114
120
  ## Commands
@@ -152,7 +158,9 @@ action <name> [--param key=value …] call any CKAN action (generic)
152
158
  > **Blank values are refused.** `ckan search ""`, `--fq ""`, `tags --query ""` or
153
159
  > `action … --param q=` exit 1 before any request, instead of silently running an unfiltered search.
154
160
  > So does a forgotten value (`--fq --rows 5`): a value of `--fq`, `--facet`,
155
- > `--sort` or `tags --query` cannot start with `--`.
161
+ > `--sort` or `tags --query` cannot start with `--`. And so does a single-value
162
+ > option given twice (`--rows 5 --rows 50`, a second `--base-url`): only `--fq`,
163
+ > `--facet` and `--param` repeat.
156
164
 
157
165
  ### Paging and list flags
158
166
 
@@ -163,7 +171,9 @@ action <name> [--param key=value …] call any CKAN action (generic)
163
171
 
164
172
  > **`--all-fields` lists are complete.** CKAN caps them at 25 entries
165
173
  > (`ckan.group_and_organization_list_all_fields_max`) and drops the rest
166
- > silently, even for a larger `--limit`; `ckan` pages through them for you.
174
+ > silently, even for a larger `--limit`; `ckan` pages through them for you,
175
+ > 0.1 s apart. A server whose pages never end is stopped after 400 pages
176
+ > (10,000 entries, far more than any portal lists) with an error (exit 1).
167
177
  | `tags` | `--query <substring>` |
168
178
 
169
179
  ### `action`
@@ -187,7 +197,7 @@ before any request is sent.
187
197
  matches nothing, and a full word such as `verwaltungsvorschriften` over-matches.
188
198
  List the values that work, with their counts:
189
199
  `ckan --compact search --rows 0 --facet extras_registerobject_type --facet-limit -1`.
190
- The [Glossary](GLOSSARY.md) has the details.
200
+ The [Glossary](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/GLOSSARY.md) has the details.
191
201
  - **Several filters work.** Hamburg's API page says only one filter can be
192
202
  used; with `fq_list` (repeat `--fq`) they combine:
193
203
  `ckan search --fq extras_registerobject_type:verwaltungsvorschrift --fq organization:workflows`.
@@ -206,6 +216,9 @@ Found by running the same commands against many portals:
206
216
  - **The default sort differs** (ZBW sorts by `metadata_created desc`).
207
217
  - **Field types vary**: licence ids are URIs on GovData, and some portals send
208
218
  booleans as the strings `"True"`/`"False"`.
219
+ - **A purely negative query matches nothing**: `ckan search -- -elbe` answers
220
+ `count: 0` on Hamburg rather than "everything but elbe". Negate a filter instead
221
+ (`--fq -organization:allris`), next to a positive query or none.
209
222
  - **Solr syntax errors** come back as HTTP 409; `ckan` prints Solr's own reason,
210
223
  e.g. `Search Error: Cannot parse 'title:(': Encountered "<EOF>" …`.
211
224
  - **A site that is not (or no longer) a CKAN** answers with a 404, an HTML page or
@@ -218,6 +231,25 @@ Every command prints the **unwrapped `result`** as pretty JSON to stdout.
218
231
  Errors go to stderr, so piping stdout into `jq` stays clean. `--compact`
219
232
  prints one line. Every global option works before or after the command.
220
233
 
234
+ Each line on stderr is a **log record**: a timestamp (UTC), a level (`ERROR`, `WARN`,
235
+ `INFO`) and a topic, the program and the area it comes from (`ckan.cli` for usage
236
+ errors, `ckan.api` for the portal's answers, a malformed one included, `ckan.http` for the connection,
237
+ `ckan.output` when stdout can't be written). By default
238
+ it is written log4j style; `--log-format jsonl` writes one JSON object per line instead.
239
+ A record is always one line: a line break, a control character or a bidi control in a
240
+ message (a server's text, a value you typed) is written as an escape (`\n`, `\u001b`,
241
+ `\u202e`), so it can neither split a record nor forge another one, nor steer the terminal;
242
+ a message longer than 4000 characters is cut and ends in `… (N more characters)`:
243
+
244
+ ```text
245
+ 2026-10-09T14:03:12.481Z WARN [ckan.http] requests to ckan.example are sent unencrypted (http:, not https:)
246
+ 2026-10-09T14:03:12.902Z ERROR [ckan.api] HTTP 404 for GET http://ckan.example/api/3/action/status_show: Not Found Error: Not found
247
+ ```
248
+
249
+ ```bash
250
+ ckan --log-format jsonl package no-such-id 2>log.jsonl # {"ts":"…","level":"ERROR","topic":"ckan.api","msg":"HTTP 404 …"}
251
+ ```
252
+
221
253
  ```bash
222
254
  # How many records a portal has
223
255
  ckan search --rows 0 | jq '.count'
@@ -234,10 +266,13 @@ ckan package bezirk-wandsbek-drucksache-22-3573-2 | jq '.resources[] | {format,
234
266
  | Code | Meaning |
235
267
  | --- | --- |
236
268
  | `0` | success (also `--help`, `help [command]`, `--version` and a bare `ckan`) |
237
- | `4` | not found (HTTP `404`): an unknown id on `package`/`resource`/`organization`/`group`, or, on any command, no CKAN at this URL (a wrong `--base-url`, a portal that left CKAN) |
238
- | `1` | any other error — bad usage, validation error (`409`), unknown action (`400`), CKAN `success:false`, network failure |
269
+ | `4` | not found (HTTP `404`, or CKAN's `Not Found Error` sent with HTTP `200`): an unknown id on `package`/`resource`/`organization`/`group`, or, on any command, no CKAN at this URL (a wrong `--base-url`, a portal that left CKAN) |
270
+ | `1` | any other error — bad usage (an unknown command, also after `help`), validation error (`409`), unknown action (`400`), any other CKAN `success:false`, network failure |
239
271
 
240
- CKAN's error text is shown on stderr, e.g.
272
+ A reader that stops early (`| head`) ends `ckan` quietly with exit 0; a failed
273
+ run keeps its exit code even when nothing reads stderr.
274
+
275
+ CKAN's error text is shown on stderr (in an `ERROR` record of `ckan.api`), e.g.
241
276
  `Validation Error: rows: Invalid integer` or
242
277
  `Bad request - Action name not known: …`.
243
278
 
@@ -254,16 +289,24 @@ CKAN's error text is shown on stderr, e.g.
254
289
  | `-V, --version` | Print the version number |
255
290
  | `-h, --help` | Show help for the program or a command |
256
291
  | `--compact` | Print JSON on a single line instead of pretty-printed |
292
+ | `--log-format <format>` | How errors, warnings and notes are written to stderr: `text` (default; log4j style, `2026-10-09T14:03:12.481Z WARN [ckan.http] …`) or `jsonl` (one JSON object per line: `ts`, `level`, `topic`, `msg`). stdout is not affected |
257
293
  | `--base-url <url>` | CKAN site URL (env `CKAN_BASE_URL`; default `https://suche.transparenz.hamburg.de`) |
258
294
  | `--portal <id>` | a known portal by id (`ckan portals`); cannot be combined with `--base-url` |
259
295
  | `--timeout <ms>` | Per-request timeout (default `30000`; at most `2147483647`) |
260
296
  | `--user-agent <ua>` | `User-Agent` header value |
261
- | `--max-retries <n>` | Retries for transient `429`/`503` responses (`0`–`10`, default `2`). Each retry waits the server's `Retry-After` (up to 30 s; a longer one is not retried) or else backs off linearly |
297
+ | `--max-retries <n>` | Retries for transient `429`/`503` responses (`0`–`10`, default `2`). Each retry backs off linearly (0.2 s, 0.4 s, …) or waits the server's `Retry-After` when that is longer, up to 30 s; a longer one is not retried, and the error names the wait. A reset or other network error is not retried |
262
298
  | `--max-response-bytes <n>` | Cap response body size in bytes (`0` = unlimited; default 100 MiB) |
263
299
 
300
+ A base URL on plain `http:` (from `--base-url` or `CKAN_BASE_URL`) to a host other than
301
+ loopback (`localhost`, `127.0.0.0/8`, `::1`) works, but the CLI writes one `WARN` record of
302
+ `ckan.http` to stderr before the first request, e.g.
303
+ `… WARN [ckan.http] requests to ckan.example are sent unencrypted (http:, not https:)`, or
304
+ `… WARN [ckan.http] the base URL's credentials are sent unencrypted to ckan.example (http:, not https:)`
305
+ when it carries a `user:password@` (never printed). stdout and the exit code are unchanged.
306
+
264
307
  ## Claude Code skills
265
308
 
266
- Three [Agent Skills](SKILLS.md) teach Claude Code to use `ckan` for real questions: find
309
+ Three [Agent Skills](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/SKILLS.md) teach Claude Code to use `ckan` for real questions: find
267
310
  which portals have data on a topic and rank the hits (**ckan-dataset-finder**), build
268
311
  publisher/format/licence statistics (**ckan-catalogue-stats**), and search the Hamburg
269
312
  Transparenzportal's contracts, Gutachten and Senate papers
@@ -274,7 +317,7 @@ Transparenzportal's contracts, Gutachten and Senate papers
274
317
  /plugin install ckan@maschinenlesbar
275
318
  ```
276
319
 
277
- Real runs of each skill are in [EXAMPLE.md](EXAMPLE.md).
320
+ Real runs of each skill are in [EXAMPLE.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/EXAMPLE.md).
278
321
 
279
322
  ## Data license
280
323
 
@@ -2,7 +2,7 @@
2
2
  // and tags.
3
3
  import { InvalidArgumentError } from "commander";
4
4
  import { facetLimitProblem } from "../../client/validate.js";
5
- import { action, collectNonEmpty, parseBoundedInt, parseIntArg, parseNonEmpty, parseOptionValue, renderJson, } from "../shared.js";
5
+ import { action, collectNonEmpty, once, parseBoundedInt, parseIntArg, parseNonEmpty, parseOptionValue, renderJson, } from "../shared.js";
6
6
  /**
7
7
  * commander value-parser for a list --limit: 1 or more. CKAN reads `limit=0` as
8
8
  * "no limit" and would send the whole list; leave --limit out for that.
@@ -24,12 +24,12 @@ export function registerCatalogueCommands(program, deps) {
24
24
  .command("search")
25
25
  .description("Search datasets (Solr query syntax)")
26
26
  .argument("[query]", "full-text query, e.g. elbe or title:haushalt", parseNonEmpty)
27
- .option("--rows <n>", "max results (servers cap this, usually at 1000)", parseIntArg)
28
- .option("--start <n>", "offset for paging", parseIntArg)
29
- .option("--sort <expr>", 'e.g. "metadata_modified desc"', parseOptionValue)
27
+ .option("--rows <n>", "max results (servers cap this, usually at 1000)", once(parseIntArg))
28
+ .option("--start <n>", "offset for paging", once(parseIntArg))
29
+ .option("--sort <expr>", 'e.g. "metadata_modified desc"', once(parseOptionValue))
30
30
  .option("--fq <filter>", "filter query, e.g. organization:allris (repeatable; all must match)", collectNonEmpty)
31
31
  .option("--facet <field>", "count values of a field, e.g. res_format (repeatable)", collectNonEmpty)
32
- .option("--facet-limit <n>", "max values per facet (default 50; -1 = all)", parseFacetLimit)
32
+ .option("--facet-limit <n>", "max values per facet (default 50; -1 = all)", once(parseFacetLimit))
33
33
  .action(action(deps, async ({ client, global, opts }, [query]) => {
34
34
  renderJson(deps, global, await client.packageSearch({
35
35
  q: query,
@@ -72,8 +72,8 @@ export function registerCatalogueCommands(program, deps) {
72
72
  program
73
73
  .command("packages")
74
74
  .description("List dataset names")
75
- .option("--limit <n>", "max names (1 or more; omit for all)", parseLimit)
76
- .option("--offset <n>", "offset for paging", parseIntArg)
75
+ .option("--limit <n>", "max names (1 or more; omit for all)", once(parseLimit))
76
+ .option("--offset <n>", "offset for paging", once(parseIntArg))
77
77
  .action(action(deps, async ({ client, global, opts }) => {
78
78
  renderJson(deps, global, await client.packageList({
79
79
  limit: opts["limit"],
@@ -84,8 +84,8 @@ export function registerCatalogueCommands(program, deps) {
84
84
  .command("organizations")
85
85
  .description("List organizations (data publishers)")
86
86
  .option("--all-fields", "return full objects instead of names")
87
- .option("--limit <n>", "max entries (1 or more; omit for all)", parseLimit)
88
- .option("--offset <n>", "offset for paging", parseIntArg)
87
+ .option("--limit <n>", "max entries (1 or more; omit for all)", once(parseLimit))
88
+ .option("--offset <n>", "offset for paging", once(parseIntArg))
89
89
  .action(action(deps, async ({ client, global, opts }) => {
90
90
  renderJson(deps, global, await client.organizationList({
91
91
  all_fields: opts["allFields"],
@@ -97,8 +97,8 @@ export function registerCatalogueCommands(program, deps) {
97
97
  .command("groups")
98
98
  .description("List groups (themes/categories)")
99
99
  .option("--all-fields", "return full objects instead of names")
100
- .option("--limit <n>", "max entries (1 or more; omit for all)", parseLimit)
101
- .option("--offset <n>", "offset for paging", parseIntArg)
100
+ .option("--limit <n>", "max entries (1 or more; omit for all)", once(parseLimit))
101
+ .option("--offset <n>", "offset for paging", once(parseIntArg))
102
102
  .action(action(deps, async ({ client, global, opts }) => {
103
103
  renderJson(deps, global, await client.groupList({
104
104
  all_fields: opts["allFields"],
@@ -109,7 +109,7 @@ export function registerCatalogueCommands(program, deps) {
109
109
  program
110
110
  .command("tags")
111
111
  .description("List tags")
112
- .option("--query <substring>", "only tags containing this substring", parseOptionValue)
112
+ .option("--query <substring>", "only tags containing this substring", once(parseOptionValue))
113
113
  .action(action(deps, async ({ client, global, opts }) => {
114
114
  renderJson(deps, global, await client.tagList({ query: opts["query"] }));
115
115
  }));
@@ -4,6 +4,15 @@ import { action, renderJson, toEngineOptions } from "../shared.js";
4
4
  import { checkPortals } from "../../client/portals.js";
5
5
  import { PORTALS } from "../../client/portals-list.js";
6
6
  import { isBlank } from "../../client/validate.js";
7
+ import { cutForMessage, redactUrl } from "../../client/errors.js";
8
+ /**
9
+ * A typed value as a message quotes it: cut at MAX_MESSAGE_VALUE_LENGTH, after its
10
+ * credentials are redacted (a cut could otherwise leave part of a password without the
11
+ * "@" the argv redaction keys on).
12
+ */
13
+ function shown(value) {
14
+ return cutForMessage(redactUrl(value));
15
+ }
7
16
  /** commander accumulator for repeatable `key=value` pairs into a record. */
8
17
  function collectKeyValue(value, previous = {}) {
9
18
  const eq = value.indexOf("=");
@@ -13,18 +22,18 @@ function collectKeyValue(value, previous = {}) {
13
22
  const key = value.slice(0, Math.max(eq, 0));
14
23
  // A blank key (`=x`, ` =x`) names no parameter: the library refuses it too.
15
24
  if (eq < 0 || isBlank(key)) {
16
- throw new InvalidArgumentError(`Invalid --param "${value}". Expected key=value.`);
25
+ throw new InvalidArgumentError(`Invalid --param "${shown(value)}". Expected key=value.`);
17
26
  }
18
27
  // Reject a duplicated key rather than silently overwriting the earlier value.
19
28
  if (Object.prototype.hasOwnProperty.call(previous, key)) {
20
- throw new InvalidArgumentError(`Duplicate --param key "${key}".`);
29
+ throw new InvalidArgumentError(`Duplicate --param key "${shown(key)}".`);
21
30
  }
22
31
  // A blank value is refused by the library as well (CKAN reads an empty
23
32
  // parameter as unset, so `--param q=` would run the action unfiltered); here it
24
33
  // is a usage error that names the flag.
25
34
  const paramValue = value.slice(eq + 1);
26
35
  if (isBlank(paramValue)) {
27
- throw new InvalidArgumentError(`Invalid --param "${value}". The value must not be blank.`);
36
+ throw new InvalidArgumentError(`Invalid --param "${shown(value)}". The value must not be blank.`);
28
37
  }
29
38
  // A computed key in a literal is an own data property, even for `__proto__`.
30
39
  return { ...previous, [key]: paramValue };
@@ -1,7 +1,12 @@
1
1
  #!/usr/bin/env node
2
2
  // Binary entry point. Thin shim around run(); all logic lives in run.ts/program.ts.
3
3
  import { handleOutputErrors } from "./io.js";
4
- import { run } from "./run.js";
5
- handleOutputErrors();
6
- const exitCode = await run(process.argv.slice(2));
4
+ import { installWarningLog } from "./log.js";
5
+ import { processLogger, run } from "./run.js";
6
+ const argv = process.argv.slice(2);
7
+ // What happens outside run() is logged too, in the format argv asks for.
8
+ const log = processLogger(argv);
9
+ installWarningLog(process, log);
10
+ handleOutputErrors(process, undefined, log);
11
+ const exitCode = await run(argv);
7
12
  process.exitCode = exitCode;
@@ -1,5 +1,6 @@
1
1
  import type { CkanClient } from "../client/client.js";
2
2
  import type { EngineOptions } from "../client/engine.js";
3
+ import { type Logger } from "./log.js";
3
4
  export interface CliIO {
4
5
  out(text: string): void;
5
6
  err(text: string): void;
@@ -10,7 +11,16 @@ export interface CliDeps {
10
11
  env: Record<string, string | undefined>;
11
12
  /** Build a client from the resolved global options (injectable for tests). */
12
13
  createClient(options: EngineOptions): CkanClient;
14
+ /**
15
+ * Where diagnostics go: one record per line on stderr, in the `--log-format`
16
+ * (`log.ts`). `run()` sets it from argv; deps without it log text through `io.err`.
17
+ */
18
+ log?: Logger;
19
+ /** The clock the log's timestamps come from. Unset, the real one. */
20
+ now?: () => Date;
13
21
  }
22
+ /** The deps' logger, or one that writes text records through `io.err`. */
23
+ export declare function logOf(deps: CliDeps): Logger;
14
24
  /** The two process streams, as far as `handleOutputErrors` needs them. */
15
25
  export interface OutputStreams {
16
26
  stdout: Pick<NodeJS.WriteStream, "on">;
@@ -22,10 +32,30 @@ export interface OutputStreams {
22
32
  *
23
33
  * A reader that stops early — `| head`, `| jq` exiting on the first match, a closed
24
34
  * pager — closes the pipe while the CLI is still writing, and the next write fails
25
- * with EPIPE. That is ordinary use, so the process exits 0 at once, quietly. Any
26
- * other stdout error prints one `Output error: <message>` line to stderr and exits
27
- * 1; any other stderr error exits 1 silently (there is nowhere left to report it).
28
- * The bin shim installs this once, before `run()`.
35
+ * with EPIPE (ENOTCONN when stdout is a socket whose peer has gone, as when a Node
36
+ * parent spawns the CLI with piped stdio on macOS). That is ordinary use, so the
37
+ * process exits 0 at once, quietly. Any
38
+ * other stdout error is an ERROR record of `ckan.output` (`Could not write to stdout:
39
+ * <message>`, through `log`, in the run's format) and exits 1. On stderr an EPIPE is
40
+ * ignored, so a failed run keeps its exit code; any other stderr error exits 1 silently
41
+ * (there is nowhere left to report it).
42
+ * The bin shim installs this once, before `run()`, with a logger for the format argv
43
+ * asks for (`processLogger`).
29
44
  */
30
- export declare function handleOutputErrors(streams?: OutputStreams, exit?: (code: number) => void): void;
45
+ export declare function handleOutputErrors(streams?: OutputStreams, exit?: (code: number) => void, log?: Pick<Logger, "error">): void;
46
+ /** What `stderrAfterStdout` needs of stdout: its backlog, and the events that end one. */
47
+ export interface StdoutBacklog {
48
+ readonly writableLength: number;
49
+ on(event: "drain" | "close" | "error", listener: () => void): unknown;
50
+ }
51
+ /**
52
+ * A stderr writer that waits for stdout. With both streams on one pipe (`2>&1 |`) and a
53
+ * slow reader, stdout's data is still queued in the process while a record is written
54
+ * to stderr at once, so the record landed inside the JSON (at 64 KiB). Here a text is
55
+ * held while `stdout.writableLength > 0` and written, in order, once the backlog is
56
+ * gone: on stdout's `drain`, `close` or `error`, or at the latest when a short poll sees
57
+ * it empty (`drain` only follows a write that returned false). The poll keeps the
58
+ * process alive until then, so a held record is never lost at exit.
59
+ */
60
+ export declare function stderrAfterStdout(stdout: StdoutBacklog, write: (text: string) => void): (text: string) => void;
31
61
  export declare const defaultIO: CliIO;
@@ -1,29 +1,85 @@
1
1
  // I/O seam for the CLI. Everything the CLI writes goes through a CliIO object so
2
2
  // tests can capture output instead of hitting the real stdout/stderr, and the
3
3
  // environment is injected so tests never depend on the real process.env.
4
+ import { createLogger } from "./log.js";
5
+ /** The deps' logger, or one that writes text records through `io.err`. */
6
+ export function logOf(deps) {
7
+ return deps.log ?? createLogger({ format: "text", write: (line) => deps.io.err(line), ...(deps.now === undefined ? {} : { now: deps.now }) });
8
+ }
4
9
  /**
5
10
  * Handle write errors on stdout/stderr, which Node otherwise reports as an
6
11
  * unhandled 'error' event: a raw stack trace and exit 1.
7
12
  *
8
13
  * A reader that stops early — `| head`, `| jq` exiting on the first match, a closed
9
14
  * pager — closes the pipe while the CLI is still writing, and the next write fails
10
- * with EPIPE. That is ordinary use, so the process exits 0 at once, quietly. Any
11
- * other stdout error prints one `Output error: <message>` line to stderr and exits
12
- * 1; any other stderr error exits 1 silently (there is nowhere left to report it).
13
- * The bin shim installs this once, before `run()`.
15
+ * with EPIPE (ENOTCONN when stdout is a socket whose peer has gone, as when a Node
16
+ * parent spawns the CLI with piped stdio on macOS). That is ordinary use, so the
17
+ * process exits 0 at once, quietly. Any
18
+ * other stdout error is an ERROR record of `ckan.output` (`Could not write to stdout:
19
+ * <message>`, through `log`, in the run's format) and exits 1. On stderr an EPIPE is
20
+ * ignored, so a failed run keeps its exit code; any other stderr error exits 1 silently
21
+ * (there is nowhere left to report it).
22
+ * The bin shim installs this once, before `run()`, with a logger for the format argv
23
+ * asks for (`processLogger`).
14
24
  */
15
- export function handleOutputErrors(streams = process, exit = (code) => process.exit(code)) {
25
+ export function handleOutputErrors(streams = process, exit = (code) => process.exit(code), log = createLogger({ format: "text", write: (line) => process.stderr.write(line + "\n") })) {
16
26
  streams.stdout.on("error", (err) => {
17
- if (err.code === "EPIPE")
27
+ if (readerGone(err))
18
28
  return exit(0);
19
- process.stderr.write(`Output error: ${err.message}\n`);
29
+ log.error("output", `Could not write to stdout: ${err.message}`);
20
30
  exit(1);
21
31
  });
32
+ // stderr's reader going away doesn't make a failed run a success: ignore EPIPE there
33
+ // and let the run's own exit code stand (`2>&1 | true` used to turn a usage error into 0).
22
34
  streams.stderr.on("error", (err) => {
23
- exit(err.code === "EPIPE" ? 0 : 1);
35
+ if (!readerGone(err))
36
+ exit(1);
24
37
  });
25
38
  }
39
+ /** True for the write errors that mean the reader has gone: EPIPE, or ENOTCONN on a socket. */
40
+ function readerGone(err) {
41
+ return err.code === "EPIPE" || err.code === "ENOTCONN";
42
+ }
43
+ /** How often a held record checks whether stdout's backlog is gone (ms). */
44
+ const BACKLOG_POLL_MS = 10;
45
+ /**
46
+ * A stderr writer that waits for stdout. With both streams on one pipe (`2>&1 |`) and a
47
+ * slow reader, stdout's data is still queued in the process while a record is written
48
+ * to stderr at once, so the record landed inside the JSON (at 64 KiB). Here a text is
49
+ * held while `stdout.writableLength > 0` and written, in order, once the backlog is
50
+ * gone: on stdout's `drain`, `close` or `error`, or at the latest when a short poll sees
51
+ * it empty (`drain` only follows a write that returned false). The poll keeps the
52
+ * process alive until then, so a held record is never lost at exit.
53
+ */
54
+ export function stderrAfterStdout(stdout, write) {
55
+ const held = [];
56
+ let poll;
57
+ const flush = () => {
58
+ if (poll !== undefined)
59
+ clearInterval(poll);
60
+ poll = undefined;
61
+ for (const text of held.splice(0))
62
+ write(text);
63
+ };
64
+ let listening = false;
65
+ return (text) => {
66
+ if (held.length === 0 && stdout.writableLength === 0)
67
+ return write(text);
68
+ held.push(text);
69
+ if (!listening) {
70
+ listening = true;
71
+ for (const event of ["drain", "close", "error"])
72
+ stdout.on(event, flush);
73
+ }
74
+ poll ??= setInterval(() => {
75
+ if (stdout.writableLength === 0)
76
+ flush();
77
+ }, BACKLOG_POLL_MS);
78
+ };
79
+ }
80
+ const stderrLine = stderrAfterStdout(process.stdout, (text) => process.stderr.write(text + "\n"));
26
81
  export const defaultIO = {
27
82
  out: (text) => process.stdout.write(text + "\n"),
28
- err: (text) => process.stderr.write(text + "\n"),
83
+ // A record never lands inside the data when both streams share a pipe.
84
+ err: stderrLine,
29
85
  };
@@ -0,0 +1,95 @@
1
+ /** The log formats `--log-format` takes. */
2
+ export declare const LOG_FORMATS: readonly ["text", "jsonl"];
3
+ export type LogFormat = (typeof LOG_FORMATS)[number];
4
+ /** The format without `--log-format`. */
5
+ export declare const DEFAULT_LOG_FORMAT: LogFormat;
6
+ export type LogLevel = "ERROR" | "WARN" | "INFO";
7
+ /** The program's name in every topic: `ckan.<area>`. */
8
+ export declare const LOG_PROGRAM = "ckan";
9
+ export interface LogRecord {
10
+ /** ISO 8601, UTC, milliseconds. */
11
+ ts: string;
12
+ level: LogLevel;
13
+ /** `<program>.<area>`, like `ckan.http`. */
14
+ topic: string;
15
+ msg: string;
16
+ }
17
+ /**
18
+ * `text` with every character that could split a record, forge a second one or steer
19
+ * the terminal written as an escape: CR as `\r`, LF as `\n`, any other C0 control but
20
+ * TAB, DEL and C1 as `\u00XX`, U+2028, U+2029 and the bidi controls (U+061C, U+200E,
21
+ * U+200F, U+202A–U+202E, U+2066–U+2069) as `\uXXXX`. Backslashes stay as they are. On
22
+ * the output of `JSON.stringify` (no raw C0 left) every escape it adds is valid JSON,
23
+ * so the same helper serves both formats. Checked by char code, so the source stays
24
+ * free of those characters.
25
+ */
26
+ export declare function escapeForRecord(text: string): string;
27
+ /**
28
+ * The longest message (in characters) a record carries. A longer one is cut at a
29
+ * code-point boundary and ends in `… (N more characters)`: one huge server text or typed
30
+ * value cannot flood stderr or a log store with a line of megabytes. Own messages that
31
+ * quote server or user data are bounded at their source too (`cutForMessage`); this
32
+ * is the backstop for every path.
33
+ */
34
+ export declare const MAX_RECORD_MESSAGE = 4000;
35
+ /**
36
+ * One record as one line, whatever the message holds: `escapeForRecord` runs over the
37
+ * message (text) or over the whole JSON object (jsonl), so no text that reaches a
38
+ * record can split it, forge another one, or reach the terminal as a control sequence.
39
+ */
40
+ export declare function formatLogRecord(record: LogRecord, format: LogFormat): string;
41
+ export interface Logger {
42
+ /**
43
+ * The format records are written in. Mutable for one reason: it is read from argv
44
+ * before commander parses (for commander's own errors), then set again from what
45
+ * commander parsed, before the action runs (`run()`'s preAction hook).
46
+ */
47
+ format: LogFormat;
48
+ error(area: string, msg: string): void;
49
+ warn(area: string, msg: string): void;
50
+ info(area: string, msg: string): void;
51
+ }
52
+ /**
53
+ * A logger that writes each record, formatted, to `write` (stderr). `redact` replaces the
54
+ * secrets of the run in the message, before it is cut and escaped: the frame (time,
55
+ * level, topic) is never touched, so a secret equal to a year or a topic cannot corrupt
56
+ * it, and a secret holding DEL, C1 or bidi characters is found in its raw form.
57
+ */
58
+ export declare function createLogger(options: {
59
+ format: LogFormat;
60
+ write: (line: string) => void;
61
+ now?: () => Date;
62
+ redact?: (text: string) => string;
63
+ }): Logger;
64
+ /** Why `value` is not a log format, or undefined. */
65
+ export declare function logFormatProblem(value: string): string | undefined;
66
+ /**
67
+ * The `--log-format` in `argv`, read before commander parses it: commander's own
68
+ * usage errors are logged too, and they happen while parsing. A missing or unknown
69
+ * value gives the default here; commander then reports an unknown one. The first
70
+ * `--log-format` counts: commander keeps it and rejects a second one (`once`), so that
71
+ * error is logged in the first one's format. This scan is only for the records of a
72
+ * parse error: once commander has parsed argv, its value is the format (the program's
73
+ * first preAction hook), so `--user-agent --log-format=jsonl` (a User-Agent) logs text.
74
+ * `valueOptions` names the program's options that take a value (`--user-agent`): the
75
+ * token after one is its value, never an option, as commander reads it, so `--user-agent
76
+ * --log-format jsonl` agrees with commander in a parse error too. A subcommand's value
77
+ * option (`--rows`) does not count: commander takes the program's own options out of
78
+ * argv first, so `search --rows --log-format jsonl` is jsonl and a `--rows` without its
79
+ * value.
80
+ */
81
+ export declare function logFormatFromArgv(argv: readonly string[], valueOptions?: ReadonlySet<string>): LogFormat;
82
+ /** What `installWarningLog` needs of the process. */
83
+ export interface WarningSource {
84
+ removeAllListeners(event: "warning"): unknown;
85
+ on(event: "warning", listener: (warning: Error) => void): unknown;
86
+ }
87
+ /**
88
+ * Node's own process warnings (`(node:PID) Warning: …`, e.g. for
89
+ * `NODE_TLS_REJECT_UNAUTHORIZED=0`) as WARN records of `ckan.cli`:
90
+ * `(node) <name>: <message>`, in the log's format and with its redaction. Node's default
91
+ * listener, which prints the plain line, is removed (checked on Node 22, 24 and 26: that
92
+ * silences it; `--no-warnings` is not needed). The bin shim installs it once, before
93
+ * `run()`; a warning Node emits while it starts up, before any code runs, stays Node's.
94
+ */
95
+ export declare function installWarningLog(source: WarningSource, log: Pick<Logger, "warn">): void;