@maschinenlesbar.org/ckan-cli 0.2.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,78 @@
1
+ # Data license
2
+
3
+ > **This tool does not include, host, or redistribute any data.**
4
+ > `ckan-cli` is a *client*. It only accesses records served live by whichever
5
+ > CKAN portal you point it at. Those records are governed by **that portal's**
6
+ > terms, and often by a licence set per dataset. The license of this CLI's own
7
+ > source code is a separate matter — see [LICENSING.md](LICENSING.md).
8
+
9
+ > [!IMPORTANT]
10
+ > **The terms depend on the portal and usually on the record.** A generic CKAN
11
+ > client cannot know them in advance. Every CKAN record carries its own
12
+ > `license_id` / `license_title` / `license_url`; check them before reusing a
13
+ > record's contents. `ckan licenses` lists the licences a portal offers, and
14
+ > `ckan search --rows 0 --facet license_id` shows how often each is used.
15
+
16
+ ## Default portal: Transparenzportal Hamburg
17
+
18
+ | | |
19
+ |---|---|
20
+ | **Data provider** | Freie und Hansestadt Hamburg; each record is the responsibility of its publishing body |
21
+ | **API / source** | `https://suche.transparenz.hamburg.de/api/3/action` (CKAN Action API) · portal: https://transparenz.hamburg.de/ |
22
+ | **Legal basis** | Hamburgisches Transparenzgesetz (HmbTG), § 10 Abs. 3: *"Die Nutzung, Weiterverwendung und Verbreitung der Informationen ist frei, sofern höherrangiges Recht oder spezialgesetzliche Regelungen nichts anderes bestimmen."* |
23
+ | **Portal statement** | *"Die Dokumente und Daten aus dem Transparenzportal Hamburg dürfen frei verwendet werden."* Records are published under two licences that allow free reuse, *"ggf. unter Nennung des Namens"*: Datenlizenz Deutschland – Namensnennung 2.0 and – Zero 2.0. |
24
+ | **Licence** | **Per record**, in `license_id`. As of 2026-09-19: `dl-de-by-2.0` (242,632 records), `cc-zero` (2,203), `dl-zero-de-2.0` / `dl-de-zero-2.0` (262), `odbl` (59), `geoNutz-20130319` (48). |
25
+ | **Attribution** | Per record, in the `terms_of_use` extra (`attribution_text`), e.g. *"Namensnennung: Freie und Hansestadt Hamburg, Bezirksversammlung Wandsbek"*. The wording names the publishing body, so it varies. |
26
+ | **Metadata** | Not licensed separately by the portal. |
27
+ | **Automated access** | Invited: the API page describes it as the way to fetch *"Inhalte und Metadaten aus dem Transparenzportal automatisiert"*. No rate limits or API terms are published; HmbTG § 10 Abs. 4 makes access *"kostenlos und anonym"*. |
28
+
29
+ ### Watch for
30
+
31
+ - **`dl-de-by-2.0`** (the vast majority) requires attribution. Use the record's
32
+ own `attribution_text`, and link the licence
33
+ (https://www.govdata.de/dl-de/by-2-0).
34
+ - **`odbl`** records are share-alike: derived databases must stay under the ODbL.
35
+ - **`geoNutz-20130319`** is Hamburg's older geodata usage licence, not a
36
+ standard open licence. Read its terms before reusing those records.
37
+ - **Third-party rights.** HmbTG § 10 Abs. 3 S. 3 only obliges authorities to
38
+ *try* to obtain reuse rights for third-party works (reports, studies), and
39
+ § 8 Abs. 1 lets intellectual-property protection override publication. A
40
+ document written by a third party can carry rights the record's licence does
41
+ not grant.
42
+ - **Personal data** is redacted before publication (HmbTG § 4), but some names
43
+ are published on purpose (contract partners, authors of reports, subsidy
44
+ recipients). Data-protection law still applies to what you do with them.
45
+ - **The website itself** (layout, graphics, editorial pages) is copyright
46
+ protected per the Impressum; that concerns transparenz.hamburg.de, not the
47
+ register records served by the API.
48
+
49
+ ### Attribution
50
+
51
+ ```
52
+ Source: Transparenzportal Hamburg (https://transparenz.hamburg.de/),
53
+ <attribution_text of the record>, licensed under <license_title of the record>
54
+ (<license_url of the record>).
55
+ ```
56
+
57
+ ## Other portals
58
+
59
+ With `--portal`, `--base-url` or `CKAN_BASE_URL` you are subject to that portal's
60
+ terms. Being in the built-in list (`ckan portals`) says only that a portal runs a
61
+ working CKAN, nothing about its terms.
62
+ Look them up on the portal itself; do not assume they match Hamburg's. For
63
+ GovData, see the `DATA_LICENSE.md` of
64
+ [govdata-cli](https://github.com/maschinenlesbar-org/govdata-cli).
65
+
66
+ ## Sources
67
+
68
+ - https://transparenz.hamburg.de/verwendung-der-daten-aus-dem-transparenzportal-796520 — free use, the two licences
69
+ - https://transparenz.hamburg.de/api-796358 — API notes, automated access
70
+ - https://transparenz.hamburg.de/gesetzestext-des-hmbtg-796514 — HmbTG §§ 4, 7, 8, 10
71
+ - https://transparenz.hamburg.de/impressum-796468 — responsibility per publishing body, website copyright
72
+ - https://www.govdata.de/dl-de/by-2-0 — Datenlizenz Deutschland – Namensnennung – 2.0
73
+ - Record counts: `ckan search --rows 0 --facet license_id` against the live API, 2026-09-19
74
+
75
+ ---
76
+
77
+ *Good-faith summary compiled 2026-09-19; not legal advice. The portal's terms
78
+ and each record's licence are authoritative and can change.*
package/README.md CHANGED
@@ -24,7 +24,7 @@ search, inspect, filter and pipe straight into [`jq`](https://jqlang.github.io/j
24
24
  `status`, `portals`, and a generic `action` escape hatch for everything else.
25
25
 
26
26
  > Want to use this as a TypeScript library or understand how it's built?
27
- > See **[DEVELOPING.md](DEVELOPING.md)**.
27
+ > See **[DEVELOPING.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/DEVELOPING.md)**.
28
28
 
29
29
  ## Install
30
30
 
@@ -114,7 +114,7 @@ The list is built into the CLI, so it needs no outside service:
114
114
 
115
115
  Counts as of the last check, 2026-09-19. The list is refreshed from Wikidata,
116
116
  the CKAN project's instance registry and GovData's harvest sources, keeping only
117
- portals that pass a live check (see [DEVELOPING.md](DEVELOPING.md#the-portal-list)).
117
+ portals that pass a live check (see [DEVELOPING.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/DEVELOPING.md#the-portal-list)).
118
118
  Many portals those lists name have left CKAN (Köln, Düsseldorf, Bonn, …).
119
119
 
120
120
  ## Commands
@@ -197,7 +197,7 @@ before any request is sent.
197
197
  matches nothing, and a full word such as `verwaltungsvorschriften` over-matches.
198
198
  List the values that work, with their counts:
199
199
  `ckan --compact search --rows 0 --facet extras_registerobject_type --facet-limit -1`.
200
- The [Glossary](GLOSSARY.md) has the details.
200
+ The [Glossary](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/GLOSSARY.md) has the details.
201
201
  - **Several filters work.** Hamburg's API page says only one filter can be
202
202
  used; with `fq_list` (repeat `--fq`) they combine:
203
203
  `ckan search --fq extras_registerobject_type:verwaltungsvorschrift --fq organization:workflows`.
@@ -231,6 +231,25 @@ Every command prints the **unwrapped `result`** as pretty JSON to stdout.
231
231
  Errors go to stderr, so piping stdout into `jq` stays clean. `--compact`
232
232
  prints one line. Every global option works before or after the command.
233
233
 
234
+ Each line on stderr is a **log record**: a timestamp (UTC), a level (`ERROR`, `WARN`,
235
+ `INFO`) and a topic, the program and the area it comes from (`ckan.cli` for usage
236
+ errors, `ckan.api` for the portal's answers, a malformed one included, `ckan.http` for the connection,
237
+ `ckan.output` when stdout can't be written). By default
238
+ it is written log4j style; `--log-format jsonl` writes one JSON object per line instead.
239
+ A record is always one line: a line break, a control character or a bidi control in a
240
+ message (a server's text, a value you typed) is written as an escape (`\n`, `\u001b`,
241
+ `\u202e`), so it can neither split a record nor forge another one, nor steer the terminal;
242
+ a message longer than 4000 characters is cut and ends in `… (N more characters)`:
243
+
244
+ ```text
245
+ 2026-10-09T14:03:12.481Z WARN [ckan.http] requests to ckan.example are sent unencrypted (http:, not https:)
246
+ 2026-10-09T14:03:12.902Z ERROR [ckan.api] HTTP 404 for GET http://ckan.example/api/3/action/status_show: Not Found Error: Not found
247
+ ```
248
+
249
+ ```bash
250
+ ckan --log-format jsonl package no-such-id 2>log.jsonl # {"ts":"…","level":"ERROR","topic":"ckan.api","msg":"HTTP 404 …"}
251
+ ```
252
+
234
253
  ```bash
235
254
  # How many records a portal has
236
255
  ckan search --rows 0 | jq '.count'
@@ -247,13 +266,13 @@ ckan package bezirk-wandsbek-drucksache-22-3573-2 | jq '.resources[] | {format,
247
266
  | Code | Meaning |
248
267
  | --- | --- |
249
268
  | `0` | success (also `--help`, `help [command]`, `--version` and a bare `ckan`) |
250
- | `4` | not found (HTTP `404`): an unknown id on `package`/`resource`/`organization`/`group`, or, on any command, no CKAN at this URL (a wrong `--base-url`, a portal that left CKAN) |
251
- | `1` | any other error — bad usage (an unknown command, also after `help`), validation error (`409`), unknown action (`400`), CKAN `success:false`, network failure |
269
+ | `4` | not found (HTTP `404`, or CKAN's `Not Found Error` sent with HTTP `200`): an unknown id on `package`/`resource`/`organization`/`group`, or, on any command, no CKAN at this URL (a wrong `--base-url`, a portal that left CKAN) |
270
+ | `1` | any other error — bad usage (an unknown command, also after `help`), validation error (`409`), unknown action (`400`), any other CKAN `success:false`, network failure |
252
271
 
253
272
  A reader that stops early (`| head`) ends `ckan` quietly with exit 0; a failed
254
273
  run keeps its exit code even when nothing reads stderr.
255
274
 
256
- CKAN's error text is shown on stderr, e.g.
275
+ CKAN's error text is shown on stderr (in an `ERROR` record of `ckan.api`), e.g.
257
276
  `Validation Error: rows: Invalid integer` or
258
277
  `Bad request - Action name not known: …`.
259
278
 
@@ -270,16 +289,24 @@ CKAN's error text is shown on stderr, e.g.
270
289
  | `-V, --version` | Print the version number |
271
290
  | `-h, --help` | Show help for the program or a command |
272
291
  | `--compact` | Print JSON on a single line instead of pretty-printed |
292
+ | `--log-format <format>` | How errors, warnings and notes are written to stderr: `text` (default; log4j style, `2026-10-09T14:03:12.481Z WARN [ckan.http] …`) or `jsonl` (one JSON object per line: `ts`, `level`, `topic`, `msg`). stdout is not affected |
273
293
  | `--base-url <url>` | CKAN site URL (env `CKAN_BASE_URL`; default `https://suche.transparenz.hamburg.de`) |
274
294
  | `--portal <id>` | a known portal by id (`ckan portals`); cannot be combined with `--base-url` |
275
295
  | `--timeout <ms>` | Per-request timeout (default `30000`; at most `2147483647`) |
276
296
  | `--user-agent <ua>` | `User-Agent` header value |
277
- | `--max-retries <n>` | Retries for transient `429`/`503` responses (`0`–`10`, default `2`). Each retry backs off linearly (0.2 s, 0.4 s, …) or waits the server's `Retry-After` when that is longer, up to 30 s; a longer one is not retried, and the error names the wait. A reset or other network error is not retried |
297
+ | `--max-retries <n>` | Retries for transient `429`/`503` responses (`0`–`10`, default `2`). Each retry backs off linearly (0.2 s, 0.4 s, …) or waits the server's `Retry-After` when that is longer, up to 30 s; a longer one is not retried, and the error names the wait. A reset or other network error is not retried. Each retry logs one WARN record of `ckan.http` before it waits (`HTTP 503 from host: retry 1 of 3 in 2 s`). |
278
298
  | `--max-response-bytes <n>` | Cap response body size in bytes (`0` = unlimited; default 100 MiB) |
279
299
 
300
+ A base URL on plain `http:` (from `--base-url` or `CKAN_BASE_URL`) to a host other than
301
+ loopback (`localhost`, `127.0.0.0/8`, `::1`) works, but the CLI writes one `WARN` record of
302
+ `ckan.http` to stderr before the first request, e.g.
303
+ `… WARN [ckan.http] requests to ckan.example are sent unencrypted (http:, not https:)`, or
304
+ `… WARN [ckan.http] the base URL's credentials are sent unencrypted to ckan.example (http:, not https:)`
305
+ when it carries a `user:password@` (never printed). stdout and the exit code are unchanged.
306
+
280
307
  ## Claude Code skills
281
308
 
282
- Three [Agent Skills](SKILLS.md) teach Claude Code to use `ckan` for real questions: find
309
+ Three [Agent Skills](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/SKILLS.md) teach Claude Code to use `ckan` for real questions: find
283
310
  which portals have data on a topic and rank the hits (**ckan-dataset-finder**), build
284
311
  publisher/format/licence statistics (**ckan-catalogue-stats**), and search the Hamburg
285
312
  Transparenzportal's contracts, Gutachten and Senate papers
@@ -290,7 +317,7 @@ Transparenzportal's contracts, Gutachten and Senate papers
290
317
  /plugin install ckan@maschinenlesbar
291
318
  ```
292
319
 
293
- Real runs of each skill are in [EXAMPLE.md](EXAMPLE.md).
320
+ Real runs of each skill are in [EXAMPLE.md](https://github.com/maschinenlesbar-org/ckan-cli/blob/main/EXAMPLE.md).
294
321
 
295
322
  ## Data license
296
323
 
@@ -4,6 +4,15 @@ import { action, renderJson, toEngineOptions } from "../shared.js";
4
4
  import { checkPortals } from "../../client/portals.js";
5
5
  import { PORTALS } from "../../client/portals-list.js";
6
6
  import { isBlank } from "../../client/validate.js";
7
+ import { cutForMessage, redactUrl } from "../../client/errors.js";
8
+ /**
9
+ * A typed value as a message quotes it: cut at MAX_MESSAGE_VALUE_LENGTH, after its
10
+ * credentials are redacted (a cut could otherwise leave part of a password without the
11
+ * "@" the argv redaction keys on).
12
+ */
13
+ function shown(value) {
14
+ return cutForMessage(redactUrl(value));
15
+ }
7
16
  /** commander accumulator for repeatable `key=value` pairs into a record. */
8
17
  function collectKeyValue(value, previous = {}) {
9
18
  const eq = value.indexOf("=");
@@ -13,18 +22,18 @@ function collectKeyValue(value, previous = {}) {
13
22
  const key = value.slice(0, Math.max(eq, 0));
14
23
  // A blank key (`=x`, ` =x`) names no parameter: the library refuses it too.
15
24
  if (eq < 0 || isBlank(key)) {
16
- throw new InvalidArgumentError(`Invalid --param "${value}". Expected key=value.`);
25
+ throw new InvalidArgumentError(`Invalid --param "${shown(value)}". Expected key=value.`);
17
26
  }
18
27
  // Reject a duplicated key rather than silently overwriting the earlier value.
19
28
  if (Object.prototype.hasOwnProperty.call(previous, key)) {
20
- throw new InvalidArgumentError(`Duplicate --param key "${key}".`);
29
+ throw new InvalidArgumentError(`Duplicate --param key "${shown(key)}".`);
21
30
  }
22
31
  // A blank value is refused by the library as well (CKAN reads an empty
23
32
  // parameter as unset, so `--param q=` would run the action unfiltered); here it
24
33
  // is a usage error that names the flag.
25
34
  const paramValue = value.slice(eq + 1);
26
35
  if (isBlank(paramValue)) {
27
- throw new InvalidArgumentError(`Invalid --param "${value}". The value must not be blank.`);
36
+ throw new InvalidArgumentError(`Invalid --param "${shown(value)}". The value must not be blank.`);
28
37
  }
29
38
  // A computed key in a literal is an own data property, even for `__proto__`.
30
39
  return { ...previous, [key]: paramValue };
@@ -1,7 +1,12 @@
1
1
  #!/usr/bin/env node
2
2
  // Binary entry point. Thin shim around run(); all logic lives in run.ts/program.ts.
3
3
  import { handleOutputErrors } from "./io.js";
4
- import { run } from "./run.js";
5
- handleOutputErrors();
6
- const exitCode = await run(process.argv.slice(2));
4
+ import { installWarningLog } from "./log.js";
5
+ import { processLogger, run } from "./run.js";
6
+ const argv = process.argv.slice(2);
7
+ // What happens outside run() is logged too, in the format argv asks for.
8
+ const log = processLogger(argv);
9
+ installWarningLog(process, log);
10
+ handleOutputErrors(process, undefined, log);
11
+ const exitCode = await run(argv);
7
12
  process.exitCode = exitCode;
@@ -1,5 +1,6 @@
1
1
  import type { CkanClient } from "../client/client.js";
2
2
  import type { EngineOptions } from "../client/engine.js";
3
+ import { type Logger } from "./log.js";
3
4
  export interface CliIO {
4
5
  out(text: string): void;
5
6
  err(text: string): void;
@@ -10,7 +11,16 @@ export interface CliDeps {
10
11
  env: Record<string, string | undefined>;
11
12
  /** Build a client from the resolved global options (injectable for tests). */
12
13
  createClient(options: EngineOptions): CkanClient;
14
+ /**
15
+ * Where diagnostics go: one record per line on stderr, in the `--log-format`
16
+ * (`log.ts`). `run()` sets it from argv; deps without it log text through `io.err`.
17
+ */
18
+ log?: Logger;
19
+ /** The clock the log's timestamps come from. Unset, the real one. */
20
+ now?: () => Date;
13
21
  }
22
+ /** The deps' logger, or one that writes text records through `io.err`. */
23
+ export declare function logOf(deps: CliDeps): Logger;
14
24
  /** The two process streams, as far as `handleOutputErrors` needs them. */
15
25
  export interface OutputStreams {
16
26
  stdout: Pick<NodeJS.WriteStream, "on">;
@@ -22,11 +32,30 @@ export interface OutputStreams {
22
32
  *
23
33
  * A reader that stops early — `| head`, `| jq` exiting on the first match, a closed
24
34
  * pager — closes the pipe while the CLI is still writing, and the next write fails
25
- * with EPIPE. That is ordinary use, so the process exits 0 at once, quietly. Any
26
- * other stdout error prints one `Output error: <message>` line to stderr and exits
27
- * 1. On stderr an EPIPE is ignored, so a failed run keeps its exit code; any other
28
- * stderr error exits 1 silently (there is nowhere left to report it).
29
- * The bin shim installs this once, before `run()`.
35
+ * with EPIPE (ENOTCONN when stdout is a socket whose peer has gone, as when a Node
36
+ * parent spawns the CLI with piped stdio on macOS). That is ordinary use, so the
37
+ * process exits 0 at once, quietly. Any
38
+ * other stdout error is an ERROR record of `ckan.output` (`Could not write to stdout:
39
+ * <message>`, through `log`, in the run's format) and exits 1. On stderr an EPIPE is
40
+ * ignored, so a failed run keeps its exit code; any other stderr error exits 1 silently
41
+ * (there is nowhere left to report it).
42
+ * The bin shim installs this once, before `run()`, with a logger for the format argv
43
+ * asks for (`processLogger`).
30
44
  */
31
- export declare function handleOutputErrors(streams?: OutputStreams, exit?: (code: number) => void): void;
45
+ export declare function handleOutputErrors(streams?: OutputStreams, exit?: (code: number) => void, log?: Pick<Logger, "error">): void;
46
+ /** What `stderrAfterStdout` needs of stdout: its backlog, and the events that end one. */
47
+ export interface StdoutBacklog {
48
+ readonly writableLength: number;
49
+ on(event: "drain" | "close" | "error", listener: () => void): unknown;
50
+ }
51
+ /**
52
+ * A stderr writer that waits for stdout. With both streams on one pipe (`2>&1 |`) and a
53
+ * slow reader, stdout's data is still queued in the process while a record is written
54
+ * to stderr at once, so the record landed inside the JSON (at 64 KiB). Here a text is
55
+ * held while `stdout.writableLength > 0` and written, in order, once the backlog is
56
+ * gone: on stdout's `drain`, `close` or `error`, or at the latest when a short poll sees
57
+ * it empty (`drain` only follows a write that returned false). The poll keeps the
58
+ * process alive until then, so a held record is never lost at exit.
59
+ */
60
+ export declare function stderrAfterStdout(stdout: StdoutBacklog, write: (text: string) => void): (text: string) => void;
32
61
  export declare const defaultIO: CliIO;
@@ -1,33 +1,85 @@
1
1
  // I/O seam for the CLI. Everything the CLI writes goes through a CliIO object so
2
2
  // tests can capture output instead of hitting the real stdout/stderr, and the
3
3
  // environment is injected so tests never depend on the real process.env.
4
+ import { createLogger } from "./log.js";
5
+ /** The deps' logger, or one that writes text records through `io.err`. */
6
+ export function logOf(deps) {
7
+ return deps.log ?? createLogger({ format: "text", write: (line) => deps.io.err(line), ...(deps.now === undefined ? {} : { now: deps.now }) });
8
+ }
4
9
  /**
5
10
  * Handle write errors on stdout/stderr, which Node otherwise reports as an
6
11
  * unhandled 'error' event: a raw stack trace and exit 1.
7
12
  *
8
13
  * A reader that stops early — `| head`, `| jq` exiting on the first match, a closed
9
14
  * pager — closes the pipe while the CLI is still writing, and the next write fails
10
- * with EPIPE. That is ordinary use, so the process exits 0 at once, quietly. Any
11
- * other stdout error prints one `Output error: <message>` line to stderr and exits
12
- * 1. On stderr an EPIPE is ignored, so a failed run keeps its exit code; any other
13
- * stderr error exits 1 silently (there is nowhere left to report it).
14
- * The bin shim installs this once, before `run()`.
15
+ * with EPIPE (ENOTCONN when stdout is a socket whose peer has gone, as when a Node
16
+ * parent spawns the CLI with piped stdio on macOS). That is ordinary use, so the
17
+ * process exits 0 at once, quietly. Any
18
+ * other stdout error is an ERROR record of `ckan.output` (`Could not write to stdout:
19
+ * <message>`, through `log`, in the run's format) and exits 1. On stderr an EPIPE is
20
+ * ignored, so a failed run keeps its exit code; any other stderr error exits 1 silently
21
+ * (there is nowhere left to report it).
22
+ * The bin shim installs this once, before `run()`, with a logger for the format argv
23
+ * asks for (`processLogger`).
15
24
  */
16
- export function handleOutputErrors(streams = process, exit = (code) => process.exit(code)) {
25
+ export function handleOutputErrors(streams = process, exit = (code) => process.exit(code), log = createLogger({ format: "text", write: (line) => process.stderr.write(line + "\n") })) {
17
26
  streams.stdout.on("error", (err) => {
18
- if (err.code === "EPIPE")
27
+ if (readerGone(err))
19
28
  return exit(0);
20
- process.stderr.write(`Output error: ${err.message}\n`);
29
+ log.error("output", `Could not write to stdout: ${err.message}`);
21
30
  exit(1);
22
31
  });
23
32
  // stderr's reader going away doesn't make a failed run a success: ignore EPIPE there
24
33
  // and let the run's own exit code stand (`2>&1 | true` used to turn a usage error into 0).
25
34
  streams.stderr.on("error", (err) => {
26
- if (err.code !== "EPIPE")
35
+ if (!readerGone(err))
27
36
  exit(1);
28
37
  });
29
38
  }
39
+ /** True for the write errors that mean the reader has gone: EPIPE, or ENOTCONN on a socket. */
40
+ function readerGone(err) {
41
+ return err.code === "EPIPE" || err.code === "ENOTCONN";
42
+ }
43
+ /** How often a held record checks whether stdout's backlog is gone (ms). */
44
+ const BACKLOG_POLL_MS = 10;
45
+ /**
46
+ * A stderr writer that waits for stdout. With both streams on one pipe (`2>&1 |`) and a
47
+ * slow reader, stdout's data is still queued in the process while a record is written
48
+ * to stderr at once, so the record landed inside the JSON (at 64 KiB). Here a text is
49
+ * held while `stdout.writableLength > 0` and written, in order, once the backlog is
50
+ * gone: on stdout's `drain`, `close` or `error`, or at the latest when a short poll sees
51
+ * it empty (`drain` only follows a write that returned false). The poll keeps the
52
+ * process alive until then, so a held record is never lost at exit.
53
+ */
54
+ export function stderrAfterStdout(stdout, write) {
55
+ const held = [];
56
+ let poll;
57
+ const flush = () => {
58
+ if (poll !== undefined)
59
+ clearInterval(poll);
60
+ poll = undefined;
61
+ for (const text of held.splice(0))
62
+ write(text);
63
+ };
64
+ let listening = false;
65
+ return (text) => {
66
+ if (held.length === 0 && stdout.writableLength === 0)
67
+ return write(text);
68
+ held.push(text);
69
+ if (!listening) {
70
+ listening = true;
71
+ for (const event of ["drain", "close", "error"])
72
+ stdout.on(event, flush);
73
+ }
74
+ poll ??= setInterval(() => {
75
+ if (stdout.writableLength === 0)
76
+ flush();
77
+ }, BACKLOG_POLL_MS);
78
+ };
79
+ }
80
+ const stderrLine = stderrAfterStdout(process.stdout, (text) => process.stderr.write(text + "\n"));
30
81
  export const defaultIO = {
31
82
  out: (text) => process.stdout.write(text + "\n"),
32
- err: (text) => process.stderr.write(text + "\n"),
83
+ // A record never lands inside the data when both streams share a pipe.
84
+ err: stderrLine,
33
85
  };
@@ -0,0 +1,95 @@
1
+ /** The log formats `--log-format` takes. */
2
+ export declare const LOG_FORMATS: readonly ["text", "jsonl"];
3
+ export type LogFormat = (typeof LOG_FORMATS)[number];
4
+ /** The format without `--log-format`. */
5
+ export declare const DEFAULT_LOG_FORMAT: LogFormat;
6
+ export type LogLevel = "ERROR" | "WARN" | "INFO";
7
+ /** The program's name in every topic: `ckan.<area>`. */
8
+ export declare const LOG_PROGRAM = "ckan";
9
+ export interface LogRecord {
10
+ /** ISO 8601, UTC, milliseconds. */
11
+ ts: string;
12
+ level: LogLevel;
13
+ /** `<program>.<area>`, like `ckan.http`. */
14
+ topic: string;
15
+ msg: string;
16
+ }
17
+ /**
18
+ * `text` with every character that could split a record, forge a second one or steer
19
+ * the terminal written as an escape: CR as `\r`, LF as `\n`, any other C0 control but
20
+ * TAB, DEL and C1 as `\u00XX`, U+2028, U+2029 and the bidi controls (U+061C, U+200E,
21
+ * U+200F, U+202A–U+202E, U+2066–U+2069) as `\uXXXX`. Backslashes stay as they are. On
22
+ * the output of `JSON.stringify` (no raw C0 left) every escape it adds is valid JSON,
23
+ * so the same helper serves both formats. Checked by char code, so the source stays
24
+ * free of those characters.
25
+ */
26
+ export declare function escapeForRecord(text: string): string;
27
+ /**
28
+ * The longest message (in characters) a record carries. A longer one is cut at a
29
+ * code-point boundary and ends in `… (N more characters)`: one huge server text or typed
30
+ * value cannot flood stderr or a log store with a line of megabytes. Own messages that
31
+ * quote server or user data are bounded at their source too (`cutForMessage`); this
32
+ * is the backstop for every path.
33
+ */
34
+ export declare const MAX_RECORD_MESSAGE = 4000;
35
+ /**
36
+ * One record as one line, whatever the message holds: `escapeForRecord` runs over the
37
+ * message (text) or over the whole JSON object (jsonl), so no text that reaches a
38
+ * record can split it, forge another one, or reach the terminal as a control sequence.
39
+ */
40
+ export declare function formatLogRecord(record: LogRecord, format: LogFormat): string;
41
+ export interface Logger {
42
+ /**
43
+ * The format records are written in. Mutable for one reason: it is read from argv
44
+ * before commander parses (for commander's own errors), then set again from what
45
+ * commander parsed, before the action runs (`run()`'s preAction hook).
46
+ */
47
+ format: LogFormat;
48
+ error(area: string, msg: string): void;
49
+ warn(area: string, msg: string): void;
50
+ info(area: string, msg: string): void;
51
+ }
52
+ /**
53
+ * A logger that writes each record, formatted, to `write` (stderr). `redact` replaces the
54
+ * secrets of the run in the message, before it is cut and escaped: the frame (time,
55
+ * level, topic) is never touched, so a secret equal to a year or a topic cannot corrupt
56
+ * it, and a secret holding DEL, C1 or bidi characters is found in its raw form.
57
+ */
58
+ export declare function createLogger(options: {
59
+ format: LogFormat;
60
+ write: (line: string) => void;
61
+ now?: () => Date;
62
+ redact?: (text: string) => string;
63
+ }): Logger;
64
+ /** Why `value` is not a log format, or undefined. */
65
+ export declare function logFormatProblem(value: string): string | undefined;
66
+ /**
67
+ * The `--log-format` in `argv`, read before commander parses it: commander's own
68
+ * usage errors are logged too, and they happen while parsing. A missing or unknown
69
+ * value gives the default here; commander then reports an unknown one. The first
70
+ * `--log-format` counts: commander keeps it and rejects a second one (`once`), so that
71
+ * error is logged in the first one's format. This scan is only for the records of a
72
+ * parse error: once commander has parsed argv, its value is the format (the program's
73
+ * first preAction hook), so `--user-agent --log-format=jsonl` (a User-Agent) logs text.
74
+ * `valueOptions` names the program's options that take a value (`--user-agent`): the
75
+ * token after one is its value, never an option, as commander reads it, so `--user-agent
76
+ * --log-format jsonl` agrees with commander in a parse error too. A subcommand's value
77
+ * option (`--rows`) does not count: commander takes the program's own options out of
78
+ * argv first, so `search --rows --log-format jsonl` is jsonl and a `--rows` without its
79
+ * value.
80
+ */
81
+ export declare function logFormatFromArgv(argv: readonly string[], valueOptions?: ReadonlySet<string>): LogFormat;
82
+ /** What `installWarningLog` needs of the process. */
83
+ export interface WarningSource {
84
+ removeAllListeners(event: "warning"): unknown;
85
+ on(event: "warning", listener: (warning: Error) => void): unknown;
86
+ }
87
+ /**
88
+ * Node's own process warnings (`(node:PID) Warning: …`, e.g. for
89
+ * `NODE_TLS_REJECT_UNAUTHORIZED=0`) as WARN records of `ckan.cli`:
90
+ * `(node) <name>: <message>`, in the log's format and with its redaction. Node's default
91
+ * listener, which prints the plain line, is removed (checked on Node 22, 24 and 26: that
92
+ * silences it; `--no-warnings` is not needed). The bin shim installs it once, before
93
+ * `run()`; a warning Node emits while it starts up, before any code runs, stays Node's.
94
+ */
95
+ export declare function installWarningLog(source: WarningSource, log: Pick<Logger, "warn">): void;