@maschinenlesbar.org/ckan-cli 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -5
- package/dist/src/cli/commands/catalogue.js +12 -12
- package/dist/src/cli/io.d.ts +2 -1
- package/dist/src/cli/io.js +6 -2
- package/dist/src/cli/program.js +16 -9
- package/dist/src/cli/run.d.ts +27 -0
- package/dist/src/cli/run.js +82 -1
- package/dist/src/cli/shared.d.ts +8 -0
- package/dist/src/cli/shared.js +16 -0
- package/dist/src/client/client.d.ts +23 -3
- package/dist/src/client/client.js +123 -22
- package/dist/src/client/engine.d.ts +48 -9
- package/dist/src/client/engine.js +305 -45
- package/dist/src/client/errors.d.ts +37 -1
- package/dist/src/client/errors.js +87 -5
- package/dist/src/client/http.d.ts +23 -0
- package/dist/src/client/http.js +14 -1
- package/dist/src/client/index.d.ts +2 -2
- package/dist/src/client/index.js +2 -2
- package/dist/src/client/portals.d.ts +1 -1
- package/dist/src/client/portals.js +38 -3
- package/dist/src/client/validate.d.ts +13 -1
- package/dist/src/client/validate.js +47 -1
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -32,7 +32,7 @@ search, inspect, filter and pipe straight into [`jq`](https://jqlang.github.io/j
|
|
|
32
32
|
npm i -g @maschinenlesbar.org/ckan-cli
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
This installs the **`ckan`** command. Requires **Node.js
|
|
35
|
+
This installs the **`ckan`** command. Requires **Node.js 22.12+**.
|
|
36
36
|
|
|
37
37
|
## Quickstart
|
|
38
38
|
|
|
@@ -79,10 +79,16 @@ sub-path keeps it (`https://open.canada.ca/data`). `ckan status` is a good first
|
|
|
79
79
|
call on an unfamiliar portal: it shows the CKAN version and the installed
|
|
80
80
|
extensions.
|
|
81
81
|
|
|
82
|
+
A portal behind a login takes its user name and password in the URL
|
|
83
|
+
(`https://user:password@host`). They go out as HTTP Basic auth to that site only:
|
|
84
|
+
a redirect to another host, port or scheme drops them, and no message, help text
|
|
85
|
+
or error shows them.
|
|
86
|
+
|
|
82
87
|
### Known portals
|
|
83
88
|
|
|
84
89
|
`ckan portals` lists the CKAN portals in Germany this version knows, with the
|
|
85
90
|
result of their last check; `ckan portals --check` checks them all live now.
|
|
91
|
+
Neither uses `CKAN_BASE_URL`, so both work even when the variable is broken.
|
|
86
92
|
The list is built into the CLI, so it needs no outside service:
|
|
87
93
|
|
|
88
94
|
| `--portal` | Portal | Datasets |
|
|
@@ -152,7 +158,9 @@ action <name> [--param key=value …] call any CKAN action (generic)
|
|
|
152
158
|
> **Blank values are refused.** `ckan search ""`, `--fq ""`, `tags --query ""` or
|
|
153
159
|
> `action … --param q=` exit 1 before any request, instead of silently running an unfiltered search.
|
|
154
160
|
> So does a forgotten value (`--fq --rows 5`): a value of `--fq`, `--facet`,
|
|
155
|
-
> `--sort` or `tags --query` cannot start with `--`.
|
|
161
|
+
> `--sort` or `tags --query` cannot start with `--`. And so does a single-value
|
|
162
|
+
> option given twice (`--rows 5 --rows 50`, a second `--base-url`): only `--fq`,
|
|
163
|
+
> `--facet` and `--param` repeat.
|
|
156
164
|
|
|
157
165
|
### Paging and list flags
|
|
158
166
|
|
|
@@ -163,7 +171,9 @@ action <name> [--param key=value …] call any CKAN action (generic)
|
|
|
163
171
|
|
|
164
172
|
> **`--all-fields` lists are complete.** CKAN caps them at 25 entries
|
|
165
173
|
> (`ckan.group_and_organization_list_all_fields_max`) and drops the rest
|
|
166
|
-
> silently, even for a larger `--limit`; `ckan` pages through them for you
|
|
174
|
+
> silently, even for a larger `--limit`; `ckan` pages through them for you,
|
|
175
|
+
> 0.1 s apart. A server whose pages never end is stopped after 400 pages
|
|
176
|
+
> (10,000 entries, far more than any portal lists) with an error (exit 1).
|
|
167
177
|
| `tags` | `--query <substring>` |
|
|
168
178
|
|
|
169
179
|
### `action`
|
|
@@ -206,6 +216,9 @@ Found by running the same commands against many portals:
|
|
|
206
216
|
- **The default sort differs** (ZBW sorts by `metadata_created desc`).
|
|
207
217
|
- **Field types vary**: licence ids are URIs on GovData, and some portals send
|
|
208
218
|
booleans as the strings `"True"`/`"False"`.
|
|
219
|
+
- **A purely negative query matches nothing**: `ckan search -- -elbe` answers
|
|
220
|
+
`count: 0` on Hamburg rather than "everything but elbe". Negate a filter instead
|
|
221
|
+
(`--fq -organization:allris`), next to a positive query or none.
|
|
209
222
|
- **Solr syntax errors** come back as HTTP 409; `ckan` prints Solr's own reason,
|
|
210
223
|
e.g. `Search Error: Cannot parse 'title:(': Encountered "<EOF>" …`.
|
|
211
224
|
- **A site that is not (or no longer) a CKAN** answers with a 404, an HTML page or
|
|
@@ -235,7 +248,10 @@ ckan package bezirk-wandsbek-drucksache-22-3573-2 | jq '.resources[] | {format,
|
|
|
235
248
|
| --- | --- |
|
|
236
249
|
| `0` | success (also `--help`, `help [command]`, `--version` and a bare `ckan`) |
|
|
237
250
|
| `4` | not found (HTTP `404`): an unknown id on `package`/`resource`/`organization`/`group`, or, on any command, no CKAN at this URL (a wrong `--base-url`, a portal that left CKAN) |
|
|
238
|
-
| `1` | any other error — bad usage, validation error (`409`), unknown action (`400`), CKAN `success:false`, network failure |
|
|
251
|
+
| `1` | any other error — bad usage (an unknown command, also after `help`), validation error (`409`), unknown action (`400`), CKAN `success:false`, network failure |
|
|
252
|
+
|
|
253
|
+
A reader that stops early (`| head`) ends `ckan` quietly with exit 0; a failed
|
|
254
|
+
run keeps its exit code even when nothing reads stderr.
|
|
239
255
|
|
|
240
256
|
CKAN's error text is shown on stderr, e.g.
|
|
241
257
|
`Validation Error: rows: Invalid integer` or
|
|
@@ -258,7 +274,7 @@ CKAN's error text is shown on stderr, e.g.
|
|
|
258
274
|
| `--portal <id>` | a known portal by id (`ckan portals`); cannot be combined with `--base-url` |
|
|
259
275
|
| `--timeout <ms>` | Per-request timeout (default `30000`; at most `2147483647`) |
|
|
260
276
|
| `--user-agent <ua>` | `User-Agent` header value |
|
|
261
|
-
| `--max-retries <n>` | Retries for transient `429`/`503` responses (`0`–`10`, default `2`). Each retry waits the server's `Retry-After`
|
|
277
|
+
| `--max-retries <n>` | Retries for transient `429`/`503` responses (`0`–`10`, default `2`). Each retry backs off linearly (0.2 s, 0.4 s, …) or waits the server's `Retry-After` when that is longer, up to 30 s; a longer one is not retried, and the error names the wait. A reset or other network error is not retried |
|
|
262
278
|
| `--max-response-bytes <n>` | Cap response body size in bytes (`0` = unlimited; default 100 MiB) |
|
|
263
279
|
|
|
264
280
|
## Claude Code skills
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
// and tags.
|
|
3
3
|
import { InvalidArgumentError } from "commander";
|
|
4
4
|
import { facetLimitProblem } from "../../client/validate.js";
|
|
5
|
-
import { action, collectNonEmpty, parseBoundedInt, parseIntArg, parseNonEmpty, parseOptionValue, renderJson, } from "../shared.js";
|
|
5
|
+
import { action, collectNonEmpty, once, parseBoundedInt, parseIntArg, parseNonEmpty, parseOptionValue, renderJson, } from "../shared.js";
|
|
6
6
|
/**
|
|
7
7
|
* commander value-parser for a list --limit: 1 or more. CKAN reads `limit=0` as
|
|
8
8
|
* "no limit" and would send the whole list; leave --limit out for that.
|
|
@@ -24,12 +24,12 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
24
24
|
.command("search")
|
|
25
25
|
.description("Search datasets (Solr query syntax)")
|
|
26
26
|
.argument("[query]", "full-text query, e.g. elbe or title:haushalt", parseNonEmpty)
|
|
27
|
-
.option("--rows <n>", "max results (servers cap this, usually at 1000)", parseIntArg)
|
|
28
|
-
.option("--start <n>", "offset for paging", parseIntArg)
|
|
29
|
-
.option("--sort <expr>", 'e.g. "metadata_modified desc"', parseOptionValue)
|
|
27
|
+
.option("--rows <n>", "max results (servers cap this, usually at 1000)", once(parseIntArg))
|
|
28
|
+
.option("--start <n>", "offset for paging", once(parseIntArg))
|
|
29
|
+
.option("--sort <expr>", 'e.g. "metadata_modified desc"', once(parseOptionValue))
|
|
30
30
|
.option("--fq <filter>", "filter query, e.g. organization:allris (repeatable; all must match)", collectNonEmpty)
|
|
31
31
|
.option("--facet <field>", "count values of a field, e.g. res_format (repeatable)", collectNonEmpty)
|
|
32
|
-
.option("--facet-limit <n>", "max values per facet (default 50; -1 = all)", parseFacetLimit)
|
|
32
|
+
.option("--facet-limit <n>", "max values per facet (default 50; -1 = all)", once(parseFacetLimit))
|
|
33
33
|
.action(action(deps, async ({ client, global, opts }, [query]) => {
|
|
34
34
|
renderJson(deps, global, await client.packageSearch({
|
|
35
35
|
q: query,
|
|
@@ -72,8 +72,8 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
72
72
|
program
|
|
73
73
|
.command("packages")
|
|
74
74
|
.description("List dataset names")
|
|
75
|
-
.option("--limit <n>", "max names (1 or more; omit for all)", parseLimit)
|
|
76
|
-
.option("--offset <n>", "offset for paging", parseIntArg)
|
|
75
|
+
.option("--limit <n>", "max names (1 or more; omit for all)", once(parseLimit))
|
|
76
|
+
.option("--offset <n>", "offset for paging", once(parseIntArg))
|
|
77
77
|
.action(action(deps, async ({ client, global, opts }) => {
|
|
78
78
|
renderJson(deps, global, await client.packageList({
|
|
79
79
|
limit: opts["limit"],
|
|
@@ -84,8 +84,8 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
84
84
|
.command("organizations")
|
|
85
85
|
.description("List organizations (data publishers)")
|
|
86
86
|
.option("--all-fields", "return full objects instead of names")
|
|
87
|
-
.option("--limit <n>", "max entries (1 or more; omit for all)", parseLimit)
|
|
88
|
-
.option("--offset <n>", "offset for paging", parseIntArg)
|
|
87
|
+
.option("--limit <n>", "max entries (1 or more; omit for all)", once(parseLimit))
|
|
88
|
+
.option("--offset <n>", "offset for paging", once(parseIntArg))
|
|
89
89
|
.action(action(deps, async ({ client, global, opts }) => {
|
|
90
90
|
renderJson(deps, global, await client.organizationList({
|
|
91
91
|
all_fields: opts["allFields"],
|
|
@@ -97,8 +97,8 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
97
97
|
.command("groups")
|
|
98
98
|
.description("List groups (themes/categories)")
|
|
99
99
|
.option("--all-fields", "return full objects instead of names")
|
|
100
|
-
.option("--limit <n>", "max entries (1 or more; omit for all)", parseLimit)
|
|
101
|
-
.option("--offset <n>", "offset for paging", parseIntArg)
|
|
100
|
+
.option("--limit <n>", "max entries (1 or more; omit for all)", once(parseLimit))
|
|
101
|
+
.option("--offset <n>", "offset for paging", once(parseIntArg))
|
|
102
102
|
.action(action(deps, async ({ client, global, opts }) => {
|
|
103
103
|
renderJson(deps, global, await client.groupList({
|
|
104
104
|
all_fields: opts["allFields"],
|
|
@@ -109,7 +109,7 @@ export function registerCatalogueCommands(program, deps) {
|
|
|
109
109
|
program
|
|
110
110
|
.command("tags")
|
|
111
111
|
.description("List tags")
|
|
112
|
-
.option("--query <substring>", "only tags containing this substring", parseOptionValue)
|
|
112
|
+
.option("--query <substring>", "only tags containing this substring", once(parseOptionValue))
|
|
113
113
|
.action(action(deps, async ({ client, global, opts }) => {
|
|
114
114
|
renderJson(deps, global, await client.tagList({ query: opts["query"] }));
|
|
115
115
|
}));
|
package/dist/src/cli/io.d.ts
CHANGED
|
@@ -24,7 +24,8 @@ export interface OutputStreams {
|
|
|
24
24
|
* pager — closes the pipe while the CLI is still writing, and the next write fails
|
|
25
25
|
* with EPIPE. That is ordinary use, so the process exits 0 at once, quietly. Any
|
|
26
26
|
* other stdout error prints one `Output error: <message>` line to stderr and exits
|
|
27
|
-
* 1
|
|
27
|
+
* 1. On stderr an EPIPE is ignored, so a failed run keeps its exit code; any other
|
|
28
|
+
* stderr error exits 1 silently (there is nowhere left to report it).
|
|
28
29
|
* The bin shim installs this once, before `run()`.
|
|
29
30
|
*/
|
|
30
31
|
export declare function handleOutputErrors(streams?: OutputStreams, exit?: (code: number) => void): void;
|
package/dist/src/cli/io.js
CHANGED
|
@@ -9,7 +9,8 @@
|
|
|
9
9
|
* pager — closes the pipe while the CLI is still writing, and the next write fails
|
|
10
10
|
* with EPIPE. That is ordinary use, so the process exits 0 at once, quietly. Any
|
|
11
11
|
* other stdout error prints one `Output error: <message>` line to stderr and exits
|
|
12
|
-
* 1
|
|
12
|
+
* 1. On stderr an EPIPE is ignored, so a failed run keeps its exit code; any other
|
|
13
|
+
* stderr error exits 1 silently (there is nowhere left to report it).
|
|
13
14
|
* The bin shim installs this once, before `run()`.
|
|
14
15
|
*/
|
|
15
16
|
export function handleOutputErrors(streams = process, exit = (code) => process.exit(code)) {
|
|
@@ -19,8 +20,11 @@ export function handleOutputErrors(streams = process, exit = (code) => process.e
|
|
|
19
20
|
process.stderr.write(`Output error: ${err.message}\n`);
|
|
20
21
|
exit(1);
|
|
21
22
|
});
|
|
23
|
+
// stderr's reader going away doesn't make a failed run a success: ignore EPIPE there
|
|
24
|
+
// and let the run's own exit code stand (`2>&1 | true` used to turn a usage error into 0).
|
|
22
25
|
streams.stderr.on("error", (err) => {
|
|
23
|
-
|
|
26
|
+
if (err.code !== "EPIPE")
|
|
27
|
+
exit(1);
|
|
24
28
|
});
|
|
25
29
|
}
|
|
26
30
|
export const defaultIO = {
|
package/dist/src/cli/program.js
CHANGED
|
@@ -9,7 +9,7 @@ import { CkanClient } from "../client/client.js";
|
|
|
9
9
|
import { DEFAULT_BASE_URL, MAX_RETRIES, MAX_RETRY_AFTER_MS } from "../client/engine.js";
|
|
10
10
|
import { MAX_TIMEOUT_MS } from "../client/http.js";
|
|
11
11
|
import { redactUrl } from "../client/errors.js";
|
|
12
|
-
import { parseBaseUrl, parseBoundedInt, parseHeaderValue, parseIntArg, parsePortal } from "./shared.js";
|
|
12
|
+
import { once, parseBaseUrl, parseBoundedInt, parseHeaderValue, parseIntArg, parsePortal } from "./shared.js";
|
|
13
13
|
import { registerCatalogueCommands } from "./commands/catalogue.js";
|
|
14
14
|
import { registerPortalCommands } from "./commands/portal.js";
|
|
15
15
|
/**
|
|
@@ -43,17 +43,17 @@ export function buildProgram(deps = defaultDeps) {
|
|
|
43
43
|
.description("CLI for any CKAN open-data portal (Action API v3)")
|
|
44
44
|
.version(VERSION)
|
|
45
45
|
.addOption(new Option("--base-url <url>", "CKAN site URL (env CKAN_BASE_URL)")
|
|
46
|
-
.argParser(parseBaseUrl)
|
|
46
|
+
.argParser(once(parseBaseUrl))
|
|
47
47
|
// The help shows the default without userinfo: a password in
|
|
48
48
|
// CKAN_BASE_URL must not end up in `--help` output or CI logs.
|
|
49
49
|
.default(baseUrlDefault, JSON.stringify(redactUrl(baseUrlDefault))))
|
|
50
50
|
.addOption(new Option("--portal <id>", "a known portal by id (see `ckan portals`)")
|
|
51
|
-
.argParser(parsePortal)
|
|
51
|
+
.argParser(once(parsePortal))
|
|
52
52
|
.conflicts("baseUrl"))
|
|
53
|
-
.option("--timeout <ms>", "per-request timeout in milliseconds", parseBoundedInt(0, MAX_TIMEOUT_MS))
|
|
54
|
-
.option("--user-agent <ua>", "User-Agent header value", parseHeaderValue)
|
|
55
|
-
.option("--max-retries <n>", `retries for transient 429/503 responses (0..${MAX_RETRIES}; each waits
|
|
56
|
-
.option("--max-response-bytes <n>", "cap response body size in bytes (0 = unlimited; default 100 MiB)", parseIntArg)
|
|
53
|
+
.option("--timeout <ms>", "per-request timeout in milliseconds", once(parseBoundedInt(0, MAX_TIMEOUT_MS)))
|
|
54
|
+
.option("--user-agent <ua>", "User-Agent header value", once(parseHeaderValue))
|
|
55
|
+
.option("--max-retries <n>", `retries for transient 429/503 responses (0..${MAX_RETRIES}; each backs off linearly, or waits a longer Retry-After up to ${MAX_RETRY_AFTER_MS / 1000} s)`, once(parseBoundedInt(0, MAX_RETRIES)))
|
|
56
|
+
.option("--max-response-bytes <n>", "cap response body size in bytes (0 = unlimited; default 100 MiB)", once(parseIntArg))
|
|
57
57
|
.option("--compact", "print JSON on a single line instead of pretty-printed")
|
|
58
58
|
.showHelpAfterError();
|
|
59
59
|
// A bare invocation (no subcommand) should print help to stdout and exit 0,
|
|
@@ -73,8 +73,15 @@ export function buildProgram(deps = defaultDeps) {
|
|
|
73
73
|
program.help();
|
|
74
74
|
});
|
|
75
75
|
// commander runs value parsers on flags but not on defaults, so a base URL
|
|
76
|
-
// taken from CKAN_BASE_URL is checked here, before
|
|
77
|
-
|
|
76
|
+
// taken from CKAN_BASE_URL is checked here, before a command that uses it runs.
|
|
77
|
+
// The commands that never use it skip the check: help (the root action prints
|
|
78
|
+
// help or names an unknown command, `help` prints help) and `portals`, which
|
|
79
|
+
// prints the built-in list or checks each portal at its own URL — a user whose
|
|
80
|
+
// variable is broken is exactly the one looking for a working portal id.
|
|
81
|
+
const offline = new Set(["help", "portals"]);
|
|
82
|
+
program.hook("preAction", (_program, actionCommand) => {
|
|
83
|
+
if (actionCommand === program || offline.has(actionCommand.name()))
|
|
84
|
+
return;
|
|
78
85
|
if (program.getOptionValueSource("baseUrl") !== "default" || program.opts()["portal"] !== undefined)
|
|
79
86
|
return;
|
|
80
87
|
try {
|
package/dist/src/cli/run.d.ts
CHANGED
|
@@ -1,2 +1,29 @@
|
|
|
1
|
+
import { type Command } from "commander";
|
|
1
2
|
import type { CliDeps } from "./io.js";
|
|
3
|
+
/**
|
|
4
|
+
* Replace the userinfo of every URL in `text` with `***`, the form `redactUrl` gives
|
|
5
|
+
* (`https://user:secret@host` becomes `https://***@host`). Text-based, so it also
|
|
6
|
+
* covers a URL that does not parse; a fallback behind the exact strings of
|
|
7
|
+
* `withRedactedOutput`.
|
|
8
|
+
*/
|
|
9
|
+
export declare function redactUserinfo(text: string): string;
|
|
10
|
+
/**
|
|
11
|
+
* `deps` with an `io` that redacts the credentials of every argument and of
|
|
12
|
+
* CKAN_BASE_URL from everything it prints. Commander echoes rejected values in its
|
|
13
|
+
* errors (`--base-url`, `--portal`, a URL typed where the command goes), the
|
|
14
|
+
* client's own messages name the values they refuse, and help shows the base URL's
|
|
15
|
+
* default: whatever path a credential takes to stdout or stderr, the exact userinfo
|
|
16
|
+
* (as `credentialsIn` finds it, plus its JSON-escaped and percent-encoded forms) is
|
|
17
|
+
* replaced by `***`. A pattern alone can't delimit a password with spaces, quotes,
|
|
18
|
+
* `#`, `?` or `/`; the exact strings can. Without credentials the output passes
|
|
19
|
+
* through unchanged.
|
|
20
|
+
*/
|
|
21
|
+
export declare function withRedactedOutput(deps: CliDeps, argv: readonly string[]): CliDeps;
|
|
22
|
+
/**
|
|
23
|
+
* The index of a `help` command whose topic names no command (`ckan help nonexistent`),
|
|
24
|
+
* or undefined. Commander answers that with the root help on stderr and exit 1, but
|
|
25
|
+
* no line saying what was wrong. Global options before `help` are skipped with their
|
|
26
|
+
* values; scanning stops at `--`.
|
|
27
|
+
*/
|
|
28
|
+
export declare function unknownHelpTopic(program: Command, argv: readonly string[]): number | undefined;
|
|
2
29
|
export declare function run(argv: string[], deps?: CliDeps): Promise<number>;
|
package/dist/src/cli/run.js
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// captured output and exit code without spawning a subprocess.
|
|
4
4
|
import { CommanderError } from "commander";
|
|
5
5
|
import { buildProgram, defaultDeps } from "./program.js";
|
|
6
|
-
import { CkanApiError, CkanError, CkanValidationError } from "../client/errors.js";
|
|
6
|
+
import { CkanApiError, CkanError, CkanValidationError, credentialsIn, redactCredentials, } from "../client/errors.js";
|
|
7
7
|
/** The exit code of a usage error: commander's own for a rejected option value. */
|
|
8
8
|
const USAGE_EXIT = 1;
|
|
9
9
|
/**
|
|
@@ -20,9 +20,90 @@ function configureTree(command, deps) {
|
|
|
20
20
|
for (const child of command.commands)
|
|
21
21
|
configureTree(child, deps);
|
|
22
22
|
}
|
|
23
|
+
/**
|
|
24
|
+
* Replace the userinfo of every URL in `text` with `***`, the form `redactUrl` gives
|
|
25
|
+
* (`https://user:secret@host` becomes `https://***@host`). Text-based, so it also
|
|
26
|
+
* covers a URL that does not parse; a fallback behind the exact strings of
|
|
27
|
+
* `withRedactedOutput`.
|
|
28
|
+
*/
|
|
29
|
+
export function redactUserinfo(text) {
|
|
30
|
+
return text.replace(/\b([a-z][a-z0-9+.-]*:\/\/)[^\s/?#']*@/gi, "$1***@");
|
|
31
|
+
}
|
|
32
|
+
/**
|
|
33
|
+
* `deps` with an `io` that redacts the credentials of every argument and of
|
|
34
|
+
* CKAN_BASE_URL from everything it prints. Commander echoes rejected values in its
|
|
35
|
+
* errors (`--base-url`, `--portal`, a URL typed where the command goes), the
|
|
36
|
+
* client's own messages name the values they refuse, and help shows the base URL's
|
|
37
|
+
* default: whatever path a credential takes to stdout or stderr, the exact userinfo
|
|
38
|
+
* (as `credentialsIn` finds it, plus its JSON-escaped and percent-encoded forms) is
|
|
39
|
+
* replaced by `***`. A pattern alone can't delimit a password with spaces, quotes,
|
|
40
|
+
* `#`, `?` or `/`; the exact strings can. Without credentials the output passes
|
|
41
|
+
* through unchanged.
|
|
42
|
+
*/
|
|
43
|
+
export function withRedactedOutput(deps, argv) {
|
|
44
|
+
// An `--option=value` token is echoed as its value alone.
|
|
45
|
+
const values = argv.map((token) => (token.startsWith("-") && token.includes("=") ? token.slice(token.indexOf("=") + 1) : token));
|
|
46
|
+
const sources = [...argv, ...values, deps.env["CKAN_BASE_URL"] ?? ""];
|
|
47
|
+
const secrets = new Set();
|
|
48
|
+
const encoded = new Set();
|
|
49
|
+
for (const source of sources) {
|
|
50
|
+
for (const secret of credentialsIn(source)) {
|
|
51
|
+
secrets.add(secret);
|
|
52
|
+
secrets.add(JSON.stringify(secret).slice(1, -1));
|
|
53
|
+
// A URL sent as a parameter (`package <url>`) is echoed percent-encoded in a
|
|
54
|
+
// 404 message: `id=https%3A%2F%2Fuser%3Apw%40host`.
|
|
55
|
+
encoded.add(encodeURIComponent(`${secret}@`));
|
|
56
|
+
}
|
|
57
|
+
}
|
|
58
|
+
if (secrets.size === 0)
|
|
59
|
+
return deps;
|
|
60
|
+
const list = [...secrets];
|
|
61
|
+
const redact = (text) => {
|
|
62
|
+
let out = redactUserinfo(redactCredentials(text, list));
|
|
63
|
+
for (const form of encoded)
|
|
64
|
+
out = out.split(form).join("***%40");
|
|
65
|
+
return out;
|
|
66
|
+
};
|
|
67
|
+
return { ...deps, io: { out: (text) => deps.io.out(redact(text)), err: (text) => deps.io.err(redact(text)) } };
|
|
68
|
+
}
|
|
69
|
+
/**
|
|
70
|
+
* The index of a `help` command whose topic names no command (`ckan help nonexistent`),
|
|
71
|
+
* or undefined. Commander answers that with the root help on stderr and exit 1, but
|
|
72
|
+
* no line saying what was wrong. Global options before `help` are skipped with their
|
|
73
|
+
* values; scanning stops at `--`.
|
|
74
|
+
*/
|
|
75
|
+
export function unknownHelpTopic(program, argv) {
|
|
76
|
+
for (let i = 0; i < argv.length; i++) {
|
|
77
|
+
const token = argv[i];
|
|
78
|
+
if (token === "--")
|
|
79
|
+
return undefined;
|
|
80
|
+
if (token.startsWith("-")) {
|
|
81
|
+
if (token.includes("="))
|
|
82
|
+
continue;
|
|
83
|
+
const option = program.options.find((o) => o.long === token || o.short === token);
|
|
84
|
+
if (option?.required)
|
|
85
|
+
i++;
|
|
86
|
+
continue;
|
|
87
|
+
}
|
|
88
|
+
if (token !== "help")
|
|
89
|
+
return undefined;
|
|
90
|
+
const topic = argv[i + 1];
|
|
91
|
+
if (topic === undefined || topic.startsWith("-"))
|
|
92
|
+
return undefined;
|
|
93
|
+
const known = program.commands.some((c) => c.name() === topic || c.aliases().includes(topic));
|
|
94
|
+
return known ? undefined : i;
|
|
95
|
+
}
|
|
96
|
+
return undefined;
|
|
97
|
+
}
|
|
23
98
|
export async function run(argv, deps = defaultDeps) {
|
|
99
|
+
deps = withRedactedOutput(deps, argv);
|
|
24
100
|
const program = buildProgram(deps);
|
|
25
101
|
configureTree(program, deps);
|
|
102
|
+
// `help <unknown>` is answered like `<unknown>`: "error: unknown command '<unknown>'",
|
|
103
|
+
// the help after it, exit 1 — not the help alone with a failure code.
|
|
104
|
+
const helpAt = unknownHelpTopic(program, argv);
|
|
105
|
+
if (helpAt !== undefined)
|
|
106
|
+
argv = argv.filter((_, i) => i !== helpAt);
|
|
26
107
|
try {
|
|
27
108
|
await program.parseAsync(argv, { from: "user" });
|
|
28
109
|
return 0;
|
package/dist/src/cli/shared.d.ts
CHANGED
|
@@ -22,6 +22,14 @@ export declare function parseHeaderValue(value: string): string;
|
|
|
22
22
|
* (a negated Solr filter, `-organization:x`) stays allowed.
|
|
23
23
|
*/
|
|
24
24
|
export declare function parseOptionValue(value: string): string;
|
|
25
|
+
/**
|
|
26
|
+
* Wrap a commander value-parser for a single-valued option so that a second
|
|
27
|
+
* occurrence is a usage error. commander otherwise keeps only the last value, so
|
|
28
|
+
* `--rows 5 --rows 50` (or a second `--base-url`) silently dropped the first. The
|
|
29
|
+
* wrapper keeps its own count, so it also works for an option with a default (which
|
|
30
|
+
* commander hands the parser as `previous`); build it per program (`buildProgram`).
|
|
31
|
+
*/
|
|
32
|
+
export declare function once<T>(parse: (value: string) => T): (value: string) => T;
|
|
25
33
|
/** commander accumulator for a repeatable option whose values are checked by parseOptionValue. */
|
|
26
34
|
export declare function collectNonEmpty(value: string, previous?: string[]): string[];
|
|
27
35
|
/** Build a commander value-parser for a non-negative integer within [min, max] (the library's intRangeProblem). */
|
package/dist/src/cli/shared.js
CHANGED
|
@@ -57,6 +57,22 @@ export function parseOptionValue(value) {
|
|
|
57
57
|
}
|
|
58
58
|
return parseNonEmpty(value);
|
|
59
59
|
}
|
|
60
|
+
/**
|
|
61
|
+
* Wrap a commander value-parser for a single-valued option so that a second
|
|
62
|
+
* occurrence is a usage error. commander otherwise keeps only the last value, so
|
|
63
|
+
* `--rows 5 --rows 50` (or a second `--base-url`) silently dropped the first. The
|
|
64
|
+
* wrapper keeps its own count, so it also works for an option with a default (which
|
|
65
|
+
* commander hands the parser as `previous`); build it per program (`buildProgram`).
|
|
66
|
+
*/
|
|
67
|
+
export function once(parse) {
|
|
68
|
+
let seen = false;
|
|
69
|
+
return (value) => {
|
|
70
|
+
if (seen)
|
|
71
|
+
throw new InvalidArgumentError("Given more than once; this option takes a single value.");
|
|
72
|
+
seen = true;
|
|
73
|
+
return parse(value);
|
|
74
|
+
};
|
|
75
|
+
}
|
|
60
76
|
/** commander accumulator for a repeatable option whose values are checked by parseOptionValue. */
|
|
61
77
|
export function collectNonEmpty(value, previous = []) {
|
|
62
78
|
return previous.concat([parseOptionValue(value)]);
|
|
@@ -1,6 +1,19 @@
|
|
|
1
1
|
import { type EngineOptions } from "./engine.js";
|
|
2
2
|
import type { QueryParams } from "./query.js";
|
|
3
3
|
import type { Group, GroupListParams, JsonValue, License, ListParams, Organization, Package, PackageSearchParams, PackageSearchResult, Resource, Status, TagListParams } from "./types.js";
|
|
4
|
+
/**
|
|
5
|
+
* Most pages one `all_fields` list call requests (10,000 entries at CKAN's page size
|
|
6
|
+
* of 25). Real CKAN portals list a few hundred organizations or groups at most; a
|
|
7
|
+
* server whose pages keep returning new entries (a broken or hostile one) would
|
|
8
|
+
* otherwise be paged forever. Past it the call fails with a CkanParseError naming it.
|
|
9
|
+
*/
|
|
10
|
+
export declare const MAX_ALL_FIELDS_PAGES = 400;
|
|
11
|
+
/**
|
|
12
|
+
* Pause between two `all_fields` pages, in milliseconds. A long list costs a few
|
|
13
|
+
* tenths of a second more; a server that never ends the list gets 10 requests a
|
|
14
|
+
* second at most instead of thousands.
|
|
15
|
+
*/
|
|
16
|
+
export declare const ALL_FIELDS_PAGE_DELAY_MS = 100;
|
|
4
17
|
/**
|
|
5
18
|
* The site root of a CKAN instance, from what a user is likely to paste: portals
|
|
6
19
|
* document their API as `<site>/api/3/action`, so that suffix (or `/api/3`) is
|
|
@@ -10,9 +23,7 @@ import type { Group, GroupListParams, JsonValue, License, ListParams, Organizati
|
|
|
10
23
|
*/
|
|
11
24
|
export declare function siteRoot(baseUrl: string): string;
|
|
12
25
|
export declare class CkanClient {
|
|
13
|
-
private
|
|
14
|
-
/** The site this client talks to, for error messages. */
|
|
15
|
-
private readonly site;
|
|
26
|
+
#private;
|
|
16
27
|
constructor(options?: EngineOptions);
|
|
17
28
|
/** Call any CKAN action by name and return its unwrapped `result`. */
|
|
18
29
|
action<T = JsonValue>(name: string, params?: QueryParams): Promise<T>;
|
|
@@ -68,6 +79,15 @@ export declare class CkanClient {
|
|
|
68
79
|
* some entries), so only an empty page ends the list. Entries are deduplicated
|
|
69
80
|
* (by `id`), and a page that adds nothing new ends the loop too, so a server
|
|
70
81
|
* that ignores `offset` cannot keep it going.
|
|
82
|
+
*
|
|
83
|
+
* Berlin counts hidden entries in its `limit`/`offset` window but leaves them
|
|
84
|
+
* out of the answer, so a window smaller than a full page (the last one before
|
|
85
|
+
* `limit` is reached) can come back empty although more entries follow. Such a
|
|
86
|
+
* miss is asked again at the same offset with a full page before it ends the list.
|
|
87
|
+
*
|
|
88
|
+
* Bounds: pages are `ALL_FIELDS_PAGE_DELAY_MS` apart, and after
|
|
89
|
+
* `MAX_ALL_FIELDS_PAGES` pages without reaching the end (a server whose pages
|
|
90
|
+
* keep returning new entries) the call fails with a CkanParseError.
|
|
71
91
|
*/
|
|
72
92
|
private groupOrOrgList;
|
|
73
93
|
/**
|