@maschinenlesbar.org/govdata-cli 0.0.1 → 0.0.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +172 -130
  2. package/package.json +1 -1
package/README.md CHANGED
@@ -1,187 +1,229 @@
1
1
  # govdata-cli
2
2
 
3
- A TypeScript **API client** and **command-line interface** for the open
4
- [GovData](https://www.govdata.de/) **CKAN Action API** (`ckan.govdata.de`) — the
5
- central German open-data catalogue (Bund/Länder/Kommunen): search datasets,
6
- inspect packages, organizations, groups, tags and resources.
7
-
8
- - **Zero runtime HTTP dependencies** — built on Node's built-in `http`/`https` (no axios, no fetch polyfill).
9
- - **One small dependency** for the CLI: [`commander`](https://github.com/tj/commander.js).
10
- - **Strongly typed** — typed CKAN envelope, search result and parameter objects, plus a generic `action` escape hatch.
11
- - **Well tested** — unit tests on Node's built-in test runner (`node --test`), every HTTP response mocked.
12
- - **Read-only, no auth** — only CKAN read actions are wrapped; no key required.
13
-
14
- ## Requirements
15
-
16
- - Node.js **>= 20** (uses the stable built-in test runner, ESM and top-level `await`).
3
+ [![CI](https://github.com/maschinenlesbar-org/govdata-cli/actions/workflows/ci.yml/badge.svg)](https://github.com/maschinenlesbar-org/govdata-cli/actions/workflows/ci.yml)
4
+ [![Release](https://github.com/maschinenlesbar-org/govdata-cli/actions/workflows/release.yml/badge.svg)](https://github.com/maschinenlesbar-org/govdata-cli/actions/workflows/release.yml)
5
+ [![npm](https://img.shields.io/npm/v/@maschinenlesbar.org/govdata-cli)](https://www.npmjs.com/package/@maschinenlesbar.org/govdata-cli)
6
+
7
+ Browse Germany's central open-data catalogue from your terminal. `govdata` is a
8
+ command-line tool over the [GovData CKAN Action API](https://www.govdata.de/)
9
+ (`ckan.govdata.de`) — the national portal that federates open datasets from
10
+ federal government, federal states and municipalities: search, inspect, filter
11
+ and pipe straight into [`jq`](https://jqlang.github.io/jq/).
12
+
13
+ - **Works out of the box** — no account, no API key, no configuration. Install
14
+ and explore.
15
+ - **Clean JSON output** — the CKAN envelope is unwrapped for you; `--compact`
16
+ for one-line/scripting.
17
+ - **Ten commands** — `search`, `package`, `packages`, `organizations`,
18
+ `organization`, `groups`, `group`, `tags`, `resource`, and a generic `action`
19
+ escape hatch.
20
+ - **Everything is open** — every dataset this tool reaches is publicly licensed;
21
+ nothing to register for.
22
+
23
+ > Want to use this as a TypeScript library or understand how it's built?
24
+ > See **[DEVELOPING.md](DEVELOPING.md)**.
17
25
 
18
26
  ## Install
19
27
 
20
28
  ```bash
21
- npm install
22
- npm run build # compiles TypeScript to dist/
29
+ npm i -g @maschinenlesbar.org/govdata-cli
23
30
  ```
24
31
 
25
- Run the CLI without a global install:
32
+ This installs the **`govdata`** command. Requires **Node.js 20+**.
33
+
34
+ Check it works:
26
35
 
27
36
  ```bash
28
- node dist/src/cli/index.js --help
29
- # or, after `npm link` / global install:
30
37
  govdata --help
31
38
  ```
32
39
 
33
- ---
40
+ ## Quickstart
34
41
 
35
- ## CLI usage
42
+ No setup needed — the API is open and requires no key. Your first search:
36
43
 
37
- CKAN wraps every response in `{ help, success, result }`; this CLI prints the
38
- unwrapped **`result`** (and exits non-zero if `success` is false). `--compact` for
39
- a single line.
44
+ ```bash
45
+ govdata search Haushalt --rows 5
46
+ ```
40
47
 
41
- ### Global options
48
+ The result is unwrapped from CKAN's `{ help, success, result }` envelope — you
49
+ get the search object directly: `{ count, results, sort, … }`. Pull out just
50
+ the titles with `jq`:
42
51
 
43
- | Option | Description |
44
- | --- | --- |
45
- | `--base-url <url>` | API base URL (default `https://ckan.govdata.de`) |
46
- | `--timeout <ms>` | Per-request timeout (default `30000`) |
47
- | `--user-agent <ua>` | `User-Agent` header value |
48
- | `--max-retries <n>` | Retries for transient `429`/`503` responses (default `2`) |
49
- | `--max-response-bytes <n>` | Cap response body size in bytes (`0` = unlimited; default 100 MiB) |
50
- | `--compact` | Print JSON on a single line |
52
+ ```bash
53
+ govdata search Haushalt --rows 5 | jq '{count, titles: [.results[].title]}'
54
+ ```
55
+
56
+ Take a name from those results and fetch the full dataset record:
51
57
 
52
- Global options may be given before or after the command, e.g.
53
- `govdata --compact search Haushalt` or `govdata search Haushalt --compact`.
58
+ ```bash
59
+ govdata package luftqualitat
60
+ ```
54
61
 
55
- ### Commands
62
+ ## Commands
56
63
 
57
64
  ```text
58
- search [query] [--rows <n>] [--start <n>] [--sort <expr>] [--fq <filter> ...]
59
- package <id> show one dataset
60
- packages [--limit] [--offset] list dataset names
61
- organizations [--all-fields] list organizations (publishers)
62
- organization <id> show one organization
63
- groups [--all-fields] list groups (themes)
64
- group <id> show one group
65
- tags [--query <substring>] list tags
66
- resource <id> show one resource (distribution)
67
- action <name> [--param key=value ...] call any CKAN action (generic)
65
+ search [query] [filters…] search datasets
66
+ package <id> show one dataset by id or name
67
+ packages [--limit <n>] [--offset <n>] list dataset names
68
+ organizations [--all-fields] list organizations (publishers)
69
+ organization <id> show one organization
70
+ groups [--all-fields] list groups (themes/categories)
71
+ group <id> show one group
72
+ tags [--query <substring>] list tags
73
+ resource <id> show one resource (distribution)
74
+ action <name> [--param key=value …] call any CKAN action (generic)
68
75
  ```
69
76
 
70
- ### Examples
77
+ ### `search` filters
71
78
 
72
- ```bash
73
- # Search datasets, newest first
74
- govdata search Haushalt --rows 5 --sort "metadata_modified desc"
79
+ | Flag | Meaning |
80
+ | --- | --- |
81
+ | `[query]` | free-text Solr query, e.g. `Haushalt` or `title:Klimaschutz` |
82
+ | `--fq <filter>` | Solr filter query, e.g. `organization:destatis` (repeatable) |
83
+ | `--rows <n>` | max results to return |
84
+ | `--start <n>` | zero-based offset for paging |
85
+ | `--sort <expr>` | Solr sort expression, e.g. `metadata_modified desc` |
75
86
 
76
- # Filter by organization and format
77
- govdata search --fq organization:destatis --fq res_format:CSV
87
+ ### `packages` flags
78
88
 
79
- # A specific dataset
80
- govdata package <dataset-id-or-name>
89
+ | Flag | Meaning |
90
+ | --- | --- |
91
+ | `--limit <n>` | max names to return |
92
+ | `--offset <n>` | number of records to skip |
81
93
 
82
- # Publishers
83
- govdata organizations
94
+ ### `organizations` / `groups` flag
84
95
 
85
- # Any action not wrapped above
86
- govdata action package_search --param q=Klima --param rows=3
87
- ```
96
+ | Flag | Meaning |
97
+ | --- | --- |
98
+ | `--all-fields` | return full objects instead of bare names |
99
+
100
+ ### `tags` flag
101
+
102
+ | Flag | Meaning |
103
+ | --- | --- |
104
+ | `--query <substring>` | filter tags by substring |
88
105
 
89
- Exit codes: `0` success (and for `--help`/`--version`), `4` on a `404` from the API, `1` for any other error (incl. a CKAN `success:false`) and for commander usage/parse errors.
106
+ ### `action` flag
90
107
 
91
- ---
108
+ | Flag | Meaning |
109
+ | --- | --- |
110
+ | `--param <key=value>` | query parameter (repeatable; duplicate keys are rejected) |
92
111
 
93
- ## Library usage
112
+ The **[Glossary](GLOSSARY.md)** decodes every CKAN term and search-parameter
113
+ name.
94
114
 
95
- ```ts
96
- import { GovDataClient, GovDataError } from "@maschinenlesbar.org/govdata-cli";
115
+ ## Common tasks
97
116
 
98
- const client = new GovDataClient(); // defaults to https://ckan.govdata.de
117
+ A few recipes to get going — see **[Usage.md](Usage.md)** for the full,
118
+ use-case-driven set.
99
119
 
100
- const hits = await client.packageSearch({ q: "Haushalt", rows: 5 });
101
- const dataset = await client.packageShow(hits.results[0]!.id as string);
102
- const orgs = await client.organizationList();
120
+ ```bash
121
+ # Newest datasets first
122
+ govdata search Klima --rows 10 --sort "metadata_modified desc"
103
123
 
104
- // Generic escape hatch for any read action:
105
- const tags = await client.action<string[]>("tag_list", { query: "energie" });
124
+ # Filter by publisher and file format
125
+ govdata search --fq organization:destatis --fq res_format:CSV
106
126
 
107
- try {
108
- await client.packageShow("does-not-exist");
109
- } catch (err) {
110
- if (err instanceof GovDataError) console.error(err.message);
111
- }
112
- ```
127
+ # Full dataset with all its resources (distributions)
128
+ govdata package luftqualitat | jq '.resources[] | {name, format, url}'
129
+
130
+ # All data publishers (short names)
131
+ govdata organizations
113
132
 
114
- ### Client options
115
-
116
- ```ts
117
- new GovDataClient({
118
- baseUrl: "https://ckan.govdata.de",
119
- timeoutMs: 15_000,
120
- maxRetries: 3, // 429 / 503 are retried with linear backoff
121
- maxResponseBytes: 50 << 20, // abort responses larger than 50 MiB (0 = unlimited)
122
- userAgent: "my-app/1.0",
123
- transport: customTransport, // inject your own HTTP transport
124
- });
133
+ # Full publisher objects, then drill into one
134
+ govdata organizations --all-fields | jq '.[] | {name, title, packages: .package_count}'
135
+ govdata organization statistisches-bundesamt | jq '{title, package_count}'
136
+
137
+ # Tags matching a substring
138
+ govdata tags --query energie
139
+
140
+ # Any CKAN action not covered by a dedicated command
141
+ govdata action package_search --param q=Verkehr --param rows=3
125
142
  ```
126
143
 
127
- ### Methods
144
+ ## Output & scripting
128
145
 
129
- `packageSearch`, `packageShow`, `packageList`, `organizationList`, `organizationShow`,
130
- `groupList`, `groupShow`, `tagList`, `resourceShow`, and the generic `action(name, params)`.
146
+ Every command prints the **unwrapped `result`** as pretty JSON to stdout.
147
+ Errors and diagnostics go to stderr, so piping stdout into `jq` stays clean.
131
148
 
132
- ---
149
+ ```bash
150
+ # Total datasets in the catalogue
151
+ govdata action package_search --param rows=0 | jq '.count'
133
152
 
134
- ## Architecture
153
+ # Resource format and download URL from a dataset
154
+ govdata package luftqualitat | jq '.resources[] | {format, url}'
135
155
 
156
+ # Discover a valid dataset name from a search hit
157
+ govdata search Luftqualität --rows 1 | jq -r '.results[0].name'
136
158
  ```
137
- src/
138
- client/
139
- types.ts # CkanEnvelope, PackageSearchResult + parameter objects
140
- query.ts # dependency-free query-string builder
141
- http.ts # the Transport interface + default node:http/https transport
142
- engine.ts # URL building, retry/backoff, redirects, JSON decoding, error mapping
143
- errors.ts # GovDataError / GovDataApiError / GovDataNetworkError / GovDataParseError
144
- client.ts # GovDataClient — CKAN actions over the engine (with result-unwrapping)
145
- cli/
146
- io.ts # injectable I/O seam (stdout/stderr)
147
- shared.ts # option parsers, global-option resolver, JSON renderer
148
- commands/ # search / package / organizations / groups / tags / resource / action
149
- program.ts # assembles the commander program from injectable deps
150
- run.ts # parses argv -> exit code (no process.exit; testable)
151
- index.ts # #! bin shim
159
+
160
+ Use `--compact` for single-line JSON in pipelines and logs:
161
+
162
+ ```bash
163
+ govdata --compact search Haushalt --rows 5 | jq -c '.results[].title'
152
164
  ```
153
165
 
154
- **Design notes**
166
+ `--compact` (and every global option) works **before or after** the command —
167
+ both `govdata --compact search …` and `govdata search … --compact` do the same
168
+ thing.
155
169
 
156
- - The HTTP layer is a single `Transport` function (`(req) => Promise<HttpResponse>`). The default
157
- uses `node:http`/`node:https`; tests inject a mock. This keeps the client free of any HTTP framework.
158
- - The client unwraps CKAN's `{ help, success, result }` envelope and raises `GovDataError`
159
- when `success` is false, so callers work directly with `result`.
160
- - A generic `action(name, params)` exposes every read action even where there is no typed convenience method. The action name is validated against `^[a-z0-9_]+$` and URL-encoded, so it cannot inject extra path segments, query string, or fragments into the request URL.
161
- - Redirects are followed up to `maxRedirects`; if a redirect crosses origin, the request headers are dropped so nothing (e.g. a future auth/cookie header) leaks to another host.
170
+ **Exit codes** make the CLI easy to use in scripts:
162
171
 
163
- ---
172
+ | Code | Meaning |
173
+ | --- | --- |
174
+ | `0` | success (also `--help` / `--version`) |
175
+ | `4` | dataset/resource not found (`404`) |
176
+ | `1` | any other error — bad usage, CKAN `success:false`, network failure |
164
177
 
165
- ## Testing
178
+ ## Troubleshooting
166
179
 
167
- ```bash
168
- npm test # builds, then runs `node --test` over dist/test
169
- ```
180
+ - **`command not found: govdata`** — the global npm bin directory isn't on your
181
+ `PATH`. Run `npm bin -g` to find it and add it, or run via
182
+ `npx @maschinenlesbar.org/govdata-cli …`.
183
+ - **Exit `4` / "not found"** — the dataset or resource id doesn't exist or has
184
+ been removed. Re-run a `search` to get a fresh name/id.
185
+ - **Exit `1` / CKAN `success:false`** — the catalogue rejected the request
186
+ (malformed filter, unknown action name, etc.). Check your `--fq` syntax or
187
+ `--param` values.
188
+ - **Network failure / timeout** — connectivity or a slow server. Try again, or
189
+ raise the limit with `--timeout 60000`.
190
+ - **Empty `results`** — the search matched nothing; broaden the keyword, drop
191
+ an `--fq` filter, or check spelling.
192
+
193
+ ## Global options
194
+
195
+ These apply to every command and may be given before *or* after it:
196
+
197
+ | Option | Description |
198
+ | --- | --- |
199
+ | `-V, --version` | Print the version number |
200
+ | `-h, --help` | Show help for the program or a command |
201
+ | `--compact` | Print JSON on a single line instead of pretty-printed |
202
+ | `--base-url <url>` | API base URL (default `https://ckan.govdata.de`) |
203
+ | `--timeout <ms>` | Per-request timeout (default `30000`) |
204
+ | `--user-agent <ua>` | `User-Agent` header value |
205
+ | `--max-retries <n>` | Retries for transient `429`/`503` responses (default `2`) |
206
+ | `--max-response-bytes <n>` | Cap response body size in bytes (`0` = unlimited; default 100 MiB) |
207
+
208
+ ## Learn more
170
209
 
171
- - **`query.test.ts`** — query-string serialisation.
172
- - **`http.test.ts`** — the default transport against a real loopback `http.createServer`.
173
- - **`engine.test.ts`** — URL building, JSON decoding, error mapping, 429/503 retry, redirects — mocked transport.
174
- - **`client.test.ts`** — action URL/param mapping, result unwrapping, success:false handling — mocked transport.
175
- - **`cli.test.ts`** — end-to-end command parsing, `--param`/`--fq` handling and exit codes — mocked client.
210
+ - **[Usage.md](Usage.md)** — full use-case-driven cookbook.
211
+ - **[GLOSSARY.md](GLOSSARY.md)** — every CKAN term, search parameter and domain
212
+ concept explained.
213
+ - **[DEVELOPING.md](DEVELOPING.md)** — TypeScript library usage, architecture,
214
+ testing, CI.
215
+ - **[SKILLS.md](SKILLS.md)** — Claude Code Agent Skills bundled with this repo
216
+ (dataset finder, catalogue stats, resource harvest), installable as a plugin.
176
217
 
177
- ## Continuous integration
218
+ ## Data license
178
219
 
179
- GitHub Actions workflows under `.github/workflows/`:
220
+ This CLI is a **client** — it accesses data it does not own or redistribute. The
221
+ upstream data is © its provider and licensed **separately from this tool's code**.
222
+ See **[DATA_LICENSE.md](DATA_LICENSE.md)**.
180
223
 
181
- - **ci.yml** — type-check, build and test on Node 20/22/24 for every push and PR.
182
- - **release.yml** — on a `v*` tag: verify the tag matches `package.json`, test, `npm pack`, and create a GitHub Release with the tarball.
183
- - **publish.yml** — manual dispatch: publish to npm via OIDC **Trusted Publishing** (no stored `NPM_TOKEN`) with provenance.
184
- - **docs.yml** — build TypeDoc API docs and deploy to GitHub Pages on each `v*` tag.
224
+ > **GovData** — catalogue *metadata* is Datenlizenz Deutschland **Zero** 2.0 (≈ CC0,
225
+ > no attribution). Each linked **dataset has its own license** set by its publisher
226
+ > — always check the dataset's `dct:license` before reusing its contents.
185
227
 
186
228
  ## License
187
229
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@maschinenlesbar.org/govdata-cli",
3
- "version": "0.0.1",
3
+ "version": "0.0.3",
4
4
  "description": "TypeScript API client and CLI for the open GovData CKAN catalogue API (ckan.govdata.de)",
5
5
  "type": "module",
6
6
  "bin": {