@aiaiai-pt/frankctl 0.4.1 → 0.4.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/CHANGELOG.md +8 -0
  2. package/dist/index.js +136 -136
  3. package/package.json +1 -1
  4. package/skills/frank-cli/SKILL.md +249 -204
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@aiaiai-pt/frankctl",
3
- "version": "0.4.1",
3
+ "version": "0.4.3",
4
4
  "description": "Terminal-first client for the Frank Low-Code Pipeline platform",
5
5
  "homepage": "https://franksdocs.nomadriver.co",
6
6
  "repository": {
package/skills/frank-cli/SKILL.md CHANGED
@@ -1,170 +1,242 @@
1
1
  ---
2
2
  name: frank-cli
3
- description: Use the frankctl CLI to interact with the Frank API — auth, config, pipelines, sources, transforms, runs
3
+ description: Use the frankctl CLI to provision Frank pipelines declaratively — apply/scaffold/export multi-doc YAML — plus auth, sources, transforms, backing datasets, and runs
4
4
  ---
5
5
 
6
- # frankctl (Frank CLI) — Phase 1
6
+ # frankctl — the Frank CLI
7
7
 
8
- ## 1. Overview
8
+ `frankctl` is the command-line client for the Frank platform: a scriptable, repeatable
9
+ way to build and operate end-to-end data pipelines — Sources (Extract/Load into an
10
+ Iceberg bronze lakehouse), Transforms/Pipelines (bronze → silver/gold), and Backing
11
+ Datasets (silver → ontology entities).
9
12
 
10
- `frankctl` is a TypeScript CLI that gives humans and agents scriptable access to the Frank API: authentication against Keycloak, config/profile management, and the core data-flow surface — pipelines, sources, transforms, and runs. Version: `0.2.0`.
13
+ It is built for two audiences: developers scripting Frank from a terminal or CI, and
14
+ agents composing and self-checking pipelines without a UI. Every command speaks the
15
+ public Frank API and authenticates against Keycloak.
11
16
 
12
- ## 2. When to use
17
+ The heart of the tool is **declarative provisioning**: describe a whole pipeline in one
18
+ YAML file and `apply` it idempotently. Reach for the imperative subcommands
19
+ (`sources`, `transforms`, …) for inspection, one-off actions, and debugging — but let
20
+ YAML be your source of truth.
13
21
 
14
- - Before activating a pipeline: `frankctl pipelines validate <id>` to run it end-to-end in sandbox mode, stream step badges to stderr, emit a final JSON summary on stdout.
15
- - Scripting against Frank from bash/CI: `$(frankctl auth token)` injects a fresh access token into any `curl` call.
16
- - Agents composing pipelines via Martha can self-check their work without going through the UI.
17
-
18
- ## 3. Installation & binary location
22
+ ## Install
19
23
 
20
24
  ```bash
21
- cd /home/ork/harnasso/frank/frank-cli
22
- npm install
23
- npm run build
24
- npm link # installs `frankctl` on PATH
25
+ npm install -g @aiaiai-pt/frankctl
26
+ frankctl --version
25
27
  ```
26
28
 
27
- The built entry point is `dist/index.js` (ESM, esbuild-bundled, `#!/usr/bin/env node` shebang). Node ≥18 required.
29
+ Node ≥18. The binary is `frankctl`. Docs: https://franksdocs.nomadriver.co
28
30
 
29
- ## 4. First-run setup
31
+ ## Connect & authenticate
30
32
 
31
33
  ```bash
32
- frankctl status # shows defaults, unauthenticated
33
- frankctl auth login # browser PKCE flow (default)
34
- frankctl auth status # confirms subject + tenant
35
- frankctl pipelines list
34
+ frankctl status # effective config + whether you're logged in
35
+ frankctl auth login # browser PKCE flow (opens Keycloak)
36
+ frankctl auth status # confirms your subject + tenant
36
37
  ```
37
38
 
38
- Credentials are written to `~/.frankctl/credentials.json` (0o600).
39
+ Point it at an environment by setting the API URL once (or per-invocation with
40
+ `--api-url`):
39
41
 
40
- ## 5. Global options
42
+ ```bash
43
+ frankctl config set profiles.default.apiUrl https://api.frank.nomadriver.co
44
+ frankctl config set profiles.default.keycloakUrl https://auth.frank.nomadriver.co
45
+ ```
41
46
 
42
- Available on every subcommand:
47
+ Auth modes:
48
+ - **PKCE (default)** — `frankctl auth login`. Browser-based, best for humans.
49
+ - **Headless** — `frankctl auth login --headless --username … --password …`. For containers/CI.
50
+ - **Service account** — `frankctl auth login --service-account --client-id … --client-secret …`.
43
51
 
44
- | Flag | Purpose |
45
- |------|---------|
46
- | `-p, --profile <name>` | Select config profile (default: `default`) |
47
- | `--json` | Emit machine-readable JSON to stdout |
48
- | `-v, --verbose` | Verbose logging (stderr) |
49
- | `-q, --quiet` | Suppress non-error output |
50
- | `--no-color` | Disable ANSI colors |
51
- | `--api-url <url>` | Override API base URL for this invocation |
52
+ Tokens are stored per-profile at `~/.frankctl/credentials.json` (0600) and refresh
53
+ automatically before expiry. In scripts, inject a fresh token instead of reading the
54
+ file:
52
55
 
53
- ## 6. Environment variables
56
+ ```bash
57
+ TOKEN=$(frankctl auth token)
58
+ curl -H "Authorization: Bearer $TOKEN" https://api.frank.nomadriver.co/api/v1/pipelines
59
+ ```
54
60
 
55
- | Var | Effect |
56
- |-----|--------|
57
- | `FRANKCTL_API_URL` | Override `apiUrl` |
58
- | `FRANKCTL_KEYCLOAK_URL` | Override `keycloakUrl` |
59
- | `FRANKCTL_KEYCLOAK_REALM` | Override `realm` |
60
- | `FRANKCTL_CLIENT_ID` | Override `clientId` |
61
- | `FRANKCTL_CLIENT_SECRET` | Service-account secret |
62
- | `FRANKCTL_PROFILE` | Active profile |
63
- | `FRANKCTL_TOKEN` | Bypass token store; use this raw JWT |
64
- | `FRANK_DEV_MODE=true` + `FRANKCTL_TENANT_ID` | Forward `X-Tenant-ID` header |
61
+ ---
65
62
 
66
- ## 7. Profiles & `~/.frankctl/config.yaml`
63
+ ## Declarative pipelines — the repeatable path
67
64
 
68
- ```yaml
69
- defaultProfile: default
70
- profiles:
71
- default:
72
- apiUrl: http://localhost:8000
73
- keycloakUrl: http://localhost:8180
74
- realm: frank
75
- clientId: frank-low-code
76
- prod:
77
- apiUrl: https://frank.prod.example
78
- keycloakUrl: https://kc.prod.example
79
- realm: frank
80
- clientId: frank-low-code
81
- ```
65
+ A pipeline is one multi-document YAML file describing three kinds of object under the
66
+ `frank.platform/v1` API version:
82
67
 
83
- Precedence (top wins): CLI flag → env → YAML → built-in default.
68
+ | `kind` | Role |
69
+ |--------|------|
70
+ | `Source` | Extract/Load config — connector `pattern_id`, streams, schedule → Iceberg **bronze** |
71
+ | `Pipeline` | Transform steps (SQL/Python) turning bronze → **silver/gold** |
72
+ | `BackingDataset` | Maps a silver table's columns to **ontology** entity properties |
84
73
 
85
- ## 8. Auth flows
74
+ Objects reference each other by `metadata.name`, so one file is self-contained and
75
+ version-controllable. Commit these files to git — they *are* your infrastructure.
86
76
 
87
- - **PKCE (default)** — browser-based, best for humans. Requires a free port in `[5173, 4173, 5174]` (matches the `frank-low-code` Keycloak client's registered redirect URIs).
88
- - **Headless (ROPC)** — `frankctl auth login --headless --username ... --password ...`. Uses `frank-low-code` public client.
89
- - **Service account** — `frankctl auth login --service-account --client-id frank-api --client-secret $S`. Uses `client_credentials` grant.
77
+ ### The loop: scaffold → edit → apply
90
78
 
91
- ## 9. Token handling
79
+ **1. Scaffold** a starter manifest from a source you've already synced. Frank reads the
80
+ real physical bronze schema and pre-fills the SQL with the actual Trino `ROW` refs:
92
81
 
93
- - Tokens refresh automatically 30s before expiry.
94
- - Service-account tokens re-acquire via `client_credentials` when `FRANKCTL_CLIENT_SECRET` is set.
95
- - Agents should use `TOKEN=$(frankctl auth token)` rather than reading the credentials file.
82
+ ```bash
83
+ frankctl pipelines scaffold --source ipma_indice_uv -o pipeline.yaml
84
+ ```
96
85
 
97
- ## 10. `frankctl status`
86
+ **2. Edit** `pipeline.yaml`. Usually the only required change is naming the ontology
87
+ entity (`BackingDataset.entity_type_id` / `entity_type_name`); optionally refine the
88
+ `primary_key_column`, add casts, or tune the SELECT.
98
89
 
99
- Prints version, active profile, URLs, and authentication state. `--json` mirrors all fields.
90
+ **3. Apply** it idempotently. Re-running is safe — Frank reconciles to the declared
91
+ state rather than creating duplicates:
100
92
 
101
- ## 11. `frankctl auth`
93
+ ```bash
94
+ frankctl pipelines apply -f pipeline.yaml
95
+ ```
102
96
 
103
- - `login [--headless | --service-account] [--username --password --client-id --client-secret --no-browser]`
104
- - `logout` — deletes credentials for active profile.
105
- - `status` — exits 3 if not logged in.
106
- - `token` — prints a valid access token to stdout.
97
+ Add `--wait` to drive the whole chain to completion and let a non-zero exit mean "it
98
+ didn't land": sync Sources → bronze, materialize Pipeline steps → silver, sync Backing
99
+ Datasets → ontology.
107
100
 
108
- ## 12. `frankctl config`
101
+ ```bash
102
+ frankctl pipelines apply -f pipeline.yaml --wait --timeout 600
103
+ ```
109
104
 
110
- - `config set <key> <value>` — dotted key (e.g. `profiles.default.apiUrl`).
111
- - `config get <key>` — returns `{value, source}` where source ∈ `flag | env | config | default`.
112
- - `config list` — merged effective profile.
105
+ ### Preflight before you apply
113
106
 
114
- ## 13. `frankctl pipelines list`
107
+ | Flag | What it does |
108
+ |------|--------------|
109
+ | `--dry-run` | Client-side parse + cross-ref resolve. Catches malformed YAML and broken references. No server call. |
110
+ | `--dry-run-server` | Server-side preflight against the catalog + DB without persisting. Catches unknown `pattern_id`s, would-be immutable-diff conflicts, and unresolved refs. |
115
111
 
116
- `GET /api/v1/pipelines` with optional `--status`, `--search`, `--page`, `--page-size`. Table output by default; `--json` returns the full `PipelineListResponse`.
112
+ Run `--dry-run-server` in CI on every change; run `apply --wait` to promote.
117
113
 
118
- ## 14. `frankctl pipelines get <id>`
114
+ ### Immutable state & when to force
119
115
 
120
- `GET /api/v1/pipelines/{id}`. Optional `--include-version` to include the active version payload.
116
+ Backing Datasets carry operational state (ontology cursor, sync history). If a change
117
+ would require destroying that state, `apply` returns `409 immutable_diff` and stops —
118
+ it will not silently discard your data. Two explicit opt-ins exist for when you *do*
119
+ mean it:
121
120
 
122
- ## 15. `frankctl pipelines validate <id>`
121
+ - `--allow-recreate` — DELETE the conflicting row and re-create. **Loses** cursor,
122
+ ledger, snapshot ids, and run history. Use knowingly.
123
+ - `--allow-deprecate` — permit an `ensure_schema` that narrows the ontology schema.
123
124
 
124
- `POST /api/v1/pipelines/{id}/sandbox` then polls `/sandbox/{wf}/status` every 2500 ms (`--poll-interval` to override). Step badges stream to **stderr**; the final status JSON is written to **stdout**.
125
+ Without these, `apply` is fully idempotent.
125
126
 
126
- Flags: `--sample-limit <n=1000> --timeout <sec=600> --no-wait`.
127
+ ### Export existing work back to YAML
127
128
 
128
- Exit codes: `completed` → 0, `partial_failure | failed` → 5.
129
+ Already built a pipeline in the UI? Pull it — plus its Sources and Backing Datasets —
130
+ into a manifest you can commit and re-apply elsewhere:
129
131
 
130
- ## 16. JSON output contracts
132
+ ```bash
133
+ frankctl pipelines export <pipeline-id> > pipeline.yaml
134
+ ```
131
135
 
132
- - `--json` on `list`/`get`/`status`/`auth status`/`config *` emits a single JSON object/array on stdout.
133
- - `validate` always emits the final sandbox status as JSON on stdout, regardless of `--json`.
136
+ ### Minimal manifest shape
134
137
 
135
- ## 17. Exit codes
138
+ ```yaml
139
+ apiVersion: frank.platform/v1
140
+ kind: Source
141
+ metadata:
142
+ name: ipma_indice_uv
143
+ spec:
144
+ pattern_id: ipma
145
+ source_config:
146
+ streams: [uv_index]
147
+ streams:
148
+ - name: ipma_uv_index
149
+ sync_mode: full_refresh
150
+ write_disposition: replace
151
+ schedule_type: cron
152
+ schedule_value: "0 6 * * *"
153
+ ---
154
+ apiVersion: frank.platform/v1
155
+ kind: Pipeline
156
+ metadata:
157
+ name: ipma_indice_uv_pipeline
158
+ spec:
159
+ source_ids: [ipma_indice_uv]
160
+ steps:
161
+ - name: ipma_indice_uv_silver
162
+ kind: custom_sql
163
+ sources: [bronze.tenant_00000000_ipma_indice_uv.ipma_uv_index]
164
+ emits_to: backing_dataset
165
+ config: { output_layer: silver }
166
+ params:
167
+ sql: |-
168
+ SELECT o."date" AS date, o."uv_index" AS uv_index, o."record_key" AS record_key
169
+ FROM tenant_00000000_ipma_indice_uv.ipma_uv_index o
170
+ ---
171
+ apiVersion: frank.platform/v1
172
+ kind: BackingDataset
173
+ metadata:
174
+ name: uv_index_forecast
175
+ spec:
176
+ entity_type_id: uv_index_forecast
177
+ entity_type_name: UV Index Forecast
178
+ pipeline: ipma_indice_uv_pipeline
179
+ sync_mode: on_materialization
180
+ primary_key_column: record_key
181
+ property_mappings:
182
+ - { column: date, property: date, type: date }
183
+ - { column: uv_index, property: uv_index, type: long }
184
+ ```
136
185
 
137
- | Code | Meaning |
186
+ ---
187
+
188
+ ## Global options
189
+
190
+ Available on every subcommand:
191
+
192
+ | Flag | Purpose |
138
193
  |------|---------|
139
- | 0 | Success |
140
- | 1 | Generic error |
141
- | 2 | Usage error |
142
- | 3 | Authentication error |
143
- | 4 | API error (non-2xx) |
144
- | 5 | Validation failed |
194
+ | `-p, --profile <name>` | Select config profile (default: `default`) |
195
+ | `--json` | Emit machine-readable JSON to stdout |
196
+ | `-v, --verbose` | Verbose logging (stderr) |
197
+ | `-q, --quiet` | Suppress non-error output |
198
+ | `--no-color` | Disable ANSI colors |
199
+ | `--api-url <url>` | Override the API base URL for this invocation |
200
+
201
+ ### Environment variables
145
202
 
146
- ## 18. Errors & troubleshooting
203
+ | Var | Effect |
204
+ |-----|--------|
205
+ | `FRANKCTL_API_URL` | Override `apiUrl` |
206
+ | `FRANKCTL_KEYCLOAK_URL` | Override `keycloakUrl` |
207
+ | `FRANKCTL_KEYCLOAK_REALM` | Override `realm` |
208
+ | `FRANKCTL_CLIENT_ID` / `FRANKCTL_CLIENT_SECRET` | Client id / service-account secret |
209
+ | `FRANKCTL_PROFILE` | Active profile |
210
+ | `FRANKCTL_TOKEN` | Use this raw JWT instead of the token store |
147
211
 
148
- - FastAPI returns `{detail: str}` or `{detail: [...]}` — both shapes are normalized.
149
- - `401` triggers a single token refresh + retry before surfacing.
150
- - Port binding failure on PKCE prints the candidate list — use `--headless` in containers.
212
+ ### Profiles (`~/.frankctl/config.yaml`)
151
213
 
152
- ## 19. Dev mode
214
+ ```yaml
215
+ defaultProfile: default
216
+ profiles:
217
+ default:
218
+ apiUrl: https://api.frank.nomadriver.co
219
+ keycloakUrl: https://auth.frank.nomadriver.co
220
+ realm: frank
221
+ clientId: frank-low-code
222
+ ```
153
223
 
154
- Setting `FRANK_DEV_MODE=true` + `FRANKCTL_TENANT_ID=<uuid>` forwards `X-Tenant-ID` to bypass JWT-based tenant resolution (dev only).
224
+ Precedence (top wins): CLI flag → env var → YAML → built-in default.
225
+ `config get <key>` reports `{value, source}` so you can see which layer won;
226
+ `config list` prints the merged effective profile.
155
227
 
156
- ## 20. Relationship to `frank-pattern` (Python)
228
+ ---
157
229
 
158
- The existing Python `frank` CLI in `utils/frank_cli/` handles pattern authoring (`init|validate|test|build`) and is being renamed to `frank-pattern` in a follow-up PR on the `frank-shared-utils` submodule. The TS `frankctl` and Python `frank-pattern` are orthogonal: different domains, different binaries.
230
+ ## Command reference
159
231
 
160
- ## 20b. Data-flow subtrees (WEC-133)
232
+ Use these to inspect and act on what `apply` provisioned, and to debug.
161
233
 
162
- ### `sources`
234
+ ### Sources (Extract/Load → bronze)
163
235
 
164
236
  ```bash
165
237
  frankctl sources list [--type <pattern-id>] [--status <s>] [--search <q>]
166
238
  frankctl sources get <id> [--no-streams]
167
- frankctl sources create -f source.yaml [--name ...] [--type ...]
239
+ frankctl sources create -f source.yaml # {name, pattern_id, source_config}
168
240
  frankctl sources update <id> -f patch.yaml
169
241
  frankctl sources delete <id> --yes
170
242
  frankctl sources discover <id> [--no-wait] [--timeout <s>]
@@ -175,149 +247,122 @@ frankctl sources history <id> [--limit N] [--offset N]
175
247
  frankctl sources streams list <source-id> [--enabled-only]
176
248
  frankctl sources streams set <source-id> -f streams.yaml
177
249
  frankctl sources streams refresh-schema <source-id> --from-discovery
178
- frankctl sources streams refresh-schema <source-id> -f discovery.json
179
250
  ```
180
251
 
181
- `-f` accepts YAML, JSON, or `-` (stdin). `sources create` requires `{name, pattern_id, source_config}`. `sources streams set` body can be a bare array or `{streams: [...]}`.
252
+ `-f` accepts YAML, JSON, or `-` (stdin). `streams set` takes a bare array or `{streams: […]}`.
182
253
 
183
- ### `transforms`
254
+ ### Pipelines & Transforms (bronze → silver/gold)
184
255
 
185
256
  ```bash
257
+ frankctl pipelines list [--status <s>] [--search <q>]
258
+ frankctl pipelines get <id> [--include-version]
259
+ frankctl pipelines validate <id> # sandbox run; badges → stderr, JSON summary → stdout
260
+ frankctl pipelines delete <id> # does not delete silver tables
261
+
186
262
  frankctl transforms list [--status <s>] [--pipeline-id <id>]
187
263
  frankctl transforms get <id>
188
264
  frankctl transforms runs <id> [--status] [--limit N]
189
265
  frankctl transforms logs <id> <run-id> [-f] [--level]
190
- frankctl transforms trigger <id> # UI "Run Now" path
266
+ frankctl transforms trigger <id> # "Run Now"
191
267
  ```
192
268
 
193
- ### `runs`
269
+ `pipelines validate` runs the pipeline end-to-end in sandbox mode against a sample and
270
+ streams step results — use it as a pre-activation check.
271
+
272
+ ### Backing Datasets (silver → ontology)
273
+
274
+ ```bash
275
+ frankctl bd list
276
+ frankctl bd get <id>
277
+ frankctl bd create -f bd.yaml
278
+ frankctl bd update <id> -f patch.yaml
279
+ frankctl bd delete <id> --yes # does not delete underlying data
280
+ frankctl bd sync <id> # trigger ontology sync, poll to terminal
281
+ frankctl bd entities <id> # read entities back out — confirm values landed
282
+ ```
283
+
284
+ `bd entities` closes the authoring loop: it reads the ontology to prove your rows
285
+ actually materialized as entities, not just that the sync reported success.
286
+
287
+ ### Runs
194
288
 
195
289
  ```bash
196
290
  frankctl runs get <workflow-id>
197
291
  frankctl runs wait <workflow-id> [--timeout <s>] [--poll-interval <ms>]
198
- frankctl runs cancel <id> --yes # UUID cascades through transform-runs → sync-runs; non-UUID hits temporal
292
+ frankctl runs cancel <id> --yes
199
293
  ```
200
294
 
201
- ## 20c. AI subtree (WEC-134)
295
+ ### Datasets (browse the lakehouse)
202
296
 
203
297
  ```bash
204
- frankctl ai compose-pipeline -f spec.yaml [--no-wait] [--timeout <s>] [--poll-interval <ms>]
205
- frankctl ai generate-transform -f spec.yaml
206
- frankctl ai review-sql -f spec.yaml
207
- frankctl ai fix-ci-failure -f spec.yaml
208
- frankctl ai suggest target-schema -f spec.yaml
209
- frankctl ai suggest field-mappings -f spec.yaml
210
- frankctl ai suggest pattern-params -f spec.yaml
211
- frankctl ai publish-transform -f spec.yaml
298
+ frankctl datasets list [--layer bronze|silver|gold] [--namespace <ns>] [--page N]
299
+ frankctl datasets preview <layer.namespace.table> [--limit 20]
300
+ frankctl datasets snapshots <layer.namespace.table>
212
301
  ```
213
302
 
214
- All AI commands emit JSON. Every response includes `available: bool`; when `false`, the CLI exits 4 with `AI service unavailable (Martha offline).`.
215
-
216
- **compose-pipeline quirks** (observed against the dev stack):
217
- - With incomplete/unknown `source_tables`, the backend returns `{available:true, execution_id:null, status:"error"}` synchronously. The CLI detects `execution_id==null && !processing` and exits 5 immediately (no poll).
218
- - With valid inputs, `execution_id` is returned and the CLI polls `/api/v1/ai/compose-pipeline/{id}/status` until `{completed, failed, error, cancelled}` or `--timeout` elapses. Exit 0 on `completed`, 5 on other terminal, 4 on timeout.
303
+ Dataset ids are the 3-part `layer.namespace.table` dotted form used directly in the API
304
+ path. `--json` on `preview` returns `{dataset_id, columns, rows, row_count}`.
219
305
 
220
- ## 20d. Patterns subtree (WEC-134)
306
+ ### Patterns & schedules
221
307
 
222
308
  ```bash
223
309
  frankctl patterns list [--category <c>] [--search <q>]
224
310
  frankctl patterns get <id>
225
311
  frankctl patterns validate-params <pattern-id> -f params.yaml
226
- frankctl patterns register -f webhook-payload.json # CI only
227
- ```
228
-
229
- - `validate-params` POSTs `{params: <file>}` to `/api/v1/transform-patterns/{id}/validate`; file contents are a bare params object. Exits 5 if `valid: false`.
230
- - `patterns register` HMAC-signs the payload with `X-Hub-Signature-256: sha256=<hex>` using `PATTERN_WEBHOOK_SECRET`. **This command is CI-only** — pattern publish workflows in `frank-transforms` invoke it after pushing a digest-pinned image to GHCR. If `PATTERN_WEBHOOK_SECRET` is unset the CLI exits **6** with a message pointing at the env var; there is no override flag because unsigned payloads are rejected by the backend with `403` regardless.
231
312
 
232
- ## 20e. Schedules subtree (WEC-134, source-scoped)
233
-
234
- ```bash
235
313
  frankctl schedules list
236
314
  frankctl schedules get <source-id>
237
- frankctl schedules set <source-id> -f sched.yaml # schedule_value required
315
+ frankctl schedules set <source-id> -f sched.yaml # schedule_value required, even for manual
238
316
  frankctl schedules pause|resume|trigger <source-id>
239
317
  frankctl schedules delete <source-id> --yes
240
318
  ```
241
319
 
242
- Backend expects `schedule_value` even for `schedule_type: manual` (it 422s on omission), so the zod schema requires it. Example:
320
+ ### AI assistance
243
321
 
244
- ```yaml
245
- schedule_type: cron
246
- schedule_value: "0 */6 * * *"
247
- ```
248
-
249
- **Transform schedules are a follow-up.** The backend surface exists (`/api/v1/transforms/{id}/schedule[/pause|resume|trigger]`) but we did not add a parallel `frankctl transform-schedules` tree in WEC-134 to keep scope tight; tracked for a future phase.
250
-
251
- ## 20f. Datasets subtree (WEC-134)
252
-
253
- ```bash
254
- frankctl datasets list [--layer bronze|silver|gold] [--namespace <ns>] [--page N] [--page-size N]
255
- frankctl datasets preview <layer.namespace.table> [--limit 20]
256
- frankctl datasets snapshots <layer.namespace.table>
257
- ```
258
-
259
- - Dataset IDs are the 3-part `layer.namespace.table` dotted form the backend uses directly in the path (FastAPI `{dataset_id:path}` converter accepts dots verbatim).
260
- - `preview` renders the payload as a table of column headers + row values with cells capped at 60 chars; `--json` returns `{dataset_id, columns, rows, row_count}` unmodified.
261
-
262
- ## 20a. Running E2E tests
263
-
264
- E2E tests exercise the compiled CLI against a live Frank stack. They are gated on `FRANK_E2E=1` so `npm test` stays hermetic.
265
-
266
- Preconditions:
267
- - Frank stack running (`docker-compose up -d` from repo root; `curl localhost:8002/health` → 200).
268
- - Test user provisioned (`testuser` / `testuser123` in the `frank` realm).
269
- - Dev tenant id: `550e8400-e29b-41d4-a716-446655440001`.
270
-
271
- Recipe:
272
- ```bash
273
- cd frank-cli
274
- export FRANKCTL_API_URL=http://localhost:8002
275
- export FRANK_DEV_MODE=true
276
- export FRANKCTL_TENANT_ID=550e8400-e29b-41d4-a716-446655440001
277
- ./dist/index.js auth login --headless --username testuser --password testuser123
278
- export FRANKCTL_TOKEN=$(./dist/index.js auth token)
279
- npm run build
280
- npm run test:e2e
281
- ```
282
-
283
- Expect ≥24 tests green in ~90s on a healthy stack (new AI file is skipped unless Martha is available). Run a single file:
284
322
  ```bash
285
- FRANK_E2E=1 npx vitest run --config vitest-e2e.config.ts tests/e2e/sources.e2e.test.ts
323
+ frankctl ai compose-pipeline -f spec.yaml [--no-wait] [--timeout <s>]
324
+ frankctl ai generate-transform -f spec.yaml
325
+ frankctl ai review-sql -f spec.yaml
326
+ frankctl ai suggest target-schema -f spec.yaml
327
+ frankctl ai suggest field-mappings -f spec.yaml
328
+ frankctl ai suggest pattern-params -f spec.yaml
286
329
  ```
287
330
 
288
- Fixtures live in `tests/e2e/fixtures/` (bash scripts). Each test wraps its seed in try/finally so a failed assertion never leaks state. Config is `vitest-e2e.config.ts` (serial, 120s timeouts).
289
-
290
- Notes on dev-stack quirks the suite tolerates:
291
- - `sources discover --wait` against real postgres can exceed 45s — test accepts exit 0 OR clean timeout (exit 4/5).
292
- - `sources sync` kicks off a Temporal workflow but `sync_run` rows may not materialize synchronously in dev — test asserts sync acceptance + well-formed history response, not a populated row.
293
- - `transforms trigger` returns 500 when Dagster code location is unreachable — test accepts 2xx JSON OR exit 4/5 with a stderr message.
294
-
295
- ## 21. Known limitations
331
+ AI commands emit JSON. Every response carries `available: bool`; when the AI service is
332
+ offline the command exits 4 with a clear message. `compose-pipeline` returns a manifest
333
+ you can review, edit, and feed straight into `pipelines apply`.
296
334
 
297
- - Prod tenant resolution requires `FRANK_DEV_MODE=true` + `FRANKCTL_TENANT_ID` (backend expects `X-Tenant-ID` today — tracked as follow-up WEC-135).
298
- - `runs cancel` against a raw temporal workflow-id currently 501s (backend gap).
299
- - No `pipelines create/update`.
300
- - No `transform-schedules` tree (follow-up). No transform-run cancel.
301
- - PKCE redirect ports are fixed by Keycloak realm JSON.
302
-
303
- ## 22. Roadmap
335
+ ---
304
336
 
305
- Phase 3+: JWT-claim tenant resolution (removes `FRANK_DEV_MODE` requirement), transform-schedules subtree, shell completion bundle, `pipelines create/update`.
337
+ ## Output & exit codes
306
338
 
307
- ## 23. Exit code reference
339
+ - `--json` makes `list`/`get`/`status`/`config`/`auth status` emit a single JSON
340
+ object or array on stdout.
341
+ - `pipelines validate` and the `ai` commands always emit their result as JSON.
342
+ - FastAPI error shapes (`{detail: …}`) are normalized; a `401` triggers one automatic
343
+ token refresh + retry before surfacing.
308
344
 
309
345
  | Code | Meaning |
310
346
  |------|---------|
311
347
  | 0 | Success |
312
348
  | 1 | Generic error |
313
349
  | 2 | Usage error |
314
- | 3 | Authentication |
315
- | 4 | API (incl. Martha unavailable) |
316
- | 5 | Validation (sandbox/compose-pipeline terminal failure, validate-params invalid) |
317
- | 6 | Config (e.g. `PATTERN_WEBHOOK_SECRET` unset for `patterns register`) |
350
+ | 3 | Authentication error |
351
+ | 4 | API error (non-2xx, incl. AI service unavailable) |
352
+ | 5 | Validation failed (sandbox/compose terminal failure, invalid params) |
353
+ | 6 | Config error (missing required env/secret) |
354
+
355
+ ## Troubleshooting
318
356
 
319
- ## 24. Changelog
357
+ - **PKCE can't bind a port** (containers/headless): use `frankctl auth login --headless`.
358
+ - **401 after login**: your token expired and refresh failed — `frankctl auth login` again.
359
+ - **`409 immutable_diff` on apply**: a Backing Dataset change would destroy state; see
360
+ `--allow-recreate` above and decide deliberately.
361
+ - **`apply` did nothing**: that's idempotency — the declared state already matches. Use
362
+ `-v` to see the reconcile plan, or `--dry-run-server` to preview.
320
363
 
321
- - `0.3.0` — WEC-134: ai (compose-pipeline + generate/review/fix-ci/suggest/publish), patterns (list/get/validate-params/register with HMAC), schedules (source-scoped list/get/set/delete/pause/resume/trigger), datasets (list/preview/snapshots). Adds ExitCode 6 (Config).
322
- - `0.2.0` — WEC-133: sources CRUD + discover/sync/streams/logs/history, transforms list/get/runs/logs/trigger, runs get/wait/cancel.
323
- - `0.1.0` — WEC-132: status, auth (PKCE/ROPC/client-credentials), config, pipelines list/get/validate.
364
+ ## Print this reference
365
+
366
+ ```bash
367
+ frankctl skill # prints this document to stdout
368
+ ```