@aiaiai-pt/frankctl 0.4.0

package/package.json

@@ -0,0 +1,65 @@
+{
+  "name": "@aiaiai-pt/frankctl",
+  "version": "0.4.0",
+  "description": "Terminal-first client for the Frank Low-Code Pipeline platform",
+  "homepage": "https://frank-docs-4k3.pages.dev",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/westeuropeco/frank-low-code.git",
+    "directory": "frank-cli"
+  },
+  "bugs": {
+    "url": "https://github.com/westeuropeco/frank-low-code/issues"
+  },
+  "license": "MIT",
+  "author": "aiaiai-pt",
+  "keywords": [
+    "frank",
+    "frankctl",
+    "cli",
+    "elt",
+    "iceberg",
+    "data-pipeline"
+  ],
+  "type": "module",
+  "bin": {
+    "frankctl": "./dist/index.js"
+  },
+  "engines": {
+    "node": ">=18.0.0"
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "CHANGELOG.md",
+    "skills/frank-cli/SKILL.md"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "build": "node scripts/build.mjs",
+    "dev": "node --import tsx src/index.ts",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "test:e2e": "FRANK_E2E=1 vitest run --config vitest-e2e.config.ts",
+    "coverage": "vitest run --coverage",
+    "typecheck": "tsc --noEmit",
+    "prepack": "node scripts/build.mjs && mkdir -p skills/frank-cli && cp ../skills/frank-cli/SKILL.md skills/frank-cli/SKILL.md"
+  },
+  "dependencies": {
+    "chalk": "^5.4.1",
+    "commander": "^12.1.0",
+    "ora": "^8.2.0",
+    "yaml": "^2.8.0",
+    "zod": "^4.3.6"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "@vitest/coverage-v8": "^2.1.0",
+    "esbuild": "^0.24.0",
+    "tsx": "^4.19.0",
+    "typescript": "~5.7.0",
+    "vitest": "^2.1.0"
+  }
+}

package/skills/frank-cli/SKILL.md

@@ -0,0 +1,323 @@
+---
+name: frank-cli
+description: Use the frankctl CLI to interact with the Frank API — auth, config, pipelines, sources, transforms, runs
+---
+
+# frankctl (Frank CLI) — Phase 1
+
+## 1. Overview
+
+`frankctl` is a TypeScript CLI that gives humans and agents scriptable access to the Frank API: authentication against Keycloak, config/profile management, and the core data-flow surface — pipelines, sources, transforms, and runs. Version: `0.2.0`.
+
+## 2. When to use
+
+- Before activating a pipeline: `frankctl pipelines validate <id>` to run it end-to-end in sandbox mode, stream step badges to stderr, emit a final JSON summary on stdout.
+- Scripting against Frank from bash/CI: `$(frankctl auth token)` injects a fresh access token into any `curl` call.
+- Agents composing pipelines via Martha can self-check their work without going through the UI.
+
+## 3. Installation & binary location
+
+```bash
+cd /home/ork/harnasso/frank/frank-cli
+npm install
+npm run build
+npm link    # installs `frankctl` on PATH
+```
+
+The built entry point is `dist/index.js` (ESM, esbuild-bundled, `#!/usr/bin/env node` shebang). Node ≥18 required.
+
+## 4. First-run setup
+
+```bash
+frankctl status                    # shows defaults, unauthenticated
+frankctl auth login                # browser PKCE flow (default)
+frankctl auth status               # confirms subject + tenant
+frankctl pipelines list
+```
+
+Credentials are written to `~/.frankctl/credentials.json` (0o600).
+
+## 5. Global options
+
+Available on every subcommand:
+
+| Flag | Purpose |
+|------|---------|
+| `-p, --profile <name>` | Select config profile (default: `default`) |
+| `--json` | Emit machine-readable JSON to stdout |
+| `-v, --verbose` | Verbose logging (stderr) |
+| `-q, --quiet` | Suppress non-error output |
+| `--no-color` | Disable ANSI colors |
+| `--api-url <url>` | Override API base URL for this invocation |
+
+## 6. Environment variables
+
+| Var | Effect |
+|-----|--------|
+| `FRANKCTL_API_URL` | Override `apiUrl` |
+| `FRANKCTL_KEYCLOAK_URL` | Override `keycloakUrl` |
+| `FRANKCTL_KEYCLOAK_REALM` | Override `realm` |
+| `FRANKCTL_CLIENT_ID` | Override `clientId` |
+| `FRANKCTL_CLIENT_SECRET` | Service-account secret |
+| `FRANKCTL_PROFILE` | Active profile |
+| `FRANKCTL_TOKEN` | Bypass token store; use this raw JWT |
+| `FRANK_DEV_MODE=true` + `FRANKCTL_TENANT_ID` | Forward `X-Tenant-ID` header |
+
+## 7. Profiles & `~/.frankctl/config.yaml`
+
+```yaml
+defaultProfile: default
+profiles:
+  default:
+    apiUrl: http://localhost:8000
+    keycloakUrl: http://localhost:8180
+    realm: frank
+    clientId: frank-low-code
+  prod:
+    apiUrl: https://frank.prod.example
+    keycloakUrl: https://kc.prod.example
+    realm: frank
+    clientId: frank-low-code
+```
+
+Precedence (top wins): CLI flag → env → YAML → built-in default.
+
+## 8. Auth flows
+
+- **PKCE (default)** — browser-based, best for humans. Requires a free port in `[5173, 4173, 5174]` (matches the `frank-low-code` Keycloak client's registered redirect URIs).
+- **Headless (ROPC)** — `frankctl auth login --headless --username ... --password ...`. Uses `frank-low-code` public client.
+- **Service account** — `frankctl auth login --service-account --client-id frank-api --client-secret $S`. Uses `client_credentials` grant.
+
+## 9. Token handling
+
+- Tokens refresh automatically 30s before expiry.
+- Service-account tokens re-acquire via `client_credentials` when `FRANKCTL_CLIENT_SECRET` is set.
+- Agents should use `TOKEN=$(frankctl auth token)` rather than reading the credentials file.
+
+## 10. `frankctl status`
+
+Prints version, active profile, URLs, and authentication state. `--json` mirrors all fields.
+
+## 11. `frankctl auth`
+
+- `login [--headless | --service-account] [--username --password --client-id --client-secret --no-browser]`
+- `logout` — deletes credentials for active profile.
+- `status` — exits 3 if not logged in.
+- `token` — prints a valid access token to stdout.
+
+## 12. `frankctl config`
+
+- `config set <key> <value>` — dotted key (e.g. `profiles.default.apiUrl`).
+- `config get <key>` — returns `{value, source}` where source ∈ `flag | env | config | default`.
+- `config list` — merged effective profile.
+
+## 13. `frankctl pipelines list`
+
+`GET /api/v1/pipelines` with optional `--status`, `--search`, `--page`, `--page-size`. Table output by default; `--json` returns the full `PipelineListResponse`.
+
+## 14. `frankctl pipelines get <id>`
+
+`GET /api/v1/pipelines/{id}`. Optional `--include-version` to include the active version payload.
+
+## 15. `frankctl pipelines validate <id>`
+
+`POST /api/v1/pipelines/{id}/sandbox` then polls `/sandbox/{wf}/status` every 2500 ms (`--poll-interval` to override). Step badges stream to **stderr**; the final status JSON is written to **stdout**.
+
+Flags: `--sample-limit <n=1000> --timeout <sec=600> --no-wait`.
+
+Exit codes: `completed` → 0, `partial_failure | failed` → 5.
+
+## 16. JSON output contracts
+
+- `--json` on `list`/`get`/`status`/`auth status`/`config *` emits a single JSON object/array on stdout.
+- `validate` always emits the final sandbox status as JSON on stdout, regardless of `--json`.
+
+## 17. Exit codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | Success |
+| 1 | Generic error |
+| 2 | Usage error |
+| 3 | Authentication error |
+| 4 | API error (non-2xx) |
+| 5 | Validation failed |
+
+## 18. Errors & troubleshooting
+
+- FastAPI returns `{detail: str}` or `{detail: [...]}` — both shapes are normalized.
+- `401` triggers a single token refresh + retry before surfacing.
+- Port binding failure on PKCE prints the candidate list — use `--headless` in containers.
+
+## 19. Dev mode
+
+Setting `FRANK_DEV_MODE=true` + `FRANKCTL_TENANT_ID=<uuid>` forwards `X-Tenant-ID` to bypass JWT-based tenant resolution (dev only).
+
+## 20. Relationship to `frank-pattern` (Python)
+
+The existing Python `frank` CLI in `utils/frank_cli/` handles pattern authoring (`init|validate|test|build`) and is being renamed to `frank-pattern` in a follow-up PR on the `frank-shared-utils` submodule. The TS `frankctl` and Python `frank-pattern` are orthogonal: different domains, different binaries.
+
+## 20b. Data-flow subtrees (WEC-133)
+
+### `sources`
+
+```bash
+frankctl sources list [--type <pattern-id>] [--status <s>] [--search <q>]
+frankctl sources get <id> [--no-streams]
+frankctl sources create -f source.yaml [--name ...] [--type ...]
+frankctl sources update <id> -f patch.yaml
+frankctl sources delete <id> --yes
+frankctl sources discover <id> [--no-wait] [--timeout <s>]
+frankctl sources sync <id> [--streams a,b] [--force-full-refresh] [--no-wait]
+frankctl sources logs <id> <run-id> [-f] [--level INFO|ERROR]
+frankctl sources history <id> [--limit N] [--offset N]
+
+frankctl sources streams list <source-id> [--enabled-only]
+frankctl sources streams set <source-id> -f streams.yaml
+frankctl sources streams refresh-schema <source-id> --from-discovery
+frankctl sources streams refresh-schema <source-id> -f discovery.json
+```
+
+`-f` accepts YAML, JSON, or `-` (stdin). `sources create` requires `{name, pattern_id, source_config}`. `sources streams set` body can be a bare array or `{streams: [...]}`.
+
+### `transforms`
+
+```bash
+frankctl transforms list [--status <s>] [--pipeline-id <id>]
+frankctl transforms get <id>
+frankctl transforms runs <id> [--status] [--limit N]
+frankctl transforms logs <id> <run-id> [-f] [--level]
+frankctl transforms trigger <id>       # UI "Run Now" path
+```
+
+### `runs`
+
+```bash
+frankctl runs get <workflow-id>
+frankctl runs wait <workflow-id> [--timeout <s>] [--poll-interval <ms>]
+frankctl runs cancel <id> --yes        # UUID cascades through transform-runs → sync-runs; non-UUID hits temporal
+```
+
+## 20c. AI subtree (WEC-134)
+
+```bash
+frankctl ai compose-pipeline -f spec.yaml [--no-wait] [--timeout <s>] [--poll-interval <ms>]
+frankctl ai generate-transform -f spec.yaml
+frankctl ai review-sql -f spec.yaml
+frankctl ai fix-ci-failure -f spec.yaml
+frankctl ai suggest target-schema  -f spec.yaml
+frankctl ai suggest field-mappings -f spec.yaml
+frankctl ai suggest pattern-params -f spec.yaml
+frankctl ai publish-transform -f spec.yaml
+```
+
+All AI commands emit JSON. Every response includes `available: bool`; when `false`, the CLI exits 4 with `AI service unavailable (Martha offline).`.
+
+**compose-pipeline quirks** (observed against the dev stack):
+- With incomplete/unknown `source_tables`, the backend returns `{available:true, execution_id:null, status:"error"}` synchronously. The CLI detects `execution_id==null && !processing` and exits 5 immediately (no poll).
+- With valid inputs, `execution_id` is returned and the CLI polls `/api/v1/ai/compose-pipeline/{id}/status` until `{completed, failed, error, cancelled}` or `--timeout` elapses. Exit 0 on `completed`, 5 on other terminal, 4 on timeout.
+
+## 20d. Patterns subtree (WEC-134)
+
+```bash
+frankctl patterns list [--category <c>] [--search <q>]
+frankctl patterns get <id>
+frankctl patterns validate-params <pattern-id> -f params.yaml
+frankctl patterns register -f webhook-payload.json       # CI only
+```
+
+- `validate-params` POSTs `{params: <file>}` to `/api/v1/transform-patterns/{id}/validate`; file contents are a bare params object. Exits 5 if `valid: false`.
+- `patterns register` HMAC-signs the payload with `X-Hub-Signature-256: sha256=<hex>` using `PATTERN_WEBHOOK_SECRET`. **This command is CI-only** — pattern publish workflows in `frank-transforms` invoke it after pushing a digest-pinned image to GHCR. If `PATTERN_WEBHOOK_SECRET` is unset the CLI exits **6** with a message pointing at the env var; there is no override flag because unsigned payloads are rejected by the backend with `403` regardless.
+
+## 20e. Schedules subtree (WEC-134, source-scoped)
+
+```bash
+frankctl schedules list
+frankctl schedules get <source-id>
+frankctl schedules set <source-id> -f sched.yaml   # schedule_value required
+frankctl schedules pause|resume|trigger <source-id>
+frankctl schedules delete <source-id> --yes
+```
+
+Backend expects `schedule_value` even for `schedule_type: manual` (it 422s on omission), so the zod schema requires it. Example:
+
+```yaml
+schedule_type: cron
+schedule_value: "0 */6 * * *"
+```
+
+**Transform schedules are a follow-up.** The backend surface exists (`/api/v1/transforms/{id}/schedule[/pause|resume|trigger]`) but we did not add a parallel `frankctl transform-schedules` tree in WEC-134 to keep scope tight; tracked for a future phase.
+
+## 20f. Datasets subtree (WEC-134)
+
+```bash
+frankctl datasets list [--layer bronze|silver|gold] [--namespace <ns>] [--page N] [--page-size N]
+frankctl datasets preview <layer.namespace.table> [--limit 20]
+frankctl datasets snapshots <layer.namespace.table>
+```
+
+- Dataset IDs are the 3-part `layer.namespace.table` dotted form the backend uses directly in the path (FastAPI `{dataset_id:path}` converter accepts dots verbatim).
+- `preview` renders the payload as a table of column headers + row values with cells capped at 60 chars; `--json` returns `{dataset_id, columns, rows, row_count}` unmodified.
+
+## 20a. Running E2E tests
+
+E2E tests exercise the compiled CLI against a live Frank stack. They are gated on `FRANK_E2E=1` so `npm test` stays hermetic.
+
+Preconditions:
+- Frank stack running (`docker-compose up -d` from repo root; `curl localhost:8002/health` → 200).
+- Test user provisioned (`testuser` / `testuser123` in the `frank` realm).
+- Dev tenant id: `550e8400-e29b-41d4-a716-446655440001`.
+
+Recipe:
+```bash
+cd frank-cli
+export FRANKCTL_API_URL=http://localhost:8002
+export FRANK_DEV_MODE=true
+export FRANKCTL_TENANT_ID=550e8400-e29b-41d4-a716-446655440001
+./dist/index.js auth login --headless --username testuser --password testuser123
+export FRANKCTL_TOKEN=$(./dist/index.js auth token)
+npm run build
+npm run test:e2e
+```
+
+Expect ≥24 tests green in ~90s on a healthy stack (new AI file is skipped unless Martha is available). Run a single file:
+```bash
+FRANK_E2E=1 npx vitest run --config vitest-e2e.config.ts tests/e2e/sources.e2e.test.ts
+```
+
+Fixtures live in `tests/e2e/fixtures/` (bash scripts). Each test wraps its seed in try/finally so a failed assertion never leaks state. Config is `vitest-e2e.config.ts` (serial, 120s timeouts).
+
+Notes on dev-stack quirks the suite tolerates:
+- `sources discover --wait` against real postgres can exceed 45s — test accepts exit 0 OR clean timeout (exit 4/5).
+- `sources sync` kicks off a Temporal workflow but `sync_run` rows may not materialize synchronously in dev — test asserts sync acceptance + well-formed history response, not a populated row.
+- `transforms trigger` returns 500 when Dagster code location is unreachable — test accepts 2xx JSON OR exit 4/5 with a stderr message.
+
+## 21. Known limitations
+
+- Prod tenant resolution requires `FRANK_DEV_MODE=true` + `FRANKCTL_TENANT_ID` (backend expects `X-Tenant-ID` today — tracked as follow-up WEC-135).
+- `runs cancel` against a raw temporal workflow-id currently 501s (backend gap).
+- No `pipelines create/update`.
+- No `transform-schedules` tree (follow-up). No transform-run cancel.
+- PKCE redirect ports are fixed by Keycloak realm JSON.
+
+## 22. Roadmap
+
+Phase 3+: JWT-claim tenant resolution (removes `FRANK_DEV_MODE` requirement), transform-schedules subtree, shell completion bundle, `pipelines create/update`.
+
+## 23. Exit code reference
+
+| Code | Meaning |
+|------|---------|
+| 0 | Success |
+| 1 | Generic error |
+| 2 | Usage error |
+| 3 | Authentication |
+| 4 | API (incl. Martha unavailable) |
+| 5 | Validation (sandbox/compose-pipeline terminal failure, validate-params invalid) |
+| 6 | Config (e.g. `PATTERN_WEBHOOK_SECRET` unset for `patterns register`) |
+
+## 24. Changelog
+
+- `0.3.0` — WEC-134: ai (compose-pipeline + generate/review/fix-ci/suggest/publish), patterns (list/get/validate-params/register with HMAC), schedules (source-scoped list/get/set/delete/pause/resume/trigger), datasets (list/preview/snapshots). Adds ExitCode 6 (Config).
+- `0.2.0` — WEC-133: sources CRUD + discover/sync/streams/logs/history, transforms list/get/runs/logs/trigger, runs get/wait/cancel.
+- `0.1.0` — WEC-132: status, auth (PKCE/ROPC/client-credentials), config, pipelines list/get/validate.