@aiaiai-pt/frankctl 0.5.0 → 0.7.0

package/dist/template-packs/cira/template-pack.yaml

@@ -0,0 +1,109 @@
+apiVersion: frank.platform/v1alpha1
+kind: PipelineTemplatePack
+metadata:
+  name: cira
+  title: CIRA source-to-ontology verticals
+  description: >-
+    All existing CIRA declarative integrations, rendered for the current
+    Frank tenant namespace and an explicitly selected ontology tenant.
+spec:
+  ontology_schema_ownership: external
+  forbidden_fields:
+    - password
+    - username
+    - api_key
+    - access_token
+    - auth_token
+    - client_secret
+    - api_token
+    - oauth_token
+    - secret
+    - secret_key
+    - private_key
+    - aws_access_key_id
+    - aws_secret_access_key
+    - aws_session_token
+    - sasl_password
+  parameters:
+    frank_tenant_prefix:
+      type: string
+      default: "00000000"
+      pattern: "^[0-9a-f]{8}$"
+      description: Eight-character Frank tenant UUID prefix used by Iceberg namespaces.
+    ontology_tenant_id:
+      type: string
+      default: cira
+      pattern: "^[a-z][a-z0-9_-]*$"
+      description: Existing ontology tenant targeted by every BackingDataset.
+  components:
+    cityfy-events:
+      title: Cityfy agenda events
+      status: live_proven
+      manifests:
+        - cityfy_agenda_events.yaml
+      required_sources: []
+      issues:
+        - 589
+    transdev-gtfs:
+      title: Historical Transdev/AveiroBus GTFS
+      status: live_proven
+      manifests:
+        - 09_transdev_aveirobus_gtfs.yaml
+      required_sources:
+        - cira-transdev-aveirobus-gtfs
+      issues:
+        - 600
+    ciclope-emergency:
+      title: CICLOPE emergency and IoT projections
+      status: live_proven
+      manifests:
+        - 10_ciclope_emergency.yaml
+      required_sources:
+        - cira-ciclope
+      issues:
+        - 601
+    regional-transport-core:
+      title: Regional transport organizations, freguesias, routes, and stops
+      status: declared_pending
+      manifests:
+        - 01_organization.yaml
+        - 02_freguesia.yaml
+        - 03_transit_route.yaml
+        - 04_transit_stop.yaml
+      required_sources:
+        - cira-bdtransportes-rede
+      issues:
+        - 577
+        - 580
+    air-quality:
+      title: Air-quality stations and observations
+      status: declared_pending
+      manifests:
+        - 05_air_quality_station.yaml
+        - 06_air_quality_observed.yaml
+      required_sources:
+        - Kunak Cloud - Ubiwhere
+        - VDL - AirQuality - Monitar
+      issues:
+        - 579
+        - 591
+    regional-transport-trips:
+      title: Regional transport schedule and trips
+      status: declared_pending
+      manifests:
+        - 07_transit_trip.yaml
+      required_sources: []
+      issues:
+        - 577
+        - 580
+        - 591
+        - 592
+    regional-transport-avl:
+      title: Regional transport current vehicle positions
+      status: declared_pending
+      manifests:
+        - 08_vehicle_position.yaml
+      required_sources: []
+      issues:
+        - 577
+        - 580

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aiaiai-pt/frankctl",
-  "version": "0.5.0",
+  "version": "0.7.0",
   "description": "Terminal-first client for the Frank Low-Code Pipeline platform",
   "homepage": "https://franksdocs.nomadriver.co",
   "repository": {
@@ -43,6 +43,7 @@
     "test": "vitest run",
     "test:watch": "vitest",
     "test:e2e": "FRANK_E2E=1 vitest run --config vitest-e2e.config.ts",
+    "test:package": "node scripts/smoke-template-pack.mjs",
     "coverage": "vitest run --coverage",
     "typecheck": "tsc --noEmit",
     "prepack": "node scripts/build.mjs && mkdir -p skills/frank-cli && cp ../skills/frank-cli/SKILL.md skills/frank-cli/SKILL.md"

package/skills/frank-cli/SKILL.md

@@ -19,6 +19,11 @@ YAML file and `apply` it idempotently. Reach for the imperative subcommands
 (`sources`, `transforms`, …) for inspection, one-off actions, and debugging — but let
 YAML be your source of truth.
 
+When a curated vertical already exists, start from its packaged template instead of
+scaffolding it again. `templates render` is local and emits the same ordinary
+`frank.platform/v1` YAML consumed by `pipelines apply`; it is not a second execution
+engine.
+
 ## Install
 
 ```bash
@@ -46,8 +51,8 @@ frankctl config set profiles.default.keycloakUrl https://auth.frank.nomadriver.c
 
 Auth modes:
 - **PKCE (default)** — `frankctl auth login`. Browser-based, best for humans.
-- **Headless** — `frankctl auth login --headless --username … --password …`. For containers/CI.
-- **Service account** — `frankctl auth login --service-account --client-id … --client-secret …`.
+- **Headless** — `frankctl auth login --headless --username … --password-stdin`, or inject `FRANKCTL_PASSWORD` without the stdin flag.
+- **Service account** — `frankctl auth login --service-account --client-id … --client-secret-stdin`, or inject `FRANKCTL_CLIENT_SECRET`.
 
 Tokens are stored per-profile at `~/.frankctl/credentials.json` (0600) and refresh
 automatically before expiry. In scripts, inject a fresh token instead of reading the
@@ -62,6 +67,24 @@ curl -H "Authorization: Bearer $TOKEN" https://api.frank.nomadriver.co/api/v1/pi
 
 ## Declarative pipelines — the repeatable path
 
+### Curated template packs
+
+```bash
+frankctl templates list
+frankctl --json templates show cira
+frankctl templates render cira -o cira-proven.yaml
+frankctl pipelines apply -f cira-proven.yaml --dry-run-server --live-ontology
+frankctl pipelines apply -f cira-proven.yaml --wait
+```
+
+`templates show` lists component status and any pre-existing Source names required by
+that component. The CIRA pack renders its live-proven components by default. Use
+`--all` to include declared-but-pending components, or repeat `--component <name>` for
+an explicit subset. Repeat `--set frank_tenant_prefix=<8-hex>` and
+`--set ontology_tenant_id=<tenant>` to target another Frank namespace and existing
+ontology tenant. The CIRA pack treats ontology schema as externally owned and therefore
+contains no `ensure_schema` mutation.
+
 A pipeline is one multi-document YAML file describing three kinds of object under the
 `frank.platform/v1` API version:
 
@@ -72,10 +95,10 @@ A pipeline is one multi-document YAML file describing three kinds of object unde
 | `BackingDataset` | Maps a silver table's columns to **ontology** entity properties |
 
 Objects reference each other by `metadata.name`, so one file can be self-contained and
-version-controllable. Commit only reviewed, secret-free declarations. Source credentials
-are currently literal `source_config` values: `frankctl` does not expand `${ENV_VAR}` and
-export preserves those values. Keep a secret-bearing Source in an untracked mode-0600
-file and commit its Pipeline/BackingDataset declarations separately.
+version-controllable. Commit only reviewed, secret-free declarations. Structurally
+declared Source credentials are stored in Vault and written with `sources credentials
+set|rotate --values-file -`; `source_config` and export contain only ordinary values plus
+the opaque tenant-owned `credential_ref`.
 
 ### The loop: scaffold → edit → apply
 
@@ -108,8 +131,9 @@ frankctl pipelines apply -f pipeline.yaml --wait --timeout 600
 Use `--json` with `--wait` for one evidence object correlated to the exact Source
 workflow/SyncRun, Dagster/TransformRun, and ontology SyncRun started by this invocation.
 Snapshot IDs are decimal strings so JavaScript cannot round them. `status: skipped`
-means the ontology was already converged and no rows were replayed; use `bd entities`
-for separate target entity readback.
+means the ontology was already converged and no rows were replayed. Ontology evidence
+prefers `snapshot_id_exact`; numeric-only evidence fails closed. Use `bd entities` for
+separate target entity readback.
 
 ### Preflight before you apply
 
@@ -149,16 +173,14 @@ Without these, `apply` is fully idempotent.
 ### Export existing work back to YAML
 
 Already built a pipeline in the UI? Export it — plus its Sources and Backing Datasets —
-to a private file for review:
+to a file for review:
 
 ```bash
-umask 077
-frankctl pipelines export <pipeline-id> > pipeline.private.yaml
+frankctl pipelines export <pipeline-id> > pipeline.yaml
 ```
 
-The export includes Source configuration and config-only stream declarations but omits
-stream runtime state. Split any secret-bearing Source document out before committing the
-remaining declarations.
+The export includes ordinary Source configuration, opaque credential references, and
+config-only stream declarations, but omits credential values and stream runtime state.
 
 ### Minimal manifest shape