@aiaiai-pt/frankctl 0.4.3 → 0.5.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aiaiai-pt/frankctl",
-  "version": "0.4.3",
+  "version": "0.5.0",
   "description": "Terminal-first client for the Frank Low-Code Pipeline platform",
   "homepage": "https://franksdocs.nomadriver.co",
   "repository": {

package/skills/frank-cli/SKILL.md

@@ -71,8 +71,11 @@ A pipeline is one multi-document YAML file describing three kinds of object unde
 | `Pipeline` | Transform steps (SQL/Python) turning bronze → **silver/gold** |
 | `BackingDataset` | Maps a silver table's columns to **ontology** entity properties |
 
-Objects reference each other by `metadata.name`, so one file is self-contained and
-version-controllable. Commit these files to git — they *are* your infrastructure.
+Objects reference each other by `metadata.name`, so one file can be self-contained and
+version-controllable. Commit only reviewed, secret-free declarations. Source credentials
+are currently literal `source_config` values: `frankctl` does not expand `${ENV_VAR}` and
+export preserves those values. Keep a secret-bearing Source in an untracked mode-0600
+file and commit its Pipeline/BackingDataset declarations separately.
 
 ### The loop: scaffold → edit → apply
 
@@ -102,15 +105,34 @@ Datasets → ontology.
 frankctl pipelines apply -f pipeline.yaml --wait --timeout 600
 ```
 
+Use `--json` with `--wait` for one evidence object correlated to the exact Source
+workflow/SyncRun, Dagster/TransformRun, and ontology SyncRun started by this invocation.
+Snapshot IDs are decimal strings so JavaScript cannot round them. `status: skipped`
+means the ontology was already converged and no rows were replayed; use `bd entities`
+for separate target entity readback.
+
 ### Preflight before you apply
 
 | Flag | What it does |
 |------|--------------|
 | `--dry-run` | Client-side parse + cross-ref resolve. Catches malformed YAML and broken references. No server call. |
 | `--dry-run-server` | Server-side preflight against the catalog + DB without persisting. Catches unknown `pattern_id`s, would-be immutable-diff conflicts, and unresolved refs. |
+| `--dry-run-server --live-ontology` | Adds an opt-in, GET-only check of ontology tenant, type, properties, relationships, keys, and declared carrier types. It never changes ontology schema or entities. |
 
 Run `--dry-run-server` in CI on every change; run `apply --wait` to promote.
 
+`Source.spec.streams` is desired state. Reapply creates or patches declared streams by
+name while preserving IDs, cursors, iterator chunks, counters, and run history. Omitted
+streams are preserved and reported as unmanaged; declare `is_enabled: false` to disable
+one explicitly. When streams are declared, `--wait` executes only their enabled names.
+`spec.max_items_per_chunk` sets an iterator chunk budget; omit it for the connector or
+cluster default and use explicit `null` in a Source PATCH to clear an override.
+
+For relationship mappings, add a matching `metadata.dependsOn` entry to the
+BackingDataset and set `spec.sync_mode: manual`. `--wait` topologically orders same-file
+dependencies and requires external dependencies in the same ontology tenant to be
+already `synced`. Cycles and self-referential relationships fail before execution.
+
 ### Immutable state & when to force
 
 Backing Datasets carry operational state (ontology cursor, sync history). If a change
@@ -126,13 +148,18 @@ Without these, `apply` is fully idempotent.
 
 ### Export existing work back to YAML
 
-Already built a pipeline in the UI? Pull it — plus its Sources and Backing Datasets —
-into a manifest you can commit and re-apply elsewhere:
+Already built a pipeline in the UI? Export it — plus its Sources and Backing Datasets —
+to a private file for review:
 
 ```bash
-frankctl pipelines export <pipeline-id> > pipeline.yaml
+umask 077
+frankctl pipelines export <pipeline-id> > pipeline.private.yaml
 ```
 
+The export includes Source configuration and config-only stream declarations but omits
+stream runtime state. Split any secret-bearing Source document out before committing the
+remaining declarations.
+
 ### Minimal manifest shape
 
 ```yaml
@@ -247,9 +274,19 @@ frankctl sources history <id> [--limit N] [--offset N]
 frankctl sources streams list <source-id> [--enabled-only]
 frankctl sources streams set <source-id> -f streams.yaml
 frankctl sources streams refresh-schema <source-id> --from-discovery
+frankctl sources streams refresh-schema <source-id> -f discovery-result.yaml
 ```
 
 `-f` accepts YAML, JSON, or `-` (stdin). `streams set` takes a bare array or `{streams: […]}`.
+Schema refresh consumes a complete discovery snapshot. It changes discovered schema and
+supported modes while preserving namespace/destination routing, operator choices, and
+runtime state; streams absent from a non-empty snapshot are removed as stale.
+
+After a successful `full_refresh` + `replace`, `rows_synced` is the current table
+cardinality rather than a lifetime sum. Append and incremental modes remain cumulative.
+A connector-confirmed, fully exhausted zero-row result can atomically replace an existing
+table only for `full_refresh` + `replace`; unknown, partial, failed, append, merge, and
+incremental empties cannot erase it.
 
 ### Pipelines & Transforms (bronze → silver/gold)
 
@@ -277,12 +314,24 @@ frankctl bd get <id>
 frankctl bd create -f bd.yaml
 frankctl bd update <id> -f patch.yaml
 frankctl bd delete <id> --yes          # does not delete underlying data
-frankctl bd sync <id>                  # trigger ontology sync, poll to terminal
+frankctl bd sync <id> [--force] [--no-wait]  # trigger ontology sync, poll exact run
 frankctl bd entities <id>              # read entities back out — confirm values landed
 ```
 
 `bd entities` closes the authoring loop: it reads the ontology to prove your rows
 actually materialized as entities, not just that the sync reported success.
+`bd sync --force` full-replays through REST even when the snapshot and effective
+mapping spec are unchanged. Normal output includes replay reason, row/snapshot
+evidence, and safe attempted/applied SyncSpec fingerprint prefixes. Human output
+prefers the precision-safe decimal string `snapshot_id_exact`; JSON exposes it
+alongside the legacy numeric `snapshot_id`.
+
+For an ordinary sync, the CLI falls back to BackingDataset status polling when
+an older API does not return exact-run evidence, and clearly marks replay
+evidence unavailable. `--force` does not fall back: it checks
+`/api/v1/backing-datasets/capabilities` before mutation and sends no sync POST
+unless version 2, exact-run polling, and force replay are explicitly supported.
+After the POST, it also requires `force_replay: true` confirmation.
 
 ### Runs