@aiaiai-pt/frankctl 0.7.2 → 0.8.1

package/dist/template-packs/cira/11_stop_time.yaml

@@ -0,0 +1,109 @@
+# Stage 11: all 237,808 regional schedule rows projected to the existing B3
+# stop_time type. The source _record_id is generated once from the declared
+# composite stream key (trip_id, Ordem, IdParagem), persisted in Bronze, and
+# passed through unchanged here. Point-in-time profile: 237,808 non-null and
+# distinct _record_id values, 8,964 trip keys, 2,411 stop keys, and zero
+# relationship misses. These counts are evidence, never row limits.
+
+apiVersion: frank.platform/v1
+kind: IdentityPolicy
+metadata:
+  name: cira-stop-time-external-id
+spec:
+  name: cira-stop-time-external-id
+  strategy: passthrough
+  source_fields:
+    - _record_id
+  normalizers: []
+  emit_format: "cira:bdtransportes:stop_time:{resolved_key}"
+  collision_policy: error
+---
+apiVersion: frank.platform/v1
+kind: Pipeline
+metadata:
+  name: cira_transport_stop_times
+spec:
+  name: cira_transport_stop_times
+  description: Complete CIRA regional stop times with stable source identity and existing trip/stop links.
+  source_ids:
+    - cira-bdtransportes-horarios
+  schedule_config:
+    type: manual
+  steps:
+    - name: stop_time_rows
+      kind: field_mapping
+      params:
+        field_mappings:
+          - target_column: external_id
+            mapping_kind: identity
+            identity_policy:
+              name: cira-stop-time-external-id
+            field_order: 0
+          - source_column: schedule
+            target_column: arrival_time
+            mapping_kind: source_expression
+            field_order: 1
+          - source_column: schedule
+            target_column: departure_time
+            mapping_kind: source_expression
+            field_order: 2
+          - source_column: ordem
+            target_column: stop_sequence
+            mapping_kind: source_expression
+            custom_sql: "try_cast(${source} AS integer)"
+            field_order: 3
+          - source_column: trip_id
+            target_column: trip_id
+            mapping_kind: source_expression
+            field_order: 4
+          - source_column: id_paragem
+            target_column: stop_key
+            mapping_kind: source_expression
+            custom_sql: "concat('stop:',cast(try_cast(${source} AS bigint) AS varchar))"
+            field_order: 5
+      depends_on: []
+      emits_to: backing_dataset
+      config:
+        output_layer: silver
+---
+apiVersion: frank.platform/v1
+kind: BackingDataset
+metadata:
+  name: cira_transport_stop_times
+  dependsOn:
+    - entityTypeId: transit_trip
+      ontologyTenantId: cira
+    - entityTypeId: transit_stop
+      ontologyTenantId: cira
+spec:
+  entity_type_id: stop_time
+  entity_type_name: Stop Time
+  ontology_tenant_id: cira
+  pipeline: cira_transport_stop_times
+  sync_mode: manual
+  primary_key_column: external_id
+  title_key_column: external_id
+  property_mappings:
+    - column: external_id
+      property: external_id
+      type: varchar
+      is_primary_key: true
+    - column: arrival_time
+      property: arrival_time
+      type: varchar
+    - column: departure_time
+      property: departure_time
+      type: varchar
+    - column: stop_sequence
+      property: stop_sequence
+      type: integer
+    - column: trip_id
+      property: trip
+      is_relationship: true
+      target_type: transit_trip
+      target_key: service_id
+    - column: stop_key
+      property: stop
+      is_relationship: true
+      target_type: transit_stop
+      target_key: code

package/dist/template-packs/cira/template-pack.yaml

@@ -98,6 +98,16 @@ spec:
         - 580
         - 591
         - 592
+    regional-transport-stop-times:
+      title: Regional transport stop times with declarative identity
+      status: declared_pending
+      manifests:
+        - 11_stop_time.yaml
+      required_sources:
+        - cira-bdtransportes-horarios
+      issues:
+        - 592
+        - 650
     regional-transport-avl:
       title: Regional transport current vehicle positions
       status: live_proven

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aiaiai-pt/frankctl",
-  "version": "0.7.2",
+  "version": "0.8.1",
   "description": "Terminal-first client for the Frank Low-Code Pipeline platform",
   "homepage": "https://franksdocs.nomadriver.co",
   "repository": {

package/skills/frank-cli/SKILL.md

@@ -85,20 +85,22 @@ an explicit subset. Repeat `--set frank_tenant_prefix=<8-hex>` and
 ontology tenant. The CIRA pack treats ontology schema as externally owned and therefore
 contains no `ensure_schema` mutation.
 
-A pipeline is one multi-document YAML file describing three kinds of object under the
+A pipeline is one multi-document YAML file describing four kinds of object under the
 `frank.platform/v1` API version:
 
 | `kind` | Role |
 |--------|------|
 | `Source` | Extract/Load config — connector `pattern_id`, streams, schedule → Iceberg **bronze** |
+| `IdentityPolicy` | Stable-key recipe reconciled and pinned before a consuming transform |
 | `Pipeline` | Transform steps (SQL/Python) turning bronze → **silver/gold** |
 | `BackingDataset` | Maps a silver table's columns to **ontology** entity properties |
 
 Objects reference each other by `metadata.name`, so one file can be self-contained and
-version-controllable. Commit only reviewed, secret-free declarations. Structurally
-declared Source credentials are stored in Vault and written with `sources credentials
-set|rotate --values-file -`; `source_config` and export contain only ordinary values plus
-the opaque tenant-owned `credential_ref`.
+version-controllable. Credential behavior follows explicit `SOURCE_CREDENTIAL_MODE`:
+current base Compose uses `legacy_inline`, where write-only values persist in redacted
+`source_config` and `sources credentials set|rotate` is unavailable; immutable release
+`vault` mode uses those commands and keeps only the opaque `credential_ref` in exports.
+Commit only reviewed, secret-free declarations in either mode.
 
 ### The loop: scaffold → edit → apply
 
@@ -139,8 +141,8 @@ separate target entity readback.
 
 | Flag | What it does |
 |------|--------------|
-| `--dry-run` | Client-side parse + cross-ref resolve. Catches malformed YAML and broken references. No server call. |
-| `--dry-run-server` | Server-side preflight against the catalog + DB without persisting. Catches unknown `pattern_id`s, would-be immutable-diff conflicts, and unresolved refs. |
+| `--dry-run` | Client-side parse + local cross-ref validation. Catches malformed YAML and broken same-file references. No server call. |
+| `--dry-run-server` | Server-side preflight against the catalog + DB without persisting. Catches unknown `pattern_id`s, identity-policy create/noop/patch/new-version state, would-be immutable-diff conflicts, and unresolved refs. |
 | `--dry-run-server --live-ontology` | Adds an opt-in, GET-only check of ontology tenant, type, properties, relationships, keys, and declared carrier types. It never changes ontology schema or entities. |
 
 Run `--dry-run-server` in CI on every change; run `apply --wait` to promote.
@@ -157,6 +159,60 @@ BackingDataset and set `spec.sync_mode: manual`. `--wait` topologically orders s
 dependencies and requires external dependencies in the same ontology tenant to be
 already `synced`. Cycles and self-referential relationships fail before execution.
 
+### Stable identity in declarative pipelines
+
+Declare a tenant-owned recipe once, then reference it by name from an identity field
+mapping. `frankctl` applies in `Source → IdentityPolicy → Pipeline → BackingDataset`
+order and replaces the symbolic reference with the exact returned policy ID/version;
+UUIDs do not belong in tracked YAML.
+
+```yaml
+apiVersion: frank.platform/v1
+kind: IdentityPolicy
+metadata: { name: source-record-id }
+spec:
+  name: source-record-id
+  strategy: passthrough
+  source_fields: [_record_id]
+  normalizers: []
+  emit_format: "my-provider:my-type:{resolved_key}"
+  collision_policy: error
+---
+apiVersion: frank.platform/v1
+kind: Pipeline
+metadata: { name: my-pipeline }
+spec:
+  source_ids: [my-source]
+  steps:
+    - name: project
+      kind: field_mapping
+      params:
+        field_mappings:
+          - target_column: external_id
+            mapping_kind: identity
+            identity_policy: { name: source-record-id }
+            field_order: 0
+```
+
+Same-file references contain only `name`; external policies require exact `{name, tier,
+version}`. Never author raw UUID/version/hash pins: `frankctl` resolves symbolic YAML
+and carries the server-returned recipe hash as an internal optimistic precondition.
+
+Policy reconciliation reports `create`, `noop`, `patch`, or `new-version`. First use freezes
+a recipe globally; later changes version it. Reads compute tenant-live `used_in`, including
+for system policies, without overwriting a shared system cache. Multi-document apply is
+sequential, not atomic: correct a failure and reapply the complete declaration.
+
+Pipeline activation locks Pipeline, affected Transforms in deterministic UUID order with
+refreshed mapping/source snapshots, then old+new policies in UUID/version order. Direct
+Transform PUT/DELETE locks Transform, then sorted policies. This prevents deadlock and
+stale `used_in`. A changed recipe still returns `409 identity_policy_recipe_changed`;
+reapply the complete file. Dry-run observes state but does not reserve it.
+
+Migration `s43_identity_policy_frozen` scans only historical `mapping_kind: identity` pins.
+It recognizes canonical/uppercase/hyphenless/braced/URN UUIDs and normalized positive
+versions without casting malformed JSON. Author symbolically; never mutate ontology schema.
+
 ### Immutable state & when to force
 
 Backing Datasets carry operational state (ontology cursor, sync history). If a change
@@ -179,8 +235,10 @@ to a file for review:
 frankctl pipelines export <pipeline-id> > pipeline.yaml
 ```
 
-The export includes ordinary Source configuration, opaque credential references, and
-config-only stream declarations, but omits credential values and stream runtime state.
+The export includes ordinary Source configuration, opaque credential references,
+config-only stream declarations, complete Pipeline steps, and portable symbolic
+IdentityPolicy references. It omits credential values, stream runtime state, and
+tenant-local policy UUIDs.
 
 ### Minimal manifest shape
 
@@ -429,6 +487,9 @@ you can review, edit, and feed straight into `pipelines apply`.
 - **401 after login**: your token expired and refresh failed — `frankctl auth login` again.
 - **`409 immutable_diff` on apply**: a Backing Dataset change would destroy state; see
   `--allow-recreate` above and decide deliberately.
+- **`409 identity_policy_recipe_changed` on apply**: replay the complete manifest so
+  policy reconciliation returns a fresh pin; do not add runtime pins to YAML or retry
+  only the Pipeline request.
 - **`apply` did nothing**: that's idempotency — the declared state already matches. Use
   `-v` to see the reconcile plan, or `--dry-run-server` to preview.