@alvera-ai/platform-sdk 0.10.0-rc.2 → 0.10.0-rc.21

package/.agent/data_activation_clients.md

@@ -0,0 +1,557 @@
+# Data activation clients
+
+A **data activation client** is the pipeline join-point that
+binds a **tool** (the transport — see `tools.md`), a **data
+source** (the endpoint — see `data_sources.md`), and one or
+more **interoperability contracts** (the row-to-resource
+transforms — see `interoperability_contracts.md`) into a single
+runnable ingestion pipeline.
+
+SDK namespace: `api.dataActivationClients`. The same namespace
+carries both the management surface (CRUD on the binding row,
+§§1–5) and the runtime surface (the verbs that actually move
+data through the configured pipeline, §6). There is no separate
+"data activation runs" resource — runs are verbs on this
+namespace, not a sibling entity.
+
+Data activation clients are **Datalake-DB-resident** — the
+parent datalake must be `status: 'ready'` and all referenced
+resources (tool, data source, contracts) must exist before
+POST.
+
+## 1. Wire shape
+
+```typescript
+import type {
+  DataActivationClientRequestWritable,
+  DataActivationClientResponse,
+} from '@alvera-ai/platform-sdk'
+
+const { data: created } = await api.dataActivationClients.create(
+  tenantSlug,
+  datalakeSlug,
+  {
+    name: 'Inbound CSV pipeline',
+    description: 'Manual CSV upload feeding the patient and appointment contracts',
+    tool_id: manualUploadToolId,
+    data_source_id: inboundDataSourceId,
+    tool_call: { tool_call_type: 'manual_upload' },
+    interoperability_contract_ids: [patientContractId, appointmentContractId],
+    cron_expressions: [],              // omit for on-demand; e.g. ['0 7,19 * * *']
+    loop_over: [],                     // 'services' | 'locations' | 'providers'
+    row_filter: undefined,             // Liquid pre-filter; see §2
+    downstream_connection_ids: [],     // DAC chaining; cycle-checked
+  },
+)
+// created.id, created.slug              — server-derived
+// created.tool_id                       === body.tool_id
+// created.interoperability_contracts    — embedded list of the bound contracts
+// created.is_default                    — true for platform-auto-created defaults
+```
+
+## 2. Rules the type cannot encode
+
+### `tool_call.tool_call_type` must align with the bound tool's `tool_body_type`
+
+The activation client's `tool_call` discriminator selects which
+call shape will be sent to the tool at run time. The valid
+`tool_call_type` is determined by the bound tool's
+`tool_body_type`:
+
+```
+tool_body_type: 'manual_upload'  → tool_call_type: 'manual_upload'
+tool_body_type: 'sftp'           → tool_call_type: 'sftp_fetch' | ...
+tool_body_type: 'rest_api'       → tool_call_type: 'rest_api_request'
+tool_body_type: 'sns'            → tool_call_type: 'sms_request'
+```
+
+(See `tools.md` §5 for the polymorphic call shape per tool body
+type.) A mismatch — e.g. a `manual_upload` tool bound with an
+`sftp_fetch` call — is rejected at create time as a 422 on
+`/tool_call/tool_call_type`.
+
+### `interoperability_contract_ids` is a virtual many-to-many array
+
+The field is **caller-supplied** on `Request[Writable]` as a
+`string[]` of contract UUIDs, but the **response** embeds the
+bound contracts as an `interoperability_contracts` array of
+full contract response objects. On PUT, re-supply the
+`interoperability_contract_ids` array verbatim — omitting it
+disassociates ALL bound contracts.
+
+Contracts and the activation client's tool / data source must
+all live under the same `datalake_slug`; cross-datalake
+references reject with a 404 on the offending id field.
+
+### `interoperability_contract_ids` composition is consumer-decided
+
+A single activation client can bind:
+
+- one or more `custom` / `system` contracts (e.g. patient +
+  appointment for a multi-resource ingestion), OR
+- a single `identity` contract bound to a generic table
+  (the auto-created identity contract from
+  `interoperability_contracts.md` §5).
+
+Mixing `identity` and non-identity contracts on one activation
+client is semantically unusual — fan-out treats each contract
+as a separate transformation lane, and identity contracts pass
+rows through unchanged. The platform doesn't reject the mix,
+but consumers typically split into two activation clients
+(one per output table).
+
+### `row_filter` has INVERTED semantics
+
+A `row_filter` is a Liquid template that decides whether a row
+enters the activation pipeline AFTER the contract-level
+`filter_template` runs. Like `filter_template`, the semantics
+are inverted from most filter DSLs:
+
+- **Empty render → PASS** (row enters the pipeline).
+- **Non-empty render → SKIP** (row is dropped; the rendered
+  string is captured as the skip reason).
+
+Liquid context includes `msg.row` (the inbound row) and
+`msg.client` (the activation client's template vars). If you
+omit `row_filter`, every row passes through.
+
+Anchor on the empty-render-passes rule, not on the Liquid
+keyword — `{% if ... %}...{% endif %}` and
+`{% unless ... %}...{% endunless %}` are equivalent.
+
+## 3. Field ownership
+
+**Server-derived (Response-only).** Universal set from
+`type_naming.md`, plus:
+
+```
+interoperability_contracts   array of bound contract response
+                              objects (mirror of the virtual
+                              *_ids array on the request)
+is_default                   boolean — true for platform-auto-
+                              created default DACs (one per resource
+                              type at datalake migration); cannot
+                              be deleted, can be PUT-updated
+```
+
+**Caller-supplied (round-trip).**
+
+```
+name                            required string
+description                     optional string
+tool_id                         required UUID — bound tool
+data_source_id                  required UUID — bound data source
+tool_call                       required embed — { tool_call_type, ... }
+interoperability_contract_ids   required string[] — virtual many-to-many
+cron_expressions                optional string[] — Crontab syntax (e.g.
+                                  ['0 */6 * * *']); empty array = on-demand
+loop_over                       optional enum[] — 'services' | 'locations' |
+                                  'providers'; fan-out dimensions per run
+row_filter                      optional string — Liquid pre-filter; see §2
+                                  for inverted semantics
+downstream_connection_ids       optional UUID[] — DACs triggered after this
+                                  one completes; cycles rejected at create
+```
+
+**Write-only (Request-only).** None — every caller-supplied
+field round-trips on the response (the virtual array
+round-trips under `interoperability_contracts` on response).
+
+## 4. Error envelopes
+
+Standard JSON:API envelopes per `errors.md`. Common rejections:
+
+| `source.pointer`                          | Cause                                              |
+|-------------------------------------------|----------------------------------------------------|
+| `/tool_id` / `/data_source_id`            | resource doesn't exist (or wrong datalake)         |
+| `/tool_call/tool_call_type`               | mismatch with the tool's `tool_body_type`          |
+| `/interoperability_contract_ids/0`        | contract id doesn't exist or wrong datalake        |
+| `/name`                                   | uniqueness within datalake (POST and PUT — rename can collide) |
+| `/cron_expressions/0`                     | invalid crontab syntax                              |
+| `/cron_expressions`                       | every-minute cron is rejected — **409 Conflict**, not 422; minimum interval enforced platform-side |
+
+## 5. Lifecycle
+
+### Create
+
+Synchronous. The activation client is immediately runnable —
+there's no async "deploy" phase. The next step is invoking one
+of the runtime verbs in §6.
+
+### Read shapes
+
+```
+.list(tenantSlug, datalakeSlug)              Paged: { data, meta }
+.get(tenantSlug, datalakeSlug, slug)         One row, by slug
+.metadata(tenantSlug, datalakeSlug, slug)    Agent-facing metadata
+```
+
+### Update
+
+`PUT` replays the full body — no PATCH. Pointer fields
+(`tool_id`, `data_source_id`, `interoperability_contract_ids`)
+can be re-pointed.
+
+**The `slug` regenerates from `name` on every PUT.** The URL
+path uses the OLD slug for lookup (so the PUT URL is valid),
+but the response body carries the NEW slug derived from the
+updated `name`. Examples:
+
+```
+   pre-update:   name: 'Original',   slug: 'original'
+   PUT body:     name: 'Renamed', ...
+   PUT URL:      .../data-activation-clients/original   ← old slug works for lookup
+   PUT response: name: 'Renamed',    slug: 'renamed'    ← new slug derived
+```
+
+**Consequence**: any downstream resource that references the
+DAC by slug (workflow templates, action bodies referencing
+`{{ ... }}` paths derived from the DAC slug, monitoring
+dashboards, etc.) breaks the moment the DAC is renamed.
+Capture the post-PUT slug from the response, NOT the URL the
+PUT was issued against. If you rename a DAC mid-deployment,
+update every binding that references it.
+
+A PUT that keeps `name` unchanged regenerates to the same
+slug (slugification is deterministic), so cosmetic-only
+updates don't break bindings.
+
+### Delete
+
+`DELETE` removes the activation client. In-flight runs
+decouple their lifecycle from the activation client row;
+deleting one mid-run is generally rejected with a 422.
+
+## 6. Runtime
+
+Three run shapes exist, selected by the bound tool's
+`tool_body_type`:
+
+```
+   inline JSON     →   .ingest(slug, { data })         (small payloads)
+   uploaded file   →   .ingestFile(slug, { key })       (bulk; async)
+   pull-based      →   .runManually(slug)               (e.g. SFTP, REST poll)
+```
+
+All three produce **log rows** observable through
+`api.dataActivationClients.logs.*` (§6.4), and the canonical way
+to **verify** that ingestion worked is the two-step
+dataset-search pattern in §6.5.
+
+### 6.1 Inline JSON ingestion — `.ingest`
+
+For activation clients bound to a `manual_upload` tool, the
+smallest run shape is a single JSON row submitted inline:
+
+```typescript
+const { data } = await api.dataActivationClients.ingest(
+  tenantSlug, datalakeSlug, activationClientSlug,
+  { data: rowAsJson },                  // a single row, shaped per the CSV/JSON the contracts expect
+)
+// data.batch_id   — UUID; the row's batch identifier (use to find log rows)
+// data.key        — storage key for the row's archived ndjson
+// data.jobs_count — number of downstream jobs the platform enqueued
+```
+
+The response returns **immediately after enqueueing**, not
+after ingestion completes. To know when the row has actually
+been written, poll the logs (§6.4) or use the verification
+pattern (§6.5).
+
+The `batch_id` is the most important field on the response —
+every downstream log row and dataset row carries the same
+`batch_id`, so it's the join key for run-specific verification.
+
+### 6.2 Bulk file ingestion — `.ingestFile`
+
+For larger payloads, upload the file via a presigned URL first,
+then trigger ingestion by the returned key:
+
+```typescript
+// 1. Mint a presigned upload URL (see datalakes.md).
+const { data: link } = await api.datalakes.createUploadLink(
+  tenantSlug, datalakeSlug,
+  { content_type: 'text/csv', filename: 'inbound.csv' },
+)
+// link.url — presigned PUT URL (signature + expiry baked in)
+// link.key — storage key the platform will reference
+
+// 2. PUT the file body to the presigned URL (raw HTTP, NOT via the SDK).
+await fetch(link.url, {
+  method: 'PUT',
+  headers: { 'Content-Type': 'text/csv' },
+  body: csvBytes,
+})
+
+// 3. Enqueue the ingest-file job by key.
+const { data: job } = await api.dataActivationClients.ingestFile(
+  tenantSlug, datalakeSlug, activationClientSlug,
+  { key: link.key },
+)
+// job.key    — echoes link.key
+// job.job_id — numeric id of the enqueued background job
+```
+
+This path is **fully async** — the response acknowledges that
+the worker has been enqueued, but no row has been written yet.
+The worker reads the file, allocates a fresh `batch_id`, and
+fans out per-row jobs. Discover the assigned `batch_id` by
+diffing the activation client's log rows before vs after the
+call (§6.4).
+
+### 6.3 Manual run — `.runManually`
+
+For activation clients bound to pull-based tools (`sftp`,
+`rest_api`, etc.), there is no caller-supplied payload — the
+tool itself fetches data from the external system. The SDK
+verb:
+
+```typescript
+const { data } = await api.dataActivationClients.runManually(
+  tenantSlug, datalakeSlug, activationClientSlug,
+)
+```
+
+Returns an acknowledgement; the actual fetch runs in a
+background worker. As with `.ingestFile`, discover the assigned
+`batch_id` by polling the logs.
+
+#### One-shot `tool_call` override
+
+`.runManually` accepts an optional `tool_call` body that
+overrides the DAC's persisted `tool_call` FOR THIS RUN ONLY.
+Omitting the body (or omitting just the `tool_call` field) runs
+with the persisted `tool_call`. Supplying a `tool_call` map
+does NOT mutate the DAC row — it's a transient per-invocation
+override, useful for one-off backfills with a different path,
+payload, or fetch parameters.
+
+```typescript
+await api.dataActivationClients.runManually(
+  tenantSlug, datalakeSlug, activationClientSlug,
+  { tool_call: { tool_call_type: 'rest_api_request', path: '/v1/backfill', ... } },
+)
+```
+
+### 6.4 Logs subresource — `.logs.list` / `.logs.get`
+
+Every ingestion run produces one or more **log rows** — one per
+top-level interoperability contract that fired, plus one per
+cascade target (a nested resource produced by a template's
+upserts). The log row carries:
+
+```
+batch_id            UUID — the run's batch identifier
+dataset_table       string — the target dataset/table name
+rows_ingested       number — count of top-level rows the contract emitted
+                              (cascade rows carry 0)
+dataset_updated     number — count of dataset rows whose batch_id this run wrote
+                              (load-bearing: see Gotcha 7)
+output_files        string[] — merged ndjson archive paths
+                                (empty until the post-batch merge worker commits)
+```
+
+```typescript
+// Paginated list of log rows for an activation client.
+const { data } = await api.dataActivationClients.logs.list(
+  tenantSlug, datalakeSlug, activationClientSlug,
+)
+// data.data — array of log rows; filter by batch_id to find this run
+
+// Fetch one log row by id.
+const { data: entry } = await api.dataActivationClients.logs.get(
+  tenantSlug, datalakeSlug, activationClientSlug, logId,
+)
+```
+
+#### Top-level rows vs cascade rows
+
+A multi-resource contract (e.g. an inbound row that produces a
+patient AND an appointment AND a related person) fires one
+log row **per dataset_table** the contract touched. The
+top-level contracts carry `rows_ingested > 0`; cascade rows
+(nested upserts inside a template) carry `rows_ingested = 0`.
+Filter on `rows_ingested > 0` when counting "how many top-level
+contracts fired".
+
+#### `output_files` as the completion signal
+
+A log row appears as soon as the per-row job acknowledges the
+write, but `output_files` stays empty until the post-batch
+merge worker has committed the ndjson archive. Use
+`output_files.length > 0` as the "this batch's rows are fully
+durable" signal — not just the presence of the log row.
+
+### 6.5 Verifying ingestion: the two-step dataset search
+
+The canonical way to **prove a row landed in the datalake** is
+to query the dataset directly. This is a two-step pattern:
+
+The dataset endpoints are datalake-slug-scoped: every call
+takes `(tenantSlug, datalakeSlug, dataset, …)` as leading args.
+There is no `datalake_id` query param and no tenant-derived
+default — the datalake is mandatory and explicit, mirroring
+`tools`, `workflows`, `dataActivationClients`, etc.
+
+```typescript
+// (1) POST creates a user search and runs the underlying SQL
+//     synchronously against the regulated schema. The result
+//     set is materialised server-side and identified by the
+//     returned UUID.
+const { data: search } = await api.datasets.createUserSearch(
+  tenantSlug,
+  datalakeSlug,
+  'patient',                                              // dataset name
+  {
+    search_query: `ri.value = '${externalPatientId}'`     // SQL WHERE-clause fragment
+                   + ` AND rp.batch_id = '${batchId}'`,
+  },
+)
+// search.id            — UUID of the materialised result set
+// search.status        — 'completed' on success; 'failed' surfaces error_message
+// search.results_count — ALWAYS null on the POST response (see Gotcha 5)
+
+// (2) GET paginates rows from the materialised result set.
+//     dataAccessMode selects the storage tier:
+//       'regulated'   — raw values verbatim (PHI/PII intact)
+//       'unregulated' — tokenized columns return as
+//                       { redacted_data, token, type } objects;
+//                       plain columns return as their primitive type
+const { data: page } = await api.datasets.search(
+  tenantSlug,
+  datalakeSlug,
+  'patient',
+  { userSearchId: search.id, dataAccessMode: 'unregulated' },
+)
+// page.data — array of rows
+// page.meta — pagination
+```
+
+#### SQL aliases
+
+`createUserSearch` accepts a WHERE-clause fragment against a
+fixed set of table aliases. Each dataset has a canonical base
+alias for its regulated table, plus `ri` for the
+regulated_identifiers join:
+
+```
+patient        → rp  (regulated_patients) + ri
+appointment    → ra  (regulated_appointments) + ri
+customer       → rc  (regulated_customers) + ri
+...            (one alias per regulated dataset)
+```
+
+The `ri.value` predicate filters on external identifiers (the
+values your inbound rows carried); the `<base_alias>.batch_id`
+predicate scopes the search to one run.
+
+#### Tokenized vs plain column shape
+
+In `dataAccessMode: 'unregulated'`, tokenized columns surface
+as objects, NOT primitives:
+
+```typescript
+{
+  redacted_data: '*****',    // server-redacted preview
+  token:         'tk_...',   // stable opaque token
+  type:          'string'    // original column's logical type — one of:
+               | 'text'      //   'string' / 'text': free-form text content
+               | 'email'     //   'email': RFC 5322 email
+               | 'phone'     //   'phone': E.164 / display phone number
+               | 'date'      //   'date': ISO 8601 date
+               | 'image',    //   'image': image file reference
+}
+```
+
+Plain columns (non-tokenized) surface as the same primitive
+the regulated schema stores. A consumer code path that
+unconditionally treats every field as a string will crash on
+tokenized columns — type-check `typeof row.field === 'object'`
+first.
+
+## 7. Gotchas
+
+1. **`tool_call_type` must match the tool's `tool_body_type`.**
+   An activation client is a contract between the tool and the
+   pipeline; sending a call shape the tool doesn't accept is
+   rejected at create. See `tools.md` §5 for the per-body
+   call-shape table.
+
+2. **`interoperability_contract_ids` on PUT replaces the bound
+   set entirely.** Omitting the field disassociates ALL
+   contracts; passing a subset removes the others. Always
+   re-supply the full id array on update, even if you're only
+   changing `description`.
+
+3. **All referenced resources live under one datalake.** The
+   activation client, its tool, its data source, and every
+   bound contract must share the same parent datalake.
+   Cross-datalake references return 404 on the offending id.
+
+4. **Custom-dataset identity contracts get their own
+   activation client.** When you ingest into a generic table,
+   bind the table's auto-created identity contract (see
+   `interoperability_contracts.md` §5) to a dedicated activation
+   client. Don't mix it with non-identity contracts on the
+   same client unless you specifically want the fan-out
+   semantics.
+
+5. **`createUserSearch.results_count` is always null.** The
+   POST response confirms the search materialised but does
+   NOT report row counts. The actual rows live in the
+   server-side result set and only become visible via the
+   paginated `.search` GET. A common mistake is reading
+   `results_count` and concluding "no rows matched" when the
+   POST simply doesn't populate that field.
+
+6. **`.ingest` returns BEFORE the row is durable.** The
+   response carries `batch_id` + `jobs_count`, signalling that
+   downstream jobs have been enqueued — not that they've
+   completed. Poll logs (or use the search verification
+   pattern in §6.5) before treating the row as ingested.
+
+7. **`dataset_updated > 0` is the load-bearing freshness
+   gate.** `rows_ingested` flips as soon as the per-row job
+   completes, but the dataset row's `batch_id` column is
+   updated by a downstream trigger that can lag. Filtering a
+   `.createUserSearch` by `<base_alias>.batch_id = '<this>'`
+   BEFORE that trigger fires returns an empty result set,
+   and there's no way to refresh the materialised result —
+   you'd have to POST a new search. Poll for
+   `dataset_updated > 0` before issuing the search.
+
+8. **`output_files` empty ≠ batch failed.** A log row with
+   `rows_ingested > 0` but `output_files: []` is mid-merge,
+   not broken. The merge worker commits the ndjson archives
+   asynchronously; wait for `output_files.length > 0` if your
+   downstream code reads the archives directly.
+
+9. **Bulk uploads PUT to a presigned URL, NOT through the
+   SDK.** Step 2 of the bulk flow uses raw `fetch()` against
+   the presigned URL the platform returns; the SDK is NOT
+   involved in the upload itself. The SDK round-trip only
+   covers minting the URL (`createUploadLink`) and
+   triggering ingestion by key (`ingestFile`).
+
+10. **`runManually` and `.ingest` produce structurally
+    identical log rows.** Once a `batch_id` is allocated,
+    every run flavor merges into the same log/dataset row
+    shape. Consumers that diff or aggregate runs don't need to
+    branch on which verb produced the batch.
+
+11. **`row_filter` is empty-passes, non-empty-skips.** Same
+    inverted semantics as InteroperabilityContract.filter_template.
+    A correctly-shaped filter renders to a non-empty string when
+    the row should be SKIPPED. Always anchor on the
+    empty-render-passes rule, not on the Liquid keyword choice.
+
+12. **`downstream_connection_ids` is cycle-checked at create.**
+    The platform rejects DAC chains that would form a cycle. Make
+    sure both endpoints of the chain exist before adding the
+    pointer; cross-datalake chaining is also rejected.
+
+13. **`is_default: true` DACs cannot be deleted.** Platform-
+    auto-created default DACs (one per resource type at datalake
+    migration time) are protected. Filter them out
+    (`!d.is_default`) when listing for management purposes; you
+    can still PUT-update them (e.g. to bind additional contracts).