@alvera-ai/platform-sdk 0.10.0-rc.9 → 0.12.0

package/.agent/workflows.md

@@ -0,0 +1,617 @@
+# Workflows
+
+A **workflow** is the event-driven decision pipeline that fires on
+top of ingested dataset rows. Every workflow has the same shape:
+
+```
+   dataset row event
+         │
+         ▼
+   filter        — does this row qualify? (Liquid expression → boolean)
+         │
+         ▼
+   enrichment    — optional AI step that derives extra context (see §3)
+         │
+         ▼
+   decision      — which action key fires? (Liquid array of strings, OR
+                   an AI agent's output)
+         │
+         ▼
+   action(s)     — scheduled tool calls keyed by decision_key
+                   (SMS, email, voice, export, ...)
+```
+
+SDK namespace: `api.workflows`. Sub-namespaces for run-time logs:
+
+```
+api.workflows.batchLogs       — one log row per workflow run (batch)
+api.workflows.workflowLogs    — one log row per dataset row processed
+                                (workflow execution logs)
+```
+
+Workflows are **Datalake-DB-resident** — the parent datalake must
+be `status: 'ready'` and every referenced resource (tools,
+connected apps, AI agents) must exist before you `create()`.
+
+## 1. Wire shape
+
+```typescript
+import type {
+  WorkflowRequestWritable,
+  WorkflowResponse,
+} from '@alvera-ai/platform-sdk'
+
+const { data: created } = await api.workflows.create(
+  tenantSlug,
+  datalakeSlug,
+  {
+    name: 'Appointment Review SMS',
+    description: 'Send a review-request SMS after a fulfilled appointment',
+    dataset_type: 'appointment',            // the dataset this workflow listens on
+    status: 'live',                         // 'live' | 'draft'
+    filter_config: {
+      type: 'custom',
+      body: `{% if appointment.source_uri == "12345.example.com" %}true{% endif %}`,
+      output_schema: '{"type":"boolean"}',
+    },
+    decision_config: {
+      type: 'custom',
+      body: `["send_appointment_review_sms"]`,
+      output_schema: '{"type":"array","items":{"type":"string"}}',
+    },
+    context_datasets: [
+      {
+        dataset_type: 'message',
+        where_clause: `rm.patient_id = '{{ patient_id }}' AND rm.sent_at > NOW() - INTERVAL '6 months'`,
+        limit: 1,
+        position: 0,
+      },
+    ],
+    actions: [
+      {
+        action_type: 'sms',
+        tool_id: smsToolId,
+        decision_key: 'send_appointment_review_sms',
+        position: 0,
+        trigger_template: 'now',                // 'now' = next available slot
+        idempotency_template:
+          '{{ patient_id }}-{{ appointment.unregulated_appointment_id }}-{{ decision_key }}',
+        connected_app_id: connectedAppId,
+        connected_app_route: '/forms/review',
+        connected_app_metadata_template:
+          '{"appointment_id":"{{ appointment.unregulated_appointment_id }}"}',
+        tool_call: {
+          tool_call_type: 'sms_request',
+          to:   { type: 'custom', body: '{{ patient_phone }}' },
+          body: { type: 'custom', body: 'Hi {{ patient_first_name }}, please share your feedback at {{ connected_app_form_url }}' },
+          sms_type: 'transactional',
+        },
+      },
+    ],
+  },
+)
+```
+
+`actions`, `context_datasets`, and `workflow_ai_agents` are all
+inline arrays — one `create()` builds the entire workflow body, agents
+included (see §3).
+
+## 2. Rules the type cannot encode
+
+### `filter_config` / `decision_config` are Liquid expressions with typed output
+
+Both configs follow the same shape: `{ type, body, output_schema }`.
+
+- `filter_config.body` renders to a boolean — the literal string
+  `"true"` (non-empty) means PASS; an empty render means SKIP.
+  (Same inverted-semantics rule as `interoperability_contracts.md`
+  §2 `filter_template`.)
+- `decision_config.body` renders to a JSON array of decision keys.
+  Each rendered key fires the matching `action` (by
+  `action.decision_key`).
+
+`output_schema` is a JSON-schema string that the platform uses
+to validate the rendered output before invoking the next stage.
+
+### `update()` replaces inline arrays whole
+
+`actions` and `context_datasets` are replaced whole by `update()`.
+`update()` replaces the whole array — resupply the full set; any
+item you omit is removed. Always re-supply the full array on every
+update, even when you're only changing `description`.
+
+Factor the workflow body construction into a helper that
+accepts the changing fields and returns the full body — same
+pattern used for datalake updates (see `datalakes.md` §5).
+
+### `idempotency_template` prevents duplicate action firings
+
+An action's `idempotency_template` renders to a string per
+dataset row; the platform deduplicates so the same idempotency
+key never fires twice. Typical recipe:
+
+```
+'{{ <subject_id> }}-{{ <event_id> }}-{{ decision_key }}'
+```
+
+The placeholder names depend on the workflow's `dataset_type`
+(patient_id + appointment.unregulated_appointment_id for an
+appointment-driven workflow, customer_id + invoice.id for an
+invoice-driven workflow, etc.).
+
+### `trigger_template` controls scheduling
+
+A Liquid template rendering to a datetime or one of these
+literals:
+
+```
+'now'      — schedule at the next available slot at-or-after current time
+''         — same as 'now' (empty string also routes to the now-slot)
+ISO 8601   — schedule at the named instant
+```
+
+The action is queued through the platform's scheduling layer
+(action windows, throttling), so even `'now'` may resolve to a
+slot some minutes in the future under load.
+
+### Action-window + queue-throttle render `'now'` non-literal
+
+Avoid templates like `{{ "" | now | date: "%Y-%m-%d..." }}` —
+they render a literal current timestamp, which then collides
+with the action-window clamp + queue throttle and pushes the
+slot forward unpredictably. Use the literal `'now'` sentinel
+when you want "fire as soon as possible".
+
+## 3. Agent-driven workflows: AI enrichment + AI-emitted decisions
+
+When `decision_config.type` references an AI agent (instead of
+`'custom'` with a literal Liquid body), the workflow gains an
+**enrichment** step between filter and decision:
+
+```
+   filter
+     │
+     ▼
+   enrichment    — each ai_agent runs, producing typed output
+                   that the decision template can reference
+     │
+     ▼
+   decision      — AI agent emits the decision_key array
+                   (or a custom decision template references
+                    agent outputs via Liquid)
+     │
+     ▼
+   action(s)
+```
+
+Agents are **nested directly on the workflow body** — the create and
+update body accept a `workflow_ai_agents` array. There is
+no separate attach/detach endpoint. The agents **fold into the
+workflow's own checksum** — there is no per-join checksum. `position` is
+required, and workflow joins **omit `output_schema`** (the server pins
+it from the agent's `input_schema`). The agent create body (prompt
+config, `input_schema` / `llm_response_schema`, runtime failure codes)
+lives in `ai_agents.md` §3. A workflow `get()` surfaces the nested agents as
+`workflow_ai_agents: [...]`.
+
+`update()` replaces the whole array: the body must
+re-supply every agent you want to keep — agents absent from the array
+are removed, and re-supplied ones are repositioned.
+
+```typescript
+await api.workflows.create(tenantSlug, datalakeSlug, {
+  name: 'Contact-Us Triage',
+  dataset_type: 'generic_table',
+  // …other workflow fields…
+  workflow_ai_agents: [
+    {
+      ai_agent_id: aiAgentId,                      // UUID from api.aiAgents.create
+      position: 0,                                 // ordering when multiple agents fire; REQUIRED
+      context_mapping_config: {
+        type: 'custom',
+        // Liquid renders to a JSON object whose keys match the referenced
+        // agent's input_schema. Values reference the inbound row's fields
+        // via `event_dataset.<field>`. Workflow joins OMIT output_schema
+        // — the server pins it from the agent's input_schema.
+        body: JSON.stringify({
+          msg:           '{{ event_dataset.message }}',
+          submission_id: '{{ event_dataset.submission_id }}',
+        }),
+      },
+    },
+  ],
+})
+```
+
+The agent's parsed output is exposed in the workflow's runtime
+context at `additional_context.<agent_slug>.<field>` (property
+paths match the agent's `llm_response_schema`). Decision and
+action templates reference it via Liquid:
+
+```liquid
+{{ additional_context.contact-us-triage-categorizer.category }}
+```
+
+A workflow with no attached agents observably runs the enrichment
+stage but emits `{ status: 'skipped' }` in the per-step artifact
+(see §6).
+
+### `event_dataset` is the workflow accessor for the inbound row
+
+Workflow templates (filter, context mappings, action bodies)
+reference the inbound dataset row via the `event_dataset`
+variable. Field paths match the columns of the workflow's
+`dataset_type`:
+
+```
+   dataset_type: 'appointment'  → event_dataset.id, event_dataset.start, ...
+   dataset_type: 'generic_table' → event_dataset.<column_name> for each
+                                   column on the bound table
+```
+
+For generic-table-bound workflows, `event_dataset` carries the
+table's columns directly (no separate `row` indirection).
+
+### `skip_mdm_resolution: true` for generic-table workflows
+
+Workflows bound to a `dataset_type: 'generic_table'` typically
+set `skip_mdm_resolution: true` on the workflow body. The flag
+opts out of the platform's entity-resolution step on the inbound
+row, which doesn't apply when the row IS the subject (no canonical
+entity to resolve to). Workflows on entity-anchored datasets
+(`patient`, `customer`, `legal_entity`, etc.) leave the flag at
+its default `false`.
+
+## 4. Field ownership
+
+**Server-derived (Response-only).** Universal set from
+`type_naming.md`, plus:
+
+```
+actions[*].id              UUID per inline action row
+context_datasets[*].id     UUID per inline context-dataset row
+```
+
+**Caller-supplied (round-trip).**
+
+```
+name                         required string
+description                  optional string
+dataset_type                  required string — the dataset this listens on
+status                       required enum — 'live' | 'draft'
+generic_table_id              optional UUID — required iff dataset_type='generic_table'
+skip_mdm_resolution           optional bool — default false; set true for
+                                              generic-table-bound workflows
+filter_config                required embed — { type, body, output_schema }
+decision_config              required embed — same shape
+context_datasets             required array — inline; replace-on-PUT
+actions                      required array — inline; replace-on-PUT
+workflow_ai_agents          optional array — inline; replace-on-PUT (see §3)
+```
+
+**Write-only (Request-only).** None.
+
+## 5. Lifecycle
+
+### Create
+
+Synchronous. The workflow is immediately runnable when
+`status: 'live'`. `status: 'draft'` is a save-without-arming
+flag — drafts can be re-saved via `update()` but won't fire on
+dataset events.
+
+### Read shapes
+
+```
+.list(tenantSlug, datalakeSlug)              Paged: { data, meta }
+.get(tenantSlug, datalakeSlug, idOrSlug)     One row
+.metadata(tenantSlug, datalakeSlug, idOrSlug) Agent-facing metadata
+```
+
+### Run
+
+`.run` evaluates the workflow against a SQL-selected row set:
+
+```typescript
+const { data: run } = await api.workflows.run(
+  tenantSlug, workflowSlug,
+  {
+    sql_where_clause: `ra.id = '${appointmentId}'`,  // run against the regulated tables
+    mode: 'live',                                    // 'live' | 'dry_run'
+    manual_override: false,                          // see below
+  },
+)
+// run.batch_id              — tagged string (e.g. "manual:<uuid>");
+//                              the prefix distinguishes manual runs from
+//                              cron-driven ones
+// run.workflow_run_log_id   — UUID — the batch log row (poll via batchLogs)
+// run.enqueued_count        — number of rows the WHERE clause matched
+```
+
+`sql_where_clause` accepts a fragment against the dataset's
+regulated tables (`ra` for appointments, `rp` for patients,
+etc. — same alias scheme as `data_activation_clients.md` §6.5).
+The id column is ambiguous across joins; always prefix
+(`ra.id`, `rp.id`).
+
+`manual_override: true` **skips BOTH the filter expression AND
+the idempotency check** — the workflow fires regardless of
+filter outcome and regardless of whether the idempotency tuple
+already fired for this row. Useful for forced re-runs in test
+contexts; avoid in production.
+
+### Execute (per-row, per-action)
+
+`.execute` skips filter + decision evaluation entirely and
+fires one specific action against one specific dataset row:
+
+```typescript
+const { data: result } = await api.workflows.execute(
+  tenantSlug, workflowSlug,
+  {
+    dataset_id: unregulatedAppointmentId,           // UNREGULATED id (not regulated)
+    decision_key: 'send_appointment_review_sms',
+    mode: 'live',                                   // 'live' | 'dry_run' (same enum as .run)
+    manual_override: true,                          // load-bearing on re-fires
+  },
+)
+// result.status                       — 'pending' | 'completed' | 'filtered' | 'failed'
+// result.scheduled_count              — number of action jobs scheduled
+// result.workflow_execution_log_id    — UUID of the execution-log row
+```
+
+Note that `dataset_id` is the **unregulated** id, not the
+regulated one. The unregulated tier is what the workflow
+runtime reads its action context from; passing the regulated
+id will fail to resolve. Issue a `dataAccessMode: 'unregulated'`
+search to obtain the right id.
+
+`manual_override: true` on `.execute` sets the unique-key
+constraint to `false` on the per-row job, allowing re-fires
+on rows that have already been processed (otherwise the new
+log row sits in `:pending` forever, deduplicated by the
+sampled-event uniqueness check).
+
+### Update
+
+`update()` replaces the whole resource — resupply the full
+body; there's no partial update. The whole-array replace applies
+to all inline arrays (see §2).
+
+### Delete
+
+`delete()` removes the workflow. In-flight runs decouple their
+lifecycle from the workflow row; deleting one mid-run rejects
+with a 422.
+
+## 6. Run-time logs
+
+### Batch logs — `api.workflows.batchLogs`
+
+One row per workflow run (per `.run` call). Lifecycle:
+
+```
+   :pending → :completed | :failed | :partial
+```
+
+- `:completed` — every row processed terminally
+- `:failed` — every row's processing errored
+- `:partial` — mixed (some rows succeeded, some filtered, some failed)
+
+Methods:
+
+```
+.list(tenantSlug, workflowSlug)                  Paged
+.get(tenantSlug, workflowSlug, runLogId)         One row
+.refresh(tenantSlug, workflowSlug, runLogId)     Force a status recompute
+                                                  (per-row jobs are async; status
+                                                   only flips after explicit refresh)
+.start(tenantSlug, workflowSlug, runLogId)       Begin polling for this batch
+.stop(tenantSlug, workflowSlug, runLogId)        Stop polling
+```
+
+The `.refresh` verb is **load-bearing** for synchronous
+progress checks — `.get` returns the cached status, but the
+underlying per-row jobs may have completed since the last
+refresh. Call `.refresh` in your poll loop.
+
+### Workflow execution logs — `api.workflows.workflowLogs`
+
+One row per dataset row evaluated. Status enum:
+
+```
+:pending  → :executing → :completed | :filtered | :failed
+```
+
+Each log carries:
+
+- `id`, `workflow_id`, `batch_id` — join keys
+- `status` — terminal state
+- `actions_total` — number of actions wired on the workflow
+- `actions_pending` — actions scheduled but not yet completed by
+  their background jobs
+- `actions_completed` — actions whose background job has finished
+- `actions_skipped` — siblings of the matched action that didn't
+  fire (the decision template returned one key; other actions
+  with non-matching `decision_key` get :skipped)
+- `error_message` — coarse failure summary (e.g.
+  `"AI enrichment failed: <agent_name>"`); rich detail lives
+  in `error.json` (§ Per-step artifacts below)
+- `action_execution_logs` — nested rows, one per action that fired
+  (`message_body` virtual field is rendered when read via the
+  `dataAccessMode: 'regulated'` mode)
+
+Methods:
+
+```
+.list(tenantSlug, workflowSlug)                    Paged
+.get(tenantSlug, workflowSlug, logId, { dataAccessMode? })
+.download(tenantSlug, workflowSlug, logId)         Downloads the run's artifact
+```
+
+The `dataAccessMode` query parameter on `.get` controls how
+`message_body` is rendered:
+- `'regulated'` — raw rendered body (with the resolved short
+  path `/t/<token>`)
+- `'unregulated'` — tokenised display body
+
+### Per-step artifacts (event, filter, enrichment, error)
+
+For each workflow execution, the platform writes JSON artifacts
+to the datalake's regulated cloud-storage bucket. The keys
+follow a fixed convention:
+
+```
+workflows/<workflow_id>/executions/<execution_log_id>/<step>.json
+```
+
+Where `<step>` is one of: `event` | `filter` | `enrichment` |
+`error`.
+
+Fetch them via the datalake's `createDownloadLink`:
+
+```typescript
+const { data: link } = await api.datalakes.createDownloadLink(
+  tenantSlug, datalakeSlug,
+  { bucket: regulatedBucket, key: 'workflows/.../filter.json' },
+)
+const artifact = await (await fetch(link.url)).json()
+```
+
+Artifact bodies:
+
+```
+event.json       { event_dataset: { ... full inbound row snapshot ... } }
+filter.json      { filter_expression, filter_result: boolean }
+enrichment.json  Success-with-agents:
+                   {
+                     status: 'completed',
+                     <agent_slug_1>: { status: 'completed',
+                                       output: { ...matches llm_response_schema... } },
+                     <agent_slug_2>: { ... },
+                   }
+                 Skipped (no ai_agents on the workflow):
+                   { status: 'skipped' }
+error.json       Only present on :failed runs:
+                   {
+                     error_code: string,
+                     stage: string,                   // 'enrichment' | 'decision' | 'action'
+                     ai_agent_slug?: string,          // present for enrichment failures
+                     error_message: string,           // coarse summary (mirrors WEL.error_message)
+                     detail: string,                  // rich provider context (URL, HTTP status, ...)
+                                                      // ONLY in error.json, NOT mirrored to the WEL row
+                   }
+```
+
+The PASS / REJECT branches of the filter both emit
+`filter.json` — the body just differs in `filter_result`.
+
+### `error.json` error codes
+
+When `stage: 'enrichment'`, the agent-related codes are
+documented in `ai_agents.md` §7. The full set of `error_code`
+values across stages:
+
+```
+tool_execution_failed     enrichment    LLM tool's HTTP call failed
+json_decode_failed        enrichment    provider returned non-JSON
+context_mapping_failed    enrichment    mapping output didn't match agent's input_schema
+ai_agent_disabled         enrichment    agent's enabled: false
+ai_agent_not_found        enrichment    agent was deleted but workflow still references it
+decision_template_failed  decision      Liquid render or schema validation failed
+action_template_failed    action        action body Liquid render failed
+```
+
+The `detail` field carries rich context (URL, HTTP status,
+transport error) that lives ONLY in `error.json`. The
+workflow execution log row's `error_message` is intentionally
+coarse so the database column doesn't grow with arbitrary
+provider responses. Always read `error.json` for diagnostics.
+
+## 7. Connected apps as action sinks
+
+Workflows that send messages (SMS, email) typically link to a
+**connected app** — a registered consumer of the action's
+rendered URL. See `api.connectedApps`:
+
+```typescript
+await api.connectedApps.create(tenantSlug, datalakeSlug, {
+  name: 'Review form',
+  description: '...',
+  mode: 'self_hosted',                               // 'self_hosted' | 'managed'
+  urls: [{ url: 'https://app.example.com', is_primary: true, label: 'production' }],
+})
+```
+
+Action wiring on the workflow:
+- `connected_app_id` — UUID of the bound connected app
+- `connected_app_route` — path on the app (e.g. `/forms/review`)
+- `connected_app_metadata_template` — Liquid template rendering
+  a JSON object that the recipient can introspect after
+  resolving the token
+
+At fire time, the action's message body's `{{ connected_app_form_url }}`
+expands to `https://<app-primary>/t/<short_path>`. The recipient
+hits that URL; the connected app calls
+`api.connectedApps.resolvePage` with the `short_path` to fetch
+the metadata, then `api.connectedApps.updateMessageTracking` to
+record `opened_at` / `form_submitted_at`.
+
+## 8. Gotchas
+
+1. **`filter_config` follows the same inverted-render rule as
+   interoperability contracts: non-empty PASSES, empty SKIPS.**
+   Same wording, same trap. See
+   `interoperability_contracts.md` §2 for the worked example.
+
+2. **`update()` silently drops omitted arrays.** Workflow
+   updates must re-supply `actions` and `context_datasets`
+   verbatim — otherwise the existing rows are deleted. Factor the
+   body construction so create and update share one builder.
+   (Nested `workflow_ai_agents` follow the same whole-array-replace
+   rule — re-supply the full set on every `update()`; see §3.)
+
+3. **`sql_where_clause` ids require an alias prefix.** `id` is
+   ambiguous across the multi-JOIN base query; use
+   `<base_alias>.id` (`ra.id`, `rp.id`, `rc.id`, etc.).
+
+4. **`.execute` takes the UNREGULATED dataset id, `.run` takes
+   the REGULATED id.** This asymmetry traps consumers who
+   reuse the id they got from a regulated-mode search. Issue
+   a fresh search with `dataAccessMode: 'unregulated'` before
+   calling `.execute`.
+
+5. **`manual_override: true` on `.execute` is required for
+   re-fires.** Without it, the per-row job dedupes against the
+   sampled-event uniqueness check and the new log row sits in
+   `:pending` indefinitely. The override flips the job's
+   unique-constraint behavior so the re-fire is allowed.
+
+6. **`batchLogs.get` returns cached status; call `.refresh`.**
+   Per-row jobs are async; `.get` does not recompute. Poll
+   `.refresh(runLogId)` in your wait loop.
+
+7. **`trigger_template: 'now'` (literal) is different from
+   `'{{ "" | now | date: ... }}'`.** The literal sentinel
+   routes to the next available scheduling slot directly;
+   the rendered timestamp gets clamped against the action
+   window + queue throttle and pushed forward unpredictably.
+
+8. **Enrichment runs for every workflow, even without agents.**
+   For standard (non-agent) workflows, `enrichment.json` lands
+   with `{ status: 'skipped' }`. Don't treat the artifact's
+   absence as the "no AI" signal — read its body.
+
+9. **Use `| to_json`, NEVER `| json`, in Liquid bodies.** When
+   serializing nested values into a JSON payload (most commonly
+   in an agent's `context_mapping_config`, but also any
+   TemplateConfig body that must produce valid JSON), the
+   correct filter is `| to_json`. `| json` is NOT in the
+   platform's allowlist of custom filters (see `ai_sandbox.md`
+   §2 for the closed eleven-filter set). Using it produces
+   structurally-invalid JSON output silently — the downstream
+   JSON parse step then rejects the rendered string with a
+   generic decode error far from the Liquid template's location,
+   making it expensive to diagnose. Always write
+   `{{ event_dataset | to_json }}`, not
+   `{{ event_dataset | json }}`.