@alvera-ai/platform-sdk 0.10.0-rc.8 → 0.11.0

package/.agent/interoperability_contracts.md

@@ -0,0 +1,417 @@
+# Interoperability contracts
+
+An **interoperability contract** is the transformation step between
+a raw inbound row (CSV, JSON, etc.) and a typed datalake resource.
+Each contract carries:
+
+- a `resource_type` (the target shape — `patient`, `customer`,
+  `payment_account`, `legal_entity`, `generic_table`, etc.)
+- a `template_config` (the Liquid template that renders the inbound
+  row into the target shape)
+- an `mdm_input_config` (the Liquid template that renders the
+  matching subject identity for entity resolution — see `mdm.md`
+  for the full MDM surface, including the `api.mdm.verify` verb
+  invoked at workflow action time)
+- an optional `filter_template` (a Liquid pre-filter that decides
+  whether a row enters the pipeline at all)
+
+SDK namespace: `api.interoperabilityContracts`.
+
+The contract is **Datalake-DB-resident** — `create()` requires the
+parent datalake at `status: 'ready'` (see `datalakes.md` §5).
+
+## 1. Wire shape
+
+```typescript
+import type {
+  InteroperabilityContractRequestWritable,
+  InteroperabilityContractResponse,
+} from '@alvera-ai/platform-sdk'
+
+const { data: created } = await api.interoperabilityContracts.create(
+  tenantSlug,
+  datalakeSlug,
+  {
+    name: 'Inbound CRM → Customer',
+    description: 'Vendor CRM rows → AccountsReceivable Customer',
+    resource_type: 'customer',
+    filter_template: undefined,                  // optional
+    template_config: { type: 'custom', body: liquidBody },
+    mdm_input_config: { type: 'custom', body: mdmBody },
+    generic_table_id: null,                      // see §2
+  },
+)
+// created.id, created.slug    — server-derived
+// created.type                — mirrors template_config.type
+// created.resource_type       === body.resource_type
+```
+
+Both polymorphic embeds use the SAME shared `TemplateConfig`
+shape with discriminator `type`:
+
+```
+template_config.type      'system' | 'custom' | 'identity' | 'null'
+mdm_input_config.type     'system' | 'custom' | 'identity' | 'null'
+```
+
+Default for both is `'null'`. The wire accepts all four values
+on each surface; semantic narrowing is a runtime concern, not
+a schema one:
+
+- `custom` — caller-supplied `body` string (Liquid template).
+  Pair with `body`; leave `path` null.
+- `system` — references a platform-shipped template by
+  filesystem path. Pair with `path` (required); leave `body`
+  null. See `templates.md`.
+- `identity` — pass-through. The inbound row is forwarded
+  unchanged. Used by the auto-created identity contracts for
+  generic tables. Leave both `body` and `path` null.
+- `null` — disabled / "row IS the subject" on the mdm surface.
+  Leave both `body` and `path` null.
+
+### TemplateConfig embed fields
+
+Both `template_config` and `mdm_input_config` are instances of
+the shared `TemplateConfig` embed:
+
+```
+type            required enum   — 'system' | 'custom' | 'identity' | 'null' (default 'null')
+body            optional string — Liquid template body; REQUIRED iff type === 'custom'
+path            optional string — filesystem path; REQUIRED iff type === 'system'
+output_schema   server-injected — JSON Schema { type: 'object' } for this surface; not caller-set
+```
+
+Server validation enforces the body/path pairing — supplying
+`body` with `type: 'system'`, or `path` with `type: 'custom'`,
+returns 422.
+
+## 2. Rules the type cannot encode
+
+### `generic_table_id` is conditionally required
+
+When `resource_type === 'generic_table'`, the body MUST include a
+non-null `generic_table_id` identifying the target table. For
+every other `resource_type`, the field must be `null`. The
+TypeScript type permits either value; the validator and server
+enforce the conditional rule.
+
+### `resource_type` is industry-scoped
+
+The valid set of `resource_type` values is determined by the
+parent datalake's `data_domain`. Generic concepts like
+`'generic_table'` are accepted everywhere; domain-anchored
+concepts (`'patient'`, `'customer'`, `'payment_account'`,
+`'legal_entity'`, ...) only resolve within their owning industry.
+Don't hardcode a fixed enum across industries — discover the
+valid set at runtime via the platform's domain catalog (see
+`datalakes.md` §5 `.systemDatasets`).
+
+### `filter_template` has INVERTED semantics
+
+This is the single highest-value gotcha on this surface. A
+`filter_template` is a Liquid template that decides whether a
+row enters the pipeline:
+
+- **Empty render → PASS** (row enters the pipeline).
+- **Non-empty render → SKIP** (row is dropped).
+
+This is the opposite of most filter DSLs ("truthy = keep"). A
+correctly-shaped filter looks like:
+
+```liquid
+{% if msg.row.should_drop == 'Y' %}drop{% endif %}
+```
+
+Renders `"drop"` when the condition fires (SKIP); renders empty
+otherwise (PASS). Inverting `{% unless %}...{% endunless %}` is
+equivalent and stylistically common — but always anchor on the
+empty-render-passes rule, not on the Liquid keyword.
+
+If you omit `filter_template`, every row passes through.
+
+### `template_config` body is Liquid, with row data on `msg`
+
+Templates receive the inbound row on the `msg` variable. The
+exact accessor depends on the source shape — CSV-shaped sources
+typically use `msg.<column>` (with `msg.row.<column>` available
+for nested cases); JSON-shaped sources mirror the inbound JSON
+under `msg.<field>`. The render output is JSON — the platform
+parses the rendered string as JSON and treats it as the target
+resource body.
+
+A render that produces invalid JSON fails at sandbox-run with
+`stage: 'failed'` (see §5). Always sandbox-run a contract before
+binding it to a live data activation client.
+
+## 3. Field ownership
+
+**Server-derived (Response-only).** Universal set from
+`type_naming.md`, plus:
+
+```
+type           string — mirrors template_config.type at create time
+```
+
+**Caller-supplied (round-trip).**
+
+```
+name              required string — agent-facing label
+description       optional string — narrative
+resource_type     required string — target shape (industry-scoped)
+filter_template   optional string — Liquid pre-filter (inverted semantics)
+template_config   required embed  — { type, body?, path? }
+mdm_input_config  optional embed  — { type, body?, path? }; omit equals { type: 'null' }
+generic_table_id  required UUID|null — required iff resource_type='generic_table'
+```
+
+**Write-only (Request-only).** None — every caller-supplied field
+round-trips on the response.
+
+## 4. Error envelopes
+
+Standard JSON:API envelopes per `errors.md`. Common rejections:
+
+| `source.pointer`             | Cause                                            |
+|------------------------------|--------------------------------------------------|
+| `/resource_type`             | value not valid for the parent datalake's domain |
+| `/template_config/type`      | enum mismatch                                    |
+| `/template_config/body`      | empty when `type === 'custom'`                   |
+| `/mdm_input_config/type`     | enum mismatch                                    |
+| `/generic_table_id`          | missing when resource_type='generic_table',      |
+|                              | or non-null when not                             |
+| `/name`                      | uniqueness within datalake                       |
+| `errors.base`                | "System-created contracts cannot be edited" (PUT) or "...deleted" (DELETE) — see §5 |
+
+## 5. Lifecycle
+
+### Create
+
+Synchronous. The create response is immediately usable — no async
+deployment phase. Templates are compiled lazily at sandbox-run
+time and at activation-client fan-out time, so a `body`
+containing invalid Liquid syntax will surface at first use, not
+at create. Always sandbox-run before binding to a live data
+activation client.
+
+### Sandbox run — `.run(slug, msg)`
+
+The contract's templates can be exercised against a sample row
+WITHOUT actually ingesting:
+
+```typescript
+const { data: result } = await api.interoperabilityContracts.run(
+  tenantSlug, datalakeSlug, contractSlug,
+  { /* sample row, same shape as inbound CSV/JSON */ },
+)
+// result.stage          === 'completed' | 'failed'
+// result.filter_result  === 'pass' | 'skip'
+// result.transformed    object | null — the rendered resource body
+```
+
+Use this verb as a CI / pre-bind verification step: produce a
+representative sample row, run it through `.run`, and assert
+the `transformed` output shape matches the consumer's
+expectations. If `filter_result === 'skip'`, the row was filtered
+out before rendering — useful for verifying the filter semantics
+on a known-skip example.
+
+### Generic-table contracts: auto-created identity contracts
+
+When you create a generic table (see `generic_tables.md`), the
+platform automatically creates an **identity contract** bound
+to that table:
+
+```
+new generic table (T) → auto-created contract with
+                         resource_type: 'generic_table'
+                         generic_table_id: T.id
+                         template_config.type: 'identity'
+```
+
+Discover this contract via `.list()` + filter on
+`generic_table_id`:
+
+```typescript
+const { data: all } = await api.interoperabilityContracts.list(
+  tenantSlug, datalakeSlug,
+)
+const identityContract = all.data?.find(
+  (c) => c.generic_table_id === tableId && c.type === 'identity',
+)
+```
+
+Multiple contracts MAY bind to the same generic table — one
+identity (auto) plus any number of caller-created `custom`
+contracts. Distinguish by `type` when iterating. The platform
+enforces at most ONE identity contract per generic table;
+attempts to create a second `type: 'identity'` contract for
+the same table are rejected.
+
+### Read shapes
+
+```
+.list(tenantSlug, datalakeSlug)              Paged: { data, meta }
+.get(tenantSlug, datalakeSlug, slug)         One row, by slug
+.metadata(tenantSlug, datalakeSlug, slug)    Markdown string — one contract
+.run(tenantSlug, datalakeSlug, slug, msg)    Sandbox runner (above)
+```
+
+### Delete
+
+`delete()` removes the contract. Contracts referenced by active
+data activation clients reject with a 422 naming the dependency.
+
+**System-created contracts cannot be edited or deleted.** The
+contract record carries a `system_created` flag (server-derived).
+When `system_created === true`, both `update()` and `delete()`
+reject with 422 on `/base` and a literal message:
+
+```
+   PUT     →  "System-created contracts cannot be edited"
+   DELETE  →  "System-created contracts cannot be deleted"
+```
+
+System-created contracts include the auto-generated identity
+contracts the platform provisions when a generic table is
+created (see §5 "Generic-table contracts: auto-created identity
+contracts") and any industry-built templates the platform ships
+with the datalake's data_domain. User-created contracts of any
+`type` (including caller-created `type: 'identity'` rows bound
+to a generic table) ARE editable and deletable — the gate is
+`system_created`, not `type`.
+
+The flag IS exposed on the `InteroperabilityContractResponse`,
+so consumer agents can pre-filter before attempting a mutation:
+
+```typescript
+const { data: list } = await api.interoperabilityContracts.list(
+  tenantSlug, datalakeSlug,
+)
+const editable = list.data.filter((c) => !c.system_created)
+```
+
+## 6. Nesting AI agents on the contract body
+
+A contract can run AI-agent **enrichment** during ingestion. Agents
+are **nested directly on the contract body** — the create/update body
+accepts an `interoperability_contract_ai_agents` array.
+There is no separate attach/detach endpoint, and **no per-join
+checksum**: the nested agents fold into the contract's own checksum.
+
+```typescript
+// Nest on create. `position` is REQUIRED.
+const { data: contract } = await api.interoperabilityContracts.create(
+  tenantSlug, datalakeSlug,
+  {
+    name: 'MOA Enrichment Contract',
+    resource_type: 'legal_entity',
+    template_config: { type: 'custom', body: '{{ name }}' },
+    mdm_input_config: { type: 'null' },
+    // …other contract fields…
+    interoperability_contract_ai_agents: [
+      {
+        ai_agent_id: createdAgentId,
+        position: 0,                       // ordering when multiple agents fire; REQUIRED
+        context_mapping_config: {
+          type: 'custom',
+          // Liquid renders to a JSON object whose keys match the agent's
+          // input_schema; values reference the inbound row via `msg.<field>`.
+          body: JSON.stringify({ name: '{{ msg.company_name }}' }),
+          // A contract join AUTHORS output_schema — unlike a workflow
+          // join, the server does NOT pin it here.
+          output_schema: '{"type":"object","properties":{"lead":{"type":"string"}}}',
+        },
+      },
+    ],
+  },
+)
+contract.checksum   // the PARENT checksum — the nested agents fold into it
+```
+
+Adding, repositioning, and removing agents are all expressed by
+re-supplying the full set on the contract's `update()` — `update()`
+replaces the whole array, and any item you omit is removed:
+
+| Operation  | How                                              | Result |
+|------------|--------------------------------------------------|--------|
+| add / nest | include in the array on create or update         | contract checksum reflects the agents |
+| reposition | `update()` with the same agent at a new `position`| full-set replace |
+| detach     | `update()` omitting that agent (or `[]` to clear all)| the omitted agent is removed |
+| drift      | contract `checksum()` endpoint over the full body| the would-be contract checksum (no persist) |
+
+A contract `get()` surfaces its nested agents as
+`interoperability_contract_ai_agents: [...]`. The array is writable on
+the contract body (create + update).
+
+**Runtime.** Enrichment fires during ingestion,
+**not** on the sandbox `.run`. Attached agents run in `position`
+order: each agent's `context_mapping_config` renders against the row,
+the result is validated against the agent's `input_schema`, sent to
+the bound `llm_enrichment` tool, and the parsed response is merged
+under `msg.enrichment.<agent_slug>` for the contract's
+`template_config` to read. So attaching an agent to a DAC-wired
+contract makes ingestion run enrichment per row — keep agents off
+contracts whose only job is a straight-through upsert.
+
+See `ai_agents.md` §3 for the agent create body and the shared attach
+surface, and `workflows.md` for the workflow side of the same join.
+
+## 7. Gotchas
+
+1. **`filter_template` is empty-passes, non-empty-skips.**
+   Already documented in §2 — repeated here because it's the
+   #1 source of "ingests nothing / ingests too much" silent
+   failures. Always anchor on this rule.
+
+2. **`mdm_input_config` is optional; omitting equals `{ type:
+   'null' }`.** The shared TemplateConfig embed defaults `type`
+   to `'null'`, so a contract that omits `mdm_input_config`
+   behaves as "the row IS the subject — no entity resolution
+   runs". Declaring `{ type: 'null' }` explicitly is allowed
+   for readability but not required.
+
+3. **`resource_type` is industry-scoped.** A datalake with
+   `data_domain: 'foundation'` accepts `'legal_entity'` but
+   rejects `'patient'`. The error surfaces as
+   `/resource_type` 422 at create time. Don't ship a fixed
+   enum across industries.
+
+4. **Sandbox-run is not optional in practice.** A template that
+   compiles but renders structurally-invalid JSON, or that
+   silently drops fields, won't surface until live ingestion —
+   and at that point the failures are batched and harder to
+   debug. Run `.run(slug, sampleRow)` against a representative
+   row for every new contract before binding it to a data
+   activation client.
+
+5. **Generic-table identity contracts are auto-created; don't
+   create them yourself.** Creating a generic table fires a
+   side-effect that creates a `type: 'identity'` contract bound
+   to the new table. Creating a second identity contract for the
+   same table is rejected. To add custom transformation on top
+   of a generic table, create an additional `type: 'custom'`
+   contract bound to the same `generic_table_id`.
+
+6. **Templates use `msg` to access the row.** Liquid receives
+   the row under the `msg` variable. CSV columns are typically
+   `msg.<column>`; nested cases use `msg.row.<column>`. JSON
+   sources expose the inbound fields under `msg.<field>`. Read
+   any platform-shipped template (see `templates.md`) to learn
+   the exact convention for your row shape.
+
+7. **No partial updates.** `update()` replaces the whole resource
+   — resupply the full body (see `mutations.md`). Re-supply
+   `template_config` / `mdm_input_config` verbatim if not
+   changing them.
+
+8. **Custom Liquid filters are a closed allowlist of eleven.**
+   The platform exposes exactly eleven custom filters beyond
+   stock Liquid (`e164`, `age`, `date`, `to_json`,
+   `json_escape`, `now`, `uuid`, `parse_date`, `convert_time`,
+   `random`, `tz_offset`). Other filter names — including the
+   Shopify-style `| json` — fall through to stock Liquid and
+   produce malformed output silently. See `ai_sandbox.md` §2
+   for the full per-filter reference and the safety rationale
+   for the closed set. The allowlist is the same in
+   `template_config.body`, `mdm_input_config.body`, and
+   `filter_template`.