@alvera-ai/platform-sdk 0.10.0-rc.2 → 0.10.0-rc.21

package/.agent/interoperability_contracts.md

@@ -0,0 +1,351 @@
+# Interoperability contracts
+
+An **interoperability contract** is the transformation step between
+a raw inbound row (CSV, JSON, etc.) and a typed datalake resource.
+Each contract carries:
+
+- a `resource_type` (the target shape — `patient`, `customer`,
+  `payment_account`, `legal_entity`, `generic_table`, etc.)
+- a `template_config` (the Liquid template that renders the inbound
+  row into the target shape)
+- an `mdm_input_config` (the Liquid template that renders the
+  matching subject identity for entity resolution — see `mdm.md`
+  for the full MDM surface, including the `api.mdm.verify` verb
+  invoked at workflow action time)
+- an optional `filter_template` (a Liquid pre-filter that decides
+  whether a row enters the pipeline at all)
+
+SDK namespace: `api.interoperabilityContracts`.
+
+The contract is **Datalake-DB-resident** — POSTing requires the
+parent datalake at `status: 'ready'` (see `datalakes.md` §5).
+
+## 1. Wire shape
+
+```typescript
+import type {
+  InteroperabilityContractRequestWritable,
+  InteroperabilityContractResponse,
+} from '@alvera-ai/platform-sdk'
+
+const { data: created } = await api.interoperabilityContracts.create(
+  tenantSlug,
+  datalakeSlug,
+  {
+    name: 'Inbound CRM → Customer',
+    description: 'Vendor CRM rows → AccountsReceivable Customer',
+    resource_type: 'customer',
+    filter_template: undefined,                  // optional
+    template_config: { type: 'custom', body: liquidBody },
+    mdm_input_config: { type: 'custom', body: mdmBody },
+    generic_table_id: null,                      // see §2
+  },
+)
+// created.id, created.slug    — server-derived
+// created.type                — mirrors template_config.type
+// created.resource_type       === body.resource_type
+```
+
+Both polymorphic embeds use the SAME shared `TemplateConfig`
+shape with discriminator `type`:
+
+```
+template_config.type      'system' | 'custom' | 'identity' | 'null'
+mdm_input_config.type     'system' | 'custom' | 'identity' | 'null'
+```
+
+Default for both is `'null'`. The wire accepts all four values
+on each surface; semantic narrowing is a runtime concern, not
+a schema one:
+
+- `custom` — caller-supplied `body` string (Liquid template).
+  Pair with `body`; leave `path` null.
+- `system` — references a platform-shipped template by
+  filesystem path. Pair with `path` (required); leave `body`
+  null. See `templates.md`.
+- `identity` — pass-through. The inbound row is forwarded
+  unchanged. Used by the auto-created identity contracts for
+  generic tables. Leave both `body` and `path` null.
+- `null` — disabled / "row IS the subject" on the mdm surface.
+  Leave both `body` and `path` null.
+
+### TemplateConfig embed fields
+
+Both `template_config` and `mdm_input_config` are instances of
+the shared `TemplateConfig` embed:
+
+```
+type            required enum   — 'system' | 'custom' | 'identity' | 'null' (default 'null')
+body            optional string — Liquid template body; REQUIRED iff type === 'custom'
+path            optional string — filesystem path; REQUIRED iff type === 'system'
+output_schema   server-injected — JSON Schema { type: 'object' } for this surface; not caller-set
+```
+
+Server validation enforces the body/path pairing — supplying
+`body` with `type: 'system'`, or `path` with `type: 'custom'`,
+returns 422.
+
+## 2. Rules the type cannot encode
+
+### `generic_table_id` is conditionally required
+
+When `resource_type === 'generic_table'`, the body MUST include a
+non-null `generic_table_id` identifying the target table. For
+every other `resource_type`, the field must be `null`. The
+TypeScript type permits either value; the validator and server
+enforce the conditional rule.
+
+### `resource_type` is industry-scoped
+
+The valid set of `resource_type` values is determined by the
+parent datalake's `data_domain`. Generic concepts like
+`'generic_table'` are accepted everywhere; domain-anchored
+concepts (`'patient'`, `'customer'`, `'payment_account'`,
+`'legal_entity'`, ...) only resolve within their owning industry.
+Don't hardcode a fixed enum across industries — discover the
+valid set at runtime via the platform's domain catalog (see
+`datalakes.md` §5 `.systemDatasets`).
+
+### `filter_template` has INVERTED semantics
+
+This is the single highest-value gotcha on this surface. A
+`filter_template` is a Liquid template that decides whether a
+row enters the pipeline:
+
+- **Empty render → PASS** (row enters the pipeline).
+- **Non-empty render → SKIP** (row is dropped).
+
+This is the opposite of most filter DSLs ("truthy = keep"). A
+correctly-shaped filter looks like:
+
+```liquid
+{% if msg.row.should_drop == 'Y' %}drop{% endif %}
+```
+
+Renders `"drop"` when the condition fires (SKIP); renders empty
+otherwise (PASS). Inverting `{% unless %}...{% endunless %}` is
+equivalent and stylistically common — but always anchor on the
+empty-render-passes rule, not on the Liquid keyword.
+
+If you omit `filter_template`, every row passes through.
+
+### `template_config` body is Liquid, with row data on `msg`
+
+Templates receive the inbound row on the `msg` variable. The
+exact accessor depends on the source shape — CSV-shaped sources
+typically use `msg.<column>` (with `msg.row.<column>` available
+for nested cases); JSON-shaped sources mirror the inbound JSON
+under `msg.<field>`. The render output is JSON — the platform
+parses the rendered string as JSON and treats it as the target
+resource body.
+
+A render that produces invalid JSON fails at sandbox-run with
+`stage: 'failed'` (see §5). Always sandbox-run a contract before
+binding it to a live data activation client.
+
+## 3. Field ownership
+
+**Server-derived (Response-only).** Universal set from
+`type_naming.md`, plus:
+
+```
+type           string — mirrors template_config.type at create time
+```
+
+**Caller-supplied (round-trip).**
+
+```
+name              required string — agent-facing label
+description       optional string — narrative
+resource_type     required string — target shape (industry-scoped)
+filter_template   optional string — Liquid pre-filter (inverted semantics)
+template_config   required embed  — { type, body?, path? }
+mdm_input_config  optional embed  — { type, body?, path? }; omit equals { type: 'null' }
+generic_table_id  required UUID|null — required iff resource_type='generic_table'
+```
+
+**Write-only (Request-only).** None — every caller-supplied field
+round-trips on the response.
+
+## 4. Error envelopes
+
+Standard JSON:API envelopes per `errors.md`. Common rejections:
+
+| `source.pointer`             | Cause                                            |
+|------------------------------|--------------------------------------------------|
+| `/resource_type`             | value not valid for the parent datalake's domain |
+| `/template_config/type`      | enum mismatch                                    |
+| `/template_config/body`      | empty when `type === 'custom'`                   |
+| `/mdm_input_config/type`     | enum mismatch                                    |
+| `/generic_table_id`          | missing when resource_type='generic_table',      |
+|                              | or non-null when not                             |
+| `/name`                      | uniqueness within datalake                       |
+| `errors.base`                | "System-created contracts cannot be edited" (PUT) or "...deleted" (DELETE) — see §5 |
+
+## 5. Lifecycle
+
+### Create
+
+Synchronous. The create response is immediately usable — no async
+deployment phase. Templates are compiled lazily at sandbox-run
+time and at activation-client fan-out time, so a `body`
+containing invalid Liquid syntax will surface at first use, not
+at create. Always sandbox-run before binding to a live data
+activation client.
+
+### Sandbox run — `.run(slug, msg)`
+
+The contract's templates can be exercised against a sample row
+WITHOUT actually ingesting:
+
+```typescript
+const { data: result } = await api.interoperabilityContracts.run(
+  tenantSlug, datalakeSlug, contractSlug,
+  { /* sample row, same shape as inbound CSV/JSON */ },
+)
+// result.stage          === 'completed' | 'failed'
+// result.filter_result  === 'pass' | 'skip'
+// result.transformed    object | null — the rendered resource body
+```
+
+Use this verb as a CI / pre-bind verification step: produce a
+representative sample row, run it through `.run`, and assert
+the `transformed` output shape matches the consumer's
+expectations. If `filter_result === 'skip'`, the row was filtered
+out before rendering — useful for verifying the filter semantics
+on a known-skip example.
+
+### Generic-table contracts: auto-created identity contracts
+
+When you create a generic table (see `generic_tables.md`), the
+platform automatically creates an **identity contract** bound
+to that table:
+
+```
+new generic table (T) → auto-created contract with
+                         resource_type: 'generic_table'
+                         generic_table_id: T.id
+                         template_config.type: 'identity'
+```
+
+Discover this contract via `.list()` + filter on
+`generic_table_id`:
+
+```typescript
+const { data: all } = await api.interoperabilityContracts.list(
+  tenantSlug, datalakeSlug,
+)
+const identityContract = all.data?.find(
+  (c) => c.generic_table_id === tableId && c.type === 'identity',
+)
+```
+
+Multiple contracts MAY bind to the same generic table — one
+identity (auto) plus any number of caller-created `custom`
+contracts. Distinguish by `type` when iterating. The platform
+enforces at most ONE identity contract per generic table;
+attempts to create a second `type: 'identity'` contract for
+the same table are rejected.
+
+### Read shapes
+
+```
+.list(tenantSlug, datalakeSlug)              Paged: { data, meta }
+.get(tenantSlug, datalakeSlug, slug)         One row, by slug
+.metadata(tenantSlug, datalakeSlug, slug)    Markdown string — one contract
+.run(tenantSlug, datalakeSlug, slug, msg)    Sandbox runner (above)
+```
+
+### Delete
+
+`DELETE` removes the contract. Contracts referenced by active
+data activation clients reject with a 422 naming the dependency.
+
+**System-created contracts cannot be edited or deleted.** The
+contract record carries a `system_created` flag (server-derived).
+When `system_created === true`, both PUT and DELETE reject with
+422 on `/base` and a literal message:
+
+```
+   PUT     →  "System-created contracts cannot be edited"
+   DELETE  →  "System-created contracts cannot be deleted"
+```
+
+System-created contracts include the auto-generated identity
+contracts the platform provisions when a generic table is
+created (see §5 "Generic-table contracts: auto-created identity
+contracts") and any industry-built templates the platform ships
+with the datalake's data_domain. User-created contracts of any
+`type` (including caller-created `type: 'identity'` rows bound
+to a generic table) ARE editable and deletable — the gate is
+`system_created`, not `type`.
+
+The flag IS exposed on the `InteroperabilityContractResponse`,
+so consumer agents can pre-filter before attempting a mutation:
+
+```typescript
+const { data: list } = await api.interoperabilityContracts.list(
+  tenantSlug, datalakeSlug,
+)
+const editable = list.data.filter((c) => !c.system_created)
+```
+
+## 6. Gotchas
+
+1. **`filter_template` is empty-passes, non-empty-skips.**
+   Already documented in §2 — repeated here because it's the
+   #1 source of "ingests nothing / ingests too much" silent
+   failures. Always anchor on this rule.
+
+2. **`mdm_input_config` is optional; omitting equals `{ type:
+   'null' }`.** The shared TemplateConfig embed defaults `type`
+   to `'null'`, so a contract that omits `mdm_input_config`
+   behaves as "the row IS the subject — no entity resolution
+   runs". Declaring `{ type: 'null' }` explicitly is allowed
+   for readability but not required.
+
+3. **`resource_type` is industry-scoped.** A datalake with
+   `data_domain: 'foundation'` accepts `'legal_entity'` but
+   rejects `'patient'`. The error surfaces as
+   `/resource_type` 422 at create time. Don't ship a fixed
+   enum across industries.
+
+4. **Sandbox-run is not optional in practice.** A template that
+   compiles but renders structurally-invalid JSON, or that
+   silently drops fields, won't surface until live ingestion —
+   and at that point the failures are batched and harder to
+   debug. Run `.run(slug, sampleRow)` against a representative
+   row for every new contract before binding it to a data
+   activation client.
+
+5. **Generic-table identity contracts are auto-created; don't
+   create them yourself.** Creating a generic table fires a
+   side-effect that creates a `type: 'identity'` contract bound
+   to the new table. Posting a second identity contract for the
+   same table is rejected. To add custom transformation on top
+   of a generic table, create an additional `type: 'custom'`
+   contract bound to the same `generic_table_id`.
+
+6. **Templates use `msg` to access the row.** Liquid receives
+   the row under the `msg` variable. CSV columns are typically
+   `msg.<column>`; nested cases use `msg.row.<column>`. JSON
+   sources expose the inbound fields under `msg.<field>`. Read
+   any platform-shipped template (see `templates.md`) to learn
+   the exact convention for your row shape.
+
+7. **No partial updates.** PUT replays the full body per
+   `mutations.md`; there's no PATCH affordance. Re-supply
+   `template_config` / `mdm_input_config` verbatim if not
+   changing them.
+
+8. **Custom Liquid filters are a closed allowlist of eleven.**
+   The platform exposes exactly eleven custom filters beyond
+   stock Liquid (`e164`, `age`, `date`, `to_json`,
+   `json_escape`, `now`, `uuid`, `parse_date`, `convert_time`,
+   `random`, `tz_offset`). Other filter names — including the
+   Shopify-style `| json` — fall through to stock Liquid and
+   produce malformed output silently. See `ai_sandbox.md` §2
+   for the full per-filter reference and the safety rationale
+   for the closed set. The allowlist is the same in
+   `template_config.body`, `mdm_input_config.body`, and
+   `filter_template`.

package/.agent/mdm.md

@@ -0,0 +1,293 @@
+# MDM — entity resolution + identity verification
+
+Master Data Management is the platform's entity-resolution layer.
+It binds every inbound row to a canonical **subject** record (the
+domain's primary entity — e.g. `patient` for healthcare-domain
+datalakes, `legal_entity` for foundation-domain datalakes) so
+that downstream datasets, workflows, and agent prompts reference
+a single, deduplicated identity rather than the raw inbound
+identifiers each upstream system carries.
+
+SDK namespace: `api.mdm`. Today the wire exposes a single verb:
+
+```
+   api.mdm.verify(tenantSlug, datalakeSlug, body)
+```
+
+Other MDM operations (subject resolution during ingestion,
+record merging, conflict reconciliation) run inside the platform
+on consumer-supplied inputs but are not directly SDK-callable —
+see §5 Lifecycle.
+
+MDM is **Datalake-DB-resident** — the parent datalake must be
+`status: 'ready'` before any MDM verb is called, since
+resolution / verification both look up against the datalake's
+regulated master records.
+
+## 1. Wire shape — `.verify`
+
+Verify that a person on the other end of a workflow action or a
+connected-app form submission matches an existing subject in the
+datalake. Used at action time (workflows confirming an SMS
+recipient) and at runtime (connected apps confirming the form
+submitter before posting tracking updates).
+
+```typescript
+import type {
+  MDMVerifyRequest,
+  MDMVerifyResponse,
+} from '@alvera-ai/platform-sdk'
+
+const { data: result } = await api.mdm.verify(
+  tenantSlug,
+  datalakeSlug,
+  {
+    subject_id: subjectUuid,                          // required
+    family_name: 'Garcia',                            // optional
+    given_name:  'Maria',                             // optional
+    birth_date:  '1978-11-03',                        // optional; ISO 8601 date
+    identifiers: [                                    // optional
+      { system: 'http://hospital.org/mrn', value: 'MRN12345' },
+    ],
+  },
+)
+// result.status        === 'verified' | 'not_verified'
+// result.subject_id    === body.subject_id
+// result.verified_at   — ISO 8601 timestamp
+// when status === 'not_verified':
+//   result.errors      — { <field_name>: [reason, ...] } per-field failure detail
+```
+
+The response is **polymorphic on `status`**:
+
+```
+status === 'verified'       → { subject_id, verified_at }
+status === 'not_verified'   → { subject_id, verified_at, errors }
+```
+
+A `not_verified` response is NOT an error — it's a successful
+identity check that returned a negative result. Distinguish from
+402-level rejections (404 for missing subject, 401 for missing
+auth, 422 for malformed body).
+
+### Subject per data domain
+
+Every datalake has a single canonical **subject entity** — the
+domain's primary identity that all other rows attribute to. The
+verify call resolves against that subject. The subject's wire
+name follows the dataset's `resource_type` (see
+`interoperability_contracts.md` §2 `resource_type`):
+
+```
+   data_domain                  subject resource_type
+   ───────────                  ─────────────────────
+   healthcare                   patient
+   core_banking                 party
+   payment_risk                 legal_entity
+   accounts_receivable          customer
+   service_commerce             consumer
+   trading                      trading_account
+   foundation                   legal_entity
+```
+
+(`legal_entity` is the subject for both `payment_risk` and
+`foundation` — different domain context, same canonical entity
+shape.)
+
+Generic tables in a datalake automatically carry a foreign-key
+column to their subject (e.g. `patient_id` in a healthcare-
+domain generic table). Querying that column scopes any custom
+dataset to subjects the verify call has confirmed.
+
+## 2. Rules the type cannot encode
+
+### `subject_id` is required; the identity fields are not
+
+`subject_id` (the master record to verify against) is the only
+required field. The identity fields (`family_name`, `given_name`,
+`birth_date`, `identifiers`) are individually optional, but the
+platform requires **at least one** identity field to be supplied
+at runtime — a body carrying only `subject_id` returns a 422.
+
+The runtime check is a "must have something to match against"
+rule that the schema can't express in TypeScript / JSON Schema.
+
+### Verification is per-field, with field-specific match rules
+
+Each supplied identity field is checked independently against
+the subject record. The platform uses fuzzy matching for the
+string fields and exact matching for the structured ones:
+
+```
+family_name      Jaro distance ≥ 0.85
+given_name       Jaro distance ≥ 0.80
+birth_date       component-wise fuzzy match ≥ 0.80
+identifiers      exact (system, value) match
+```
+
+All supplied fields must pass for the overall result to be
+`'verified'`. A single failing field flips the result to
+`'not_verified'` and surfaces the field name in `errors`.
+
+The exact thresholds and field set are **domain-scoped** —
+healthcare-domain datalakes use the FHIR-shaped demographics
+listed above; future domains may extend the field set with
+additional verification predicates. The verify endpoint accepts
+the union of fields the parent datalake's domain knows how to
+match against; unknown fields are rejected as a 422.
+
+### `subject_id` must exist in the datalake's regulated schema
+
+If the supplied `subject_id` doesn't resolve to a master record
+in the datalake's regulated schema, the response is a 404, not
+a `'not_verified'`. The distinction matters:
+
+- `'not_verified'` → "subject exists, identity fields didn't match"
+- 404               → "subject doesn't exist at all"
+
+A consumer that conflates the two will mis-route an identity
+challenge as a missing-record error.
+
+## 3. Field ownership
+
+**Server-derived (Response-only).**
+
+```
+status         enum   — 'verified' | 'not_verified'
+verified_at    string — ISO 8601 timestamp of the verification attempt
+errors         object — present only when status === 'not_verified';
+                         maps field name to an array of failure reasons
+```
+
+**Caller-supplied (round-trip on the verified branch).**
+
+```
+subject_id     required UUID  — the master record to verify against
+family_name    optional string — fuzzy-matched (string distance)
+given_name     optional string — fuzzy-matched (string distance)
+birth_date     optional date   — fuzzy-matched component-wise
+identifiers    optional array  — [{ system, value }] exact-matched
+```
+
+**Write-only (Request-only).** None.
+
+## 4. Error envelopes
+
+Standard JSON:API envelopes per `errors.md`. Common rejections:
+
+| `source.pointer`              | Cause                                                  |
+|-------------------------------|--------------------------------------------------------|
+| `/subject_id`                 | missing, or the subject doesn't exist in the datalake  |
+| `errors.base`                 | no identity fields supplied (subject_id alone) — 422 response with `status: "not_verified"` in the body |
+| `/family_name` etc.           | field structurally invalid (wrong type, malformed)     |
+| `/identifiers/0/system`       | missing system on an identifier entry                  |
+| `/identifiers/0/value`        | missing value on an identifier entry                   |
+
+The `not_verified` outcome surfaces in two response shapes:
+
+- **Identity mismatch (most common)** — 200 response with
+  `status: 'not_verified'` and a field-keyed `errors` map naming
+  the demographics that failed (e.g. `errors.family_name`). The
+  subject exists; the supplied identity didn't match.
+- **No identity fields supplied** — 422 response with
+  `status: 'not_verified'` in the body and `errors.base` set to
+  `["at least one verification field is required"]`. Verification
+  was structurally impossible.
+
+Pattern-match on `status` first, then on the HTTP status code to
+distinguish the two not_verified shapes. The 404 case (subject
+doesn't exist) is a separate envelope — see §6 gotcha 2.
+
+## 5. Lifecycle
+
+### `.verify` is synchronous
+
+The verify call returns the resolution outcome in the response —
+no async polling, no job id. The platform performs the fuzzy
+match against the regulated master record inline.
+
+### Other MDM operations are platform-internal
+
+Beyond `.verify`, MDM also runs at two points without a direct
+SDK surface:
+
+- **During data activation ingestion.** Each inbound row that
+  carries an `mdm_input_config` on its bound interoperability
+  contract gets routed through the platform's resolution
+  pipeline before the dataset upsert. The pipeline matches the
+  rendered MDM input against existing subjects (or creates one
+  if no match) and produces an `mdm_output` carried forward to
+  the regulated upsert. Consumers configure this implicitly via
+  the contract's `mdm_input_config` — see
+  `interoperability_contracts.md` §2 and `data_activation_clients.md`
+  §6.
+
+- **Conflict reconciliation.** When two master records turn out
+  to represent the same subject (e.g. an MRN-only record and a
+  demographics-only record that operators later confirm are the
+  same person), the platform supports merge / unmerge operations.
+  These are administrative actions surfaced in the admin console,
+  not SDK-exposed verbs.
+
+### Read shapes
+
+```
+.verify(tenantSlug, datalakeSlug, body)    Single round-trip; see §1
+```
+
+Today there is no `.list`, `.get`, or `.metadata` on the MDM
+namespace — subjects themselves are queried through the dataset
+search surface (see `data_activation_clients.md` §6.5), not
+through `api.mdm`.
+
+## 6. Gotchas
+
+1. **`'not_verified'` is usually a successful 200 response, not
+   an error — but the "no fields supplied" case is a 422 with a
+   `not_verified` body.** Pattern-match on `result.status === 'verified'`
+   to gate the downstream "act on this person" path. Throwing on
+   `'not_verified'` collapses two distinct outcomes (subject
+   verified vs identity mismatch) into one branch and loses the
+   `errors` payload that explains which fields failed. The 422
+   sub-case (no identity fields, just `subject_id`) needs HTTP-status
+   handling on top of body-status handling — see §4 above.
+
+2. **`subject_id` missing → 404, identity mismatch → 200.** A
+   verify call against a deleted or never-existed subject id
+   returns a 404 envelope, NOT a `'not_verified'` response.
+   Consumers that handle only the polymorphic 200 response will
+   surface the 404 as an unexpected exception. Wrap the verify
+   call in standard 4xx handling per `errors.md`.
+
+3. **At least one identity field is required.** A body with
+   only `subject_id` and no identity fields returns a 422 with
+   `errors.base` set to `["at least one verification field is
+   required"]` and `status: "not_verified"` in the response
+   body. The platform cannot verify identity against zero
+   attributes; this is enforced at runtime rather than in the
+   request schema, so the TypeScript type permits the
+   no-identity-field body but the wire rejects it.
+
+4. **The verification field set is domain-scoped.** Today the
+   healthcare-domain verifier accepts FHIR-shaped demographics
+   (family_name, given_name, birth_date, identifiers). Other
+   domains will expose verifier-appropriate field sets as they
+   land. Querying `api.mdm.verify` against a datalake whose
+   domain doesn't support a supplied field returns a 422 on
+   that field's pointer — don't assume the full healthcare
+   field set works on every domain.
+
+5. **MDM also runs at ingestion — and you don't call it
+   directly.** The `mdm_input_config` on each interoperability
+   contract feeds the platform's resolution pipeline at
+   ingestion time, producing the `mdm_output` carried into the
+   dataset upsert. There is no `api.mdm.resolve` SDK verb; you
+   configure resolution by shaping the contract's
+   `mdm_input_config`. See `interoperability_contracts.md` §2.
+
+6. **`identifiers` matches are exact, not fuzzy.** Whitespace,
+   case, and trailing slashes in `system` URIs all matter.
+   Compare with the master record's identifier system strings
+   character-for-character; a `'http://hospital.org/mrn'` vs
+   `'http://hospital.org/mrn/'` mismatch will flip the verifier
+   to `'not_verified'` even when the value is correct.