@alvera-ai/platform-sdk 0.10.0-rc.2 → 0.10.0-rc.21

package/.agent/datalakes.md

@@ -0,0 +1,712 @@
+# Datalakes
+
+A **datalake** is the storage layer for one tenant slice. It holds
+two databases under one roof:
+
+- an **unregulated** database for tokenized data, exposed to AI
+  workflows and downstream agents
+- a **regulated** database for raw values that never leave the
+  trust boundary
+
+Every other resource in the platform (data sources, tools, AI
+agents, workflows, etc.) scopes under a datalake. A tenant can own
+many datalakes; the canonical pattern is one per data domain (e.g.
+`healthcare`, `foundation`, `payment_risk`).
+
+```typescript
+import type { PlatformApi } from '@alvera-ai/platform-sdk'
+
+const api: PlatformApi = /* see AGENTS.md */
+const { data: created } = await api.datalakes.create(tenantSlug, body)
+const datalakeSlug = created.slug    // canonical — use for downstream scoping
+```
+
+SDK namespace: `api.datalakes`.
+
+## 1. Wire shape
+
+The request type splits across two layers: top-level database +
+identity fields, plus two polymorphic cloud-storage embeds.
+
+```typescript
+import type {
+  DatalakeRequestWritable,
+  DatalakeResponse,
+} from '@alvera-ai/platform-sdk'
+```
+
+See `type_naming.md` for the `Writable` convention and the
+universal server-derived field set. (The SDK exports types only —
+there is no runtime validator; validation is server-authoritative.)
+
+### Polymorphic cloud-storage embed
+
+`unregulated_cloud_storage` and `regulated_cloud_storage` are
+polymorphic. The discriminator field is **`cloud_storage_type`**
+(see `mutations.md` "Polymorphic discriminator naming") with three
+branches:
+
+| `cloud_storage_type` | TypeScript branch                       |
+|----------------------|-----------------------------------------|
+| `"aws"`              | `DatalakeCloudStorageAwsRequest`        |
+| `"r2"`               | `DatalakeCloudStorageR2Request`         |
+| `"custom"`           | `DatalakeCloudStorageCustomRequest`     |
+
+Each branch carries its own field set; the shared keys are
+`bucket`, `region`, `access_key_id`, `secret_access_key`. `custom`
+adds `endpoint` (used for LocalStack and self-hosted S3-compatible
+stores).
+
+The `aws` branch also carries an `auth_method` field —
+`"access_key"` (default) or `"iam_role"` — that pairs with the
+credentials the same way the DB-profile `auth_method` does. See
+§2 "AWS cloud-storage `auth_method = iam_role` forbids
+credentials" for the conditional the type cannot encode.
+
+The two embeds are independent — a datalake can pair an `aws`
+unregulated bucket with a `custom` regulated bucket. Both embeds
+ship in the same POST/PUT body.
+
+### Cloud storage is also the cold-archive destination
+
+Beyond being the platform's primary blob store for inline payloads
+(image uploads, large context payloads, presigned-URL ingest), the
+configured `regulated_cloud_storage` and `unregulated_cloud_storage`
+buckets are the **cold-archive destination** for the datalake. As
+data flows through the pipeline, two NDJSON streams land under
+well-defined key prefixes inside the bucket the credentials point at:
+
+```
+   {datalake_slug}/data-activations/{client_slug}/{batch_id}/{dataset_table}.ndjson
+   {datalake_slug}/workflows/{wf_slug}/{batch_id}/merged.ndjson
+   {datalake_slug}/workflows/{wf_slug}/{batch_id}/{decision_key}/merged.ndjson
+```
+
+The merged NDJSONs are produced by background batch-merge workers
+after each Data Activation Client batch and each workflow run
+completes. The platform never streams the merged bytes through its
+own process — the merge happens via DuckDB's `httpfs` extension
+operating directly against the bucket.
+
+The same DuckDB-over-`httpfs` pattern is available to **consumers**
+for cold-data analytics. With the datalake's cloud-storage
+credentials (or a separately-provisioned reader key on the same
+bucket), an external DuckDB process can read the archived NDJSONs
+directly:
+
+```sql
+INSTALL httpfs; LOAD httpfs;
+CREATE OR REPLACE SECRET (TYPE S3, KEY_ID '...', SECRET '...', ENDPOINT '...');
+SELECT * FROM read_ndjson_auto(
+  's3://{bucket}/{datalake_slug}/data-activations/{client_slug}/*/{dataset_table}.ndjson'
+);
+```
+
+The platform does NOT expose a query API over the archive itself —
+cold reads happen externally so the platform's processes aren't
+responsible for analytical workloads. Live data lives in the
+datalake's Postgres halves (queryable via the dataset search
+two-step pattern in `data_activation_clients.md` §6.5); historical
+data lives in the cloud-storage buckets configured here.
+
+### Canonical create body
+
+Paste-ready TypeScript body. The four database-role groups
+(unregulated writer + reader, regulated writer + reader) share
+an identical eight-field shape; the type lists all 32 explicitly
+with no role-level union:
+
+```typescript
+import type { DatalakeRequestWritable } from '@alvera-ai/platform-sdk'
+
+// Pick one of four supported industries. The body shape below is
+// identical across them; only `data_domain` and the names it
+// interpolates differ.
+const dataDomain = 'healthcare'
+//                  ^ swap to 'foundation' | 'accounts_receivable' | 'payment_risk'
+
+// Secrets resolved upstream by your code (secret store, config loader, etc.)
+const pgHost = '...'
+const pgUser = '...'
+const pgPass = '...'
+const pgReaderHost = '...'
+const pgReaderUser = '...'
+const pgReaderPass = '...'
+const awsAccessKeyId = '...'
+const awsSecretAccessKey = '...'
+
+const body: DatalakeRequestWritable = {
+  name: 'Production Lake',
+  description: `Primary ${dataDomain} datalake`,
+  data_domain: dataDomain,
+  timezone: 'America/New_York',
+  pool_size: 5,
+
+  unregulated_db_writer_host: pgHost,
+  unregulated_db_writer_port: 5432,
+  unregulated_db_writer_name: `alvera_${dataDomain}`,
+  unregulated_db_writer_schema: 'tenant_acme_unreg',
+  unregulated_db_writer_auth_method: 'password',
+  unregulated_db_writer_user: pgUser,
+  unregulated_db_writer_pass: pgPass,
+  unregulated_db_writer_enable_ssl: true,
+  unregulated_db_reader_host: pgReaderHost,
+  unregulated_db_reader_port: 5432,
+  unregulated_db_reader_name: `alvera_${dataDomain}`,
+  unregulated_db_reader_schema: 'tenant_acme_unreg',
+  unregulated_db_reader_auth_method: 'password',
+  unregulated_db_reader_user: pgReaderUser,
+  unregulated_db_reader_pass: pgReaderPass,
+  unregulated_db_reader_enable_ssl: true,
+
+  regulated_data_db_writer_host: pgHost,
+  regulated_data_db_writer_port: 5432,
+  regulated_data_db_writer_name: `alvera_${dataDomain}`,
+  regulated_data_db_writer_schema: 'tenant_acme_reg',
+  regulated_data_db_writer_auth_method: 'password',
+  regulated_data_db_writer_user: pgUser,
+  regulated_data_db_writer_pass: pgPass,
+  regulated_data_db_writer_enable_ssl: true,
+  regulated_data_db_reader_host: pgReaderHost,
+  regulated_data_db_reader_port: 5432,
+  regulated_data_db_reader_name: `alvera_${dataDomain}`,
+  regulated_data_db_reader_schema: 'tenant_acme_reg',
+  regulated_data_db_reader_auth_method: 'password',
+  regulated_data_db_reader_user: pgReaderUser,
+  regulated_data_db_reader_pass: pgReaderPass,
+  regulated_data_db_reader_enable_ssl: true,
+
+  unregulated_cloud_storage: {
+    cloud_storage_type: 'aws',
+    region: 'us-east-1',
+    bucket: `acme-${dataDomain}-unregulated`,
+    access_key_id: awsAccessKeyId,
+    secret_access_key: awsSecretAccessKey,
+  },
+  regulated_cloud_storage: {
+    cloud_storage_type: 'aws',
+    region: 'us-east-1',
+    bucket: `acme-${dataDomain}-regulated`,
+    access_key_id: awsAccessKeyId,
+    secret_access_key: awsSecretAccessKey,
+  },
+}
+
+const { data: created } = await api.datalakes.create(tenantSlug, body)
+// created.id, created.slug    — server-derived; use slug for downstream scoping
+// created.status === 'new'    — migration NOT yet started; see §5 Lifecycle
+```
+
+## 2. Rules the type cannot encode
+
+### `auth_method = "iam_role"` relaxes the password requirement
+
+All eight `*_pass` fields are typed as required strings. When a
+role's `auth_method` is `"iam_role"`, its matching `*_pass` value
+is **server-cleared at write time** — the value submitted is
+ignored. The typed validator does not encode this conditional, so
+the field is still structurally required.
+
+The convention is to supply a sentinel literal (e.g.
+`"unused-iam-role"`) so a reviewer sees intent. **Real credentials
+do not belong in this slot** — the value never leaves the
+validator.
+
+### AWS cloud-storage `auth_method = iam_role` forbids credentials
+
+Mirrors the DB `auth_method` rule above but with the **opposite
+disposition** on the credential fields. Inside an AWS cloud-storage
+embed:
+
+- `auth_method: "access_key"` (default) — `access_key_id` and
+  `secret_access_key` are **required**.
+- `auth_method: "iam_role"` — `access_key_id` and
+  `secret_access_key` MUST be omitted. Supplying either yields a
+  field-level rejection at create. Unlike the DB case, the values
+  are NOT silently server-cleared — the platform treats their
+  presence as a configuration conflict.
+
+The two embeds (`unregulated_cloud_storage` and
+`regulated_cloud_storage`) are validated independently, so a body
+can pair `iam_role` on one side with `access_key` on the other.
+
+### Database identifiers follow strict format rules
+
+The four `*_db_*_schema` and four `*_db_*_name` fields must be
+lowercase identifiers (letters, digits, underscore) — uppercase
+letters and dashes are rejected with a `/<role>_schema` or
+`/<role>_name` pointer. The eight `*_db_*_host` fields must match
+a hostname pattern; values like `"invalid_host!"` (with
+non-hostname punctuation) are rejected.
+
+This is the only field-format constraint surfaced before the
+synchronous reachability probe runs (see §6 Gotcha 9) — a
+malformed identifier never gets as far as a connection attempt.
+
+### Schema names must be unique within a database
+
+`*_db_*_schema` fields name the PostgreSQL schema the platform
+creates inside the target database. Sibling datalakes sharing a
+`*_db_*_name` (same physical database) **must** use distinct
+schema values. Collision produces a 422 at create time.
+
+The reader and writer for the same role typically share a schema
+(they're two connection profiles into the same logical schema);
+unregulated and regulated must not (different trust boundaries).
+
+### Discriminator on PUT must match
+
+Per `mutations.md`, `cloud_storage_type` on update must match the
+existing row's discriminator. Changing the embed type (e.g. `aws`
+→ `r2`) requires DELETE + recreate, not PUT.
+
+### `data_domain` is a closed enum
+
+Industry values are enumerated in the OpenAPI spec
+(`DatalakeDataDomain`); the SDK exports them as a const-style
+value (see `type_naming.md` "Enum types"). The enum drifts as
+new industries land — consult the SDK's `DatalakeDataDomain`
+const for the live set rather than hardcoding values.
+
+## 3. Field ownership
+
+Datalake's fields split across three POV-relevant categories —
+distinguishing them by whether they round-trip determines how a
+consumer constructs the PUT body (see §5 Update).
+
+**Server-derived (Response-only).** The universal set (`id`,
+`slug`, `created_at`, `updated_at`, `status`, `status_reason`)
+is documented in `type_naming.md`. Datalake adds no extras here
+— but the `status` field has a richer lifecycle than most
+resources (see §5).
+
+**Caller-supplied (round-trip).** Present on both
+`DatalakeRequestWritable` and `DatalakeResponse`:
+
+```
+identification         name, description, data_domain
+global settings        timezone, pool_size
+db profiles (×4)       <role>_{host, port, name, schema, user,
+                               auth_method, enable_ssl}
+cloud storage (×2)     <role>_cloud_storage.{cloud_storage_type,
+                                             bucket, region,
+                                             + branch-specific
+                                             fields per §1}
+```
+
+**Write-only (Request-only).** Present on
+`DatalakeRequestWritable`, absent from `DatalakeResponse`. The
+consumer must re-supply these on every PUT from their secret
+store (see §5 Update for the canonical pattern):
+
+```
+db credentials (×4)    unregulated_db_writer_pass
+                       unregulated_db_reader_pass
+                       regulated_data_db_writer_pass
+                       regulated_data_db_reader_pass
+
+cloud storage          unregulated_cloud_storage.access_key_id
+  credentials (×4)     unregulated_cloud_storage.secret_access_key
+                       regulated_cloud_storage.access_key_id
+                       regulated_cloud_storage.secret_access_key
+```
+
+The `slug` returned in the Response is the canonical reference
+for all downstream scoping. Never pre-compute it client-side —
+see `type_naming.md` "Never pre-compute the slug client-side".
+
+## 4. Error envelopes
+
+The platform returns a standardized JSON:API envelope — see
+`errors.md` for the canonical shape (`{ errors: [{ source:
+{ pointer }, title, detail }, ...] }`) and the two-layer
+validation model (structural cast → server-side semantic check).
+
+All validation errors come back from the server as a thrown 422
+`AlveraApiError` (the SDK does not validate client-side). See
+`errors.md` for the typed narrowing pattern.
+
+Datalake-specific common rejections:
+
+| `source.pointer`                                | Typical cause                                        |
+|-------------------------------------------------|------------------------------------------------------|
+| `/name`                                         | name collision within tenant (uniqueness)            |
+| `/data_domain`                                  | value not in the live enum                           |
+| `/unregulated_cloud_storage/cloud_storage_type` | discriminator value doesn't match any oneOf branch   |
+| `/unregulated_db_writer_schema`                 | schema-name collision within the target DB; or identifier format violation (uppercase / dash) |
+| `/unregulated_db_writer_name`                   | identifier format violation                          |
+| `/unregulated_db_writer_host`                   | hostname format violation, OR synchronous probe could not reach the host |
+| `/unregulated_cloud_storage`                    | synchronous probe could not reach the bucket with the supplied credentials |
+| `/unregulated_cloud_storage/access_key_id`      | supplied under `auth_method: "iam_role"` (must be omitted) |
+| `/pool_size`                                    | out of the inclusive range 1–50                      |
+| `/timezone`                                     | not a valid IANA timezone identifier                 |
+
+For polymorphic embeds, the `cloud_storage_type` value is the
+first thing to inspect — it routes downstream field validation to
+one of the three branches.
+
+## 5. Lifecycle
+
+Datalakes have the richest lifecycle of any platform resource.
+The status enum (wire-level values):
+
+```
+new         POST returned; migration not yet started
+processing  platform provisioning the per-datalake schemas and
+              supporting infrastructure
+ready       both DBs migrated; downstream Datalake-DB-resident
+              resources can be created
+```
+
+The platform console UI renders `processing` as the label
+"Migrating..." — the wire-level enum value is `processing`.
+Code that switches on status MUST use the wire values, not the
+UI labels.
+
+The status enum has NO terminal failure value. If a migration
+fails, the datalake remains in `processing` (or `new` if it
+never started); failures surface in the admin console's Migration
+Logs view, which is not exposed over the SDK. Consumers polling
+for readiness must therefore rely on a sensible timeout — see §5.
+
+### Create
+
+```typescript
+const { data: created } = await api.datalakes.create(tenantSlug, body)
+// created.status === 'new'
+// created.slug   === '<server-derived>'
+```
+
+`POST` returns immediately with `status: "new"`. **Migration is
+not auto-enqueued** — the caller explicitly invokes it.
+
+### Migrate (explicit, async)
+
+```typescript
+const { data: enqueued } = await api.datalakes.migrate(
+  tenantSlug,
+  created.slug,
+)
+// enqueued.status      === 'enqueued'
+// enqueued.job_id      === <number>
+// enqueued.datalake_id === created.id
+```
+
+`api.datalakes.migrate(...)` enqueues the schema-creation job. The
+response is the *job acknowledgement* (`status: "enqueued"`), NOT
+the datalake row — it carries `job_id` + `datalake_id` +
+`enqueued_at` only. To learn the datalake's live status, GET
+separately:
+
+```typescript
+const { data: dl } = await api.datalakes.get(tenantSlug, created.slug)
+// dl.status === 'new' | 'processing' | 'ready'
+```
+
+Use the canonical `waitUntilReady` from `async.md`:
+
+```typescript
+const dl = await waitUntilReady(
+  () => api.datalakes.get(tenantSlug, created.slug),
+  (status) => status === 'ready' ? 'ready' : 'pending',
+  { intervalMs: 15_000, timeoutMs: 5 * 60_000 },
+)
+```
+
+Local-dev migration takes seconds to a minute; remote deployments
+can run several minutes. **Most** downstream resource creates
+must wait for the datalake to reach `ready` — but the rule has
+one exception worth understanding.
+
+Downstream resources split into two categories by storage
+location:
+
+| Where the resource's state lives | Examples                                       | Wait for `ready`? |
+|----------------------------------|------------------------------------------------|-------------------|
+| Platform DB (tenant-wide)        | Data sources                                   | No                |
+| Per-datalake DB (tenant + datalake schemas) | Tools, AI agents, generic tables, interoperability contracts, data activation clients, workflows | **Yes**           |
+
+Data sources can be created against a `new`-status datalake
+because their metadata writes to the platform DB, not the
+per-datalake schemas. Everything else needs the per-datalake
+schemas in place — POSTing them against an unready datalake
+produces server-side errors (table-not-found, capability-gate,
+etc.) that are recoverable but indicate a missing wait.
+
+In practice: a sibling datalake left unmigrated is still a valid
+target for data-source creation, since data sources are
+platform-DB-resident and don't depend on per-datalake readiness.
+
+### Update
+
+`PUT` replays the full body per `mutations.md`. Common pattern:
+GET current → re-supply write-only fields from your secret store →
+mutate target fields → PUT the whole shape. Changing the
+cloud-storage discriminator is rejected — DELETE + recreate
+instead.
+
+**`DatalakeResponse` omits write-only fields.** Eight fields on
+`DatalakeRequestWritable` — the four `*_pass` fields plus both
+cloud-storage embeds' `access_key_id` / `secret_access_key` — are
+write-only: present on the request type, absent from the response
+type. A naive spread of `current` into the PUT body is therefore
+missing those 8 required fields. The server rejects this with a 422
+`AlveraApiError` pointing at the missing fields (see `errors.md`).
+Re-supply them explicitly from your secret store:
+
+```typescript
+const { data: current } = await api.datalakes.get(tenantSlug, datalakeId)
+// `current` is a DatalakeResponse — write-only fields (DB passwords +
+// cloud-storage secrets) are not present on the response type.
+
+const next = {
+  ...current,
+
+  // Re-supply every write-only field from your secret store:
+  unregulated_db_writer_pass: pgPass,
+  unregulated_db_reader_pass: pgReaderPass,
+  regulated_data_db_writer_pass: pgPass,
+  regulated_data_db_reader_pass: pgReaderPass,
+  unregulated_cloud_storage: {
+    ...current.unregulated_cloud_storage,
+    access_key_id: awsAccessKeyId,
+    secret_access_key: awsSecretAccessKey,
+  },
+  regulated_cloud_storage: {
+    ...current.regulated_cloud_storage,
+    access_key_id: awsAccessKeyId,
+    secret_access_key: awsSecretAccessKey,
+  },
+
+  // Mutate your target fields:
+  description: 'Now with revised description',
+}
+
+const { data: updated } = await api.datalakes.update(
+  tenantSlug,
+  datalakeId,
+  next,
+)
+// updated.id          === datalakeId    — path-key stable
+// updated.description === 'Now with revised description'
+```
+
+The canonical pattern: caller re-supplies the write-only DB +
+cloud-storage fields verbatim, mutates only what changes (e.g.
+`description`). The eight write-only fields are enumerated in
+§3 Field ownership.
+
+**PUT fires server-side connectivity probes before persisting.**
+Submitting a body produces a successful 200 only after two
+probes pass for each role profile:
+
+- **Cloud-storage probe** — verifies the bucket exists + the
+  supplied `access_key_id` / `secret_access_key` (or IAM role)
+  can list it.
+- **DB-connection probe** — verifies the platform can connect
+  to the supplied host:port with the supplied user + auth
+  method.
+
+A body that's structurally valid but carries stale or wrong
+credentials therefore fails at PUT time, not at later write
+time. The 422 envelope (see `errors.md`) names the failing probe
+via `source.pointer` (e.g. `/unregulated_cloud_storage` or
+`/regulated_data_db_writer_host`). This makes drift-then-PUT
+workflows fail fast and locally rather than mid-pipeline.
+
+Most field changes do NOT re-trigger migration. Schema-name or
+auth-method changes DO — the status reverts to `processing` and
+downstream operations must wait again.
+
+`api.datalakes.update(...)` accepts the datalake's `id` OR `slug`
+as the path key (same as `.get`); both are stable across migrate
+status changes.
+
+### Delete
+
+`DELETE` requires the datalake to be empty of dependent resources
+(no data sources, tools, AI agents, workflows scoped under it). A
+non-empty datalake DELETE returns a 422 with a `source.pointer`
+naming the blocking dependency type. Cleanup is operator-driven in
+most deployments.
+
+### Read shapes
+
+`api.datalakes` exposes five distinct read methods, each
+returning a different shape:
+
+| Method                               | Returns                                                     |
+|--------------------------------------|-------------------------------------------------------------|
+| `.list(tenantSlug)`                  | Paged list: `{ data: DatalakeResponse[], meta }`            |
+| `.get(tenantSlug, idOrSlug)`         | One row: `DatalakeResponse`                                 |
+| `.metadata(tenantSlug, slug)`        | **Markdown string** describing the lake                     |
+| `.indexMetadata(tenantSlug)`         | **Markdown string** listing every lake                      |
+| `.systemDatasets(tenantSlug, slug)`  | `{ datasets: string[] }` — industry-built dataset names     |
+
+The two `metadata` methods return `string`, not a structured
+object — they're agent-facing summaries (`data: string`, not `data:
+{...}`). Test the shape with `typeof data === 'string'`.
+
+`.indexMetadata(tenantSlug)` body begins with `# Datalakes on
+tenant` and lists every datalake by slug — useful for agent-side
+discovery without paging through `.list`.
+
+`.systemDatasets(...)` enumerates the dataset names the platform
+auto-creates for the datalake's `data_domain`. Each industry's
+catalog mixes **domain-anchored names** (the primary subject of
+the industry plus related entities) with **cross-cutting concepts**
+(audit logs,
+messages, documents, etc.) that the platform composes per
+industry — agents iterating the catalog should not hardcode a
+closed set of expected names; discover them at runtime via
+this call. The array is sorted, unique, and **excludes
+operator-defined generic tables** — those live under
+`api.genericTables.list(...)`. The
+literal placeholder `"generic_table"` is also absent. Each name
+can be drilled into via
+`api.datasets.metadataDetails(tenantSlug, datalakeSlug, name)` for
+agent-facing markdown, or the whole-domain catalog can be fetched
+in one shot via `api.datasets.metadata(tenantSlug, datalakeSlug)`. Pair `.systemDatasets()` with a
+`page_size: 1` probe per name as a post-migration smoke (proves
+each industry-built table is queryable; the post-ready probe
+code is in §6 Gotcha 3).
+
+```typescript
+const { data: catalog } = await api.datalakes.systemDatasets(
+  tenantSlug,
+  datalakeSlug,
+)
+// catalog.datasets: sorted, unique array of industry-built dataset names
+
+for (const name of catalog.datasets) {
+  const { data: md } = await api.datasets.metadataDetails(tenantSlug, datalakeSlug, name)
+  // md is a string — agent-facing markdown describing the dataset
+}
+```
+
+## 6. Gotchas
+
+1. **POST does NOT auto-enqueue migration.** The create response
+   returns immediately with `status: 'new'`; Datalake-DB-resident
+   downstream resources (tools, AI agents, generic tables,
+   interoperability contracts, data activation clients,
+   workflows) MUST wait for `ready` (see §5 Lifecycle for the
+   storage-location split). The canonical sequence:
+
+   ```typescript
+   const { data: created } = await api.datalakes.create(tenantSlug, body)
+   // created.status === 'new'
+
+   await api.datalakes.migrate(tenantSlug, created.slug)
+
+   const ready = await waitUntilReady(
+     () => api.datalakes.get(tenantSlug, created.slug),
+     (s) => s === 'ready' ? 'ready' : 'pending',
+     { intervalMs: 15_000, timeoutMs: 5 * 60_000 },
+   )
+   // ready.status === 'ready' — safe to POST tools, AI agents, etc.
+   ```
+
+2. **`migrate` returns the job ack, not the datalake.** The
+   `{ status: "enqueued", job_id, ... }` shape can be mistaken for
+   the datalake row. To learn live status, `.get` separately and
+   poll per `async.md`.
+
+3. **`status: "ready"` is necessary but not sufficient.** A
+   partial-migration failure can leave a datalake in `ready`
+   with some tables silently missing — a failure mode that
+   otherwise only surfaces mid-pipeline (during downstream
+   resource creation or live data ingestion), far from its
+   root cause.
+   Pair the status check with a post-ready probe:
+
+   ```typescript
+   const { data: catalog } = await api.datalakes.systemDatasets(
+     tenantSlug, datalakeSlug,
+   )
+   // catalog.datasets is the industry-built name list —
+   // discovered at runtime; don't hardcode names.
+
+   for (const dataset of catalog.datasets) {
+     const { data } = await api.datasets.createUserSearch(
+       tenantSlug,
+       datalakeSlug,
+       dataset,
+       { search_query: '1 = 1' },
+     )
+     if (data.status !== 'completed') {
+       throw new Error(
+         `post-ready probe failed: dataset=${dataset} status=${data.status}`,
+       )
+     }
+   }
+   ```
+
+4. **`id` and `slug` are interchangeable as path keys.**
+   `.get`, `.update`, `.migrate`, `.metadata` all accept either.
+   The `slug` is human-readable; the `id` is a UUID. Pick one
+   convention per code path to avoid mixed references.
+
+5. **Schema-name collisions surface at create, not migrate.** A
+   sibling datalake reusing a `*_schema` value gets a 422
+   immediately — well before the asynchronous schema-creation job
+   would have hit the conflict.
+
+6. **`iam_role` auth still requires a `*_pass` value structurally.**
+   The TypeScript type (and the server's schema) treat the field as
+   a required string. Supply a sentinel literal (e.g.
+   `"unused-iam-role"`); the platform clears it server-side.
+
+7. **`cloud_storage_type` lives inside each embed, not at the
+   top level.** A single body has two discriminators
+   (`unregulated_cloud_storage.cloud_storage_type` and
+   `regulated_cloud_storage.cloud_storage_type`); they're
+   independently validated and can carry different values.
+
+8. **DELETE is rare in practice.** Datalakes accrete dependents
+   quickly; routine cleanup is via operator-driven console
+   actions, not SDK calls. Treat `.delete` as an integration-test
+   teardown method, not a production operation.
+
+9. **Create and update run synchronous reachability probes.** Before
+   the row is written, the platform opens connections to every
+   declared endpoint — all four DB profiles (unregulated reader +
+   writer, regulated reader + writer) AND both cloud-storage
+   buckets — and only proceeds if every probe succeeds. A
+   structurally valid body can still fail with field-level
+   rejections on `*_db_*_host` (unreachable, wrong credentials, or
+   SSL-mismatch) or `*_cloud_storage` (bucket inaccessible). The
+   same probe runs on update; an update that only changes the
+   datalake's `name` still re-probes if the body re-supplies any
+   `*_host` or `*_cloud_storage` field.
+
+   Two consumer-relevant consequences:
+
+   - **Multiple probe failures are surfaced in a single envelope.**
+     Two bad reader hosts produce two field-level entries —
+     consumer agents iterating on a single response can correct
+     several fields per round-trip.
+   - **`enable_ssl: true` against a non-SSL database fails the
+     probe.** The flag looks client-side but is validated by the
+     probe attempting an SSL handshake. Match it to what the
+     target database actually supports.
+
+10. **`pool_size` is clamped to the inclusive range 1–50.** The
+    typed field accepts any integer; the server rejects values
+    outside `[1, 50]` with a `/pool_size` pointer. The error message
+    mentions a `greater_than: 0` or `less_than_or_equal_to: 50`
+    constraint depending on which side was crossed.
+
+11. **`timezone` is a closed IANA enum.** Submit a non-IANA string
+    (e.g. `"Bad/Timezone"` or `"EST"`) and the platform rejects
+    with `is not a valid IANA timezone` on `/timezone`. The set of
+    valid identifiers is not exposed over the SDK as a discovery
+    surface — consumers must source IANA tz database names
+    externally (e.g. JavaScript's `Intl.supportedValuesOf('timeZone')`,
+    or the IANA tz data directly).
+
+12. **`auth_method` defaults to `"password"` on every DB profile.**
+    Omitting `*_auth_method` or supplying `null` is equivalent to
+    `"password"` — and consequently requires both `*_user` and
+    `*_pass` to be supplied. The IAM-role relaxation in §2 applies
+    only when `auth_method` is explicitly `"iam_role"`.