@alvera-ai/platform-sdk 0.17.0 → 0.18.0

package/.agent/datalakes.md

@@ -1,17 +1,20 @@
 # Datalakes
 
-A **datalake** is the storage layer for one tenant slice. It holds
-two databases under one roof:
+A **datalake** is the storage layer for one tenant slice: one
+Postgres database — a writer profile and a reader profile onto it
+— plus one cloud-storage bucket.
 
-- an **unregulated** database for tokenized data, exposed to AI
-  workflows and downstream agents
-- a **regulated** database for raw values that never leave the
-  trust boundary
+Masking is **not** a second database under the same roof. A
+datalake carries a `type` (`raw`, `tokenized`, or `redacted`), and
+the derived tiers are *separate datalake rows* pointing back at the
+raw one through `primary_datalake_id`, filled by replication rather
+than written directly. §2b covers which tier to read, and why
+reading the wrong one is the subtlest bug in this guide.
 
 Every other resource in the platform (data sources, tools, AI
 agents, workflows, etc.) scopes under a datalake. A tenant can own
-many datalakes; the canonical pattern is one per data domain (e.g.
-`healthcare`, `foundation`, `payments`).
+many datalakes; group them however the tenant's data actually
+divides.
 
 ```typescript
 import type { PlatformApi } from '@alvera-ai/platform-sdk'
@@ -26,7 +29,7 @@ SDK namespace: `api.datalakes`.
 ## 1. Wire shape
 
 The request type splits across two layers: top-level database +
-identity fields, plus two polymorphic cloud-storage embeds.
+identity fields, plus one polymorphic cloud-storage embed.
 
 ```typescript
 import type {
@@ -41,8 +44,8 @@ there is no runtime validator; validation is server-authoritative.)
 
 ### Polymorphic cloud-storage embed
 
-`unregulated_cloud_storage` and `regulated_cloud_storage` are
-polymorphic. The discriminator field is **`cloud_storage_type`**
+`cloud_storage` is polymorphic. The discriminator field is
+**`cloud_storage_type`**
 (see `mutations.md` "Polymorphic discriminator naming") with three
 branches:
 
@@ -63,16 +66,17 @@ credentials the same way the DB-profile `auth_method` does. See
 §2 "AWS cloud-storage `auth_method = iam_role` forbids
 credentials" for the conditional the type cannot encode.
 
-The two embeds are independent — a datalake can pair an `aws`
-unregulated bucket with a `custom` regulated bucket. Both embeds
-ship in the same POST/PUT body.
+There is one embed, not a pair. The old
+`unregulated_cloud_storage` / `regulated_cloud_storage` pairing
+died with the two-tier lake — a derived lake is its own datalake
+row and carries its own `cloud_storage`.
 
 ### Cloud storage is also the cold-archive destination
 
 Beyond being the platform's primary blob store for inline payloads
 (image uploads, large context payloads, presigned-URL ingest), the
-configured `regulated_cloud_storage` and `unregulated_cloud_storage`
-buckets are the **cold-archive destination** for the datalake. As
+configured `cloud_storage` bucket is the **cold-archive
+destination** for the datalake. As
 data flows through the pipeline, two NDJSON streams land under
 well-defined key prefixes inside the bucket the credentials point at:
 
@@ -111,20 +115,13 @@ data lives in the cloud-storage buckets configured here.
 
 ### Canonical create body
 
-Paste-ready TypeScript body. The four database-role groups
-(unregulated writer + reader, regulated writer + reader) share
-an identical eight-field shape; the type lists all 32 explicitly
-with no role-level union:
+Paste-ready TypeScript body. There are two database-role groups —
+writer and reader — sharing an identical eight-field shape; the
+type lists all 16 explicitly with no role-level union.
 
 ```typescript
 import type { DatalakeRequestWritable } from '@alvera-ai/platform-sdk'
 
-// Pick one of four supported industries. The body shape below is
-// identical across them; only `data_domain` and the names it
-// interpolates differ.
-const dataDomain = 'healthcare'
-//                  ^ swap to 'foundation' | 'subscription' | 'payments'
-
 // Secrets resolved upstream by your code (secret store, config loader, etc.)
 const pgHost = '...'
 const pgUser = '...'
@@ -137,56 +134,40 @@ const awsSecretAccessKey = '...'
 
 const body: DatalakeRequestWritable = {
   name: 'Production Lake',
-  description: `Primary ${dataDomain} datalake`,
-  data_domain: dataDomain,
+  description: 'Primary datalake',
   timezone: 'America/New_York',
   pool_size: 5,
 
-  unregulated_db_writer_host: pgHost,
-  unregulated_db_writer_port: 5432,
-  unregulated_db_writer_name: `alvera_${dataDomain}`,
-  unregulated_db_writer_schema: 'tenant_acme_unreg',
-  unregulated_db_writer_auth_method: 'password',
-  unregulated_db_writer_user: pgUser,
-  unregulated_db_writer_pass: pgPass,
-  unregulated_db_writer_enable_ssl: true,
-  unregulated_db_reader_host: pgReaderHost,
-  unregulated_db_reader_port: 5432,
-  unregulated_db_reader_name: `alvera_${dataDomain}`,
-  unregulated_db_reader_schema: 'tenant_acme_unreg',
-  unregulated_db_reader_auth_method: 'password',
-  unregulated_db_reader_user: pgReaderUser,
-  unregulated_db_reader_pass: pgReaderPass,
-  unregulated_db_reader_enable_ssl: true,
-
-  regulated_data_db_writer_host: pgHost,
-  regulated_data_db_writer_port: 5432,
-  regulated_data_db_writer_name: `alvera_${dataDomain}`,
-  regulated_data_db_writer_schema: 'tenant_acme_reg',
-  regulated_data_db_writer_auth_method: 'password',
-  regulated_data_db_writer_user: pgUser,
-  regulated_data_db_writer_pass: pgPass,
-  regulated_data_db_writer_enable_ssl: true,
-  regulated_data_db_reader_host: pgReaderHost,
-  regulated_data_db_reader_port: 5432,
-  regulated_data_db_reader_name: `alvera_${dataDomain}`,
-  regulated_data_db_reader_schema: 'tenant_acme_reg',
-  regulated_data_db_reader_auth_method: 'password',
-  regulated_data_db_reader_user: pgReaderUser,
-  regulated_data_db_reader_pass: pgReaderPass,
-  regulated_data_db_reader_enable_ssl: true,
-
-  unregulated_cloud_storage: {
+  // `type` defaults to 'raw' — the lake the platform writes.
+  // A derived lake sets type: 'tokenized' | 'redacted' AND
+  // primary_datalake_id; see §2b before creating one.
+  type: 'raw',
+
+  db_writer_host: pgHost,
+  db_writer_port: 5432,
+  db_writer_name: 'alvera_production',
+  db_writer_schema: 'tenant_acme',
+  db_writer_auth_method: 'password',
+  db_writer_user: pgUser,
+  db_writer_pass: pgPass,
+  db_writer_enable_ssl: true,
+
+  db_reader_host: pgReaderHost,
+  db_reader_port: 5432,
+  db_reader_name: 'alvera_production',
+  db_reader_schema: 'tenant_acme',
+  db_reader_auth_method: 'password',
+  db_reader_user: pgReaderUser,
+  db_reader_pass: pgReaderPass,
+  db_reader_enable_ssl: true,
+
+  // One bucket. Scope multiple lakes onto a shared bucket with
+  // `base_path` rather than provisioning a bucket per lake.
+  cloud_storage: {
     cloud_storage_type: 'aws',
     region: 'us-east-1',
-    bucket: `acme-${dataDomain}-unregulated`,
-    access_key_id: awsAccessKeyId,
-    secret_access_key: awsSecretAccessKey,
-  },
-  regulated_cloud_storage: {
-    cloud_storage_type: 'aws',
-    region: 'us-east-1',
-    bucket: `acme-${dataDomain}-regulated`,
+    bucket: 'acme-platform',
+    base_path: 'production',
     access_key_id: awsAccessKeyId,
     secret_access_key: awsSecretAccessKey,
   },
@@ -226,16 +207,15 @@ embed:
   are NOT silently server-cleared — the platform treats their
   presence as a configuration conflict.
 
-The two embeds (`unregulated_cloud_storage` and
-`regulated_cloud_storage`) are validated independently, so a body
-can pair `iam_role` on one side with `access_key` on the other.
+There is a single embed, so this conditional is evaluated once
+per body.
 
 ### Database identifiers follow strict format rules
 
-The four `*_db_*_schema` and four `*_db_*_name` fields must be
+The two `db_*_schema` and two `db_*_name` fields must be
 lowercase identifiers (letters, digits, underscore) — uppercase
-letters and dashes are rejected with a `/<role>_schema` or
-`/<role>_name` pointer. The eight `*_db_*_host` fields must match
+letters and dashes are rejected with a `/db_<role>_schema` or
+`/db_<role>_name` pointer. Both `db_*_host` fields must match
 a hostname pattern; values like `"invalid_host!"` (with
 non-hostname punctuation) are rejected.
 
@@ -245,14 +225,15 @@ malformed identifier never gets as far as a connection attempt.
 
 ### Schema names must be unique within a database
 
-`*_db_*_schema` fields name the PostgreSQL schema the platform
+`db_*_schema` fields name the PostgreSQL schema the platform
 creates inside the target database. Sibling datalakes sharing a
-`*_db_*_name` (same physical database) **must** use distinct
+`db_*_name` (same physical database) **must** use distinct
 schema values. Collision produces a 422 at create time.
 
-The reader and writer for the same role typically share a schema
-(they're two connection profiles into the same logical schema);
-unregulated and regulated must not (different trust boundaries).
+The reader and writer normally share a schema — they are two
+connection profiles into the same logical schema. A derived
+(tokenized or redacted) lake is a separate datalake row and so
+brings its own schema; it must not collide with its raw parent's.
 
 ### The discriminator can't change on `update()`
 
@@ -261,13 +242,180 @@ existing row's discriminator. To change the embed type (e.g. `aws`
 → `r2`), `delete()` and `create()` fresh — you can't change body
 type on `update()`.
 
-### `data_domain` is a closed enum
+## 2b. The three lakes, and which one to read
+
+A datalake is not one database. It is three, and the other two are
+**derived**: fully managed, read-only, and written only by
+replication from the raw lake.
+
+```
+   raw          every column, every real value.        the superset.
+    │
+    ├── tokenized    columns declared privacy_requirement: "tokenize"
+    │                come back as a token; everything else passes through.
+    │
+    └── redacted     tokenize AND redact columns are scrubbed.
+```
+
+The tier is chosen per call, not per datalake — `mode` on
+`executeSql`, `dataAccessMode` on a dataset search. Nothing about
+the datalake decides it.
+
+### What decides what a reader sees
+
+**`privacy_requirement` on the column.** That single declaration is
+the whole masking policy — `tokenize` for PII, `redact` for free
+text, `none` to pass through. There is no second schema to write
+sensitive values into, and no changeset quietly masking on the way
+past: the raw value goes in, and the tier the reader asked for
+decides what comes out. Dropping or changing that declaration
+silently changes who can see what.
 
-Industry values are enumerated in the OpenAPI spec
-(`DatalakeDataDomain`); the SDK exports them as a const-style
-value (see `type_naming.md` "Enum types"). The enum drifts as
-new industries land — consult the SDK's `DatalakeDataDomain`
-const for the live set rather than hardcoding values.
+### Which tier to read
+
+**Read `raw` unless you have a reason not to.** It is the superset:
+every column is present with its real value, so any assertion about
+what landed — a row exists, a foreign key was stamped, a count is
+right — is answerable there. It also does not depend on replication
+having caught up, which matters more than it sounds like: a derived
+lake is populated asynchronously, so a read against one is a question
+about *replication* as much as about your data. Every read-back in
+the cookbook corpus reads `raw` for exactly this reason.
+
+**Read `tokenized` when the point IS the masking** — proving a
+`tokenize` column really is masked for a consumer that should not
+see it, or reproducing what an AI agent sees. An agent's
+`data_access` is a separate, stricter thing: it takes
+`'raw' | 'tokenized'` only, because an agent is never pointed at the
+redacted lake.
+
+**Read `redacted` when you are checking the strictest projection.**
+
+### Two traps
+
+**A derived lake is not there until it is provisioned.** A datalake
+is created as the raw tier alone; the tokenized and redacted copies
+are provisioned separately and migrate on their own jobs. A read
+against a tier that does not exist yet fails with a missing-relation
+error, not an empty result.
+
+**Filter and assert on a `none` column.** In `tokenized` a
+`tokenize` column returns its token, so a `WHERE` on one matches
+nothing and an equality assertion on one compares a token to a
+plaintext value. Pick a `none` column — a business key, a status —
+for anything you filter or assert on.
+
+### Has the copy actually arrived?
+
+Because a derived lake is populated asynchronously, "the copy
+exists" and "the copy is complete" are different questions.
+`derivedLakeTables` answers the second, per table.
+
+```typescript
+const { data: tables } = await api.datalakes.derivedLakeTables(
+  tenantSlug,
+  datalakeSlug,   // the PRIMARY lake — see the addressing note below
+  'tokenized',    // which copy: 'tokenized' | 'redacted'
+)
+
+for (const t of tables.data) {
+  // `caught_up` is the answer. `label` is the same state in words.
+  console.log(t.table, t.label, t.caught_up, t.resync_empties)
+}
+
+const behind = tables.data.filter((t) => !t.caught_up)
+```
+
+**You address a derived lake by naming its PRIMARY lake plus a
+mode.** A derived lake does have a slug of its own, but that slug is
+not an address: the lookup resolves primaries only and answers `404`
+for a derived slug deliberately. Passing the tokenized lake's own
+slug is not a working alternative spelling.
+
+**Read `caught_up`, never the state's name.** There are five states,
+and one of them reads as finished without being finished:
+
+| `state` | `caught_up` | what it means |
+|---|---|---|
+| `waiting_to_start` | `false` | queued |
+| `copying` | `false` | first pass running |
+| `finished_copy` | **`false`** | first pass done — changes made *during* it are NOT applied yet |
+| `catching_up` | `false` | applying those changes |
+| `caught_up` | **`true`** | arrived |
+
+`finished_copy` is how a lake gets called ready while rows are still
+missing. Only `caught_up` means arrived, and the boolean is there so
+you never have to pattern-match the word.
+
+`label` is that same state in words, identical to what the
+platform's own Tokenization screen shows. Use it verbatim rather
+than inventing a second vocabulary — a table reading "Copying" in
+one surface must not read something else in another.
+
+`resync_empties` says what repairing *that* table would empty, so a
+surface can state the cost before offering the action. It is `null`
+for tables already caught up — the platform does not spend a query
+per table answering a question about tables nobody will repair — and
+an array otherwise, with the named table first. **`null` and `[]`
+are different answers**, and `[]` never occurs, since a table is
+always in its own set. Type it as optional-nullable, not as an empty
+array default.
+
+### Repairing one table
+
+When a table is stuck or wrong, `resyncTable` empties it in the copy
+and copies it again from the start.
+
+```typescript
+const { data: repair } = await api.datalakes.resyncTable(
+  tenantSlug, datalakeSlug, 'tokenized',
+  { table: 'legal_entities' },
+)
+
+repair.emptied  // EVERY table this emptied — often more than the one asked for
+repair.status   // always 'copying'
+
+// 202 = started. There is no completion callback, so poll.
+let done = false
+while (!done) {
+  const { data: after } = await api.datalakes.derivedLakeTables(
+    tenantSlug, datalakeSlug, 'tokenized',
+  )
+  done = after.data.some((t) => t.table === repair.table && t.caught_up)
+}
+```
+
+**It can empty more than the table you named.** Postgres refuses to
+truncate any table with an inbound foreign key — whether or not the
+referencing tables hold rows — so repairing a parent empties its
+whole dependent closure, transitively, in one statement. All of them
+are re-copied by the same run. On real data one request has reached
+nine tables. `emptied` is returned rather than assumed for exactly
+this reason: a caller that emptied nine tables should be told nine.
+Read `resync_empties` first if you are going to ask someone to
+confirm.
+
+**`202` means started, not done.** The repair empties the table and
+a fresh first pass begins; it finishes later. Poll
+`derivedLakeTables` until that table reads `caught_up`.
+
+**It does not roll back.** A failure part-way leaves the table empty
+and says so. That is deliberate — the rows to put back are the ones
+that broke the copy in the first place. The recovery is to call it
+again once the cause is addressed; the endpoint handles the case
+where a previous attempt left the table off the publication.
+
+**The table name is gated.** `table` must be one the copy carries or
+one the original can send. Anything else is a `422` and **nothing is
+emptied** — that gate is what stands between a typo and an emptied
+table. Read `derivedLakeTables` for the list rather than
+constructing names.
+
+The `422` bodies here are sentences written for a person, not codes
+— a lake with no such copy, a name that is neither carried nor
+sendable, a dependent that could be emptied but not copied back
+(ending `Nothing has been emptied.`). Surface them; do not parse
+them.
 
 ## 3. Field ownership
 
@@ -285,14 +433,16 @@ resources (see §5).
 `DatalakeRequestWritable` and `DatalakeResponse`:
 
 ```
-identification         name, description, data_domain
+identification         name, description
+tiering                type, primary_datalake_id
 global settings        timezone, pool_size
-db profiles (×4)       <role>_{host, port, name, schema, user,
-                               auth_method, enable_ssl}
-cloud storage (×2)     <role>_cloud_storage.{cloud_storage_type,
-                                             bucket, region,
-                                             + branch-specific
-                                             fields per §1}
+db profiles (×2)       db_{writer,reader}_{host, port, name,
+                               schema, user, auth_method,
+                               enable_ssl}
+cloud storage          cloud_storage.{cloud_storage_type, bucket,
+                                      region, base_path,
+                                      + branch-specific fields
+                                      per §1}
 ```
 
 **Write-only (Request-only).** Present on
@@ -301,15 +451,11 @@ consumer must re-supply these on every PUT from their secret
 store (see §5 Update for the canonical pattern):
 
 ```
-db credentials (×4)    unregulated_db_writer_pass
-                       unregulated_db_reader_pass
-                       regulated_data_db_writer_pass
-                       regulated_data_db_reader_pass
-
-cloud storage          unregulated_cloud_storage.access_key_id
-  credentials (×4)     unregulated_cloud_storage.secret_access_key
-                       regulated_cloud_storage.access_key_id
-                       regulated_cloud_storage.secret_access_key
+db credentials (×2)    db_writer_pass
+                       db_reader_pass
+
+cloud storage          cloud_storage.access_key_id
+  credentials (×2)     cloud_storage.secret_access_key
 ```
 
 The `slug` returned in the Response is the canonical reference
@@ -332,13 +478,14 @@ Datalake-specific common rejections:
 | `source.pointer`                                | Typical cause                                        |
 |-------------------------------------------------|------------------------------------------------------|
 | `/name`                                         | name collision within tenant (uniqueness)            |
-| `/data_domain`                                  | value not in the live enum                           |
-| `/unregulated_cloud_storage/cloud_storage_type` | discriminator value doesn't match any oneOf branch   |
-| `/unregulated_db_writer_schema`                 | schema-name collision within the target DB; or identifier format violation (uppercase / dash) |
-| `/unregulated_db_writer_name`                   | identifier format violation                          |
-| `/unregulated_db_writer_host`                   | hostname format violation, OR synchronous probe could not reach the host |
-| `/unregulated_cloud_storage`                    | synchronous probe could not reach the bucket with the supplied credentials |
-| `/unregulated_cloud_storage/access_key_id`      | supplied under `auth_method: "iam_role"` (must be omitted) |
+| `/type`                                         | not one of `raw` / `tokenized` / `redacted`          |
+| `/primary_datalake_id`                          | missing on a derived lake, or supplied on a raw one  |
+| `/cloud_storage/cloud_storage_type`             | discriminator value doesn't match any oneOf branch   |
+| `/db_writer_schema`                             | schema-name collision within the target DB; or identifier format violation (uppercase / dash) |
+| `/db_writer_name`                               | identifier format violation                          |
+| `/db_writer_host`                               | hostname format violation, OR synchronous probe could not reach the host |
+| `/cloud_storage`                                | synchronous probe could not reach the bucket with the supplied credentials |
+| `/cloud_storage/access_key_id`                  | supplied under `auth_method: "iam_role"` (must be omitted) |
 | `/pool_size`                                    | out of the inclusive range 1–50                      |
 | `/timezone`                                     | not in the closed 8-zone US whitelist (valid IANA values like `UTC` are also rejected — see gotcha 11) |
 
@@ -470,17 +617,10 @@ const next = {
   ...current,
 
   // Re-supply every write-only field from your secret store:
-  unregulated_db_writer_pass: pgPass,
-  unregulated_db_reader_pass: pgReaderPass,
-  regulated_data_db_writer_pass: pgPass,
-  regulated_data_db_reader_pass: pgReaderPass,
-  unregulated_cloud_storage: {
-    ...current.unregulated_cloud_storage,
-    access_key_id: awsAccessKeyId,
-    secret_access_key: awsSecretAccessKey,
-  },
-  regulated_cloud_storage: {
-    ...current.regulated_cloud_storage,
+  db_writer_pass: pgPass,
+  db_reader_pass: pgReaderPass,
+  cloud_storage: {
+    ...current.cloud_storage,
     access_key_id: awsAccessKeyId,
     secret_access_key: awsSecretAccessKey,
   },
@@ -492,7 +632,7 @@ const next = {
 const { data: updated } = await api.datalakes.update(
   tenantSlug,
   datalakeId,
-  next,
+  next as unknown as DatalakeRequestWritable,   // see the note below
 )
 // updated.id          === datalakeId    — path-key stable
 // updated.description === 'Now with revised description'
@@ -500,12 +640,12 @@ const { data: updated } = await api.datalakes.update(
 
 The canonical pattern: caller re-supplies the write-only DB +
 cloud-storage fields verbatim, mutates only what changes (e.g.
-`description`). The eight write-only fields are enumerated in
+`description`). The four write-only fields are enumerated in
 §3 Field ownership.
 
 **`update()` fires server-side connectivity probes before persisting.**
 Submitting a body produces a successful 200 only after two
-probes pass for each role profile:
+probes pass:
 
 - **Cloud-storage probe** — verifies the bucket exists + the
   supplied `access_key_id` / `secret_access_key` (or IAM role)
@@ -517,8 +657,8 @@ probes pass for each role profile:
 A body that's structurally valid but carries stale or wrong
 credentials therefore fails at `update()` time, not at later
 write time. The 422 envelope (see `errors.md`) names the failing
-probe via `source.pointer` (e.g. `/unregulated_cloud_storage` or
-`/regulated_data_db_writer_host`). This makes drift-then-update
+probe via `source.pointer` (e.g. `/cloud_storage` or
+`/db_writer_host`). This makes drift-then-update
 workflows fail fast and locally rather than mid-pipeline.
 
 Most field changes do NOT re-trigger migration. Schema-name or
@@ -548,7 +688,7 @@ returning a different shape:
 | `.get(tenantSlug, id)`               | One row: `DatalakeResponse` — UUID only (slug 422s)         |
 | `.metadata(tenantSlug, slug)`        | **Markdown string** describing the lake                     |
 | `.indexMetadata(tenantSlug)`         | **Markdown string** listing every lake                      |
-| `.systemDatasets(tenantSlug, slug)`  | `{ datasets: string[] }` — industry-built dataset names     |
+| `.systemDatasets(tenantSlug, slug)`  | `{ datasets: string[] }` — platform dataset names             |
 
 The two `metadata` methods return `string`, not a structured
 object — they're agent-facing summaries (`data: string`, not `data:
@@ -559,14 +699,14 @@ tenant` and lists every datalake by slug — useful for agent-side
 discovery without paging through `.list`.
 
 `.systemDatasets(...)` enumerates the dataset names the platform
-auto-creates for the datalake's `data_domain`. Each industry's
-catalog mixes **domain-anchored names** (the primary subject of
-the industry plus related entities) with **cross-cutting concepts**
-(audit logs,
-messages, documents, etc.) that the platform composes per
-industry — agents iterating the catalog should not hardcode a
-closed set of expected names; discover them at runtime via
-this call. The array is sorted, unique, and **excludes
+auto-creates for the datalake. GH-859 removed the per-industry
+catalogs: every datalake now gets the same platform-defined set
+(`legal_entity`, `message`, `action_log`, `document`,
+`beneficial_owner`), and the subject-specific tables an industry
+used to bring are modelled as generic tables instead. Still
+discover the catalog at runtime rather than hardcoding it —
+the set is server-owned and this call is the only authority on
+it. The array is sorted, unique, and **excludes
 operator-defined generic tables** — those live under
 `api.genericTables.list(...)`. The
 literal placeholder `"generic_table"` is also absent. Each name
@@ -575,7 +715,7 @@ can be drilled into via
 agent-facing markdown, or the whole-domain catalog can be fetched
 in one shot via `api.datasets.metadata(tenantSlug, datalakeSlug)`. Pair `.systemDatasets()` with a
 `page_size: 1` probe per name as a post-migration smoke (proves
-each industry-built table is queryable; the post-ready probe
+each platform-built table is queryable; the post-ready probe
 code is in §8 Gotcha 3).
 
 ```typescript
@@ -583,7 +723,7 @@ const { data: catalog } = await api.datalakes.systemDatasets(
   tenantSlug,
   datalakeSlug,
 )
-// catalog.datasets: sorted, unique array of industry-built dataset names
+// catalog.datasets: sorted, unique array of platform dataset names
 
 for (const name of catalog.datasets) {
   const { data: md } = await api.datasets.metadataDetails(tenantSlug, datalakeSlug, name)
@@ -602,8 +742,8 @@ the SQL between the two calls.
 | `.textToSql(tenantSlug, datalakeSlug, body)`               | `{ sql, model, provider, explanation }`                   |
 | `.executeSql(tenantSlug, datalakeSlug, body, options?)`    | `{ data, meta }` — or a CSV **string** with `{ format: 'csv' }` |
 
-- **`textToSql`** body is `{ prompt, mode }` (`mode` ∈ `'regulated' | 'unregulated'`,
-  selecting which schema to target). It runs ordered multi-provider LLM failover and
+- **`textToSql`** body is `{ prompt, mode }` (`mode` ∈ `'raw' | 'tokenized' |
+  'redacted'`, selecting which lake tier to target — see §2b). It runs ordered multi-provider LLM failover and
   returns the generated `sql`, the winning `model` + `provider`, and a best-effort
   `explanation` (`null` when the explainer is unavailable). **Only the prompt + the
   schema cross the LLM boundary — never datalake rows**, so it is safe in both modes.
@@ -612,22 +752,40 @@ the SQL between the two calls.
   live — e.g. `WHERE some_boolean = 1` 422s on Postgres (booleans compare to
   `true`/`false`, never `1`). Surface the SQL to a human (or a checking step)
   and expect to edit before `executeSql`.
+- **`executeSql` returns rows as POSITIONAL ARRAYS, not objects keyed by
+  column.** `data` is `unknown[][]`; the column names are in `meta.columns`, in
+  the same order. So `row.legal_entity_id` is silently `undefined` — not an
+  error, just absent — and a probe written against it fails with a message
+  built from its own bug while the data is perfectly correct. Zip them:
+
+  ```ts
+  const { data: res } = await api.datalakes.executeSql(tenantSlug, datalakeSlug, {
+    sql, mode: 'raw',
+  })
+  const cols = res.meta.columns
+  const rows = res.data.map((r) => Object.fromEntries(cols.map((c, i) => [c, r[i]])))
+  // now rows[0].legal_entity_id is real
+  ```
+
+  Worth knowing before you write a richer check than the scaffolds ship: their
+  seeded example only asserts `data.length > 0`, which behaves identically for
+  both shapes, so nothing in the example reveals this.
 - **`executeSql`** body is `{ sql, mode, page?, page_size? }`. It runs the SQL
-  **read-only** (INSERT/UPDATE/DELETE/DDL are rejected) on the mode-appropriate
-  schema, with Flop-inspired pagination (`page_size` capped server-side). The JSON
+  **read-only** (INSERT/UPDATE/DELETE/DDL are rejected) against the
+  mode-selected lake tier, with Flop-inspired pagination (`page_size` capped server-side). The JSON
   shape is `{ data, meta }`; pass `options = { format: 'csv' }` to get the page as a
   CSV string attachment instead.
 
 ```typescript
 const { data: gen } = await api.datalakes.textToSql(tenantSlug, datalakeSlug, {
-  prompt: 'count contacts created this month',
-  mode: 'unregulated',
+  prompt: 'count legal entities created this month',
+  mode: 'raw',
 })
 // gen.sql — review / edit before running
 
 const { data: page } = await api.datalakes.executeSql(tenantSlug, datalakeSlug, {
   sql: gen.sql,
-  mode: 'unregulated',
+  mode: 'raw',
   page: 1,
   page_size: 100,
 })
@@ -636,7 +794,7 @@ const { data: page } = await api.datalakes.executeSql(tenantSlug, datalakeSlug,
 
 const { data: csv } = await api.datalakes.executeSql(
   tenantSlug, datalakeSlug,
-  { sql: gen.sql, mode: 'unregulated' },
+  { sql: gen.sql, mode: 'raw' },
   { format: 'csv' },
 ) // csv is a string, not the JSON envelope
 ```
@@ -646,15 +804,26 @@ const { data: csv } = await api.datalakes.executeSql(
 The name you author is not the name you query. Datasets are addressed
 **logically** everywhere in a manifest or SDK call, and **physically** in SQL:
 
-| You author | You query in SQL | Schema |
-|---|---|---|
-| `message` | `regulated_messages` | `app_regulated` |
-| `action_log` | `action_logs` | `app_unregulated` |
-| a generic table `clients` | `regulated_alvera_custom_clients` | `app_regulated` |
+For a **generic table**, the physical name is the `name` the server
+returns from `genericTables.create` / `.get` — query it directly, with
+no prefix and no schema qualifier:
+
+```typescript
+const { data: table } = await api.genericTables.create(tenantSlug, datalakeSlug, body)
+await api.datalakes.executeSql(tenantSlug, datalakeSlug, {
+  sql: `SELECT * FROM ${table.name} LIMIT 1`,
+  mode: 'raw',
+})
+```
+
+Do not reconstruct that name from the slug you authored — the server
+derives it, and the derivation is not part of the contract.
 
-Custom (generic) tables carry an `alvera_custom_` prefix, and the regulated side
-adds a further `regulated_` prefix on top. The schema follows `execute-sql`'s
-`--mode` (`regulated` → `app_regulated`, `unregulated` → `app_unregulated`).
+For the **platform datasets** (`message`, `action_log`, and the rest of
+§5's `systemDatasets` catalog) the physical name and schema are likewise
+server-owned. Read them, don't assume them: the physical mapping moved
+with the GH-859 tier rename and the old `app_regulated` /
+`app_unregulated` schema pair no longer follows from `mode`.
 
 **Don't derive these by hand — read them.** Guessing column and table names is
 how six turns get spent on `column "error_message" does not exist` (the real
@@ -695,10 +864,10 @@ rows downstream reveal the mismatch. Check the derivation, not the metadata.
    const { data: created } = await api.datalakes.create(tenantSlug, body)
    // created.status === 'new'
 
-   await api.datalakes.migrate(tenantSlug, created.slug)   // migrate is slug-keyed
+   await api.datalakes.migrate(tenantSlug, created.slug!)  // migrate is slug-keyed
 
    const ready = await waitUntilReady(
-     () => api.datalakes.get(tenantSlug, created.id),      // get is UUID-id-keyed
+     () => api.datalakes.get(tenantSlug, created.id!),     // get is UUID-id-keyed
      (s) => s === 'ready' ? 'ready' : 'pending',
      { intervalMs: 15_000, timeoutMs: 5 * 60_000 },
    )
@@ -722,7 +891,7 @@ rows downstream reveal the mismatch. Check the derivation, not the metadata.
    const { data: catalog } = await api.datalakes.systemDatasets(
      tenantSlug, datalakeSlug,
    )
-   // catalog.datasets is the industry-built name list —
+   // catalog.datasets is the platform dataset name list —
    // discovered at runtime; don't hardcode names.
 
    for (const dataset of catalog.datasets) {
@@ -755,11 +924,10 @@ rows downstream reveal the mismatch. Check the derivation, not the metadata.
    a required string. Supply a sentinel literal (e.g.
    `"unused-iam-role"`); the platform clears it server-side.
 
-7. **`cloud_storage_type` lives inside each embed, not at the
-   top level.** A single body has two discriminators
-   (`unregulated_cloud_storage.cloud_storage_type` and
-   `regulated_cloud_storage.cloud_storage_type`); they're
-   independently validated and can carry different values.
+7. **`cloud_storage_type` lives inside the embed, not at the
+   top level.** The discriminator is
+   `cloud_storage.cloud_storage_type` — one per body, since a
+   datalake carries one bucket.
 
 8. **`delete()` is rare in practice.** Datalakes accrete dependents
    quickly; routine cleanup is via operator-driven console
@@ -768,15 +936,14 @@ rows downstream reveal the mismatch. Check the derivation, not the metadata.
 
 9. **Create and update run synchronous reachability probes.** Before
    the row is written, the platform opens connections to every
-   declared endpoint — all four DB profiles (unregulated reader +
-   writer, regulated reader + writer) AND both cloud-storage
-   buckets — and only proceeds if every probe succeeds. A
-   structurally valid body can still fail with field-level
-   rejections on `*_db_*_host` (unreachable, wrong credentials, or
-   SSL-mismatch) or `*_cloud_storage` (bucket inaccessible). The
-   same probe runs on update; an update that only changes the
-   datalake's `name` still re-probes if the body re-supplies any
-   `*_host` or `*_cloud_storage` field.
+   declared endpoint — both DB profiles (reader and writer) AND the
+   cloud-storage bucket — and only proceeds if every probe
+   succeeds. A structurally valid body can still fail with
+   field-level rejections on `db_*_host` (unreachable, wrong
+   credentials, or SSL-mismatch) or `cloud_storage` (bucket
+   inaccessible). The same probe runs on update; an update that
+   only changes the datalake's `name` still re-probes if the body
+   re-supplies any `db_*_host` or `cloud_storage` field.
 
    Two consumer-relevant consequences: