@alvera-ai/platform-sdk 0.11.0 → 0.12.1-next.f7f8a3c

package/.agent/AGENTS.md

@@ -357,7 +357,7 @@ The `cookbook/` directory holds two kinds of recipe, both validated by
   an outcome.
 - **Capability docs** (`<capability>.md`) — one platform capability each,
   shown as the minimal call sequence that proves it (e.g. `bulk-ingest`,
-  `ai-agent-invoke`, `custom-tables`, `invite-team`). Read these to learn one
+  `ai-agent-invoke`, `generic-tables`, `invite-team`). Read these to learn one
   capability in isolation; cookbooks weave them into outcomes.
 
 ### Available cookbooks
@@ -430,14 +430,18 @@ full wire shape.
   document/image and pull structured JSON out of it (`aiAgents.invoke` with
   files). The correct file-vision path — the agent reads the file; the DAC
   ingests it.
-- [custom-tables](./cookbook/custom-tables.md) — stand up a custom table the
-  built-in datasets don't model; deploy → ingest → search.
+- [generic-tables](./cookbook/generic-tables.md) — stand up a generic table the
+  built-in datasets don't model; deploy → ingest → read back via executeSql.
 - [action-status-updaters](./cookbook/action-status-updaters.md) — reconcile
   the delivery status of messages you send, on a schedule.
 - [system-templates](./cookbook/system-templates.md) — discover the platform's
   built-in row-mapping Liquid templates.
 - [invite-team](./cookbook/invite-team.md) — invite a teammate into your tenant
   (root / tenantless / tenant-scoped sessions in one flow).
+- [talk-to-data](./cookbook/talk-to-data.md) — turn a datalake conversational:
+  natural language → reviewable SQL (`datalakes.textToSql`, data-free by
+  construction) → read-only execution returning a `{ data, meta }` row page or a
+  CSV export (`datalakes.executeSql`).
 
 ## Utility namespaces
 

package/.agent/ai_sandbox.md

@@ -179,6 +179,31 @@ via consumer-authored fragments is structurally impossible: the
 fragment is the WHERE *expression*, with values supplied via
 positional bindings the platform builds.
 
+### Datalake "talk to data" — `textToSql` + `executeSql`
+
+The datalake BI surface (`api.datalakes.textToSql` /
+`api.datalakes.executeSql`, see `datalakes.md` "Talk to data") is
+the **full-statement** sibling of the WHERE-clause lane above, and
+it sits inside the same Layer-3 boundary:
+
+- **Generation is data-free by construction.** `textToSql` sends
+  only the natural-language `prompt` plus the datalake **schema**
+  to the LLM — never rows. The model returns SQL for review; no
+  datalake data ever crosses the LLM boundary, so generation is
+  safe in both `regulated` and `unregulated` mode. The two-call
+  split (generate, then execute) exists so a human or agent can
+  inspect and edit the SQL before any data is read.
+- **Execution is read-only and mode-routed.** `executeSql` runs the
+  statement against the mode-appropriate schema with a read-only
+  connection: INSERT/UPDATE/DELETE/DDL are rejected (app-layer
+  `@deny` + a read-only DB transaction + an EXPLAIN preflight), and
+  `mode` (`regulated` vs `unregulated`) plus the session's tenant
+  (RLS) decide which schema and which rows are reachable — the same
+  Layer-1 compliance gates that govern every other read.
+- **Pagination caps blast radius.** `page` / `page_size` map to a
+  server-side `LIMIT/OFFSET` window capped at the platform default,
+  so a single call cannot exfiltrate an unbounded result set.
+
 ## How the three layers compose
 
 ```

package/.agent/cookbook/{custom-tables.md → generic-tables.md}

@@ -1,8 +1,8 @@
 ---
-title: "Capability: stand up a custom table the platform doesn't model"
-summary: A capability walk for generic tables. Create a custom table with typed, privacy-tagged columns (`api.genericTables.create`), wait for it to deploy, use the default ingestion client the platform auto-provisions for it, ingest a row (`api.dataActivationClients.ingest`), and read it back by its unique key (`api.datasets.search`). For data the built-in datasets don't cover.
+title: "Capability: stand up a generic table the platform doesn't model"
+summary: A capability walk for generic tables. Create a generic table with typed, privacy-tagged columns (`api.genericTables.create`), wait for it to deploy, use the default ingestion client the platform auto-provisions for it, ingest a row (`api.dataActivationClients.ingest`), and read it back with read-only SQL (`api.datalakes.executeSql`). For data the built-in datasets don't cover.
 industry: accounts_receivable
-slug: custom-tables
+slug: generic-tables
 vitest_source:
   - integration-tests/tests/accounts_receivable/generic-tables.test.ts
   - integration-tests/tests/accounts_receivable/bootstrap.test.ts
@@ -23,7 +23,7 @@ The lifecycle is:
    **auto-provisions a default Data Activation Client** bound to an identity
    contract — no tool, contract, or DAC to wire by hand.
 3. `dataActivationClients.ingest(...)` a row through that default client.
-4. `datasets.search(...)` it back by its unique column.
+4. `datalakes.executeSql(...)` reads it back by its unique column with read-only SQL.
 
 This is **global** to every datalake — only the table's domain meaning differs.
 See `generic_tables.md` for the column reference.
@@ -33,7 +33,7 @@ See `generic_tables.md` for the column reference.
 The `_setup/accounts_receivable.md` bootstrap left `api`, `tenantSlug`, and
 `datalakeSlug` populated.
 
-## 001 — list existing custom tables (sanity)
+## 001 — list existing generic tables (sanity)
 
 `genericTables.list` returns the paginated `{ data, meta }` envelope. A fresh
 datalake has none; the call just confirms the surface is reachable.
@@ -45,7 +45,7 @@ if (!Array.isArray(existing.data)) {
 }
 ```
 
-## 002 — create the custom table
+## 002 — create the generic table
 
 Declare the table with a `title` and `columns`. Each column has a `type`, an
 `is_unique` flag, and — load-bearing — a `privacy_requirement` that decides how
@@ -115,7 +115,7 @@ while (Date.now() < deadline && !defaultDacSlug) {
   if (!defaultDacSlug) await new Promise((r) => setTimeout(r, 1_000))
 }
 if (!defaultDacSlug) {
-  throw new Error('default DAC for the custom table was not provisioned within 30s')
+  throw new Error('default DAC for the generic table was not provisioned within 30s')
 }
 ctx.defaultDacSlug = defaultDacSlug
 ```
@@ -144,34 +144,34 @@ if (!ingest.batch_id) {
 }
 ```
 
-## 006 — read the row back by its unique key
+## 006 — read the row back with read-only SQL
 
-Search the `generic_table` dataset, scoped to this table via `generic_table_id`,
-filtering on the regulated table name (`regulated_<table_name>`) and the unique
-column. `createUserSearch` compiles the query; `search` returns rows once
-ingestion has landed — poll until the row appears.
+`executeSql` runs a read-only statement against the datalake and returns a page
+as `{ data, meta }` — `data` is an array-of-arrays aligned positionally to
+`meta.columns`. `mode: 'unregulated'` runs it against the unregulated schema, so
+you reference the table by its server-derived `name` (the `alvera_custom_…`
+value from §002 — `mode` selects the schema, so no `regulated_`/`unregulated_`
+prefix). Ingestion is async, so poll until the row lands.
 
 ```typescript
-const { data: userSearch } = await api.datasets.createUserSearch(tenantSlug, datalakeSlug, 'generic_table', {
-  search_query: `regulated_${ctx.tableName}.submission_id = '${ctx.submissionId}'`,
-  generic_table_id: genericTableId,
-})
-if (userSearch.status !== 'completed') {
-  throw new Error(`user-search did not compile: ${userSearch.status}`)
-}
-
 const deadline = Date.now() + 45_000
-let rows: Array<Record<string, unknown>> = []
-while (Date.now() < deadline && rows.length === 0) {
-  const { data: page } = await api.datasets.search(tenantSlug, datalakeSlug, 'generic_table', {
-    userSearchId: userSearch.id!,
-    dataAccessMode: 'unregulated',
+let row: Record<string, unknown> | undefined
+while (Date.now() < deadline && !row) {
+  const { data: result } = await api.datalakes.executeSql(tenantSlug, datalakeSlug, {
+    sql: `SELECT submission_id, customer_name, source_channel
+          FROM ${ctx.tableName}
+          WHERE submission_id = '${ctx.submissionId}'`,
+    mode: 'unregulated',
   })
-  rows = (page.data ?? []) as Array<Record<string, unknown>>
-  if (rows.length === 0) await new Promise((r) => setTimeout(r, 1_000))
+  // JSON envelope (a `string` only for `{ format: 'csv' }`); zip the positional
+  // row back into an object keyed by meta.columns.
+  if (typeof result !== 'string' && result.data.length > 0) {
+    row = Object.fromEntries(result.meta.columns.map((c, i) => [c, result.data[0][i]]))
+  }
+  if (!row) await new Promise((r) => setTimeout(r, 1_000))
 }
-if (!rows.some((r) => r.submission_id === ctx.submissionId)) {
-  throw new Error('ingested row not found in the custom table within 45s')
+if (!row || row.submission_id !== ctx.submissionId) {
+  throw new Error('ingested row not found in the generic table within 45s')
 }
 ```
 
@@ -182,15 +182,18 @@ if (!rows.some((r) => r.submission_id === ctx.submissionId)) {
   free text is scrubbed, `none` passes through. Dropping it silently changes who
   can see what.
 - **The table name is server-derived, never set by you.** The platform slugifies
-  the title and prefixes `alvera_custom_`. Read `name` off the create response;
-  the search query uses `regulated_<name>`.
+  the title and prefixes `alvera_custom_`. Read `name` off the create response and
+  reference it directly in `executeSql`; `mode` selects the schema, so you don't
+  add a `regulated_`/`unregulated_` prefix yourself.
 - **You don't create the ingestion client.** Deploying the table auto-provisions
   a default DAC + identity contract. Find it by name; don't build one.
-- **Two async waits.** Wait for `:deployed` before ingesting, then poll the
-  search until the row materializes — ingestion is async like every other
+- **Two async waits.** Wait for `:deployed` before ingesting, then poll
+  `executeSql` until the row materializes — ingestion is async like every other
   ingest path.
-- **`generic_table_id` on the search is required.** It routes the query to your
-  table's dataset; omitting it searches the wrong scope.
+- **`mode` picks the schema, and `tokenize` columns come back masked.** In
+  `mode: 'unregulated'` the read runs against the tokenized view, so filter and
+  assert on a `none` column (here `submission_id`); a `tokenize` column like
+  `customer_name` returns its token, not the raw value.
 
 # See also
 

package/.agent/cookbook/talk-to-data.md

@@ -0,0 +1,139 @@
+---
+title: "Capability: talk to your datalake — natural language → SQL → rows"
+summary: A capability walk for datalake text-to-SQL. Generate a read-only SQL statement from a natural-language prompt (`api.datalakes.textToSql`) — only the prompt + schema cross the LLM boundary, never rows — then run it read-only and read back the page (`api.datalakes.executeSql`), with both the JSON `{ data, meta }` envelope and a CSV export. The BI surface on a datalake.
+industry: foundation
+slug: talk-to-data
+vitest_source:
+  - integration-tests/tests/foundation/text-to-sql.test.ts
+status: green
+---
+
+# Capability
+
+**What you get:** a two-call BI surface on a datalake — describe what you want in
+plain language, get back reviewable SQL, then run it read-only and read the rows.
+
+The split is deliberate: `textToSql` *generates*, `executeSql` *runs*. A UI or
+agent can show the SQL, let a human edit it, and re-run — and the generate step
+is **data-free by construction** (only the prompt + the datalake schema reach the
+LLM, never rows), so it is safe in both `regulated` and `unregulated` mode.
+
+1. `datalakes.textToSql(...)` — natural language → `{ sql, model, provider, explanation }`.
+2. `datalakes.executeSql(...)` — read-only execution → `{ data, meta }` (data is an
+   array-of-arrays row page aligned to `meta.columns`), or a CSV string with
+   `{ format: 'csv' }`.
+
+See `datalakes.md` "Talk to data" for the wire reference and `ai_sandbox.md`
+Layer 3 for why the boundary holds.
+
+# Walkthrough
+
+The `_setup/foundation.md` bootstrap left `api`, `tenantSlug`, and `datalakeSlug`
+populated, pointing at a freshly-migrated foundation datalake.
+
+## 001 — generate SQL from a natural-language prompt
+
+`textToSql` runs ordered multi-provider LLM failover and returns the generated
+`sql` plus the winning `model` / `provider` and a best-effort `explanation`
+(`null` when the explainer is unavailable). It does **not** execute the SQL — it
+hands it back for review. Nothing but the prompt and the datalake schema is sent
+to the model, so no datalake rows leave the trust boundary.
+
+```typescript
+const { data: gen } = await api.datalakes.textToSql(tenantSlug, datalakeSlug, {
+  prompt: 'how many legal entities are there?',
+  mode: 'unregulated',
+})
+if (typeof gen.sql !== 'string' || gen.sql.trim() === '') {
+  throw new Error(`textToSql returned no SQL (got: ${JSON.stringify(gen.sql)})`)
+}
+if (typeof gen.model !== 'string' || typeof gen.provider !== 'string') {
+  throw new Error('textToSql response missing model/provider attribution')
+}
+// gen.explanation is best-effort plain-language prose, or null.
+```
+
+## 002 — run a read-only query and read back the page
+
+`executeSql` runs the SQL **read-only** (INSERT/UPDATE/DELETE/DDL are rejected)
+and returns `{ data, meta }`. `data` is an **array-of-arrays** row page — positional,
+aligned to `meta.columns`, because arbitrary SQL can have duplicate / expression
+column names. `meta` carries the structural + pagination fields. A deterministic
+`SELECT 1 AS n` keeps this step independent of the generated SQL and seeded data.
+
+The curated return is `ExecuteSqlResponse | string` (the CSV branch is a string),
+so narrow with a `typeof` guard before reading the JSON envelope.
+
+```typescript
+const jsonResult = await api.datalakes.executeSql(tenantSlug, datalakeSlug, {
+  sql: 'SELECT 1 AS n',
+  mode: 'unregulated',
+})
+if (typeof jsonResult.data === 'string') {
+  throw new Error('expected the JSON envelope, got a CSV string')
+}
+const page = jsonResult.data
+if (!Array.isArray(page.data) || !Array.isArray(page.data[0])) {
+  throw new Error('executeSql data is not an array-of-arrays')
+}
+if (page.meta.columns[0] !== 'n' || page.data[0][0] !== 1 || page.meta.num_rows !== 1) {
+  throw new Error(`unexpected SELECT 1 result: ${JSON.stringify(page)}`)
+}
+// page.meta also carries total_count, page, page_size, total_pages, duration_ms, command.
+```
+
+## 003 — export the same page as CSV
+
+Pass `{ format: 'csv' }` to get the page as a CSV **string** instead of the JSON
+envelope — the same read, content-negotiated for download.
+
+```typescript
+const csvResult = await api.datalakes.executeSql(
+  tenantSlug,
+  datalakeSlug,
+  { sql: 'SELECT 1 AS n', mode: 'unregulated' },
+  { format: 'csv' },
+)
+if (typeof csvResult.data !== 'string') {
+  throw new Error('expected a CSV string for { format: "csv" }')
+}
+const csv = csvResult.data
+if (!/\bn\b/.test(csv) || !csv.includes('1')) {
+  throw new Error(`unexpected CSV body: ${JSON.stringify(csv)}`)
+}
+```
+
+# Branches
+
+- **Generation failure.** If every configured LLM provider fails, `textToSql`
+  returns `422` (`AlveraApiError`) — `Text-to-SQL generation failed for all
+  providers (...)`. Surface it; do not retry blindly.
+- **Write SQL rejected.** `executeSql` with a non-read-only statement
+  (`INSERT`/`UPDATE`/`DELETE`/DDL) is rejected `422` — the read-only boundary is
+  enforced at the app `@deny` regex, a read-only DB transaction, and an EXPLAIN
+  preflight.
+- **Pagination.** Pass `page` / `page_size` to window large result sets;
+  `meta.total_count` / `meta.total_pages` describe the full set (`page_size` is
+  capped server-side).
+
+# Rollback
+
+Nothing to tear down — both calls are read-only (or generate-only). The
+`_setup/foundation.md` tenant + datalake are reset with `mix ecto.reset` on the
+platform; per-run `runSuffix` names avoid collisions across reruns.
+
+# Outcome
+
+A datalake becomes conversational: a prompt yields reviewable SQL with provider
+attribution and a plain-language explanation, that SQL runs read-only, and the
+page comes back either as a structured `{ data, meta }` envelope or a CSV export
+— all without any datalake row ever reaching the LLM.
+
+# See also
+
+- `.agent/datalakes.md` — "Talk to data" (`textToSql` / `executeSql` wire shape)
+  and §6 gotcha on the array-of-arrays `data`.
+- `.agent/ai_sandbox.md` — Layer 3 SQL boundary: how generation stays data-free
+  and execution stays read-only + mode-routed.
+- `integration-tests/tests/foundation/text-to-sql.test.ts` — the green vitest
+  these snippets are lifted from.

package/.agent/datalakes.md

@@ -586,6 +586,52 @@ for (const name of catalog.datasets) {
 }
 ```
 
+### Talk to data — `textToSql` + `executeSql`
+
+Two query methods turn the datalake into a BI surface: natural language → SQL →
+rows. They are **split on purpose** so an agent or UI can show, edit, and re-run
+the SQL between the two calls.
+
+| Method                                                     | Returns                                                   |
+|------------------------------------------------------------|-----------------------------------------------------------|
+| `.textToSql(tenantSlug, datalakeSlug, body)`               | `{ sql, model, provider, explanation }`                   |
+| `.executeSql(tenantSlug, datalakeSlug, body, options?)`    | `{ data, meta }` — or a CSV **string** with `{ format: 'csv' }` |
+
+- **`textToSql`** body is `{ prompt, mode }` (`mode` ∈ `'regulated' | 'unregulated'`,
+  selecting which schema to target). It runs ordered multi-provider LLM failover and
+  returns the generated `sql`, the winning `model` + `provider`, and a best-effort
+  `explanation` (`null` when the explainer is unavailable). **Only the prompt + the
+  schema cross the LLM boundary — never datalake rows**, so it is safe in both modes.
+  It returns SQL for review; it does **not** execute it.
+- **`executeSql`** body is `{ sql, mode, page?, page_size? }`. It runs the SQL
+  **read-only** (INSERT/UPDATE/DELETE/DDL are rejected) on the mode-appropriate
+  schema, with Flop-inspired pagination (`page_size` capped server-side). The JSON
+  shape is `{ data, meta }`; pass `options = { format: 'csv' }` to get the page as a
+  CSV string attachment instead.
+
+```typescript
+const { data: gen } = await api.datalakes.textToSql(tenantSlug, datalakeSlug, {
+  prompt: 'count contacts created this month',
+  mode: 'unregulated',
+})
+// gen.sql — review / edit before running
+
+const { data: page } = await api.datalakes.executeSql(tenantSlug, datalakeSlug, {
+  sql: gen.sql,
+  mode: 'unregulated',
+  page: 1,
+  page_size: 100,
+})
+// page.data  — array-of-arrays rows, aligned to page.meta.columns
+// page.meta  — { columns, num_rows, total_count, page, page_size, total_pages, duration_ms, command }
+
+const { data: csv } = await api.datalakes.executeSql(
+  tenantSlug, datalakeSlug,
+  { sql: gen.sql, mode: 'unregulated' },
+  { format: 'csv' },
+) // csv is a string, not the JSON envelope
+```
+
 ## 6. Gotchas
 
 1. **`create()` does NOT auto-enqueue migration.** The create response
@@ -712,3 +758,12 @@ for (const name of catalog.datasets) {
     `"password"` — and consequently requires both `*_user` and
     `*_pass` to be supplied. The IAM-role relaxation in §2 applies
     only when `auth_method` is explicitly `"iam_role"`.
+
+13. **`executeSql` rows are array-of-arrays, not objects.** `data` is
+    `Array<Array<unknown>>` — a positional row page with **no column
+    keys**, because arbitrary SQL can yield duplicate or expression
+    column names that have no safe object key. Read column names from
+    `meta.columns` and index rows positionally (`row[i]` ↔
+    `meta.columns[i]`); never assume `row.someColumn`. The `?format=csv`
+    branch returns a CSV **string** with no `meta` envelope — for the
+    structural metadata, use the JSON call.

package/LICENSE

@@ -0,0 +1,93 @@
+Elastic License 2.0
+
+URL: https://www.elastic.co/licensing/elastic-license
+
+## Acceptance
+
+By using the software, you agree to all of the terms and conditions below.
+
+## Copyright License
+
+The licensor grants you a non-exclusive, royalty-free, worldwide,
+non-sublicensable, non-transferable license to use, copy, distribute, make
+available, and prepare derivative works of the software, in each case subject to
+the limitations and conditions below.
+
+## Limitations
+
+You may not provide the software to third parties as a hosted or managed
+service, where the service provides users with access to any substantial set of
+the features or functionality of the software.
+
+You may not move, change, disable, or circumvent the license key functionality
+in the software, and you may not remove or obscure any functionality in the
+software that is protected by the license key.
+
+You may not alter, remove, or obscure any licensing, copyright, or other notices
+of the licensor in the software. Any use of the licensor’s trademarks is subject
+to applicable law.
+
+## Patents
+
+The licensor grants you a license, under any patent claims the licensor can
+license, or becomes able to license, to make, have made, use, sell, offer for
+sale, import and have imported the software, in each case subject to the
+limitations and conditions in this license. This license does not cover any
+patent claims that you cause to be infringed by modifications or additions to
+the software. If you or your company make any written claim that the software
+infringes or contributes to infringement of any patent, your patent license for
+the software granted under these terms ends immediately. If your company makes
+such a claim, your patent license ends immediately for work on behalf of your
+company.
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software from you
+also gets a copy of these terms.
+
+If you modify the software, you must include in any modified copies of the
+software prominent notices stating that you have modified the software.
+
+## No Other Rights
+
+These terms do not imply any licenses other than those expressly granted in
+these terms.
+
+## Termination
+
+If you use the software in violation of these terms, such use is not licensed,
+and your licenses will automatically terminate. If the licensor provides you
+with a notice of your violation, and you cease all violation of this license no
+later than 30 days after you receive that notice, your licenses will be
+reinstated retroactively. However, if you violate these terms after such
+reinstatement, any additional violation of these terms will cause your licenses
+to terminate automatically and permanently.
+
+## No Liability
+
+*As far as the law allows, the software comes as is, without any warranty or
+condition, and the licensor will not be liable to you for any damages arising
+out of these terms or the use or nature of the software, under any kind of
+legal claim.*
+
+## Definitions
+
+The **licensor** is the entity offering these terms, and the **software** is the
+software the licensor makes available under these terms, including any portion
+of it.
+
+**you** refers to the individual or entity agreeing to these terms.
+
+**your company** is any legal entity, sole proprietorship, or other kind of
+organization that you work for, plus all organizations that have control over,
+are under the control of, or are under common control with that
+organization. **control** means ownership of substantially all the assets of an
+entity, or the power to direct its management and policies by vote, contract, or
+otherwise. Control can be direct or indirect.
+
+**your licenses** are all the licenses granted to you for the software under
+these terms.
+
+**use** means anything you do with the software requiring one of your licenses.
+
+**trademark** means trademarks, service marks, and similar rights.

package/README.md

@@ -132,7 +132,7 @@ Every resource is a typed namespace on the client (`api.<resource>.<verb>`):
 | `tenants`                    | `list`, `create`                                                                                                                             |
 | `invitations`                | `list`, `create`, `accept`                                                                                                                   |
 | `datasets`                   | `search`, `metadata`, `createUserSearch`                                                                                                     |
-| `datalakes`                  | `list`, `get`, `create`, `metadata`, `migrate`, `createUploadLink`, `createDownloadLink`                                                     |
+| `datalakes`                  | `list`, `get`, `create`, `metadata`, `migrate`, `createUploadLink`, `createDownloadLink`, `textToSql`, `executeSql`                          |
 | `dataSources`                | `list`, `create`, `update`                                                                                                                   |
 | `tools`                      | `list`, `get`, `create`, `update`, `delete`, `testInvocation`                                                                                |
 | `genericTables`              | `list`, `create`                                                                                                                             |