@alvera-ai/platform-sdk 0.17.0 → 0.18.0

package/.agent/cookbook/bulk-ingest.md

@@ -1,302 +0,0 @@
----
-title: "Capability: load a whole file of records in one upload"
-summary: A capability walk for bulk file ingestion. Stand up an ingestion chain (data source → manual-upload tool → interop contract → DAC), mint a presigned upload URL (`api.datalakes.createUploadLink`), PUT a CSV to it, enqueue the file (`api.dataActivationClients.ingestFile`), discover the batch the worker allocates by diffing the logs, and verify every row landed (`api.datasets.search`). For backfills and exports — not row-at-a-time.
-industry: subscription
-slug: bulk-ingest
-vitest_source:
-  - integration-tests/tests/subscription/run-dac-bulk.test.ts
-  - integration-tests/tests/subscription/create-dac.test.ts
-  - integration-tests/tests/subscription/interoperability-contracts.test.ts
-  - integration-tests/tests/subscription/tools.test.ts
-  - integration-tests/tests/subscription/data-sources.test.ts
-  - integration-tests/tests/subscription/bootstrap.test.ts
-status: green
----
-
-# Capability
-
-**What you get:** a whole file of records — a CSV export, a backfill — turned
-into dataset rows in one upload, instead of one `.ingest` call per row.
-
-The bulk path is three calls plus a discovery step:
-
-1. `datalakes.createUploadLink(...)` → a presigned PUT URL + a storage `key`.
-2. Raw `fetch` PUT of the file bytes to that URL (**plain HTTP, not the SDK**).
-3. `dataActivationClients.ingestFile(..., { key })` → enqueues a background job.
-4. The worker reads the file, allocates a fresh `batch_id`, and fans out
-   per-row jobs. The `batch_id` is **not returned** — you discover it by diffing
-   the DAC's logs before vs. after.
-
-It runs through the same data-activation chain as inline `.ingest`; only the
-entry point differs. This is **global** to every datalake. See
-`data_activation_clients.md` §6.2 for the reference.
-
-# Walkthrough
-
-The `_setup/subscription.md` bootstrap left `api`, `tenantSlug`,
-`datalakeSlug`, and `ctx.datalakeId` populated. Steps 001–004 build the same
-ingestion chain any AR cookbook uses; 005–008 are the bulk-specific part.
-
-## 001 — create the data source
-
-The chain's first link registers where the rows originate. Its `uri` flows into
-each ingested row's `source_uri`.
-
-```typescript
-const { data: ds } = await api.dataSources.create(tenantSlug, datalakeSlug, {
-  name: `Bulk Stripe Source ${runSuffix}`,
-  uri: 'stripe.example.com',
-  description: 'Stripe billing export — origin of the bulk customer CSV.',
-  status: 'active',
-  is_default: false,
-})
-dataSourceId = ds.id!
-```
-
-## 002 — create the manual-upload tool
-
-A `manual_upload` tool backs both the inline and the file-upload ingest paths —
-the rows arrive in the upload, not by the tool fetching them.
-
-```typescript
-const { data: tool } = await api.tools.create(tenantSlug, datalakeSlug, {
-  name: `Bulk Manual Upload Tool ${runSuffix}`,
-  description: 'Manual-upload data-exchange tool backing the bulk-ingest DAC.',
-  intent: 'data_exchange',
-  status: 'active',
-  datalake_id: ctx.datalakeId,
-  data_source_id: dataSourceId,
-  body: { tool_body_type: 'manual_upload' },
-})
-ctx.manualUploadToolId = tool.id!
-```
-
-## 003 — create the customer interoperability contract
-
-The contract maps each inbound CSV row into a `Customer` upsert. Load the
-production Stripe customer + MDM Liquid templates from the vendored fixtures
-rather than inlining them.
-
-```typescript
-const { readFileSync } = await import('node:fs')
-const { join } = await import('node:path')
-const customerTemplate = readFileSync(
-  join(process.env.COOKBOOK_FIXTURES_DIR!, 'subscription/_customers_subscription_customer.liquid'),
-  'utf8',
-)
-const mdmTemplate = readFileSync(
-  join(process.env.COOKBOOK_FIXTURES_DIR!, 'subscription/_customers_subscription_mdm.liquid'),
-  'utf8',
-)
-
-const { data: contract } = await api.interoperabilityContracts.create(tenantSlug, datalakeSlug, {
-  name: `Bulk Stripe Customer Contract ${runSuffix}`,
-  description: 'Stripe customers → Subscription Customer (custom Liquid + MDM input).',
-  resource_type: 'customer',
-  template_config: { type: 'custom', body: customerTemplate },
-  mdm_input_config: { type: 'custom', body: mdmTemplate },
-  generic_table_id: null,
-})
-interopContractId = contract.id!
-```
-
-## 004 — create the manual-upload DAC
-
-The Data Activation Client binds tool + data source + contract into one
-ingestion endpoint. Its server-derived `slug` is the handle the file is ingested
-against.
-
-```typescript
-const { data: dac } = await api.dataActivationClients.create(tenantSlug, datalakeSlug, {
-  name: `Bulk Stripe Customer DAC ${runSuffix}`,
-  description: 'Manual-upload DAC — ingests the bulk Stripe customer CSV.',
-  tool_id: ctx.manualUploadToolId,
-  data_source_id: dataSourceId,
-  tool_call: { tool_call_type: 'manual_upload' },
-  interoperability_contract_ids: [interopContractId],
-})
-dacId = dac.id!
-ctx.dacSlug = dac.slug!
-```
-
-## 005 — mint a presigned upload URL and PUT the CSV
-
-`createUploadLink` returns a presigned PUT `url` and the storage `key` the
-platform will reference. Upload the file body to the URL with a **raw `fetch`** —
-this PUT goes straight to object storage, **not** through the SDK. The CSV is the
-vendored 4-row Stripe customer export.
-
-```typescript
-const { readFileSync } = await import('node:fs')
-const { join } = await import('node:path')
-const csvBody = readFileSync(
-  join(process.env.COOKBOOK_FIXTURES_DIR!, 'subscription/stripe_customers_batch1.csv'),
-  'utf8',
-)
-
-const { data: link } = await api.datalakes.createUploadLink(tenantSlug, datalakeSlug, {
-  content_type: 'text/csv',
-  filename: 'stripe_customers_batch1.csv',
-})
-ctx.uploadKey = link.key!
-
-const put = await fetch(link.url!, {
-  method: 'PUT',
-  headers: { 'Content-Type': 'text/csv' },
-  body: csvBody,
-})
-if (put.status !== 200) {
-  throw new Error(`presigned PUT failed: ${put.status}`)
-}
-```
-
-## 006 — snapshot the logs, then enqueue the file
-
-`ingestFile` is fully async — it enqueues a background job and returns a
-`job_id`, but **no `batch_id`** (the worker allocates that later). Snapshot the
-existing log batch ids first, so step 007 can spot the new one the worker
-creates.
-
-```typescript
-const { data: pre } = await api.dataActivationClients.logs.list(tenantSlug, datalakeSlug, ctx.dacSlug)
-ctx.preBatchIds = (pre.data ?? [])
-  .map((r) => (r as { batch_id?: string }).batch_id)
-  .filter((b): b is string => typeof b === 'string')
-
-const { data: job } = await api.dataActivationClients.ingestFile(tenantSlug, datalakeSlug, ctx.dacSlug, {
-  key: ctx.uploadKey,
-})
-if (!job.job_id) {
-  throw new Error('ingestFile did not return a job_id')
-}
-```
-
-## 007 — discover the batch the worker allocated
-
-Poll the logs until a **new** batch id (one not in the pre-snapshot) shows a
-fully-merged customer ingest: `rows_ingested >= 4` (the four CSV rows) and a
-non-empty `output_files` array (the merged data landed in storage).
-
-```typescript
-const pre = new Set<string>(ctx.preBatchIds)
-const deadline = Date.now() + 60_000
-let batchId: string | null = null
-while (Date.now() < deadline && !batchId) {
-  const { data } = await api.dataActivationClients.logs.list(tenantSlug, datalakeSlug, ctx.dacSlug)
-  for (const row of (data.data ?? []) as Array<Record<string, unknown>>) {
-    const b = row.batch_id
-    if (typeof b !== 'string' || pre.has(b)) continue
-    if (row.dataset_table !== 'customers') continue
-    if (typeof row.rows_ingested !== 'number' || row.rows_ingested < 4) continue
-    if (!Array.isArray(row.output_files) || row.output_files.length === 0) continue
-    batchId = b
-    break
-  }
-  if (!batchId) await new Promise((r) => setTimeout(r, 1_000))
-}
-if (!batchId) {
-  throw new Error('bulk ingest did not produce a fully-merged batch within 60s')
-}
-ctx.batchId = batchId
-```
-
-## 008 — verify every row landed
-
-Search the `customer` dataset scoped to that batch. `createUserSearch` compiles
-the query (`ra` is the customer dataset alias); `search` returns the rows once
-they materialize — poll until all four appear.
-
-```typescript
-const { data: userSearch } = await api.datasets.createUserSearch(tenantSlug, datalakeSlug, 'customer', {
-  search_query: `ra.batch_id = '${ctx.batchId}'`,
-})
-if (userSearch.status !== 'completed') {
-  throw new Error(`customer user-search did not compile: ${userSearch.status}`)
-}
-
-const deadline = Date.now() + 90_000
-let rows: Array<Record<string, unknown>> = []
-while (Date.now() < deadline && rows.length < 4) {
-  const { data: page } = await api.datasets.search(tenantSlug, datalakeSlug, 'customer', {
-    userSearchId: userSearch.id!,
-    dataAccessMode: 'unregulated',
-  })
-  rows = (page.data ?? []) as Array<Record<string, unknown>>
-  if (rows.length < 4) await new Promise((r) => setTimeout(r, 1_000))
-}
-if (rows.length < 4) {
-  throw new Error(`expected 4 ingested customers, found ${rows.length} within 90s`)
-}
-```
-
-## 009 — write the integration test
-
-End the build with a test you keep: run the bulk path in miniature —
-mint a link, PUT a tiny inline CSV, enqueue it — and assert each call's
-own acknowledgement. The probe rows are `test-`-prefixed so they are
-unmistakably synthetic wherever they surface, and the test asserts the
-enqueue ack (`job_id`), never synchronous merge completion — the worker
-owns that. This block runs live under `make validate-cookbook`.
-
-```typescript
-// Re-GET — the client must still be there and carry its ingest slug.
-const { data: dacRow } = await api.dataActivationClients.get(tenantSlug, datalakeSlug, dacId)
-if (dacRow.slug !== ctx.dacSlug) {
-  throw new Error(`DAC slug drifted on read-back: ${dacRow.slug}`)
-}
-// Behavioural probe — the three-step bulk path, each step asserted on
-// its own response.
-const probeCsv =
-  'customer_number,customer_type,status,name,email,currency\n' +
-  `test-PROBE-1-${runSuffix},individual,contracted,test-Grace Hopper,test-grace-${runSuffix}@example.com,USD\n`
-const { data: probeLink } = await api.datalakes.createUploadLink(tenantSlug, datalakeSlug, {
-  content_type: 'text/csv',
-  filename: `test-bulk-probe-${runSuffix}.csv`,
-})
-if (!probeLink.url || !probeLink.key) {
-  throw new Error('createUploadLink returned no url/key')
-}
-const probePut = await fetch(probeLink.url, {
-  method: 'PUT',
-  headers: { 'Content-Type': 'text/csv' },
-  body: probeCsv,
-})
-if (probePut.status !== 200) {
-  throw new Error(`presigned PUT failed: ${probePut.status}`)
-}
-const { data: probeJob } = await api.dataActivationClients.ingestFile(tenantSlug, datalakeSlug, ctx.dacSlug, {
-  key: probeLink.key,
-})
-// job_id is a NUMBER on the wire (an Oban job id), status "scheduled".
-if (probeJob.job_id === undefined || probeJob.job_id === null) {
-  throw new Error('ingestFile probe returned no job_id')
-}
-```
-
-If the probe fails in production, escalate with the failing call's
-response — don't retry the enqueue in a loop or reach into the worker's
-storage to "help it along".
-
-# Gotchas
-
-- **The file PUT is raw HTTP, not the SDK.** `createUploadLink` and `ingestFile`
-  are SDK calls; the upload itself is a plain `fetch` PUT to the presigned URL.
-  Don't try to route the bytes through an `api.*` method.
-- **`ingestFile` does NOT return the `batch_id`.** It returns only a `job_id`.
-  The worker allocates the `batch_id` when it reads the file — discover it by
-  diffing the DAC logs before vs. after (the snapshot in 006).
-- **Wait for merge, not just for rows.** A log row with `rows_ingested > 0` but
-  `output_files: []` is mid-merge — gate on `output_files.length > 0` before
-  trusting the batch.
-- **Search is a two-call, poll-until-ready pattern.** `createUserSearch` compiles
-  the query and returns `status: 'completed'`; the rows still take a moment to
-  materialize, so poll `search` until they appear.
-
-# See also
-
-- `data_activation_clients.md` §6.2 — `.ingestFile` reference + the bulk flow
-- `datalakes.md` — `createUploadLink` (presigned upload) reference
-- `_setup/subscription.md` — the bootstrap this walk starts from
-- `_fixtures/subscription/` — the vendored Stripe customer CSV + Liquid
-- `integration-tests/tests/subscription/run-dac-bulk.test.ts` — the green
-  test these calls are lifted from