@alvera-ai/platform-sdk 0.10.0-rc.8 → 0.11.0

package/.agent/cookbook/bulk-ingest.md

@@ -0,0 +1,254 @@
+---
+title: "Capability: load a whole file of records in one upload"
+summary: A capability walk for bulk file ingestion. Stand up an ingestion chain (data source → manual-upload tool → interop contract → DAC), mint a presigned upload URL (`api.datalakes.createUploadLink`), PUT a CSV to it, enqueue the file (`api.dataActivationClients.ingestFile`), discover the batch the worker allocates by diffing the logs, and verify every row landed (`api.datasets.search`). For backfills and exports — not row-at-a-time.
+industry: accounts_receivable
+slug: bulk-ingest
+vitest_source:
+  - integration-tests/tests/accounts_receivable/run-dac-bulk.test.ts
+  - integration-tests/tests/accounts_receivable/create-dac.test.ts
+  - integration-tests/tests/accounts_receivable/interoperability-contracts.test.ts
+  - integration-tests/tests/accounts_receivable/tools.test.ts
+  - integration-tests/tests/accounts_receivable/data-sources.test.ts
+  - integration-tests/tests/accounts_receivable/bootstrap.test.ts
+status: green
+---
+
+# Capability
+
+**What you get:** a whole file of records — a CSV export, a backfill — turned
+into dataset rows in one upload, instead of one `.ingest` call per row.
+
+The bulk path is three calls plus a discovery step:
+
+1. `datalakes.createUploadLink(...)` → a presigned PUT URL + a storage `key`.
+2. Raw `fetch` PUT of the file bytes to that URL (**plain HTTP, not the SDK**).
+3. `dataActivationClients.ingestFile(..., { key })` → enqueues a background job.
+4. The worker reads the file, allocates a fresh `batch_id`, and fans out
+   per-row jobs. The `batch_id` is **not returned** — you discover it by diffing
+   the DAC's logs before vs. after.
+
+It runs through the same data-activation chain as inline `.ingest`; only the
+entry point differs. This is **global** to every datalake. See
+`data_activation_clients.md` §6.2 for the reference.
+
+# Walkthrough
+
+The `_setup/accounts_receivable.md` bootstrap left `api`, `tenantSlug`,
+`datalakeSlug`, and `ctx.datalakeId` populated. Steps 001–004 build the same
+ingestion chain any AR cookbook uses; 005–008 are the bulk-specific part.
+
+## 001 — create the data source
+
+The chain's first link registers where the rows originate. Its `uri` flows into
+each ingested row's `source_uri`.
+
+```typescript
+const { data: ds } = await api.dataSources.create(tenantSlug, datalakeSlug, {
+  name: `Bulk Stripe Source ${runSuffix}`,
+  uri: 'stripe.example.com',
+  description: 'Stripe billing export — origin of the bulk customer CSV.',
+  status: 'active',
+  is_default: false,
+})
+dataSourceId = ds.id!
+```
+
+## 002 — create the manual-upload tool
+
+A `manual_upload` tool backs both the inline and the file-upload ingest paths —
+the rows arrive in the upload, not by the tool fetching them.
+
+```typescript
+const { data: tool } = await api.tools.create(tenantSlug, datalakeSlug, {
+  name: `Bulk Manual Upload Tool ${runSuffix}`,
+  description: 'Manual-upload data-exchange tool backing the bulk-ingest DAC.',
+  intent: 'data_exchange',
+  status: 'active',
+  datalake_id: ctx.datalakeId,
+  data_source_id: dataSourceId,
+  body: { tool_body_type: 'manual_upload' },
+})
+ctx.manualUploadToolId = tool.id!
+```
+
+## 003 — create the customer interoperability contract
+
+The contract maps each inbound CSV row into a `Customer` upsert. Load the
+production Stripe customer + MDM Liquid templates from the vendored fixtures
+rather than inlining them.
+
+```typescript
+const { readFileSync } = await import('node:fs')
+const { join } = await import('node:path')
+const customerTemplate = readFileSync(
+  join(process.env.COOKBOOK_FIXTURES_DIR!, 'accounts_receivable/_customers_accounts_receivable_customer.liquid'),
+  'utf8',
+)
+const mdmTemplate = readFileSync(
+  join(process.env.COOKBOOK_FIXTURES_DIR!, 'accounts_receivable/_customers_accounts_receivable_mdm.liquid'),
+  'utf8',
+)
+
+const { data: contract } = await api.interoperabilityContracts.create(tenantSlug, datalakeSlug, {
+  name: `Bulk Stripe Customer Contract ${runSuffix}`,
+  description: 'Stripe customers → AccountsReceivable Customer (custom Liquid + MDM input).',
+  resource_type: 'customer',
+  template_config: { type: 'custom', body: customerTemplate },
+  mdm_input_config: { type: 'custom', body: mdmTemplate },
+  generic_table_id: null,
+})
+interopContractId = contract.id!
+```
+
+## 004 — create the manual-upload DAC
+
+The Data Activation Client binds tool + data source + contract into one
+ingestion endpoint. Its server-derived `slug` is the handle the file is ingested
+against.
+
+```typescript
+const { data: dac } = await api.dataActivationClients.create(tenantSlug, datalakeSlug, {
+  name: `Bulk Stripe Customer DAC ${runSuffix}`,
+  description: 'Manual-upload DAC — ingests the bulk Stripe customer CSV.',
+  tool_id: ctx.manualUploadToolId,
+  data_source_id: dataSourceId,
+  tool_call: { tool_call_type: 'manual_upload' },
+  interoperability_contract_ids: [interopContractId],
+})
+dacId = dac.id!
+ctx.dacSlug = dac.slug!
+```
+
+## 005 — mint a presigned upload URL and PUT the CSV
+
+`createUploadLink` returns a presigned PUT `url` and the storage `key` the
+platform will reference. Upload the file body to the URL with a **raw `fetch`** —
+this PUT goes straight to object storage, **not** through the SDK. The CSV is the
+vendored 4-row Stripe customer export.
+
+```typescript
+const { readFileSync } = await import('node:fs')
+const { join } = await import('node:path')
+const csvBody = readFileSync(
+  join(process.env.COOKBOOK_FIXTURES_DIR!, 'accounts_receivable/stripe_customers_batch1.csv'),
+  'utf8',
+)
+
+const { data: link } = await api.datalakes.createUploadLink(tenantSlug, datalakeSlug, {
+  content_type: 'text/csv',
+  filename: 'stripe_customers_batch1.csv',
+})
+ctx.uploadKey = link.key!
+
+const put = await fetch(link.url!, {
+  method: 'PUT',
+  headers: { 'Content-Type': 'text/csv' },
+  body: csvBody,
+})
+if (put.status !== 200) {
+  throw new Error(`presigned PUT failed: ${put.status}`)
+}
+```
+
+## 006 — snapshot the logs, then enqueue the file
+
+`ingestFile` is fully async — it enqueues a background job and returns a
+`job_id`, but **no `batch_id`** (the worker allocates that later). Snapshot the
+existing log batch ids first, so step 007 can spot the new one the worker
+creates.
+
+```typescript
+const { data: pre } = await api.dataActivationClients.logs.list(tenantSlug, datalakeSlug, ctx.dacSlug)
+ctx.preBatchIds = (pre.data ?? [])
+  .map((r) => (r as { batch_id?: string }).batch_id)
+  .filter((b): b is string => typeof b === 'string')
+
+const { data: job } = await api.dataActivationClients.ingestFile(tenantSlug, datalakeSlug, ctx.dacSlug, {
+  key: ctx.uploadKey,
+})
+if (!job.job_id) {
+  throw new Error('ingestFile did not return a job_id')
+}
+```
+
+## 007 — discover the batch the worker allocated
+
+Poll the logs until a **new** batch id (one not in the pre-snapshot) shows a
+fully-merged customer ingest: `rows_ingested >= 4` (the four CSV rows) and a
+non-empty `output_files` array (the merged data landed in storage).
+
+```typescript
+const pre = new Set<string>(ctx.preBatchIds)
+const deadline = Date.now() + 60_000
+let batchId: string | null = null
+while (Date.now() < deadline && !batchId) {
+  const { data } = await api.dataActivationClients.logs.list(tenantSlug, datalakeSlug, ctx.dacSlug)
+  for (const row of (data.data ?? []) as Array<Record<string, unknown>>) {
+    const b = row.batch_id
+    if (typeof b !== 'string' || pre.has(b)) continue
+    if (row.dataset_table !== 'customers') continue
+    if (typeof row.rows_ingested !== 'number' || row.rows_ingested < 4) continue
+    if (!Array.isArray(row.output_files) || row.output_files.length === 0) continue
+    batchId = b
+    break
+  }
+  if (!batchId) await new Promise((r) => setTimeout(r, 1_000))
+}
+if (!batchId) {
+  throw new Error('bulk ingest did not produce a fully-merged batch within 60s')
+}
+ctx.batchId = batchId
+```
+
+## 008 — verify every row landed
+
+Search the `customer` dataset scoped to that batch. `createUserSearch` compiles
+the query (`ra` is the customer dataset alias); `search` returns the rows once
+they materialize — poll until all four appear.
+
+```typescript
+const { data: userSearch } = await api.datasets.createUserSearch(tenantSlug, datalakeSlug, 'customer', {
+  search_query: `ra.batch_id = '${ctx.batchId}'`,
+})
+if (userSearch.status !== 'completed') {
+  throw new Error(`customer user-search did not compile: ${userSearch.status}`)
+}
+
+const deadline = Date.now() + 90_000
+let rows: Array<Record<string, unknown>> = []
+while (Date.now() < deadline && rows.length < 4) {
+  const { data: page } = await api.datasets.search(tenantSlug, datalakeSlug, 'customer', {
+    userSearchId: userSearch.id!,
+    dataAccessMode: 'unregulated',
+  })
+  rows = (page.data ?? []) as Array<Record<string, unknown>>
+  if (rows.length < 4) await new Promise((r) => setTimeout(r, 1_000))
+}
+if (rows.length < 4) {
+  throw new Error(`expected 4 ingested customers, found ${rows.length} within 90s`)
+}
+```
+
+# Gotchas
+
+- **The file PUT is raw HTTP, not the SDK.** `createUploadLink` and `ingestFile`
+  are SDK calls; the upload itself is a plain `fetch` PUT to the presigned URL.
+  Don't try to route the bytes through an `api.*` method.
+- **`ingestFile` does NOT return the `batch_id`.** It returns only a `job_id`.
+  The worker allocates the `batch_id` when it reads the file — discover it by
+  diffing the DAC logs before vs. after (the snapshot in 006).
+- **Wait for merge, not just for rows.** A log row with `rows_ingested > 0` but
+  `output_files: []` is mid-merge — gate on `output_files.length > 0` before
+  trusting the batch.
+- **Search is a two-call, poll-until-ready pattern.** `createUserSearch` compiles
+  the query and returns `status: 'completed'`; the rows still take a moment to
+  materialize, so poll `search` until they appear.
+
+# See also
+
+- `data_activation_clients.md` §6.2 — `.ingestFile` reference + the bulk flow
+- `datalakes.md` — `createUploadLink` (presigned upload) reference
+- `_setup/accounts_receivable.md` — the bootstrap this walk starts from
+- `_fixtures/accounts_receivable/` — the vendored Stripe customer CSV + Liquid
+- `integration-tests/tests/accounts_receivable/run-dac-bulk.test.ts` — the green
+  test these calls are lifted from