@alvera-ai/platform-sdk 0.10.0-rc.8 → 0.11.0

package/.agent/cookbook/rest-fetch.md

@@ -0,0 +1,246 @@
+---
+title: "Capability: pull records from a third-party REST API on demand"
+summary: A capability walk for pull-based ingestion. Register a REST-API tool with its auth (Bearer here; an OAuth2 refresh-token variant is shown too), bind it to a contract and a Data Activation Client whose `tool_call` names the path to fetch, trigger a fetch (`api.dataActivationClients.runManually`), and verify the rows landed. The platform calls the API for you — no file to upload.
+industry: accounts_receivable
+slug: rest-fetch
+vitest_source:
+  - integration-tests/tests/accounts_receivable/run-dac-fetch.test.ts
+  - integration-tests/tests/foundation/oauth2-dac-fetch.test.ts
+  - integration-tests/tests/accounts_receivable/bootstrap.test.ts
+status: green
+---
+
+# Capability
+
+**What you get:** records pulled straight from a third-party REST API into your
+dataset on demand, with the platform handling auth and the HTTP call — no file
+to export and upload.
+
+A **pull-based** Data Activation Client fetches instead of receiving:
+
+- the **tool** holds the API's base URL + auth (`bearer`, or `oauth2` with a
+  refresh-token grant the platform exchanges server-side before each call),
+- the **DAC's `tool_call`** names the HTTP `method` + `path` to hit and how to
+  paginate; a `response_extractor` unwraps the API's envelope into a row array,
+- `dataActivationClients.runManually(...)` triggers a fetch and returns a
+  `batch_id`.
+
+This is **global** — only the API and its auth differ. See
+`data_activation_clients.md` §6.3 (`.runManually`) and `tools.md` (`rest_api`
+body) for the reference.
+
+# Walkthrough
+
+The `_setup/accounts_receivable.md` bootstrap left `api`, `tenantSlug`,
+`datalakeSlug`, and `ctx.datalakeId` populated. The fetch targets a mock Stripe
+API; locally that is the integration stack's WireMock on `localhost:8080`.
+
+## 001 — create the data source
+
+```typescript
+const { data: ds } = await api.dataSources.create(tenantSlug, datalakeSlug, {
+  name: `REST Stripe Source ${runSuffix}`,
+  uri: 'api.stripe.com',
+  description: 'Stripe REST API — origin of the pulled customer rows.',
+  status: 'active',
+  is_default: false,
+})
+dataSourceId = ds.id!
+```
+
+## 002 — create the REST-API tool (Bearer auth)
+
+The tool holds the API's `base_url` and credentials. `auth_method: 'bearer'`
+with a `bearer_token` is the simplest case; `status: 'active'` is required (a
+draft tool is skipped at fetch time — see Gotchas).
+
+```typescript
+const { data: tool } = await api.tools.create(tenantSlug, datalakeSlug, {
+  name: `Stripe REST Tool ${runSuffix}`,
+  description: 'WireMock-mocked Stripe REST API (AR customer fetch).',
+  intent: 'data_exchange',
+  status: 'active',
+  datalake_id: ctx.datalakeId,
+  data_source_id: dataSourceId,
+  body: {
+    tool_body_type: 'rest_api',
+    auth_method: 'bearer',
+    base_url: 'http://localhost:8080/stripe',
+    bearer_token: 'sk_test_vitest_stripe_token',
+    request_type: 'json',
+    response_type: 'json',
+    timeout_ms: 30_000,
+  },
+})
+ctx.restToolId = tool.id!
+```
+
+## 003 — create the customer interoperability contract
+
+The contract shapes each fetched customer into a `Customer` upsert. Reuse the
+vendored production Stripe customer + MDM templates.
+
+```typescript
+const { readFileSync } = await import('node:fs')
+const { join } = await import('node:path')
+const customerTemplate = readFileSync(
+  join(process.env.COOKBOOK_FIXTURES_DIR!, 'accounts_receivable/_customers_accounts_receivable_customer.liquid'),
+  'utf8',
+)
+const mdmTemplate = readFileSync(
+  join(process.env.COOKBOOK_FIXTURES_DIR!, 'accounts_receivable/_customers_accounts_receivable_mdm.liquid'),
+  'utf8',
+)
+
+const { data: contract } = await api.interoperabilityContracts.create(tenantSlug, datalakeSlug, {
+  name: `Stripe REST Customer Contract ${runSuffix}`,
+  description: 'Stripe customer fetch → AccountsReceivable Customer.',
+  resource_type: 'customer',
+  template_config: { type: 'custom', body: customerTemplate },
+  mdm_input_config: { type: 'custom', body: mdmTemplate },
+  generic_table_id: null,
+})
+interopContractId = contract.id!
+```
+
+## 004 — create the fetch DAC
+
+The Data Activation Client's `tool_call` selects the REST path: `restapi_request`
+with the HTTP `method` and the `path` to hit (`/v1/customers`, appended to the
+tool's `base_url`). `pagination_context_template` declares no further pages here.
+A top-level `response_extractor` unwraps the API's `{ data: [...] }` envelope so
+each element becomes one row.
+
+```typescript
+const { data: dac } = await api.dataActivationClients.create(tenantSlug, datalakeSlug, {
+  name: `Stripe REST Fetch DAC ${runSuffix}`,
+  description: 'Pull-based DAC — fetches Stripe customers over REST.',
+  tool_id: ctx.restToolId,
+  data_source_id: dataSourceId,
+  tool_call: {
+    tool_call_type: 'restapi_request',
+    method: 'get',
+    path: { type: 'custom', body: '/v1/customers' },
+    pagination_context_template: { type: 'custom', body: '{"has_next": false}' },
+  },
+  response_extractor: { type: 'custom', body: '{{ msg.data | to_json }}' },
+  interoperability_contract_ids: [interopContractId],
+})
+dacId = dac.id!
+ctx.dacSlug = dac.slug!
+```
+
+## 005 — trigger a fetch
+
+`runManually` enqueues a fetch and returns the `batch_id` immediately; the actual
+HTTP call + ingestion run in a background worker.
+
+```typescript
+const { data: run } = await api.dataActivationClients.runManually(tenantSlug, datalakeSlug, ctx.dacSlug)
+ctx.batchId = run.batch_id!
+if (!ctx.batchId) {
+  throw new Error('runManually did not return a batch_id')
+}
+```
+
+## 006 — verify the fetched rows landed
+
+Poll the DAC logs until the batch shows a fully-merged ingest (the mock returns 3
+customers), then search the `customer` dataset scoped to the batch.
+
+```typescript
+const deadline = Date.now() + 90_000
+let merged = false
+while (Date.now() < deadline && !merged) {
+  const { data } = await api.dataActivationClients.logs.list(tenantSlug, datalakeSlug, ctx.dacSlug)
+  merged = (data.data ?? []).some((r) => {
+    const row = r as Record<string, unknown>
+    return row.batch_id === ctx.batchId &&
+      typeof row.rows_ingested === 'number' && row.rows_ingested >= 3 &&
+      Array.isArray(row.output_files) && row.output_files.length > 0
+  })
+  if (!merged) await new Promise((r) => setTimeout(r, 1_000))
+}
+if (!merged) {
+  throw new Error('REST fetch did not produce a merged batch within 90s')
+}
+
+const { data: userSearch } = await api.datasets.createUserSearch(tenantSlug, datalakeSlug, 'customer', {
+  search_query: `ra.batch_id = '${ctx.batchId}'`,
+})
+const searchDeadline = Date.now() + 60_000
+let rows: Array<Record<string, unknown>> = []
+while (Date.now() < searchDeadline && rows.length < 3) {
+  const { data: page } = await api.datasets.search(tenantSlug, datalakeSlug, 'customer', {
+    userSearchId: userSearch.id!,
+    dataAccessMode: 'unregulated',
+  })
+  rows = (page.data ?? []) as Array<Record<string, unknown>>
+  if (rows.length < 3) await new Promise((r) => setTimeout(r, 1_000))
+}
+if (rows.length < 3) {
+  throw new Error(`expected 3 fetched customers, found ${rows.length}`)
+}
+```
+
+## 007 — variant: an OAuth2 refresh-token tool
+
+When the API uses OAuth2, the tool carries the full `oauth2_*` grant config
+instead of a static token. The platform exchanges the `oauth2_refresh_token` for
+an access token **server-side before each fetch** — your code never handles the
+token. This step creates such a tool to show the body shape (the fetch itself
+runs the same `runManually` flow once bound to a DAC + contract for that API).
+
+```typescript
+const { data: oauthTool } = await api.tools.create(tenantSlug, datalakeSlug, {
+  name: `OAuth2 REST Tool ${runSuffix}`,
+  description: 'REST tool authenticating via an OAuth2 refresh-token grant.',
+  intent: 'data_exchange',
+  status: 'active',
+  datalake_id: ctx.datalakeId,
+  data_source_id: dataSourceId,
+  body: {
+    tool_body_type: 'rest_api',
+    auth_method: 'oauth2',
+    base_url: 'http://localhost:8080/quickbooks',
+    oauth2_grant_type: 'authorization_code',
+    oauth2_client_id: 'test_client_id',
+    oauth2_client_secret: 'test_client_secret',
+    oauth2_token_url: 'http://localhost:8080/quickbooks/oauth2/v1/tokens/bearer',
+    oauth2_refresh_token: 'test_initial_refresh_token',
+    oauth2_scope: 'com.intuit.quickbooks.accounting',
+    oauth2_token_ttl: 3300,
+    request_type: 'json',
+    response_type: 'json',
+    timeout_ms: 30_000,
+  },
+})
+if (!oauthTool.id) {
+  throw new Error('OAuth2 tool create did not return an id')
+}
+```
+
+# Gotchas
+
+- **A draft tool is silently skipped at fetch time.** The worker only fetches
+  through `status: 'active'` tools — a `draft` tool produces zero log rows (no
+  error, no data). If a fetch ingests nothing, check the tool status first.
+- **The `response_extractor` unwraps the API envelope.** REST APIs wrap rows in
+  an envelope (`{ data: [...] }`); the extractor (`{{ msg.data | to_json }}`)
+  turns that into the row array the contract maps. Without it, the whole envelope
+  is treated as one row.
+- **OAuth2 token exchange is server-side.** With `auth_method: 'oauth2'`, the
+  platform swaps the refresh token for an access token before each call — your
+  code never sees a token. Bad credentials fail the exchange and produce zero log
+  rows (same silent-skip shape as a draft tool).
+- **`runManually` returns immediately.** The `batch_id` comes back before the
+  fetch runs; poll the logs to know when rows actually landed.
+
+# See also
+
+- `data_activation_clients.md` §6.3 — `.runManually` + the pull-based reference
+- `tools.md` — the `rest_api` tool body, `bearer` vs `oauth2` auth
+- `_setup/accounts_receivable.md` — the bootstrap this walk starts from
+- `integration-tests/tests/accounts_receivable/run-dac-fetch.test.ts`,
+  `integration-tests/tests/foundation/oauth2-dac-fetch.test.ts` — the green tests
+  these calls are lifted from