@deepexi/datasense-connector 0.0.0-stage → 3.0.1

package/README.md

@@ -1,3 +1,173 @@
-# Temporary Holding Version
+# @deepexi/datasense-connector
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+SDK for writing Datasense connectors, plus tools to run them on your own machine.
+
+```sh
+npm install --save-dev @deepexi/datasense-connector
+```
+
+A connector is a small TypeScript project: a `package.json` whose only dependency is
+`@deepexi/datasense-connector`, and an entry file (`main`, default `src/index.ts`) that default-exports
+`defineConnector(...)`. Upload the source directory with `ontos connector publish`; the platform bundles it and
+the connector runner executes it in an isolated VM context.
+
+## `@deepexi/datasense-connector`
+
+- `defineConnector` and the `t.*` type builders (`t.secret({ exposeToScript })` for credential fields)
+- Pull connectors: `hosts`, `db`, `rateLimit`, `check(ctx)`, `discover(ctx)` returning streams and push tables,
+  and `streams.<name>(ctx, input)` async generators yielding `ctx.record` / `ctx.remove` / `ctx.checkpoint`
+- Push connectors: named handlers `push: { <name>: { verify?, map } }`, for example `push: { default: { map } }`
+- `ConnectorError` and `errors.AuthError` / `RateLimited` / `TransientError` / `SchemaError` / `FatalError` /
+  `RejectedPayload` / `NotAvailable`
+- `validateConnectorDefinition` / `validateDiscoverResult` / `validateConfig` / `summarizeDefinition`, shared with the platform
+- `SDK_VERSION` and `SUPPORTED_SDK_MAJORS`
+
+Connector code imports only `@deepexi/datasense-connector` itself and relative files. Node or Bun built-ins,
+other packages, subpaths of this package and import attributes are rejected when the source is bundled.
+
+The `ctx` passed to `check`, `discover` and streams (`http`, `paginate`, `db`, `map`, `cache`, `crypto`, `time`,
+`log`, `signal`) is provided by the runtime. External write operations (`writes`) and `ctx.files` are not
+available yet.
+
+Pull semantics worth knowing:
+
+- Records are sent in batches of at most 1000 records or 4 MiB. Yielding `ctx.checkpoint(state)` sends the
+  pending batch at once, followed by the checkpoint, and the platform commits the checkpoint after those records
+  are written. A failed run resumes from the last committed checkpoint, so only the records yielded after it
+  are fetched again. Yield a checkpoint after each page.
+- `hosts` limits every outbound request, and `ctx.db` too. A database connector must list the database host,
+  for example `hosts: ['{config.host}']`, or its queries are denied with `EGRESS_HOST_DENIED`.
+- Secrets are handles (`ontos:secret:<field>`) that the gateway replaces on the way out. Plaintext is only
+  available for fields declared `t.secret({ exposeToScript: true })`, and reading them is logged on the run.
+- `ctx.http` waits and resends on gateway rate limits and upstream 429. Upstream 502 / 503 / 504 and gateway
+  timeouts are retried with backoff only for idempotent methods (GET, HEAD, PUT, DELETE); pass `retry: true`
+  to retry a POST / PATCH as well, or `retry: false` to never retry.
+
+```typescript
+import { defineConnector, t } from '@deepexi/datasense-connector';
+
+export default defineConnector({
+  name: 'demo_pull',
+  version: '1.0.0',
+  config: t.object({ apiKey: t.secret(), region: t.string() }),
+  hosts: ['{config.region}.api.example.com'],
+  discover: () => ({
+    streams: {
+      orders: {
+        tables: { order: { columns: { id: t.string(), updated_at: t.instant() }, primaryKey: ['id'] } },
+        syncModes: ['full', 'incremental'],
+      },
+    },
+  }),
+  streams: {
+    async *orders(ctx, { state }) {
+      const since = (state as { since?: string } | undefined)?.since;
+      for await (const page of ctx.paginate.cursor({
+        request: (cursor) => ctx.http.get<{ items: Array<{ id: string; updated_at: string }>; next?: string }>(
+          `https://${String(ctx.config.region)}.api.example.com/orders`,
+          { query: { cursor, since }, headers: { authorization: `Bearer ${ctx.secrets.apiKey}` } },
+        ),
+        rows: (res) => res.data.items,
+        nextCursor: (res) => res.data.next,
+      })) {
+        for (const row of page.rows) yield ctx.record('order', row);
+        yield ctx.checkpoint({ since: page.maxOf((row) => row.updated_at) ?? since });
+      }
+    },
+  },
+});
+```
+
+## `@deepexi/datasense-connector/testing`
+
+Runs a connector locally, with a local gateway that answers its outbound HTTP, database, crypto, cache and
+secret calls the way the platform does. Use it from your tests (for example vitest). Connector source must not
+import this entry.
+
+```typescript
+import { expect, test } from 'vitest';
+import { createLocalGateway, runStream } from '@deepexi/datasense-connector/testing';
+import connector from '../src/index';
+
+test('reads the first page', async () => {
+  const host = createLocalGateway({
+    hosts: ['eu.api.example.com'],
+    secrets: { apiKey: process.env.DEMO_API_KEY ?? '' },
+  });
+  try {
+    const result = await runStream(connector, { stream: 'orders', config: { region: 'eu' }, limit: 50, host });
+    expect(result.error).toBeNull();
+    expect(result.records.length).toBeGreaterThan(0);
+  } finally {
+    await host.close();
+  }
+});
+```
+
+- `runStream`, `runDiscover`, `runCheck` and `runPush` run an imported definition in the current process and
+  return a `LocalRunResult`: `records`, `removals`, `checkpoints`, `logs`, `rejected`, `declaration`, `stopped`
+  and `error` (`null` on success). `runStream` runs `check` and `discover` first and accepts records only for the
+  tables the stream declares. With `limit`, the run stops after that many records and `stopped` is `true`.
+- `createLocalGateway(options)`:
+  - `hosts`: the hosts the connector may reach (exact names or `*.suffix`). Anything else fails with
+    `EGRESS_HOST_DENIED`. Cloud metadata addresses are always denied.
+  - `secrets`: field to plaintext. Secret handles in URLs, headers, bodies and crypto keys are replaced on the
+    way out, and plaintext in responses, logs and error messages is replaced with `***`.
+  - `rateLimit: { perSecond, concurrency }`: requests are spaced and queued instead of failing.
+  - `db`: connection settings for `ctx.db`. Queries run in a read-only transaction with cursor paging and
+    need the optional peer dependency `postgres` (`npm install --save-dev postgres`); without it the run ends
+    with `DB_DRIVER_MISSING`.
+  - `fetch`: replaces the network in tests. `onExchange` receives every HTTP, database and crypto exchange.
+- `loadConnectorSource(dir, { typescript })` reads a connector project, checks its imports with the same rules
+  as the platform and transpiles it with your TypeScript compiler (`import ts from 'typescript'`).
+  `createVmExecutor(source).run(job, host, { limit, timeoutMs })` then runs it in a child process with an empty
+  environment and the same VM context as the platform runner, and kills the process on timeout.
+  `createInProcessExecutor(definition)` runs the same jobs without a child process.
+
+### Fixtures
+
+A fixture is a recorded run that replays without the network. Each case lives in
+`fixtures/<case>/` (case names match `^[a-z0-9][a-z0-9_-]{0,63}$`):
+
+- `case.json`: `{ kind: 'pull', stream, mode, state?, streamConfig?, config, secrets?, limit?, expect? }`,
+  `{ kind: 'push', push, config, secrets?, expect? }` or `{ kind: 'discover', config, secrets?, expect? }`.
+- `exchanges.ndjson`: one recorded HTTP, database or crypto exchange per line.
+- `requests.ndjson`: the inbox requests a push case replays (required for push cases).
+
+```typescript
+import ts from 'typescript';
+import {
+  createInProcessExecutor, createLocalGateway, createRecorder, readFixtureDir, runFixtures, runStream, serializeFixture,
+} from '@deepexi/datasense-connector/testing';
+
+// Record: attach a recorder to the local gateway, run, then write the files under fixtures/<case>/.
+const secrets = { apiKey: process.env.DEMO_API_KEY ?? '' };
+const recorder = createRecorder({ secrets });
+const host = createLocalGateway({ hosts: ['eu.api.example.com'], secrets, onExchange: recorder.onExchange });
+await runStream(connector, { stream: 'orders', config: { region: 'eu' }, limit: 20, host });
+const files = serializeFixture('orders-first-page', {
+  case: { kind: 'pull', stream: 'orders', mode: 'full', config: { region: 'eu' }, secrets, limit: 20 },
+  exchanges: recorder.exchanges(),
+});
+
+// Replay: every case must come back ok.
+const results = await runFixtures(createInProcessExecutor(connector), await readFixtureDir('.'));
+```
+
+- Recording replaces every secret value of 4 or more characters with `***`, as well as the values of credential
+  headers (`authorization`, `cookie`, `set-cookie`, `x-api-key`, ...) and of query parameters that contain a
+  secret handle. Handles such as `ontos:secret:apiKey` stay as they are. A database query read in several pages
+  is recorded as one exchange holding every row.
+- During replay, an HTTP request matches a recorded one by method, URL (query sorted by key) and body; a request
+  with no match fails with `FIXTURE_EXCHANGE_MISMATCH` and never reaches the network. `secret.reveal` answers
+  from `case.secrets`, the cache is in memory, and `hash` is computed locally when it was not recorded.
+- `runFixture` / `runFixtures` return a `FixtureCaseResult` per case with `ok`, `findings`, `stats`,
+  `declaration` and `error`. Pull cases run `check` and `discover` first; full-mode pull cases are replayed twice
+  to check that the read is repeatable. Findings: `PRIMARY_KEY_DUPLICATE`, `RECORD_SCHEMA_MISMATCH`,
+  `UNDECLARED_TABLE`, `CHECKPOINT_NOT_MONOTONIC`, `FULL_READ_NOT_REPEATABLE`, `FIXTURE_EXCHANGE_MISMATCH`,
+  `EXPECTATION_FAILED` and `FIXTURE_EXCHANGE_UNUSED`. A case is `ok` when the run succeeds and the only findings
+  are `FIXTURE_EXCHANGE_UNUSED`.
+- `checkCompliance` and `inferRecordSchema` are the same checks as plain functions.
+
+The SDK itself has no runtime dependencies. The root entry does not import Node or Bun built-ins, so it can be
+bundled into a connector and run in an isolated VM context; `./testing` needs Node 20+ or Bun.