@deepexi/datasense-connector 0.0.0-stage → 3.1.0

package/README.md

@@ -1,3 +1,192 @@
-# Temporary Holding Version
+# @deepexi/datasense-connector
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+SDK for writing Datasense connectors, plus tools to run and test them on your own machine.
+
+A connector is a source directory. Its metadata (config fields, streams and their tables, pushes, operations)
+lives in YAML descriptor files; TypeScript files hold only the implementations:
+
+```
+ontos-connector.yaml              name, version, title, description, config, hosts, db, rateLimit
+package.json                      { "dependencies": { "@deepexi/datasense-connector": "^3.0.0" } } and nothing else
+check.ts                          required when there are config fields
+lib/**                            optional shared code
+streams/<name>/stream.yaml        sync modes, params, tables (columns, primary key, extensible)
+streams/<name>/index.ts           export default defineStream('<name>', ...)
+streams/<name>/suggest-columns.ts optional, only for streams with an extensible table
+streams/<name>/tests/<case>/      test cases (case.json + exchanges.ndjson)
+pushes/<name>/push.yaml + index.ts + tests/
+operations/<name>/operation.yaml + index.ts + tests/
+.ontos/                           local only, never uploaded: descriptor.json and types.ts (written by codegen)
+```
+
+The platform parses and validates the YAML into one JSON descriptor (`ConnectorDescriptor`). The SDK never reads
+YAML: at runtime it takes the descriptor embedded in the bundle, locally it reads `.ontos/descriptor.json`.
+`ontos admin connector codegen` writes `.ontos/descriptor.json` and `.ontos/types.ts`; rerun it after every change
+to a YAML file.
+
+Connector code imports only `@deepexi/datasense-connector`, relative files and `.ontos/types`. Node or Bun
+built-ins, other packages, subpaths of this package and import attributes are rejected when the source is bundled.
+
+## `@deepexi/datasense-connector`
+
+- Authoring API, typed by the generated descriptor type `D`:
+  `defineStream`, `definePush`, `defineOperation`, `defineCheck`, `defineSuggestColumns`. Import the bound versions
+  from `.ontos/types` so table names, row types, `ctx.config`, `ctx.secrets`, operation input and output are checked.
+- Descriptor types: `ConnectorDescriptor`, `TableDescriptor`, `ColumnDescriptor`, `ConfigField`, `ValueField`,
+  `StreamDescriptor`, `PushDescriptor`, `OperationDescriptor`, `SuggestedColumn`, `LocalizedText`.
+- `DescriptorTypes<typeof literal>` derives `D` from a descriptor literal written `as const` (handy in tests).
+- `validateConfig(fields, value)`, `validateOperationValue`, `validateSuggestedColumns`, shared with the platform.
+- `ConnectorError` and the `errors` classes (the code is what the platform and the test cases see):
+
+  | Class | Code | Use |
+  | --- | --- | --- |
+  | `errors.AuthError(message)` | `AUTH_ERROR` | Bad or expired credentials. `ctx.http` throws it on a 401 |
+  | `errors.RateLimited(retryAfterMs, message?)` | `RATE_LIMITED` | The source asked to slow down; treated as transient |
+  | `errors.TransientError(message)` | `TRANSIENT_ERROR` | Retryable upstream trouble (502 / 503 / 504, timeouts) |
+  | `errors.SchemaError(message)` | `SCHEMA_ERROR` | The source's data no longer matches the contract |
+  | `errors.FatalError(message)` | `FATAL_ERROR` | Anything else that will not succeed on retry |
+  | `errors.RejectedPayload(message)` | `REJECTED_PAYLOAD` | Push: refuse one inbox request |
+  | `errors.RejectedError(message, { code? })` | `REJECTED` | Operation: the source refused the call and applied nothing |
+  | `errors.IndeterminateError(message)` | `INDETERMINATE` | Operation: you cannot tell whether the source applied the call |
+  | `errors.NotAvailable(feature)` | `NOT_AVAILABLE` | Thrown by reserved features such as `ctx.files` |
+
+- `SDK_VERSION` and `SUPPORTED_SDK_MAJORS`.
+
+```typescript
+// streams/orders/index.ts
+import { defineStream } from '../../.ontos/types';
+
+export default defineStream('orders', async function* (ctx, input) {
+  const since = (input.state as { since?: string } | undefined)?.since;
+  for await (const page of ctx.paginate.cursor({
+    request: (cursor) => ctx.http.get<{ items: Array<{ id: string; updated_at: string }>; next?: string }>(
+      `${ctx.config.baseUrl}/orders`,
+      { query: { cursor, since }, headers: { authorization: `Bearer ${ctx.secrets.apiKey}` } },
+    ),
+    rows: (res) => res.data.items,
+    nextCursor: (res) => res.data.next,
+  })) {
+    for (const row of page.rows) yield ctx.record('orders', row);
+    yield ctx.checkpoint({ since: page.maxOf((row) => row.updated_at) ?? since });
+  }
+});
+```
+
+The `ctx` (`http`, `paginate`, `db`, `map`, `cache`, `crypto`, `time`, `log`, `signal`, `record`, `remove`,
+`checkpoint`) is provided by the runtime. `ctx.files` is reserved and throws `NOT_AVAILABLE`.
+
+- Records are sent in batches of at most 1000 records or 4 MiB. Yielding `ctx.checkpoint(state)` sends the
+  pending batch at once, followed by the checkpoint, which the platform commits after those records are written.
+  A failed run resumes from the last committed checkpoint. Yield a checkpoint after each page.
+- A record field that is not a declared column is dropped by the platform, unless the table is `extensible` and
+  the connection added that field as an extension column. `ontos admin connector test` fails such a case with
+  `UNDECLARED_FIELDS` for non-extensible tables.
+- Outbound hosts are the descriptor's `hosts` plus the host of every `url` config value. Relative URLs in
+  `ctx.http` resolve against the first `url` config value.
+- Secrets are handles (`ontos:secret:<field>`) that the gateway replaces on the way out. Plaintext is only
+  available for secret fields marked `exposeToScript`.
+- `ctx.http` waits and resends on gateway rate limits and upstream 429. Upstream 502 / 503 / 504 and gateway
+  timeouts are retried with backoff only for idempotent methods; pass `retry: true` to retry a POST / PATCH.
+- `ctx.map` runs at most `rateLimit.concurrency` calls at once (the smaller of the descriptor's and the
+  connection's limit).
+
+### Connector operations
+
+An operation is one write call into the source. Its effect, risk, idempotency, input and output are declared in
+`operation.yaml`; `index.ts` implements it:
+
+```typescript
+// operations/create_order/index.ts
+import { errors } from '@deepexi/datasense-connector';
+import { defineOperation } from '../../.ontos/types';
+
+export default defineOperation('create_order', {
+  async run(ctx, params, { idempotencyKey }) {
+    const res = await ctx.http.post(`${ctx.config.baseUrl}/orders`, {
+      headers: { authorization: `Bearer ${ctx.secrets.token}`, 'idempotency-key': idempotencyKey },
+      body: params, raw: true, timeoutMs: 10_000,
+    });
+    const body = res.text ? (JSON.parse(res.text) as Record<string, unknown>) : {};
+    if (res.status === 422) throw new errors.RejectedError(String(body.message), typeof body.code === 'string' ? { code: body.code } : undefined);
+    if (res.status >= 400) throw new errors.FatalError(`upstream returned ${res.status}`);
+    return { externalId: String(body.id), status: String(body.status) };
+  },
+  simulate: (params) => ({ externalId: `simulated-${params.shopCode}`, status: 'simulated' }),
+});
+```
+
+- With `idempotency: unsupported`, `lookup(ctx, { idempotencyKey, params })` is required (a type error
+  otherwise); it answers `{ status: 'applied', output }`, `{ status: 'not_applied' }` or `{ status: 'unknown' }`.
+- The platform checks `params` against `input` before every phase (`INPUT_INVALID`) and the result against
+  `output` (`OUTPUT_INVALID`).
+- `simulate(params)` is a pure function used by dry runs and simulations.
+
+### Suggested columns
+
+A stream with an extensible table may add `suggest-columns.ts`, which reads the fields the source offers so a
+connection can pick extension columns:
+
+```typescript
+import { defineSuggestColumns } from '../../.ontos/types';
+
+export default defineSuggestColumns('customers', async (ctx, { table }) => [
+  { name: 'vip_level', type: 'string', title: 'VIP level' },
+]);
+```
+
+Names must match `^[A-Za-z][A-Za-z0-9_]{0,62}$` and must not start with `_ontos_`; suggestions that repeat a
+base column are dropped.
+
+## `@deepexi/datasense-connector/testing`
+
+Runs a connector locally with a local gateway that answers its outbound HTTP, database, crypto, cache and secret
+calls the way the platform does. Connector source must not import this entry.
+
+```typescript
+import ts from 'typescript';
+import { loadLocalConnector, runTests } from '@deepexi/datasense-connector/testing';
+
+const connector = await loadLocalConnector('.', { typescript: ts });
+const report = await runTests(connector);
+// report: { ok, cases: [{ capability, name, ok, failure }], covered, uncovered }
+```
+
+- `loadLocalConnector(dir, { typescript })` reads `.ontos/descriptor.json` (missing: `DESCRIPTOR_NOT_GENERATED`,
+  run codegen first), checks imports with the platform's rules, transpiles the sources with your TypeScript and
+  assembles the same entry module as the platform. It also collects the test case files.
+- `runTests(connector, { cases? })` replays every test case in a child process with an empty environment and the
+  platform runner's VM context. Every stream, push and operation needs at least one case; `uncovered` lists the
+  capabilities without one (such as `streams/customers`), and `ok` is false while any case fails or anything is
+  uncovered. Suggest cases run but do not count for coverage.
+- `runStream`, `runCheck`, `runPush`, `runSuggestColumns` run an imported module `{ descriptor, implementations }`
+  in the current process and return a `LocalRunResult` (`records`, `removals`, `checkpoints`, `logs`, `rejected`,
+  `suggest`, `stopped`, `error`). `runOperation(executor, input)` runs one operation phase.
+- `createLocalGateway({ hosts, secrets, rateLimit?, db?, fetch?, onExchange? })` is the local gateway; `db`
+  needs the optional peer dependency `postgres`.
+- `createMockExternalSystem(routes)` is a scriptable fake of the source; pass its `fetch` to `createLocalGateway`.
+- `createVmExecutor(connector)` / `createInProcessExecutor(module)` run guest jobs directly.
+
+### Test cases
+
+A test case is a recorded run that replays without the network. It lives in the capability's directory,
+`<streams|pushes|operations>/<name>/tests/<case>/` (case names match `^[a-z0-9][a-z0-9_-]{0,63}$`):
+
+- `case.json`: the kind and the capability name follow from the directory.
+  - stream: `{ mode, state?, streamConfig?, config, secrets?, limit?, expect? }`
+  - push: `{ config, secrets?, expect? }`, plus `requests.ndjson` with the inbox requests to replay
+  - operation: `{ params, idempotencyKey, config, secrets?, expect: { ok: <output> } | { error: '<code>' } }`
+  - suggest (under a stream): `{ "kind": "suggest", table, config, secrets?, expect: { columns } }`
+- `exchanges.ndjson`: one recorded HTTP, database or crypto exchange per line.
+
+Recording (`createRecorder` attached to the local gateway, then `serializeFixture`) replaces secret values of 4 or
+more characters, credential headers and query parameters holding a handle with `***`. During replay a request
+with no recorded match fails with `FIXTURE_EXCHANGE_MISMATCH` and never reaches the network.
+
+Findings: `UNDECLARED_FIELDS`, `PRIMARY_KEY_DUPLICATE`, `RECORD_SCHEMA_MISMATCH`, `UNDECLARED_TABLE`,
+`CHECKPOINT_NOT_MONOTONIC`, `FULL_READ_NOT_REPEATABLE` (full-mode cases are replayed twice),
+`FIXTURE_EXCHANGE_MISMATCH`, `EXPECTATION_FAILED` and `FIXTURE_EXCHANGE_UNUSED`. A case is ok when the run succeeds
+and the only findings are `FIXTURE_EXCHANGE_UNUSED`. `checkCompliance` and `inferRecordSchema` are the same checks
+as plain functions.
+
+The SDK itself has no runtime dependencies. The root entry does not import Node or Bun built-ins, so it can be
+bundled into a connector and run in an isolated VM context; `./testing` needs Node 20+ or Bun.

package/package.json

@@ -1,6 +1,35 @@
 {
   "name": "@deepexi/datasense-connector",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "3.1.0",
+  "module": "src/index.ts",
+  "type": "module",
+  "private": false,
+  "exports": {
+    ".": "./src/index.ts",
+    "./testing": "./src/testing/index.ts",
+    "./runtime": "./src/runtime/index.ts"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "build": "bun scripts/build.ts",
+    "test": "NODE_ENV=test bun test",
+    "lint": "eslint --fix",
+    "lint:check": "eslint . --max-warnings 0",
+    "ts-check": "tsc --noEmit"
+  },
+  "peerDependencies": {
+    "postgres": "^3.4.9"
+  },
+  "peerDependenciesMeta": {
+    "postgres": {
+      "optional": true
+    }
+  },
+  "devDependencies": {
+    "@mu-ontos/shared": "workspace:*",
+    "postgres": "^3.4.9",
+    "typescript": "^6.0.3"
+  }
+}

package/scripts/build.ts

@@ -0,0 +1,188 @@
+/**
+ * 构建可直接 `npm publish` 的发布目录（缺省 `dist/`，`--outdir <dir>` 可改，测试用）：
+ *
+ * 1. `tsc -p tsconfig.build.json`：ESM `.js`（去掉注释）与 `.d.ts`（保留注释，给连接器作者看）；之后把相对导入补成 `.js` / `/index.js`（Node 的 ESM 要求带扩展名）；
+ * 2. Bun.build 另出 CJS 版本：`cjs/index.cjs`、`cjs/testing.cjs`、`cjs/runtime.cjs`（`src/internal/location.ts` 换成 `__dirname` 版本）；
+ * 3. `guest.cjs`：guest 运行时连同 SDK 打成一个 CJS 串（参数与 shared 的 `bundleTrustedEntry` 相同：cjs、browser、无外部依赖、`macros: false`）；
+ * 4. `testing/child.cjs`：`createVmExecutor` 的子进程入口（Node 可直接执行）；
+ * 5. `package.json`（`exports` 的 `.`、`./testing`、`./runtime` 各有 `types` / `import` / `require`，`dependencies` 为空，
+ *    `postgres` 是可选 peerDependency）与 `README.md`。
+ *
+ * 不改写工作区的 `package.json`；`postgres` 与 `typescript` 不打进产物。
+ */
+import { existsSync } from 'node:fs';
+import { mkdir, readdir, readFile, rm, writeFile } from 'node:fs/promises';
+import path from 'node:path';
+import { bundleGuestWithBun } from '../src/runtime/guest-runtime';
+
+const PACKAGE_DIR = path.resolve(import.meta.dir, '..');
+const SRC = path.join(PACKAGE_DIR, 'src');
+
+function parseOutdir(argv: readonly string[]): string {
+  const index = argv.indexOf('--outdir');
+  const value = index >= 0 ? argv[index + 1] : undefined;
+  return value ? path.resolve(value) : path.join(PACKAGE_DIR, 'dist');
+}
+
+const OUT = parseOutdir(process.argv.slice(2));
+
+async function run(command: string[], cwd: string): Promise<void> {
+  const child = Bun.spawn(command, { cwd, stdout: 'inherit', stderr: 'inherit' });
+  const code = await child.exited;
+  if (code !== 0) throw new Error(`${command.join(' ')} exited with ${code}`);
+}
+
+async function listFiles(dir: string): Promise<string[]> {
+  const out: string[] = [];
+  for (const entry of await readdir(dir, { withFileTypes: true })) {
+    const full = path.join(dir, entry.name);
+    if (entry.isDirectory()) out.push(...await listFiles(full));
+    else out.push(full);
+  }
+  return out;
+}
+
+// ---------------------------------------------------------------------------
+// 1. ESM + 声明文件
+
+const RELATIVE_SPECIFIER = /(\bfrom\s*|\bimport\s*\(\s*|\bimport\s+)(['"])(\.{1,2}(?:\/[^'"]*)?)\2/g;
+
+/** 相对导入补扩展名：`./x` → `./x.js`（或 `./x/index.js`）；已带扩展名的不动。 */
+function rewriteSpecifiers(file: string, text: string): string {
+  const dir = path.dirname(file);
+  return text.replace(RELATIVE_SPECIFIER, (match, prefix: string, quote: string, specifier: string) => {
+    if (/\.(?:[cm]?js|json)$/.test(specifier)) return match;
+    const target = path.resolve(dir, specifier);
+    if (existsSync(`${target}.js`)) return `${prefix}${quote}${specifier}.js${quote}`;
+    if (existsSync(path.join(target, 'index.js'))) return `${prefix}${quote}${specifier.replace(/\/$/, '')}/index.js${quote}`;
+    throw new Error(`cannot resolve ${specifier} from ${path.relative(OUT, file)}`);
+  });
+}
+
+async function buildEsm(): Promise<void> {
+  // 源码注释多是平台内部的实现说明，不随包发布；第二遍只出声明，覆盖第一遍去掉了注释的 `.d.ts`。
+  await run([process.execPath, 'x', 'tsc', '-p', 'tsconfig.build.json', '--outDir', OUT, '--removeComments'], PACKAGE_DIR);
+  await run([process.execPath, 'x', 'tsc', '-p', 'tsconfig.build.json', '--outDir', OUT, '--emitDeclarationOnly'], PACKAGE_DIR);
+  for (const file of await listFiles(OUT)) {
+    if (!file.endsWith('.js') && !file.endsWith('.d.ts')) continue;
+    const text = await readFile(file, 'utf8');
+    const rewritten = rewriteSpecifiers(file, text);
+    if (rewritten !== text) await writeFile(file, rewritten);
+  }
+}
+
+// ---------------------------------------------------------------------------
+// 2–4. Bun.build 产物
+
+/** CJS 版本里 `location.ts` 的替身：dist/cjs/*.cjs 的上一级就是 dist/。 */
+const CJS_LOCATION = `import path from 'node:path';
+export const IS_SOURCE = false;
+export const PACKAGE_ROOT = path.resolve(__dirname, '..');
+export function moduleFile(relative) { return path.join(PACKAGE_ROOT, relative + '.js'); }
+`;
+
+const LOCATION_FILE = path.join(SRC, 'internal', 'location.ts');
+const CODE_FILE = /\.[cm]?[jt]sx?$/;
+
+/** 源文件自己读（Bun 1.3+ 已 import 过的文件再交给 Bun.build 会报错）；CJS 构建时替换 `location.ts`。 */
+function sourcePlugin(replaceLocation: boolean): Bun.BunPlugin {
+  return {
+    name: 'datasense-connector-build',
+    setup(build) {
+      build.onLoad({ filter: CODE_FILE }, async (args) => {
+        if (replaceLocation && path.resolve(args.path) === LOCATION_FILE) return { contents: CJS_LOCATION, loader: 'js' };
+        return { contents: await Bun.file(args.path).text(), loader: args.path.endsWith('.tsx') ? 'tsx' : 'ts' };
+      });
+    },
+  };
+}
+
+const MACROS_OFF = { macros: false } as unknown as Partial<Parameters<typeof Bun.build>[0]>;
+
+async function bundleCjs(entry: string, outFile: string): Promise<void> {
+  const result = await Bun.build({
+    entrypoints: [entry],
+    format: 'cjs',
+    target: 'node',
+    minify: false,
+    external: ['postgres', 'typescript'],
+    plugins: [sourcePlugin(true)],
+    throw: false,
+    ...MACROS_OFF,
+  });
+  const output = result.outputs.find((item) => item.kind === 'entry-point');
+  if (!result.success || !output) throw new Error(`bundle ${path.relative(PACKAGE_DIR, entry)} failed: ${result.logs.map((log) => log.message).join('; ')}`);
+  // Bun 把 `__dirname` / `__filename` 内联成构建机上的源文件路径（`var __dirname = "/…/src/internal";`）；
+  // 去掉这些声明，让代码用 Node CommonJS 包装函数给的真实值。
+  const text = (await output.text()).replace(/^var (__dirname|__filename) = "[^"\n]*";\n/gm, '');
+  await mkdir(path.dirname(outFile), { recursive: true });
+  await writeFile(outFile, text);
+}
+
+/** 产物里不得出现构建机的源码路径。 */
+async function assertNoBuildPaths(): Promise<void> {
+  for (const file of await listFiles(OUT)) {
+    if (!/\.(?:c?js|d\.ts)$/.test(file)) continue;
+    if ((await readFile(file, 'utf8')).includes(PACKAGE_DIR)) throw new Error(`${path.relative(OUT, file)} contains the build machine path ${PACKAGE_DIR}`);
+  }
+}
+
+// ---------------------------------------------------------------------------
+// 5. package.json 与 README
+
+interface WorkspacePackage {
+  name: string;
+  version: string;
+  peerDependencies?: Record<string, string>;
+  peerDependenciesMeta?: Record<string, { optional?: boolean }>;
+}
+
+function exportEntry(base: string, cjs: string): Record<string, string> {
+  return { types: `./${base}.d.ts`, import: `./${base}.js`, require: `./cjs/${cjs}.cjs` };
+}
+
+async function writeManifest(): Promise<void> {
+  const workspace = JSON.parse(await readFile(path.join(PACKAGE_DIR, 'package.json'), 'utf8')) as WorkspacePackage;
+  const manifest = {
+    name: workspace.name,
+    version: workspace.version,
+    description: 'SDK for writing Datasense connectors: defineStream / definePush / defineOperation, the guest runtime, and local testing tools.',
+    license: 'UNLICENSED',
+    type: 'module',
+    main: './cjs/index.cjs',
+    module: './index.js',
+    types: './index.d.ts',
+    exports: {
+      '.': exportEntry('index', 'index'),
+      './testing': exportEntry('testing/index', 'testing'),
+      './runtime': exportEntry('runtime/index', 'runtime'),
+      './package.json': './package.json',
+    },
+    sideEffects: false,
+    engines: { node: '>=20' },
+    dependencies: {},
+    peerDependencies: workspace.peerDependencies ?? {},
+    peerDependenciesMeta: workspace.peerDependenciesMeta ?? {},
+    publishConfig: { access: 'public' },
+  };
+  await writeFile(path.join(OUT, 'package.json'), `${JSON.stringify(manifest, null, 2)}\n`);
+  await writeFile(path.join(OUT, 'README.md'), await readFile(path.join(PACKAGE_DIR, 'README.md'), 'utf8'));
+}
+
+// ---------------------------------------------------------------------------
+
+async function main(): Promise<void> {
+  await rm(OUT, { recursive: true, force: true });
+  await mkdir(OUT, { recursive: true });
+  await buildEsm();
+  await bundleCjs(path.join(SRC, 'index.ts'), path.join(OUT, 'cjs', 'index.cjs'));
+  await bundleCjs(path.join(SRC, 'testing', 'index.ts'), path.join(OUT, 'cjs', 'testing.cjs'));
+  await bundleCjs(path.join(SRC, 'runtime', 'index.ts'), path.join(OUT, 'cjs', 'runtime.cjs'));
+  await bundleCjs(path.join(SRC, 'testing', 'child.ts'), path.join(OUT, 'testing', 'child.cjs'));
+  await writeFile(path.join(OUT, 'guest.cjs'), await bundleGuestWithBun(path.join(SRC, 'runtime', 'guest', 'index.ts')));
+  await writeManifest();
+  await assertNoBuildPaths();
+  console.log(`built ${path.relative(process.cwd(), OUT) || OUT}`);
+}
+
+await main();