@elench/testkit 0.1.17 → 0.1.19

package/README.md

@@ -1,6 +1,6 @@
 # @elench/testkit
 
-CLI that reads `runner.manifest.json` plus `testkit.config.json` from a product repo, starts the required local services, provisions the selected database backend, and runs manifest-defined suites across `k6` and Playwright.
+CLI that reads `runner.manifest.json` plus `testkit.config.json` from a product repo, starts the required local services, provisions local Postgres isolation, and runs manifest-defined suites across `k6` and Playwright.
 
 ## Prerequisites
 
@@ -9,14 +9,14 @@ sudo snap install k6
 sudo apt-get install -y jq
 ```
 
-For DAL suites, `@elench/testkit` ships its own `k6` SQL binary. For suites using a Neon branch, set `NEON_API_KEY` in the product `.env`, shell, or `.envrc`.
+For DAL suites, `@elench/testkit` ships its own `k6` SQL binary.
 
-Local database mode uses Docker-managed `postgres:16` containers. `testkit` creates and reuses template databases automatically, then clones per-worker databases from those templates for fast reruns.
+Database isolation uses Docker-managed local Postgres containers. `testkit` creates and reuses template databases automatically, then clones per-worker databases from those templates for fast reruns. The default image is `pgvector/pgvector:pg16`.
 
 ## Usage
 
 ```bash
-cd bourne
+cd my-product
 
 # Run every testkit-managed suite
 npx @elench/testkit
@@ -26,10 +26,6 @@ npx @elench/testkit int
 npx @elench/testkit dal
 npx @elench/testkit e2e
 
-# Force a database backend
-npx @elench/testkit --db-backend local
-npx @elench/testkit --db-backend neon
-
 # Filter by framework
 npx @elench/testkit --framework playwright
 npx @elench/testkit --framework k6
@@ -43,26 +39,74 @@ npx @elench/testkit --jobs 2 --shard 2/3
 
 # Specific service / suite
 npx @elench/testkit frontend e2e -s auth
-npx @elench/testkit bourne int -s health
+npx @elench/testkit api int -s health
 
 # Lifecycle
 npx @elench/testkit status
 npx @elench/testkit destroy
 ```
 
+## K6 Authoring
+
+Consumer k6 tests can import the shared authoring API directly from the package:
+
+```js
+import { defineHttpSuite } from "@elench/testkit";
+
+const suite = defineHttpSuite(({ rawReq }) => {
+  const res = rawReq("GET", "/health");
+});
+
+export const options = suite.options;
+export const setup = suite.setup;
+export default suite.exec;
+```
+
+For auth or schema-specific behavior, keep a small consumer-owned adapter next
+to the tests and pass it into the generic suite factory:
+
+```js
+import { defineHttpSuite } from "@elench/testkit";
+import { clerkSessionAuth } from "../helpers/testkit-auth.js";
+
+const suite = defineHttpSuite({ auth: clerkSessionAuth }, ({ req, setupData }) => {
+  req("GET", "/api/auth/me", setupData);
+});
+```
+
+`testkit` bundles these imports before invoking k6, so tests do not need
+generated `_testkit` files or direct package-manager path imports.
+
+Legacy compatibility:
+
+- `testkit runtime install`
+- `testkit runtime status`
+- `testkit runtime update`
+
+still exist, but direct package imports are now the preferred model.
+
+From outside the product repo, use `--dir` explicitly:
+
+```bash
+npx @elench/testkit --dir my-product int
+npx @elench/testkit --dir my-product api int -s health
+```
+
 ## How it works
 
 1. **Discovery** — reads `runner.manifest.json` for services, suites, files, and frameworks
 2. **Config** — reads `testkit.config.json` for local runtime, migration, dependency, and database settings
    Per-service `.env` files declared in config are loaded when present.
-3. **Database** — provisions the selected backend (`neon` or Docker-managed `local`) when a service declares one
+3. **Database** — provisions Docker-managed local Postgres when a service declares one
 4. **Seed** — runs optional product seed commands against the provisioned database
 5. **Runtime** — starts required local services, waits for readiness, and injects test env
-6. **Execution** — schedules file-level execution tasks across a global worker pool, reuses warm dependency graphs when possible, runs `k6` file-by-file, and batches Playwright files per worker
+6. **Execution** — schedules file-level execution tasks across a global worker pool, reuses warm dependency graphs when possible, bundles `k6` files before execution so package imports resolve cleanly, and batches Playwright files per worker
 7. **Cleanup** — stops local processes and preserves `.testkit/` state for inspection or later destroy
 
 Each run ends with a compact summary that shows per-service pass/fail status, suite totals, failed suites, and total duration.
 
+`testkit` also writes `.testkit/results/latest.json` for every run. When `testkit.config.json` includes a top-level `telemetry` block, it can best-effort upload that artifact to a configured HTTP endpoint with bearer auth.
+
 ## File roles
 
 - `runner.manifest.json`: canonical test inventory
@@ -70,11 +114,12 @@ Each run ends with a compact summary that shows per-service pass/fail status, su
 
 `testkit.config.json` can also declare:
 
+- `telemetry` for optional generic HTTP result upload
 - `envFile` / `envFiles` for service-specific environment loading
 - `databaseFrom` for dependency services that must share another service's `DATABASE_URL`
-- `database.defaultBackend` / `database.backends` to switch between Neon and local Postgres
+- `database.provider` for local Postgres settings
 - `database.template.inputs` to define the local template cache invalidation inputs
-- `migrate.backends` / `seed.backends` for backend-specific command overrides
+- `migrate.backends` / `seed.backends` for optional local-only command overrides
 
 ## Parallel execution
 
@@ -83,12 +128,12 @@ Each run ends with a compact summary that shows per-service pass/fail status, su
 `--jobs` is global for the whole run, not per service.
 
 Each worker gets its own:
-- Neon branch
-- or cloned local Postgres database
+
+- cloned local Postgres database
 - `.testkit` state subtree
 - local service ports
 
-Workers prefer to stay on the same dependency graph so service stacks can be reused across assigned work, including dependent services such as `frontend -> bourne`.
+Workers prefer to stay on the same dependency graph so service stacks can be reused across assigned work, including dependent services such as `frontend -> api`.
 
 Use `--shard <i/n>` to split the selected suite set deterministically before worker scheduling.
 
@@ -115,3 +160,18 @@ Use `--shard <i/n>` to split the selected suite set deterministically before wor
 ## Schema
 
 See [testkit-config-schema.md](testkit-config-schema.md).
+
+## Development Tests
+
+```bash
+npm test
+npm run test:unit
+npm run test:integration
+npm run test:system
+```
+
+`test:system` runs real end-to-end fixtures from `test/fixtures/system/`. It requires:
+
+- Docker with a running daemon
+- `k6` on `PATH`
+- Playwright Chromium browser assets available to `@playwright/test`

package/bin/testkit.mjs

@@ -1,3 +1,3 @@
 #!/usr/bin/env node
-import { run } from "../lib/cli.mjs";
+import { run } from "../lib/cli/index.mjs";
 run();

package/lib/bundler/index.mjs

@@ -0,0 +1,95 @@
+import fs from "fs";
+import os from "os";
+import path from "path";
+import crypto from "crypto";
+import { build } from "esbuild";
+import { fileURLToPath } from "url";
+
+const PACKAGE_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
+const ROOT_ENTRY = path.join(PACKAGE_ROOT, "lib", "index.mjs");
+const K6_ENTRY = path.join(PACKAGE_ROOT, "lib", "k6", "index.mjs");
+const K6_DIR = path.join(PACKAGE_ROOT, "lib", "k6");
+const bundleCache = new Map();
+
+export async function bundleK6File({
+  productDir,
+  serviceName,
+  sourceFile,
+}) {
+  const absoluteSource = path.resolve(productDir, sourceFile);
+  const bundleDir = path.join(productDir, ".testkit", "_bundles", serviceName || "shared");
+  fs.mkdirSync(bundleDir, { recursive: true });
+
+  const cacheKey = await buildCacheKey(absoluteSource);
+  const cached = bundleCache.get(cacheKey);
+  if (cached && fs.existsSync(cached)) {
+    return cached;
+  }
+
+  const outputFile = path.join(
+    bundleDir,
+    `${path.basename(sourceFile, path.extname(sourceFile))}-${cacheKey.slice(0, 12)}.js`
+  );
+
+  await build({
+    absWorkingDir: path.dirname(absoluteSource),
+    bundle: true,
+    entryPoints: [absoluteSource],
+    format: "esm",
+    legalComments: "none",
+    outfile: outputFile,
+    platform: "neutral",
+    sourcemap: "inline",
+    target: "es2020",
+    plugins: [testkitPackageAliasPlugin()],
+    external: [
+      "k6",
+      "k6/*",
+    ],
+  });
+
+  bundleCache.set(cacheKey, outputFile);
+  return outputFile;
+}
+
+async function buildCacheKey(sourceFile) {
+  const source = await fs.promises.readFile(sourceFile, "utf8");
+  const packageJson = await fs.promises.readFile(path.join(PACKAGE_ROOT, "package.json"), "utf8");
+  return crypto
+    .createHash("sha256")
+    .update(sourceFile)
+    .update("\0")
+    .update(source)
+    .update("\0")
+    .update(packageJson)
+    .digest("hex");
+}
+
+function testkitPackageAliasPlugin() {
+  return {
+    name: "testkit-package-alias",
+    setup(buildApi) {
+      buildApi.onResolve({ filter: /^@elench\/testkit(?:\/.*)?$/ }, (args) => {
+        return {
+          namespace: "file",
+          path: resolvePackageSubpath(args.path),
+        };
+      });
+    },
+  };
+}
+
+function resolvePackageSubpath(specifier) {
+  const subpath = specifier.slice("@elench/testkit".length);
+  if (!subpath) return ROOT_ENTRY;
+  if (subpath === "/k6") return K6_ENTRY;
+  if (subpath.startsWith("/k6/")) {
+    const rel = subpath.slice("/k6/".length);
+    const candidate = path.join(K6_DIR, `${rel}.mjs`);
+    if (fs.existsSync(candidate)) {
+      return candidate;
+    }
+  }
+
+  throw new Error(`Unsupported @elench/testkit import "${specifier}" in ${os.platform()}`);
+}

package/lib/bundler/index.test.mjs

@@ -0,0 +1,79 @@
+import fs from "fs";
+import os from "os";
+import path from "path";
+import { afterEach, describe, expect, it } from "vitest";
+import { bundleK6File } from "./index.mjs";
+
+const cleanups = [];
+
+afterEach(() => {
+  while (cleanups.length > 0) {
+    cleanups.pop()();
+  }
+});
+
+describe("k6 bundler", () => {
+  it("bundles root package imports for k6 execution", async () => {
+    const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "testkit-bundle-"));
+    cleanups.push(() => fs.rmSync(tmpDir, { force: true, recursive: true }));
+
+    const sourceFile = path.join(tmpDir, "health.js");
+    fs.writeFileSync(
+      sourceFile,
+      [
+        'import { defineHttpSuite, json } from "@elench/testkit";',
+        'import { check } from "k6";',
+        "const suite = defineHttpSuite(({ rawReq }) => {",
+        '  const res = rawReq("GET", "/health");',
+        "  check(json(res), {",
+        '    "has status": (body) => typeof body.status === "string",',
+        "  });",
+        "});",
+        "export const options = suite.options;",
+        "export const setup = suite.setup;",
+        "export default suite.exec;",
+        "",
+      ].join("\n")
+    );
+
+    const bundledFile = await bundleK6File({
+      productDir: tmpDir,
+      serviceName: "api",
+      sourceFile,
+    });
+
+    const bundled = fs.readFileSync(bundledFile, "utf8");
+    expect(bundled).toContain("defineHttpSuite");
+    expect(bundled).toContain('import { check } from "k6"');
+  });
+
+  it("bundles subpath package imports for DAL execution", async () => {
+    const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "testkit-bundle-"));
+    cleanups.push(() => fs.rmSync(tmpDir, { force: true, recursive: true }));
+
+    const sourceFile = path.join(tmpDir, "dal.js");
+    fs.writeFileSync(
+      sourceFile,
+      [
+        'import { defineDalSuite } from "@elench/testkit/k6";',
+        "const suite = defineDalSuite(({ db }) => {",
+        '  db.query("SELECT 1");',
+        "});",
+        "export const options = suite.options;",
+        "export const setup = suite.setup;",
+        "export default suite.exec;",
+        "",
+      ].join("\n")
+    );
+
+    const bundledFile = await bundleK6File({
+      productDir: tmpDir,
+      serviceName: "api",
+      sourceFile,
+    });
+
+    const bundled = fs.readFileSync(bundledFile, "utf8");
+    expect(bundled).toContain("defineDalSuite");
+    expect(bundled).toContain('import sql from "k6/x/sql"');
+  });
+});

package/lib/cli/args.mjs

@@ -0,0 +1,57 @@
+export const SUITE_TYPES = new Set(["int", "integration", "e2e", "dal", "load", "all"]);
+export const LIFECYCLE = new Set(["status", "destroy"]);
+export const RESERVED = new Set([...SUITE_TYPES, ...LIFECYCLE]);
+
+export function resolveCliSelection({ first, second, serviceNames }) {
+  let service = null;
+  let type = null;
+
+  if (first && serviceNames.has(first)) {
+    service = first;
+    type = second || null;
+  } else if (first && RESERVED.has(first)) {
+    type = first;
+  } else if (first) {
+    throw new Error(
+      `Unknown argument "${first}". Expected a service name (${[...serviceNames].join(", ") || "none found"}) ` +
+      `or suite type (int, e2e, dal, all).`
+    );
+  }
+
+  return { service, type };
+}
+
+export function validateFrameworkOption(value) {
+  if (!["all", "k6", "playwright"].includes(value)) {
+    throw new Error(
+      `Unknown framework "${value}". Expected one of: all, k6, playwright.`
+    );
+  }
+}
+
+export function parseJobsOption(value) {
+  const jobs = Number.parseInt(String(value), 10);
+  if (!Number.isInteger(jobs) || jobs <= 0) {
+    throw new Error(`Invalid --jobs value "${value}". Expected a positive integer.`);
+  }
+  return jobs;
+}
+
+export function parseShardOption(value) {
+  if (!value) return null;
+
+  const match = String(value).match(/^(\d+)\/(\d+)$/);
+  if (!match) {
+    throw new Error(
+      `Invalid --shard value "${value}". Expected the form "i/n", e.g. 1/3.`
+    );
+  }
+
+  const index = Number.parseInt(match[1], 10);
+  const total = Number.parseInt(match[2], 10);
+  if (index <= 0 || total <= 0 || index > total) {
+    throw new Error(`Invalid --shard value "${value}". Expected 1 <= i <= n.`);
+  }
+
+  return { index, total };
+}

package/lib/cli/args.test.mjs

@@ -0,0 +1,62 @@
+import { describe, expect, it } from "vitest";
+import {
+  parseJobsOption,
+  parseShardOption,
+  resolveCliSelection,
+  validateFrameworkOption,
+} from "./args.mjs";
+
+describe("cli-args", () => {
+  it("resolves a service and suite type", () => {
+    expect(
+      resolveCliSelection({
+        first: "api",
+        second: "int",
+        serviceNames: new Set(["api", "frontend"]),
+      })
+    ).toEqual({
+      service: "api",
+      type: "int",
+    });
+  });
+
+  it("resolves a reserved suite type without a service", () => {
+    expect(
+      resolveCliSelection({
+        first: "e2e",
+        second: null,
+        serviceNames: new Set(["api"]),
+      })
+    ).toEqual({
+      service: null,
+      type: "e2e",
+    });
+  });
+
+  it("rejects unknown positional arguments", () => {
+    expect(() =>
+      resolveCliSelection({
+        first: "mystery",
+        second: null,
+        serviceNames: new Set(["api"]),
+      })
+    ).toThrow('Unknown argument "mystery"');
+  });
+
+  it("validates framework names", () => {
+    expect(() => validateFrameworkOption("playwright")).not.toThrow();
+    expect(() => validateFrameworkOption("jest")).toThrow("Unknown framework");
+  });
+
+  it("parses and validates jobs", () => {
+    expect(parseJobsOption("3")).toBe(3);
+    expect(() => parseJobsOption("0")).toThrow("Invalid --jobs value");
+  });
+
+  it("parses and validates shards", () => {
+    expect(parseShardOption("2/5")).toEqual({ index: 2, total: 5 });
+    expect(parseShardOption(null)).toBeNull();
+    expect(() => parseShardOption("2-of-5")).toThrow("Invalid --shard value");
+    expect(() => parseShardOption("3/2")).toThrow("Expected 1 <= i <= n");
+  });
+});

package/lib/cli/index.mjs

@@ -0,0 +1,114 @@
+import { cac } from "cac";
+import { loadConfigs, getServiceNames } from "../config/index.mjs";
+import {
+  parseJobsOption,
+  parseShardOption,
+  RESERVED,
+  resolveCliSelection,
+  validateFrameworkOption,
+} from "./args.mjs";
+import * as runner from "../runner/index.mjs";
+import * as runtime from "../runtime/index.mjs";
+
+export function run() {
+  const cli = cac("testkit");
+
+  cli
+    .command("runtime <action>", "Install or inspect the consumer runtime bundle")
+    .option("--dir <path>", "Explicit product directory")
+    .option("--path <path>", "Target runtime path relative to the product directory")
+    .option("--strict", "Exit non-zero when runtime status is missing or drifted")
+    .action((action, options) => {
+      if (!["install", "status", "update"].includes(action)) {
+        throw new Error('Unknown runtime action. Expected one of: install, status, update.');
+      }
+
+      if (action === "status") {
+        const status = runtime.getRuntimeStatus(options);
+        console.log(runtime.formatRuntimeStatus(status));
+        if (options.strict && status.status !== "installed") {
+          process.exitCode = 1;
+        }
+        return;
+      }
+
+      const result = runtime.installRuntime(options);
+      console.log(
+        `Installed testkit runtime to ${result.relativeRuntimeDir} (${result.files.length} file${result.files.length === 1 ? "" : "s"})`
+      );
+    });
+
+  cli
+    .command("[first] [second] [third]", "Run test suites (int, e2e, dal, all)")
+    .option("-s, --suite <name>", "Run specific suite(s)", { default: [] })
+    .option("--dir <path>", "Explicit product directory")
+    .option("--jobs <n>", "Number of isolated worker stacks for the whole run", {
+      default: "1",
+    })
+    .option("--shard <i/n>", "Run only shard i of n at suite granularity")
+    .option("--framework <name>", "Filter by framework (k6, playwright, all)", {
+      default: "all",
+    })
+    .action(async (first, second, third, options) => {
+      // Resolve: service filter, suite type, and --dir.
+      //
+      // From product dir:
+      //   testkit                          → all services, all types
+      //   testkit int -s health            → all services, int, health
+      //   testkit api int                  → one service, int
+      //   testkit api                      → one service, all types
+      //
+      // From workspace root:
+      //   testkit --dir my-product int     → all services, int
+      //   testkit --dir my-product api int → one service, int
+
+      // Now resolve service vs type from remaining args
+      const serviceNames = new Set(getServiceNames(options.dir));
+      const { service, type } = resolveCliSelection({
+        first,
+        second,
+        serviceNames,
+      });
+
+      const allConfigs = loadConfigs({ dir: options.dir });
+      const configs = service
+        ? allConfigs.filter((config) => config.name === service)
+        : allConfigs;
+      if (service && configs.length === 0) {
+        const available = allConfigs.map((config) => config.name).join(", ");
+        throw new Error(`Service "${service}" not found. Available: ${available}`);
+      }
+
+      // Lifecycle commands
+      if (type === "status" || type === "destroy") {
+        for (const config of configs) {
+          if (configs.length > 1) console.log(`\n── ${config.name} ──`);
+          if (type === "status") runner.showStatus(config);
+          else await runner.destroy(config);
+        }
+        return;
+      }
+
+      validateFrameworkOption(options.framework);
+
+      const jobs = parseJobsOption(options.jobs);
+      const shard = parseShardOption(options.shard);
+
+      const suiteType = type || "all";
+      const suiteNames = Array.isArray(options.suite) ? options.suite : [options.suite].filter(Boolean);
+      await runner.runAll(
+        configs,
+        suiteType,
+        suiteNames,
+        {
+          ...options,
+          jobs,
+          shard,
+        },
+        allConfigs
+      );
+    });
+
+  cli.help();
+  cli.parse();
+}