@elench/testkit 0.1.26 → 0.1.28

package/README.md

@@ -1,12 +1,10 @@
 # @elench/testkit
 
-CLI that reads `testkit.config.json` from a product repo, discovers `*.testkit.ts` files by convention, starts the required local services, provisions local Postgres isolation, and runs HTTP, DAL, and Playwright suites.
+`@elench/testkit` discovers `*.testkit.ts` files, infers suite ownership from the
+filesystem, starts local services, provisions isolated local Postgres databases,
+and runs HTTP, DAL, and Playwright suites.
 
-## Prerequisites
-
-`@elench/testkit` ships its own execution engine for HTTP and DAL suites. Consumers do not need to install or invoke any separate load-testing binary.
-
-Database isolation uses Docker-managed local Postgres containers. `testkit` creates and reuses template databases automatically, then clones per-worker databases from those templates for fast reruns. The default image is `pgvector/pgvector:pg16`.
+The package is now driven by `testkit.setup.ts`, not `testkit.config.json`.
 
 ## Usage
 
@@ -30,14 +28,13 @@ npx @elench/testkit --jobs 3
 
 # Run a deterministic shard
 npx @elench/testkit --shard 1/3
-npx @elench/testkit --jobs 2 --shard 2/3
 
 # Specific service / suite
 npx @elench/testkit frontend e2e -s auth
 npx @elench/testkit api int -s health
 
 # Exact file
-npx @elench/testkit int --file tests/api/integration/health.int.testkit.ts
+npx @elench/testkit int --file __testkit__/health/health.int.testkit.ts
 
 # Deterministic git-trackable status snapshot
 npx @elench/testkit int --write-status
@@ -47,113 +44,145 @@ npx @elench/testkit status
 npx @elench/testkit destroy
 ```
 
+## Setup
+
+Create `testkit.setup.ts` at repo root:
+
+```ts
+import {
+  defineTestkitSetup,
+  lifecycle,
+  localDatabase,
+  nextService,
+  service,
+  tsxService,
+} from "@elench/testkit/setup";
+
+export default defineTestkitSetup({
+  services: {
+    api: service({
+      ...tsxService({
+        cwd: ".",
+        entry: "src/index.ts",
+        port: 3004,
+        readyPath: "/health",
+      }),
+      envFiles: [".env.testkit"],
+      database: localDatabase(),
+      migrate: lifecycle("npm run db:migrate", {
+        testkitCmd: "npm run db:migrate:testkit",
+      }),
+    }),
+    frontend: service({
+      ...nextService({
+        cwd: "frontend",
+        port: 3000,
+        env: {
+          NEXT_PUBLIC_API_URL: "{baseUrl:api}",
+        },
+      }),
+      dependsOn: ["api"],
+      envFiles: ["frontend/.env.testkit"],
+    }),
+  },
+});
+```
+
+`testkit.setup.ts` is optional for simple repos, but it is the primary escape hatch
+for:
+
+- multi-service graphs
+- local DB configuration
+- migrate / seed commands
+- test-local migrate / seed overrides
+- named HTTP suite profiles
+- telemetry upload configuration
+
 ## Authoring
 
-Consumer tests can import the shared authoring API directly from the package:
+HTTP suites:
 
-```js
+```ts
 import { defineHttpSuite } from "@elench/testkit";
 
 const suite = defineHttpSuite(({ rawReq }) => {
-  const res = rawReq("GET", "/health");
+  rawReq("GET", "/health");
 });
 
-export const options = suite.options;
-export const setup = suite.setup;
-export default suite.exec;
+export default suite;
 ```
 
-For auth or schema-specific behavior, keep a small consumer-owned adapter next
-to the tests and pass it into the generic suite factory:
+`testkit` suite files should default-export the suite object returned by
+`defineHttpSuite(...)` or `defineDalSuite(...)`.
+
+Named HTTP profiles live in `testkit.setup.ts` and can be referenced by name:
 
-```js
+```ts
 import { defineHttpSuite } from "@elench/testkit";
-import { clerkSessionAuth } from "../helpers/testkit-auth.js";
 
-const suite = defineHttpSuite({ auth: clerkSessionAuth }, ({ req, setupData }) => {
-  req("GET", "/api/auth/me", setupData);
+const suite = defineHttpSuite({ profile: "default-auth" }, ({ req, setupData }) => {
+  req("GET", "/api/auth/session", setupData);
 });
 ```
 
-Low-level runtime primitives are also available without exposing the underlying engine:
-
-```js
-import { check, group, http } from "@elench/testkit/runtime";
-```
-
-`testkit` bundles these imports before execution, so tests do not need
-generated `_testkit` files, direct package-manager path imports, or any separate engine installation.
-The published package also ships first-party TypeScript declarations for both
-`@elench/testkit` and `@elench/testkit/runtime`, so consumer repos do not need
-local ambient module shims for the supported authoring surface.
-
-Legacy compatibility:
-
-- `testkit runtime install`
-- `testkit runtime status`
-- `testkit runtime update`
+DAL suites:
 
-still exist, but direct package imports are now the preferred model.
+```ts
+import { defineDalSuite } from "@elench/testkit";
 
-From outside the product repo, use `--dir` explicitly:
+const suite = defineDalSuite(({ db }) => {
+  db.query("select 1");
+});
 
-```bash
-npx @elench/testkit --dir my-product int
-npx @elench/testkit --dir my-product api int -s health
+export default suite;
 ```
 
-## How it works
-
-1. **Discovery** — reads `testkit.config.json` for services, then discovers files by suffix:
-   `*.int.testkit.ts`, `*.e2e.testkit.ts`, `*.dal.testkit.ts`, `*.load.testkit.ts`, `*.pw.testkit.ts`
-2. **Config** — reads `testkit.config.json` for local runtime, migration, dependency, and database settings
-   Per-service `.env` files declared in config are loaded when present.
-3. **Database** — provisions Docker-managed local Postgres when a service declares one
-4. **Seed** — runs optional product seed commands against the provisioned database
-5. **Runtime** — starts required local services, waits for readiness, and injects test env
-6. **Execution** — schedules file-level execution tasks across a global worker pool, reuses warm dependency graphs when possible, bundles test files before execution so package imports resolve cleanly, and batches Playwright files per worker
-7. **Cleanup** — stops local processes and preserves `.testkit/` state for inspection or later destroy
+Low-level runtime primitives remain available:
 
-Each run ends with a compact summary that shows per-service pass/fail status, suite totals, failed suites, and total duration.
-
-`testkit` also writes `.testkit/results/latest.json` for every run. With `--write-status`, it writes a deterministic `testkit.status.json` snapshot that is suitable for git tracking. When `testkit.config.json` includes a top-level `telemetry` block, it can best-effort upload the richer run artifact to a configured HTTP endpoint with bearer auth.
+```ts
+import { check, group, http } from "@elench/testkit/runtime";
+```
 
-## File roles
+## Discovery
 
-- `testkit.config.json`: local execution and provisioning config
-- `testkit.status.json`: optional deterministic run snapshot written by `--write-status`
+`testkit` discovers suites from `__testkit__/` directories.
 
-`testkit.config.json` can also declare:
+Example layouts:
 
-- `telemetry` for optional generic HTTP result upload
-- no test registration in config; `testkit` discovers `*.testkit.ts` files from the filesystem automatically
-- `envFile` / `envFiles` for service-specific environment loading
-- `databaseFrom` for dependency services that must share another service's `DATABASE_URL`
-- `database.provider` for local Postgres settings
-- `database.template.inputs` to define the local template cache invalidation inputs
-- `migrate.backends` / `seed.backends` for optional local-only command overrides
+- `src/api/routes/__testkit__/auth/me.int.testkit.ts`
+- `src/db/__testkit__/sessions/count-type.dal.testkit.ts`
+- `frontend/__testkit__/navigation/navigation.pw.testkit.ts`
+- `avocado_api/internal/handler/__testkit__/repos/crud.int.testkit.ts`
 
-## Parallel execution
+`testkit` uses these suffixes automatically:
 
-`@elench/testkit` can run test work in parallel with `--jobs <n>`.
+- `*.int.testkit.ts`
+- `*.e2e.testkit.ts`
+- `*.dal.testkit.ts`
+- `*.load.testkit.ts`
+- `*.pw.testkit.ts`
 
-`--jobs` is global for the whole run, not per service.
+Ownership is inferred from:
 
-Each worker gets its own:
+- the deepest matching service root from `services.<name>.local.cwd`
+- optional `services.<name>.discovery.roots` overrides for shared-root edge cases
 
-- cloned local Postgres database
-- `.testkit` state subtree
-- local service ports
+Suite names are inferred from the colocated path:
 
-Workers prefer to stay on the same dependency graph so service stacks can be reused across assigned work, including dependent services such as `frontend -> api`.
+- `auth/__testkit__/*.int.testkit.ts` => `auth`
+- `routes/__testkit__/auth/*.int.testkit.ts` => `auth`
 
-Use `--shard <i/n>` to split the selected suite set deterministically before worker scheduling.
+## Local Databases
 
-`testkit` also writes `.testkit/timings.json` and uses those file timings on later runs for longest-first balancing.
+`@elench/testkit` provisions Docker-managed local Postgres automatically for
+services that define `database: localDatabase(...)`.
 
-## Schema
+- template databases are cached
+- worker databases are cloned from templates
+- template fingerprints are derived automatically from env files, migrate/seed
+  config, and repo contents
 
-See [testkit-config-schema.md](testkit-config-schema.md).
+Manual `template.inputs` overrides are still available for edge cases.
 
 ## Development Tests
 
@@ -163,8 +192,3 @@ npm run test:unit
 npm run test:integration
 npm run test:system
 ```
-
-`test:system` runs real end-to-end fixtures from `test/fixtures/system/`. It requires:
-
-- Docker with a running daemon
-- Playwright Chromium browser assets available to `@playwright/test`

package/lib/bundler/index.mjs

@@ -4,9 +4,11 @@ import path from "path";
 import crypto from "crypto";
 import { build } from "esbuild";
 import { fileURLToPath } from "url";
+import { findSetupFile } from "../config/setup-loader.mjs";
 
 const PACKAGE_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
 const ROOT_ENTRY = path.join(PACKAGE_ROOT, "lib", "index.mjs");
+const SETUP_ENTRY = path.join(PACKAGE_ROOT, "lib", "setup", "index.mjs");
 const RUNTIME_ENTRY = path.join(PACKAGE_ROOT, "lib", "runtime", "index.mjs");
 const bundleCache = new Map();
 
@@ -19,7 +21,8 @@ export async function bundleK6File({
   const bundleDir = path.join(productDir, ".testkit", "_bundles", serviceName || "shared");
   fs.mkdirSync(bundleDir, { recursive: true });
 
-  const cacheKey = await buildCacheKey(absoluteSource);
+  const setupFile = findSetupFile(productDir);
+  const cacheKey = await buildCacheKey(absoluteSource, setupFile);
   const cached = bundleCache.get(cacheKey);
   if (cached && fs.existsSync(cached)) {
     return cached;
@@ -29,11 +32,19 @@ export async function bundleK6File({
     bundleDir,
     `${path.basename(sourceFile, path.extname(sourceFile))}-${cacheKey.slice(0, 12)}.js`
   );
+  const entryFile = path.join(
+    bundleDir,
+    `${path.basename(sourceFile, path.extname(sourceFile))}-${cacheKey.slice(0, 12)}.entry.mjs`
+  );
+  fs.writeFileSync(entryFile, buildBundleEntryModule({
+    sourceFile: absoluteSource,
+    setupFile,
+  }));
 
   await build({
-    absWorkingDir: path.dirname(absoluteSource),
+    absWorkingDir: bundleDir,
     bundle: true,
-    entryPoints: [absoluteSource],
+    entryPoints: [entryFile],
     format: "esm",
     legalComments: "none",
     outfile: outputFile,
@@ -51,17 +62,23 @@ export async function bundleK6File({
   return outputFile;
 }
 
-async function buildCacheKey(sourceFile) {
+async function buildCacheKey(sourceFile, setupFile = null) {
   const source = await fs.promises.readFile(sourceFile, "utf8");
   const packageJson = await fs.promises.readFile(path.join(PACKAGE_ROOT, "package.json"), "utf8");
-  return crypto
+  const hash = crypto
     .createHash("sha256")
     .update(sourceFile)
     .update("\0")
     .update(source)
     .update("\0")
-    .update(packageJson)
-    .digest("hex");
+    .update(packageJson);
+
+  if (setupFile && fs.existsSync(setupFile)) {
+    hash.update("\0").update(setupFile).update("\0");
+    hash.update(await fs.promises.readFile(setupFile, "utf8"));
+  }
+
+  return hash.digest("hex");
 }
 
 function testkitPackageAliasPlugin() {
@@ -81,7 +98,58 @@ function testkitPackageAliasPlugin() {
 function resolvePackageSubpath(specifier) {
   const subpath = specifier.slice("@elench/testkit".length);
   if (!subpath) return ROOT_ENTRY;
+  if (subpath === "/setup") return SETUP_ENTRY;
   if (subpath === "/runtime") return RUNTIME_ENTRY;
 
   throw new Error(`Unsupported @elench/testkit import "${specifier}" in ${os.platform()}`);
 }
+
+function buildBundleEntryModule({ sourceFile, setupFile }) {
+  const sourceImport = JSON.stringify(sourceFile);
+  const setupRegistration = setupFile
+    ? `
+import * as repoSetupModule from ${JSON.stringify(setupFile)};
+registerRepoSetup(repoSetupModule.default || repoSetupModule || null);
+`
+    : `
+registerRepoSetup(null);
+`;
+
+  return `
+import { registerRepoSetup } from "@elench/testkit/setup";
+import * as suiteModule from ${sourceImport};
+${setupRegistration}
+const suite = normalizeTestkitSuite(suiteModule);
+export const options = suite.options;
+export function setup(...args) {
+  return suite.setup(...args);
+}
+export default function exec(...args) {
+  return suite.exec(...args);
+}
+
+function normalizeTestkitSuite(module) {
+  const candidate = module?.default;
+  if (!candidate || typeof candidate !== "object") {
+    throw new Error(
+      "testkit suite files must default-export the suite object returned by defineHttpSuite(...) or defineDalSuite(...). Example: export default defineHttpSuite(...) or const suite = defineHttpSuite(...); export default suite;"
+    );
+  }
+  if (typeof candidate.exec !== "function") {
+    throw new Error(
+      "testkit suite default export must expose an exec(setupData) function"
+    );
+  }
+
+  const setupFn = typeof candidate["setup"] === "function"
+    ? (...args) => candidate.setup(...args)
+    : () => null;
+
+  return {
+    options: candidate["options"],
+    setup: setupFn,
+    exec: (...args) => candidate.exec(...args),
+  };
+}
+`;
+}

package/lib/bundler/index.test.mjs

@@ -1,6 +1,7 @@
 import fs from "fs";
 import os from "os";
 import path from "path";
+import { pathToFileURL } from "url";
 import { afterEach, describe, expect, it } from "vitest";
 import { bundleK6File } from "./index.mjs";
 
@@ -29,9 +30,7 @@ describe("runtime bundler", () => {
         '    "has status": (body) => typeof body.status === "string",',
         "  });",
         "});",
-        "export const options = suite.options;",
-        "export const setup = suite.setup;",
-        "export default suite.exec;",
+        "export default suite;",
         "",
       ].join("\n")
     );
@@ -60,9 +59,7 @@ describe("runtime bundler", () => {
         "const suite = defineDalSuite(({ db }) => {",
         '  db.query("SELECT 1");',
         "});",
-        "export const options = suite.options;",
-        "export const setup = suite.setup;",
-        "export default suite.exec;",
+        "export default suite;",
         "",
       ].join("\n")
     );
@@ -77,4 +74,62 @@ describe("runtime bundler", () => {
     expect(bundled).toContain("defineDalSuite");
     expect(bundled).toContain('import sql from "k6/x/sql"');
   });
+
+  it("normalizes a default-exported suite object with no setup override", async () => {
+    const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "testkit-bundle-"));
+    cleanups.push(() => fs.rmSync(tmpDir, { force: true, recursive: true }));
+
+    const sourceFile = path.join(tmpDir, "no-setup.js");
+    fs.writeFileSync(
+      sourceFile,
+      [
+        "const suite = {",
+        "  options: { vus: 1, iterations: 1 },",
+        "  exec() {",
+        "    return 'ok';",
+        "  },",
+        "};",
+        "export default suite;",
+        "",
+      ].join("\n")
+    );
+
+    const bundledFile = await bundleK6File({
+      productDir: tmpDir,
+      serviceName: "api",
+      sourceFile,
+    });
+
+    const bundled = await import(`${pathToFileURL(bundledFile).href}?v=${Date.now()}`);
+    expect(typeof bundled.setup).toBe("function");
+    expect(bundled.setup()).toBeNull();
+    expect(typeof bundled.default).toBe("function");
+    expect(bundled.default()).toBe("ok");
+  });
+
+  it("throws a clear error when the default export is not a suite object", async () => {
+    const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "testkit-bundle-"));
+    cleanups.push(() => fs.rmSync(tmpDir, { force: true, recursive: true }));
+
+    const sourceFile = path.join(tmpDir, "legacy-shape.js");
+    fs.writeFileSync(
+      sourceFile,
+      [
+        "export default function exec() {",
+        "  return 'legacy';",
+        "}",
+        "",
+      ].join("\n")
+    );
+
+    const bundledFile = await bundleK6File({
+      productDir: tmpDir,
+      serviceName: "api",
+      sourceFile,
+    });
+
+    await expect(
+      import(`${pathToFileURL(bundledFile).href}?v=${Date.now()}`)
+    ).rejects.toThrow(/default-export the suite object/);
+  });
 });

package/lib/cli/index.mjs

@@ -1,9 +1,8 @@
 import { cac } from "cac";
-import { loadConfigs, getServiceNames } from "../config/index.mjs";
+import { loadConfigs } from "../config/index.mjs";
 import {
   parseJobsOption,
   parseShardOption,
-  RESERVED,
   resolveRequestedFiles,
   resolveCliSelection,
   validateFrameworkOption,
@@ -13,33 +12,6 @@ import * as runner from "../runner/index.mjs";
 export function run() {
   const cli = cac("testkit");
 
-  cli
-    .command("runtime <action>", "Install or inspect the consumer runtime bundle")
-    .option("--dir <path>", "Explicit product directory")
-    .option("--path <path>", "Target runtime path relative to the product directory")
-    .option("--strict", "Exit non-zero when runtime status is missing or drifted")
-    .action(async (action, options) => {
-      if (!["install", "status", "update"].includes(action)) {
-        throw new Error('Unknown runtime action. Expected one of: install, status, update.');
-      }
-
-      const runtime = await import("../runtime-manager/index.mjs");
-
-      if (action === "status") {
-        const status = runtime.getRuntimeStatus(options);
-        console.log(runtime.formatRuntimeStatus(status));
-        if (options.strict && status.status !== "installed") {
-          process.exitCode = 1;
-        }
-        return;
-      }
-
-      const result = runtime.installRuntime(options);
-      console.log(
-        `Installed testkit runtime to ${result.relativeRuntimeDir} (${result.files.length} file${result.files.length === 1 ? "" : "s"})`
-      );
-    });
-
   cli
     .command("[first] [second] [third]", "Run test suites (int, e2e, dal, all)")
     .option("-s, --suite <name>", "Run specific suite(s)", { default: [] })
@@ -68,14 +40,13 @@ export function run() {
       //   testkit --dir my-product api int → one service, int
 
       // Now resolve service vs type from remaining args
-      const serviceNames = new Set(getServiceNames(options.dir));
+      const allConfigs = await loadConfigs({ dir: options.dir });
+      const serviceNames = new Set(allConfigs.map((config) => config.name));
       const { service, type } = resolveCliSelection({
         first,
         second,
         serviceNames,
       });
-
-      const allConfigs = loadConfigs({ dir: options.dir });
       const configs = service
         ? allConfigs.filter((config) => config.name === service)
         : allConfigs;