@luckystack/test-runner 0.10.2 → 0.11.0

package/AGENTS.md

@@ -0,0 +1,7 @@
+# @luckystack/test-runner — agent guidance
+
+Use this package to test a running LuckyStack server against generated route contracts and security guards. It cannot substitute for project-specific business, role or tenant-isolation tests. See `README.md` for a minimal invocation and `docs/` for each layer's intent.
+
+Never expose the test-reset endpoint in production or without its token; reset affects session/rate-limit state and is not an application-database reset. Load environment files and resolve configured secret pointers in the test process before importing modules that construct clients: resolution in the server process does not propagate to the test runner. An auth guard rejection must be established before interpreting a rate-limit probe. Fuzz tests establish a shaped error response, not an absence of server errors; run expensive fuzz sweeps outside the fast PR gate.
+
+The runner must never test a route over a different path than real clients use, nor change server state its own run depends on (ADR 0064). Probes follow the project's `transport.invocation` (CSRF probes stay on HTTP, since CSRF is an HTTP-only guard); GET payloads use the same query-string encoding as the real client (ADR 0062). The runner's own resets are always scoped to rate limits, so an authenticated sweep keeps its session; sockets it opens on the caller's session close as an intentional disconnect so presence does not log that session out. Reset targets live in one list in `@luckystack/core` shared with the server.

package/CHANGELOG.md

@@ -5,6 +5,13 @@ All notable changes to `@luckystack/test-runner` are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.11.0] - 2026-10-14
+
+### Changed
+
+- Type declarations are emitted per source file by TypeScript 7 (`tsc -b`) instead of one bundled `.d.ts` per entry, so the framework build no longer runs package by package. The exported types are unchanged; relative imports inside the declarations carry `.js`, so `moduleResolution: "node16"`/`"nodenext"` projects resolve them as before.
+- Package design guidance now ships as `AGENTS.md`, not `CLAUDE.md`. Handwritten docs focus on rationale and safety; exact API contracts remain in installed declarations/source.
+
 ## [0.10.2] - 2026-10-09
 
 ### Added

package/README.md

@@ -1,88 +1,33 @@
 # @luckystack/test-runner
 
-> Generated-type-driven test layers for [LuckyStack](https://github.com/ItsLucky23/LuckyStack-v2). Walks every endpoint in your `apiMethodMap.generated.ts` and runs four progressive test layers: contract smoke (Zod-valid input), auth enforcement, rate-limit, and crash-resistance fuzz.
+Generated-route contract, auth, rate-limit, CSRF and crash-resistance checks for a running LuckyStack server, over both API and sync routes. The checks probe each route over the transport real clients use, so a green sweep says something about the path users actually hit (ADR 0064).
 
-## Install
+## Install and run
 
 ```bash
-npm install --save-dev @luckystack/test-runner zod
+npm install --save-dev @luckystack/test-runner zod socket.io-client
 ```
 
-## Quickstart
-
-Run all four layers against a running server:
+Start from the scaffolded project's `scripts/testAll.ts` and its test command; consult the installed types for a custom harness. A minimal entry passes the generated maps for **both** route kinds and the lazy config loader:
 
 ```ts
-import {
-  runAllTests,
-  resolveTestBaseUrl,
-  logRunAllSummary,
-} from '@luckystack/test-runner';
-import { apiInputSchemas, syncInputSchemas } from './src/_sockets/apiInputSchemas.generated';
-import { apiMetaMap, apiMethodMap, syncMetaMap } from './src/_sockets/apiTypes.generated';
-import { ports } from './config.ports';
-
-const baseUrl = resolveTestBaseUrl({ fallbackUrl: `http://localhost:${ports.backend}` });
 const summary = await runAllTests({
-  apiMethodMap,
-  apiMetaMap,
-  apiInputSchemas,
-  syncMetaMap,      // omit and sync routes are not swept (a warning says so)
-  syncInputSchemas,
-  baseUrl,
-  // Loaded only after .env/.env.local. If config.secretManager.url is set,
-  // pointers are resolved in THIS process before Layer-5 tests touch Prisma.
+  apiMethodMap, apiMetaMap, apiInputSchemas,
+  syncMetaMap, syncInputSchemas, // omit and sync routes are not swept (a warning says so)
+  baseUrl: resolveTestBaseUrl({ fallbackUrl: `http://localhost:${ports.backend}` }),
   loadProjectConfig: async () => (await import('./config')).default,
 });
 logRunAllSummary(summary);
 ```
 
-Each layer is independently runnable — you can skip the ones that don't apply or rearrange them.
-
-`runAllTests` always loads the normal env-file layers and requires `loadProjectConfig` as above, even in projects that currently use only local env values. A configured resolver is loaded dynamically and runs in fail-fast remote mode before custom test modules are imported. Direct `runCustomTests` callers must pass the same callback. Both public paths reject a missing loader so a later secret-manager rollout cannot silently expose pointer values. This matters because the live server resolving its own `DATABASE_URL` does not mutate the separate test process.
-
-## Layers
-
-| Layer | What it asserts |
-| --- | --- |
-| **Contract** | A Zod-valid input payload reaches `main` and returns `{ status: 'success' }` (or a typed failure status). Catches schema/route mismatches. |
-| **Auth enforcement** | Endpoints flagged `auth.login: true` reject unauthenticated requests with the framework's standard 401 shape. |
-| **Rate limit** | Endpoints with `rateLimit: <N>` reject the `(N + 1)`th request inside the window. |
-| **Fuzz** | Bad/random inputs never crash the server (still returns a typed error response, never 500 with a stack). |
-
-The fuzz layer is intentionally exhaustive and slow — wire it into nightly CI rather than every PR.
-
-## Server hooks it depends on
-
-- `/_test/reset` endpoint — served by `@luckystack/server` only when `NODE_ENV` is exactly `development` or `test`, and always gated by a non-empty `TEST_RESET_TOKEN`. `resetServerState` clears framework rate-limit state plus the default Redis session/active-user namespaces, or only the targets passed as `only` (the runner's own resets clear rate limits only, so an authenticated sweep keeps its session); it does not clear application database rows or custom `SessionAdapter` stores.
-- `apiMethodMap.generated.ts` — produced by `@luckystack/devkit` from your `_api/*` files. Defaults are read via `getApiMethodMapPath()` from `@luckystack/core`.
-
-## Public API
-
-| Export | Purpose |
-| --- | --- |
-| `walkEndpoints(apiMethodMap)` | Returns `EndpointDescriptor[]` from the generated map. |
-| `runContractCheck(input)` / `runContractTests(input)` | Single endpoint / full sweep. |
-| `runAuthEnforcementCheck(input)` / `runAuthEnforcementTests(input)` | Auth layer. |
-| `runRateLimitCheck(input)` / `runRateLimitTests(input)` | Rate-limit layer. |
-| `runFuzzCheck(input)` / `runFuzzTests(input)` | Fuzz layer. |
-| `resetServerState(input)` | POST to `/_test/reset`. |
-| `resolveTestEnvironment({ loadProjectConfig? })` | Load env layers and optional secret-manager pointers for the test process; `runAllTests` calls it automatically. |
-| `sampleSchemaInput(schema)` | Generate a Zod-valid sample payload. |
-| `logContractResult` / `logContractSummary` | Pretty-printers. |
-
-Types: `EndpointDescriptor`, `HttpMethod`, `ContractCheckResult`, `RunContractSummary`, plus per-layer input types.
-
-## Related architecture docs
+`runAllTests` writes the results file itself; an entry script should not write its own copy. A route filter is applied before any request is sent, so filtered-out routes are never called — not merely left out of the report.
 
-- [`docs/ARCHITECTURE_API.md`](../../docs/ARCHITECTURE_API.md) — the contract this runner asserts against.
-- [`docs/ARCHITECTURE_PACKAGING.md`](../../docs/ARCHITECTURE_PACKAGING.md) — how generated maps are emitted per preset.
+**Upgrading an existing entry script to 0.10.2:** import `syncMetaMap` / `syncInputSchemas` from the generated files and pass them, and remove any manual write of the summary file. In a socket-mode project, Layer-5 `ctx.callSync` now runs over Socket.io; a test that depended on HTTP-only behaviour must request HTTP for that call. See the package changelog for the full list.
 
-## Dependencies
+## Safety
 
-- Required peers: `zod@^4.0.0`, `socket.io-client@^4.8.0`.
-- Optional peers: `@luckystack/login`, `@luckystack/secret-manager` (loaded only for their corresponding features).
+A test process must load local environment layers and resolve configured secret pointers **before** importing database-backed tests. The server resolving its own secrets does not resolve the separate runner process. Never run against production or expose the test reset route there. The reset route requires a non-empty token in development/test and affects framework state, not your application database or custom session storage. The runner's own resets clear rate limits only, so an authenticated sweep keeps its session; see [rate-limit tests](./docs/rate-limit-tests.md).
 
-## License
+An authenticated sweep calls every route with its own session, including routes that destroy it (logout, account deletion, session revoke); skip those routes or every later layer runs logged out.
 
-MIT — see [LICENSE](../../LICENSE).
+The built-in checks are not a substitute for project-specific role, tenant or business assertions. Run the slower fuzz layer separately from fast PR checks. A fuzz pass means responses keep the framework envelope, **not** that no HTTP 5xx occurred; see [fuzz tests](./docs/fuzz-tests.md).

package/dist/authEnforcementCheck.d.ts

@@ -0,0 +1,12 @@
+import type { ContractCheckResult, EndpointDescriptor } from './types.js';
+import type { ProbeTransport } from './probeRequest.js';
+export interface AuthEnforcementCheckInput {
+    endpoint: EndpointDescriptor;
+    baseUrl: string;
+    inputFor?: (endpoint: EndpointDescriptor) => unknown;
+    /** Per-call request timeout in ms. Defaults to 5000. */
+    requestTimeoutMs?: number;
+    /** How the probe reaches the server. Defaults to HTTP. */
+    transport?: ProbeTransport;
+}
+export declare const runAuthEnforcementCheck: (input: AuthEnforcementCheckInput) => Promise<ContractCheckResult>;

package/dist/contractCheck.d.ts

@@ -0,0 +1,24 @@
+import type { ContractCheckResult, EndpointDescriptor } from './types.js';
+import type { ProbeTransport } from './probeRequest.js';
+export interface ContractCheckInput {
+    endpoint: EndpointDescriptor;
+    baseUrl: string;
+    /**
+     * Optional input factory. Defaults to `{}`. When an endpoint requires a
+     * specific shape to not crash, callers can override per-endpoint here.
+     */
+    inputFor?: (endpoint: EndpointDescriptor) => unknown;
+    /**
+     * Optional headers (auth tokens, cookies). Applied to every request.
+     */
+    headers?: Record<string, string>;
+    /**
+     * Per-call request timeout in ms. Defaults to 5000. Bump for slow endpoints
+     * (AI calls, large reports) where the framework default would false-fail
+     * the contract check.
+     */
+    requestTimeoutMs?: number;
+    /** How the probe reaches the server. Defaults to HTTP. */
+    transport?: ProbeTransport;
+}
+export declare const runContractCheck: (input: ContractCheckInput) => Promise<ContractCheckResult>;

package/dist/csrfEnforcementCheck.d.ts

@@ -0,0 +1,25 @@
+import type { ContractCheckResult, EndpointDescriptor } from './types.js';
+import type { ProbeTransport } from './probeRequest.js';
+export interface CsrfEnforcementCheckInput {
+    endpoint: EndpointDescriptor;
+    baseUrl: string;
+    /**
+     * A valid session Cookie header (`<name>=<token>`). The probe deliberately
+     * carries it so the request passes the auth guard and reaches the CSRF check.
+     */
+    authCookie: string;
+    inputFor?: (endpoint: EndpointDescriptor) => unknown;
+    /** Per-call request timeout in ms. Defaults to 5000. */
+    requestTimeoutMs?: number;
+    /** Error code expected on CSRF rejection. Defaults to `auth.csrfMismatch`. */
+    expectedErrorCode?: string;
+    /** HTTP status expected on CSRF rejection. Defaults to 403; pass `false` to skip. */
+    expectedHttpStatus?: number | false;
+    /**
+     * HTTP transport the probe uses (e.g. one carrying the sync `receiver`).
+     * CSRF is an HTTP-only guard, so a non-HTTP transport makes the check skip.
+     * Defaults to plain HTTP.
+     */
+    transport?: ProbeTransport;
+}
+export declare const runCsrfEnforcementCheck: (input: CsrfEnforcementCheckInput) => Promise<ContractCheckResult>;

package/dist/customTests.d.ts

@@ -0,0 +1,145 @@
+import { getPrismaClient } from '@luckystack/core';
+import { type ResolveTestEnvironmentInput } from './resolveTestEnvironment.js';
+import { type StreamChunkFrame, type StreamWatcher } from './streamWatcher.js';
+import type { ApiMethodMap } from './types.js';
+export interface CustomTestCase {
+    name: string;
+    run: (ctx: TestContext) => Promise<void>;
+    /**
+     * Mark this case as a KNOWN, accepted failure (e.g. a documented bug not
+     * yet fixed, or a scenario the current implementation cannot satisfy). The
+     * string is the reason shown in the report. A case marked `expectedToFail`
+     * that throws is reported as `xfail` — counted separately, NOT a red
+     * failure. If it unexpectedly PASSES it is reported as `xpass` so the stale
+     * marker can be removed. Leave undefined for normal cases.
+     */
+    expectedToFail?: string;
+}
+export interface CustomTestResult {
+    routePath: string;
+    caseName: string;
+    status: 'pass' | 'fail' | 'xfail' | 'xpass';
+    durationMs: number;
+    reason?: string;
+    /** Documented reason when the case carried an `expectedToFail` marker. */
+    expectedToFail?: string;
+}
+export interface RunCustomTestsSummary {
+    total: number;
+    passed: number;
+    failed: number;
+    /** Cases marked `expectedToFail` that failed as expected (known issues). */
+    xfailed: number;
+    /** Cases marked `expectedToFail` that unexpectedly passed (remove the marker). */
+    xpassed: number;
+    results: CustomTestResult[];
+}
+export interface RunCustomTestsInput {
+    baseUrl: string;
+    /** Optional cookie name for the session token. Defaults to the project's configured cookie. */
+    sessionCookieName?: string;
+    /** Filter to a subset of routes — substring match against `<page>/<name>/<version>`. */
+    filter?: string;
+    /**
+     * Generated `apiMethodMap` so the harness sends each API route its declared
+     * HTTP method (e.g. logout is DELETE). Falls back to POST when absent.
+     */
+    apiMethodMap?: ApiMethodMap;
+    /**
+     * Lazy consumer config for direct Layer-5 runs. Required so the public path
+     * cannot silently skip a configured secret manager; `runAllTests` prepares
+     * the environment separately before using the internal Layer-5 entrypoint.
+     */
+    loadProjectConfig: NonNullable<ResolveTestEnvironmentInput['loadProjectConfig']>;
+    onResult?: (result: CustomTestResult) => void;
+}
+type PreparedRunCustomTestsInput = Omit<RunCustomTestsInput, 'loadProjectConfig'>;
+interface DiscoveredTestFile {
+    /** Absolute path to the test file. */
+    filePath: string;
+    /** `api` or `sync` — which transport this route uses. */
+    kind: 'api' | 'sync';
+    /** Route descriptor parsed from the filename + directory location. */
+    route: {
+        page: string;
+        name: string;
+        version: string;
+    };
+}
+export declare const discoverCustomTestFiles: (srcDir?: string) => DiscoveredTestFile[];
+export interface CallSyncOptions {
+    /** Room the sync targets (the server requires one). */
+    receiver?: string;
+    /** Skip the sender's own socket in the fan-out (socket transport). */
+    ignoreSelf?: boolean;
+    /** Force a transport. Defaults to the project's `transport.invocation`. */
+    transport?: 'socket' | 'http';
+    /** Socket transport: join `receiver` before sending. Default `true`. */
+    joinRoom?: boolean;
+}
+export interface TestContext {
+    /**
+     * Invoke the route under test. Page / name / version are baked in.
+     *
+     * `input` travels exactly like the real client sends it: as the JSON body
+     * for POST / PUT / DELETE routes, and as the `__luckystack_data` query-string
+     * value for GET routes (a GET cannot carry a body). Either way the server
+     * validates it as the route's `data`, so a GET route with required fields
+     * receives them.
+     */
+    callApi: <TInput = unknown, TOutput = unknown>(input: TInput) => Promise<TOutput>;
+    /**
+     * Invoke the sync route under test over the SAME transport the project's
+     * real client uses (`transport.invocation`): Socket.io for `'socket'` (the
+     * default), the `/sync/*` HTTP route for `'routed-http'`. Override per call
+     * with `opts.transport`. Over the socket, the case's session socket first
+     * joins `opts.receiver` (like a real client joins its room before syncing);
+     * pass `joinRoom: false` to test a non-member.
+     */
+    callSync: <TInput = unknown, TOutput = unknown>(input: TInput, opts?: CallSyncOptions) => Promise<TOutput>;
+    /**
+     * Open a second socket connection that joins `roomCode` as a pure
+     * subscriber and exposes the incoming chunk stream for assertions.
+     *
+     * The route is auto-derived from the test file location, so a watcher
+     * only surfaces chunks for the route this test belongs to. Multiple
+     * watchers per test are allowed; all are auto-closed when the test
+     * case finishes (whether it passes or throws).
+     */
+    watchStream: <TChunk extends StreamChunkFrame = StreamChunkFrame>(roomCode: string) => Promise<StreamWatcher<TChunk>>;
+    session: {
+        login: (user?: {
+            email?: string;
+            id?: string;
+            name?: string;
+        }) => Promise<{
+            token: string;
+            userId: string;
+        }>;
+        logout: () => Promise<void>;
+        current: () => {
+            token: string | null;
+            userId: string | null;
+        };
+    };
+    prisma: ReturnType<typeof getPrismaClient>;
+    expect: TestExpect;
+}
+export interface TestExpect {
+    eq: <T>(actual: T, expected: T, message?: string) => void;
+    ok: (value: unknown, message?: string) => void;
+    throws: (fn: () => Promise<unknown>, message?: string) => Promise<Error>;
+    matches: (value: string, pattern: RegExp, message?: string) => void;
+}
+interface TestSessionState {
+    token: string | null;
+    userId: string | null;
+    csrfToken: string | null;
+    lastResponse: unknown;
+}
+/** @internal Exported for the unit test only; consumers use `TestContext.callApi`. */
+export declare const buildCallApi: (baseUrl: string, routePath: string, state: TestSessionState, cookieName: string, method: string) => TestContext['callApi'];
+/** @internal Environment is already prepared by `runAllTests`. */
+export declare const runCustomTestsAfterEnvironmentPrepared: (input: PreparedRunCustomTestsInput) => Promise<RunCustomTestsSummary>;
+export declare const runCustomTests: (input: RunCustomTestsInput) => Promise<RunCustomTestsSummary>;
+export {};

package/dist/extensionRegistry.d.ts

@@ -0,0 +1,83 @@
+export interface TestLayerInput {
+    /** Resolved route name (`api/billing/getInvoice/v1` or `sync/chat/send/v1`). */
+    endpoint: string;
+    /** Method when applicable, undefined for sync endpoints. */
+    method?: string;
+    /** Bearer token / session token to use for auth-aware checks. */
+    authToken?: string;
+    /**
+     * A Zod-valid sample `data` payload for this route — generated from the
+     * route's input schema when `runRegisteredLayers` received the generated
+     * schemas (or an `inputFor`), `{}` otherwise. The same payload the built-in
+     * contract sweep sends, so a layer can exercise the route past validation.
+     */
+    sampleInput?: unknown;
+}
+export interface TestLayerResult {
+    passed: boolean;
+    /** Short reason key — i18n-style errorCode when failed. */
+    message?: string;
+    /** Optional extras for the reporter (latency, response sample, ...). */
+    metadata?: Record<string, unknown>;
+}
+export interface TestLayer {
+    name: string;
+    run: (input: TestLayerInput) => Promise<TestLayerResult> | TestLayerResult;
+}
+export interface TestFixture<TPayload = unknown> {
+    valid: TPayload[];
+    invalid: TPayload[];
+}
+export interface TestResult {
+    layer: string;
+    endpoint: string;
+    passed: boolean;
+    message?: string;
+    durationMs?: number;
+    metadata?: Record<string, unknown>;
+}
+export interface TestSummary {
+    totalLayers: number;
+    totalEndpoints: number;
+    passed: number;
+    failed: number;
+    results: TestResult[];
+}
+export interface TestReporter {
+    /** Called once per individual TestResult as the runner progresses. */
+    onResult?: (result: TestResult) => void | Promise<void>;
+    /** Called once at the end with the aggregated summary. */
+    onSummary?: (summary: TestSummary) => void | Promise<void>;
+    /** Optional webhook the runner POSTs the summary to. */
+    webhookUrl?: string;
+    webhookAuth?: {
+        type: 'bearer';
+        token: string;
+    };
+}
+/**
+ * Register a custom test layer. Re-registering the same name replaces
+ * the previous entry. Use for CORS enforcement, business-rule checks,
+ * custom auth schemes, multi-tenant isolation, GDPR data-flow checks.
+ */
+export declare const registerTestLayer: (layer: TestLayer) => void;
+export declare const listTestLayers: () => TestLayer[];
+/**
+ * Register a realistic payload fixture for a specific type key. The fuzz
+ * layer prefers these over schema-random generated inputs when a fixture
+ * exists for the endpoint's input type.
+ *
+ * `valid` payloads are used to verify happy-path behavior; `invalid`
+ * payloads probe the validator's error responses.
+ */
+export declare const registerTestFixture: <TPayload = unknown>(typeKey: string, fixture: TestFixture<TPayload>) => void;
+export declare const getTestFixture: (typeKey: string) => TestFixture | undefined;
+/**
+ * Register a test reporter. Set `null` to unregister. The runner calls
+ * `onResult` per individual result as the run progresses and `onSummary`
+ * once at the end. When `webhookUrl` is set, the runner additionally
+ * POSTs the JSON-serialised summary to that URL.
+ */
+export declare const registerTestReporter: (newReporter: TestReporter | null) => void;
+export declare const getTestReporter: () => TestReporter | null;
+export declare const resetTestExtensionsForTests: () => void;

package/dist/fuzzCheck.d.ts

@@ -0,0 +1,12 @@
+import type { ContractCheckResult, EndpointDescriptor } from './types.js';
+import type { ProbeTransport } from './probeRequest.js';
+export interface FuzzCheckInput {
+    endpoint: EndpointDescriptor;
+    baseUrl: string;
+    headers?: Record<string, string>;
+    /** Per-call request timeout in ms. Defaults to 5000. */
+    requestTimeoutMs?: number;
+    /** How the probes reach the server. Defaults to HTTP. */
+    transport?: ProbeTransport;
+}
+export declare const runFuzzCheck: (input: FuzzCheckInput) => Promise<ContractCheckResult>;