@luckystack/test-runner 0.10.0 → 0.10.2

package/docs/auth-tests.md

@@ -26,7 +26,7 @@ interface AuthEnforcementCheckInput {
 Flow:
 
 1. Build URL the same way the contract layer does.
-2. Build body via `inputFor?.(endpoint) ?? {}`. `GET` drops the body.
+2. Build body via `inputFor?.(endpoint) ?? {}`. On `GET` it travels in the query string, as in the contract layer.
 3. **Send the request with no session cookie, no `Authorization` header, no `X-CSRF-Token`.** The runner deliberately omits every form of credential — that is the whole point of this layer. `headers` is not a supported field for this reason; passing one would defeat the test.
 4. Parse the JSON. Parse failure is a fail.
 5. Branch on `parsed.status`:
@@ -39,7 +39,7 @@ The accepted error codes are the framework's auth-rejection set, hard-coded in t
 
 ### `runAuthEnforcementTests(input)`
 
-Full sweep. Iterates `walkEndpoints(input.apiMethodMap)` and checks every endpoint that declares an authorization requirement — `apiMetaMap[page][name][version].auth.login === true` OR a non-empty `auth.additional[]` predicate list (`hasAdditional`). Truly-public endpoints (`auth.login: false` AND no `additional[]`, or missing meta) are skipped **silently** — they do not appear in the summary at all. This keeps the report focused on what this layer actually covers; public endpoints are already exercised by the contract layer.
+Full sweep. Iterates every API route (plus every sync route when `syncMetaMap` is given — its requirement is read from `syncMetaMap`), narrowed by `filter` before any request, and checks every endpoint that declares an authorization requirement — `apiMetaMap[page][name][version].auth.login === true` OR a non-empty `auth.additional[]` predicate list (`hasAdditional`). Truly-public endpoints (`auth.login: false` AND no `additional[]`, or missing meta) are skipped **silently** — they do not appear in the summary at all. This keeps the report focused on what this layer actually covers; public endpoints are already exercised by the contract layer.
 
 Signature:
 

package/docs/contract-tests.md

@@ -1,6 +1,6 @@
 # Contract Tests
 
-The contract layer is the entry point of `@luckystack/test-runner`. It walks every endpoint that the devkit emits into `apiMethodMap.generated.ts`, fires a single well-formed request, and asserts the response conforms to the framework envelope. It is the cheapest and broadest of the four layers — it does NOT prove business logic, it proves the route exists, decodes input without crashing, and replies with `{ status: 'success' | 'error', errorCode? }`.
+The contract layer is the entry point of `@luckystack/test-runner`. It walks every endpoint that the devkit emits into `apiMethodMap.generated.ts` (and every sync route in `syncMetaMap` when one is passed), fires a single well-formed request, and asserts the response conforms to the framework envelope. It is the cheapest and broadest of the four layers — it does NOT prove business logic, it proves the route exists, decodes input without crashing, and replies with `{ status: 'success' | 'error', errorCode? }`.
 
 ## When to use it
 
@@ -24,6 +24,14 @@ type ApiMethodMap = Record<string, Record<string, Record<string, string>>>;
 
 The order is deterministic: it follows `Object.entries` over the generated map, which the devkit emits sorted by page name, then api name, then version. Empty maps return `[]` (not an error).
 
+`walkSyncEndpoints(syncMap)` is the sync counterpart: it takes the generated `syncMetaMap` (or a `syncMethodMap`-shaped map), yields `fullPath: sync/<page>/<name>/<version>`, and marks each descriptor `kind: 'sync'`. A descriptor without `kind` is an API route.
+
+### Sync routes and probe transports
+
+The checks only know "this endpoint, this payload"; a `ProbeTransport` owns the wire. API routes always go over HTTP. Sync routes go over the transport the sweep passes as `syncTransport`: `runAllTests` picks Socket.io (`createSocketProbeTransport` — the same `sync` emit a real client makes) unless the project's `transport.invocation` is `routed-http` or the caller sets `syncTransport: 'http'`, in which case `createHttpProbeTransport` posts the `{ data, receiver }` envelope to `/sync/*`. The CSRF layer always probes sync over HTTP, because CSRF is an HTTP-only guard. Every sync probe in a run targets one throwaway room; with an `authToken` the sweep session joins it first, so room-membership enforcement sees a member rather than rejecting every probe. Without `syncMetaMap`, `runAllTests` warns that sync routes are not swept.
+
+Layer-5 `ctx.callSync` follows the same rule (project `transport.invocation`, per-call `opts.transport` override) — see `docs/ARCHITECTURE_TESTING.md`.
+
 ### `runContractCheck(input)`
 
 Run the contract assertion against one endpoint.
@@ -39,13 +47,14 @@ interface ContractCheckInput {
   inputFor?: (endpoint: EndpointDescriptor) => unknown;
   headers?: Record<string, string>;
   requestTimeoutMs?: number; // default 5000
+  transport?: ProbeTransport; // default: HTTP
 }
 ```
 
 Flow:
 
-1. Build URL: `${baseUrl.replace(/\/$/, '')}/${endpoint.fullPath}` so trailing slashes on `baseUrl` are tolerated.
-2. Build body: `inputFor?.(endpoint) ?? {}`. For `GET` the body is dropped entirely; otherwise it is `JSON.stringify`d.
+1. Build URL: `${baseUrl.replace(/\/$/, '')}/${endpoint.fullPath}` so trailing slashes on `baseUrl` are tolerated. (HTTP transport; a socket transport emits instead — see above.)
+2. Build body: `inputFor?.(endpoint) ?? {}`. For `GET` it rides the query string under the reserved key the real client uses (core `buildRoutedGetUrl`); otherwise it is `JSON.stringify`d.
 3. Send the request with `AbortController` wired to `requestTimeoutMs`. A timeout surfaces as a `fetch` error rather than hanging the sweep.
 4. Parse the JSON body. Any parse error is a contract failure (the framework always returns JSON).
 5. Assert `parsed.status` is exactly `'success'` or `'error'`. Anything else (`null`, `undefined`, missing key, a custom shape) fails with `reason: 'Response missing standard `status` envelope'`.
@@ -55,7 +64,7 @@ Pass criterion: a response that round-trips through the framework envelope, rega
 
 ### `runContractTests(input)`
 
-Full sweep version of `runContractCheck`. Iterates `walkEndpoints(input.apiMethodMap)` serially and aggregates results.
+Full sweep version of `runContractCheck`. Iterates every API route (plus every sync route when `syncMetaMap` is given), narrowed by `filter` before any request is sent, serially, and aggregates results.
 
 Signature:
 
@@ -64,6 +73,9 @@ runContractTests(input: RunContractTestsInput): Promise<RunContractSummary>
 
 interface RunContractTestsInput {
   apiMethodMap: ApiMethodMap;
+  syncMetaMap?: SyncMetaMap;       // sweep sync routes too
+  filter?: string;                 // substring on `<page>/<name>/<version>`
+  syncTransport?: ProbeTransport;  // HTTP when omitted
   baseUrl: string;
   skip?: string[];
   inputFor?: (endpoint: EndpointDescriptor) => unknown;
@@ -74,6 +86,8 @@ interface RunContractTestsInput {
 
 Behavior:
 
+- **Every route is called with the sweep's session — logout-type routes included.** An authenticated sweep that hits a route which destroys the current session (a logout, account deletion, session revoke) loses its session for every layer after it. Put such routes in `skip` (`TEST_SKIP` in the scaffolded script).
+
 - **Serial, not parallel.** The runner walks endpoints one by one. This avoids tripping shared rate-limit buckets, keeps log output linear, and stops endpoint-ordering bugs from being masked by races. A pool option may land later; do not assume concurrency.
 - **`skip` matching is two-tier.** `skip: ['billing/getInvoice']` skips every version of that endpoint. `skip: ['billing/getInvoice/v2']` skips only v2. Both forms compose: pass both to skip every version AND keep the specific one explicit.
 - **`onResult` is fired after every endpoint** including skipped ones, so progress reporters update in real time. Throwing inside `onResult` propagates out of the sweep — wrap your own handler in `tryCatch` if you want it isolated.
@@ -163,7 +177,8 @@ interface EndpointDescriptor {
   name: string;
   version: string;
   method: HttpMethod;
-  fullPath: string; // `api/<page>/<name>/<version>`
+  fullPath: string; // `api/<page>/<name>/<version>` or `sync/<page>/<name>/<version>`
+  kind?: 'api' | 'sync'; // absent = 'api'
 }
 
 interface ContractCheckResult {
@@ -290,7 +305,7 @@ await runContractTests({
 
 ## Edge cases and gotchas
 
-- **`GET` endpoints ignore `inputFor`.** The handler still calls `inputFor`, but the body is dropped before `fetch` sees it. Query-string fuzz is not in scope for the contract layer.
+- **`GET` endpoints receive `inputFor` through the query string.** `fetch` cannot send a GET body, so the payload is JSON-encoded under the same reserved query key the real client sends and the server reads. Before this, the payload was dropped and every GET route with a required field failed validation. Layer 5's `ctx.callApi` uses the same encoding.
 - **`headers` are merged after `Content-Type: application/json`.** If you pass `Content-Type` yourself you will override the default; the framework parses based on the header so this is intentional.
 - **`baseUrl` trailing slash is normalized once.** `http://localhost/` and `http://localhost` produce identical requests. Embedded `//` further down the path is not collapsed.
 - **`requestTimeoutMs` is per-request, not per-sweep.** A 30-endpoint sweep with the default 5000ms timeout can run for over two minutes if every endpoint stalls. The sweep itself has no overall timeout.

package/docs/extension-hooks.md

@@ -70,9 +70,10 @@ interface TestLayer {
 }
 
 interface TestLayerInput {
-  endpoint: string;          // resolved route, e.g. 'api/billing/getInvoice/v1'
+  endpoint: string;          // resolved route, e.g. 'api/billing/getInvoice/v1' or 'sync/chat/send/v1'
   method?: string;           // HTTP method, undefined for sync endpoints
   authToken?: string;        // bearer/session token for auth-aware checks
+  sampleInput?: unknown;     // Zod-valid sample payload (same one the contract sweep sends), `{}` without schemas
 }
 
 interface TestLayerResult {
@@ -100,7 +101,7 @@ Order is registration order (Map iteration order). The returned array is a fresh
 
 ### `runRegisteredLayers(input)`
 
-Run every registered layer against every endpoint in `apiMethodMap` and emit the aggregate through the registered reporter. This is the shipped entry point for the layer registry — the built-in sweeps never touch it, and it never runs the built-in sweeps.
+Run every registered layer against every endpoint in `apiMethodMap` (plus every sync route in `syncMetaMap` when given) and emit the aggregate through the registered reporter. This is the shipped entry point for the layer registry — the built-in sweeps never touch it, and it never runs the built-in sweeps.
 
 Signature:
 
@@ -109,14 +110,19 @@ runRegisteredLayers(input: RunRegisteredLayersInput): Promise<TestSummary>
 
 interface RunRegisteredLayersInput {
   apiMethodMap: ApiMethodMap;
-  authToken?: string;   // forwarded to each layer's run({ authToken })
-  skip?: string[];      // '<page>/<name>' or '<page>/<name>/<version>'
+  syncMetaMap?: SyncMetaMap;          // also run layers against every sync route
+  authToken?: string;                 // forwarded to each layer's run({ authToken })
+  skip?: string[];                    // '<page>/<name>' or '<page>/<name>/<version>'
+  filter?: string;                    // substring on '<page>/<name>/<version>', same as the sweeps
+  apiInputSchemas?: InputSchemaMap;   // source of each API route's sampleInput
+  syncInputSchemas?: InputSchemaMap;  // source of each sync route's sampleInput
+  inputFor?: (endpoint: EndpointDescriptor) => unknown; // custom sample factory; wins over the schema maps
 }
 ```
 
 Behavior (read from `src/runRegisteredLayers.ts`):
 
-- **Cartesian walk.** For each layer from `listTestLayers()` (registration order) × each endpoint from `walkEndpoints(apiMethodMap)`, it calls `layer.run({ endpoint: endpoint.fullPath, method: endpoint.method, authToken })`. `skip` uses the same two-tier matching as the built-in sweeps; skipped endpoints produce no result.
+- **Cartesian walk.** For each layer from `listTestLayers()` (registration order) × each selected endpoint (API routes, plus sync routes when `syncMetaMap` is given, narrowed by `filter`), it calls `layer.run({ endpoint, method, authToken, sampleInput })` — `method` is `undefined` for sync routes, and `sampleInput` comes from `inputFor` or the schema maps (strings tagged with the run's test-data marker), so a layer can exercise the route past validation. `skip` uses the same two-tier matching as the built-in sweeps; skipped endpoints produce no result.
 - **Error isolation.** Each `layer.run` is wrapped in `tryCatch`. A throwing layer (or one returning a falsy result) does not abort the walk — it is recorded as a failed `TestResult` whose `message` is the thrown error's message, or `'layer returned no result'`. Each result's `durationMs` is measured around the call. This is the behavior the manual `listTestLayers()` loop lacks, where a throw propagates and skips the rest.
 - **Reporter fan-out.** After each result it calls `reporter.onResult?.(result)`; after the walk it calls `reporter.onSummary?.(summary)`. Both are wrapped in `tryCatch`, so a throwing reporter callback never fails the run — unlike the built-in sweeps, whose `onResult` is called un-wrapped.
 - **Webhook.** If the reporter declares a `webhookUrl`, the summary is POSTed to it: `Content-Type: application/json`, plus `Authorization: Bearer <token>` when `webhookAuth?.type === 'bearer'`, body `JSON.stringify(summary)`, with a 10s `AbortController` timeout. The POST is best-effort (`tryCatch`-wrapped) — a failure never fails the run. A plaintext-`http:` URL pointed at a non-loopback host logs a `console.warn` (the summary carries endpoint paths, error codes, and any bearer token in clear) but is never blocked, since the URL is consumer-self-registered.
@@ -231,6 +237,7 @@ interface TestLayerInput {
   endpoint: string;
   method?: string;
   authToken?: string;
+  sampleInput?: unknown;
 }
 
 interface TestLayerResult {

package/docs/fuzz-tests.md

@@ -26,7 +26,7 @@ interface FuzzCheckInput {
 Flow:
 
 1. Iterate the built-in `JUNK_PAYLOADS` list serially.
-2. For each payload, POST/PUT/DELETE it to `${baseUrl}/${endpoint.fullPath}` (or send no body on GET). The body is `JSON.stringify(payload)` so values that lose information under stringify (e.g. `undefined` keys, `NaN`) round-trip through the wire encoding the same as in production.
+2. For each payload, POST/PUT/DELETE it to `${baseUrl}/${endpoint.fullPath}` (on GET the payload rides the query string, under the same reserved key the real client uses). The body is `JSON.stringify(payload)` so values that lose information under stringify (e.g. `undefined` keys, `NaN`) round-trip through the wire encoding the same as in production.
 3. After each request:
    - **No response (fetch error, abort, DNS):** fail with `reason: 'fuzz probe crashed (no response) with payload: ...'`. The reason includes the first 80 chars of the payload JSON.
    - **`httpStatus >= 500` WITH a valid envelope:** **pass**. A 5xx is deliberately tolerated as long as the body is still a `{ status: 'error', errorCode }` envelope — that means the framework caught the error and shaped it, which is exactly the crash-resistance property this layer checks. Only the envelope checks below can fail a probe; the status code alone never does.
@@ -38,7 +38,7 @@ The loop stops at the first failing probe. The result captures only that probe's
 
 ### `runFuzzTests(input)`
 
-Full sweep. Iterates `walkEndpoints(input.apiMethodMap)` and runs `runFuzzCheck` against each.
+Full sweep. Iterates every API route (plus every sync route when `syncMetaMap` is given), narrowed by `filter` before any request, and runs `runFuzzCheck` against each. In an authenticated (cookie) sweep, state-changing routes are reported as skipped to avoid junk-body mutations on the test database — every sync route counts as state-changing, so an authenticated sweep does not fuzz sync routes.
 
 Signature:
 
@@ -54,7 +54,7 @@ interface RunFuzzTestsInput {
 }
 ```
 
-There is no per-endpoint metadata filtering here — every endpoint is fuzzed, regardless of `auth.login` or `rateLimit`. `skip` matching follows the same two-tier convention as the other sweeps: `'<page>/<name>'` for version-agnostic, `'<page>/<name>/<version>'` for specific.
+There is no per-endpoint metadata filtering here — every endpoint (bar the authenticated-sweep skip above) is fuzzed, regardless of `auth.login` or `rateLimit`. `skip` matching follows the same two-tier convention as the other sweeps: `'<page>/<name>'` for version-agnostic, `'<page>/<name>/<version>'` for specific.
 
 `onResult` fires once per endpoint (after all junk payloads for that endpoint have run, or at the first failing probe). It is NOT called per junk payload.
 

package/docs/rate-limit-tests.md

@@ -38,7 +38,7 @@ Important details:
 
 ### `runRateLimitTests(input)`
 
-Full sweep. Reads `rateLimit` from `apiMetaMap[page][name][version].rateLimit`, skips endpoints where it is `false`, `undefined`, or above `maxRateLimitToTest`, then runs `runRateLimitCheck` against the rest.
+Full sweep. Reads `rateLimit` from `apiMetaMap[page][name][version].rateLimit` (from `syncMetaMap` for sync routes, which are swept when `syncMetaMap` is given; `filter` narrows the set before any request), skips endpoints where it is `false`, `undefined`, or above `maxRateLimitToTest`, then runs `runRateLimitCheck` against the rest.
 
 Signature:
 
@@ -63,13 +63,13 @@ Behavior notes:
 
 - **Endpoints with `rateLimit: false` are silently skipped.** No `skipped` result, no `onResult` call. The summary's `total` reflects only endpoints with a numeric rate limit.
 - **Endpoints with `rateLimit > maxRateLimitToTest` are skipped loudly.** They appear with `status: 'skipped'` and `reason: 'rateLimit ${N} exceeds maxRateLimitToTest=${max}'`. Bump `maxRateLimitToTest` to cover them, but be aware each endpoint then fires `N+1` requests serially.
-- **`resetBetweenEndpoints: true` calls `resetServerState({ baseUrl, token: resetToken })` before every endpoint** that survives the skip filter. Without this, the limiter bucket on a shared IP (the test runner's host) is not flushed between endpoints — endpoint A's drain requests bleed into endpoint B's window and the N+1 assertion fires too early or too late depending on key collisions.
+- **`resetBetweenEndpoints: true` calls `resetServerState({ baseUrl, token: resetToken, only: ['rateLimits'] })` before every endpoint** that survives the skip filter. Only limiter buckets are cleared, so an authenticated sweep keeps its session. Without this, the limiter bucket on a shared IP (the test runner's host) is not flushed between endpoints — endpoint A's drain requests bleed into endpoint B's window and the N+1 assertion fires too early or too late depending on key collisions.
 - **`resetToken` is required when `TEST_RESET_TOKEN` is set on the server.** Always set this in any non-prod environment that is reachable over the network (staging, preview). The server is fail-closed: missing token = 403, unset env var server-side = 403 (never "open").
 - **`onResult` fires for tested AND `maxRateLimitToTest`-skipped endpoints**, but not for `rateLimit: false` endpoints (those don't show up at all).
 
 ### `resetServerState(input)`
 
-POST to the server's `/_test/reset` endpoint. Used between rate-limit probes and at CI bookends to clear framework-owned limiter/default-session state; it is not a general application-database reset.
+POST to the server's `/_test/reset` endpoint. Used between rate-limit probes and at the `runAllTests` bookends to clear framework-owned limiter (and, unscoped, default-session) state; it is not an application-database reset.
 
 Signature:
 
@@ -79,9 +79,13 @@ resetServerState(input: ResetServerStateInput): Promise<boolean>
 interface ResetServerStateInput {
   baseUrl: string;
   token?: string;
+  requestTimeoutMs?: number;
+  only?: TestResetTarget[]; // sent as `?only=`; omit for the server's default reset
 }
 ```
 
+Pass `only` to clear exactly those targets. Omitting it gives the server's default reset, which also deletes every Redis session — including an authenticated sweep's own. The runner itself therefore always passes `only: ['rateLimits']`.
+
 Returns `true` when the server responds 2xx, `false` on network error, parse error, or non-2xx. The error is swallowed deliberately — the most common reason for `false` is "the route is disabled in this environment", and the caller usually wants to proceed anyway.
 
 URL: `${baseUrl}/_test/reset` (POST). Supply `token` (normally `TEST_RESET_TOKEN`); the request sends it as `X-Test-Reset-Token`. The server always requires a non-empty matching value.
@@ -97,7 +101,9 @@ Gating, in order:
 
 What it clears (when both gates pass):
 
-- **All rate-limit buckets** via `clearAllRateLimits()` from `@luckystack/core`. Every key under the rate-limit Redis namespace is dropped. Always cleared.
+**`?only=<target>[,<target>]`** clears exactly the listed targets and nothing else. The valid targets are the shared list in core `TEST_RESET_TARGETS` (rate limits, sessions, active users, hooks), so client and server cannot drift. An unknown or empty target is a `400` with `errorCode: 'test.invalidResetTarget'` — a typo must not silently reset nothing, or everything. Without `only`, the default below applies unchanged:
+
+- **All rate-limit buckets** via `clearAllRateLimits()` from `@luckystack/core`. Every key under the rate-limit Redis namespace is dropped.
 - **Default Redis session namespaces** matching `formatKey('-session', '') + ':*'` and `formatKey('-activeUsers', '') + ':*'`. They are scanned in batches of 200 and follow the registered Redis key formatter.
 - **Registered runtime hooks** when `?include=hooks` is in the query string. NOT included by default because framework-internal handlers (e.g. presence postLogout) register at boot — clearing them would require a server restart to recover.
 
@@ -139,6 +145,8 @@ interface RunRateLimitTestsInput {
 interface ResetServerStateInput {
   baseUrl: string;
   token?: string;
+  requestTimeoutMs?: number;
+  only?: TestResetTarget[];
 }
 ```
 
@@ -255,5 +263,5 @@ Endpoints with `rateLimit: 100` will appear as skipped with the reason inline.
 - **The `validation.failed` trap.** If the validator rejects every drain request before the limiter sees them, the framework still counts the request — but verifying this assumption is the user's responsibility. Use `inputFor` to provide payloads that survive validation when in doubt.
 - **`maxRateLimitToTest` accidentally too high.** With `maxRateLimitToTest: 1000` and 30 endpoints, the sweep fires `~30000` requests serially. This will time out CI long before it completes. Default 50 is deliberate.
 - **`/_test/reset` returning false from `resetServerState`.** Most common cause: `NODE_ENV !== 'development' | 'test'` on the server. Second most common: missing token. Third: route disabled via custom routing. The runner does not retry — it returns `false` and proceeds. The next endpoint's window will be polluted; consider failing the sweep early.
-- **Sessions cleared as collateral damage.** `resetServerState` always drops sessions and active users in addition to limiter buckets. If you have a long-running logged-in test alongside this layer, run the rate-limit sweep first, then re-establish the session.
+- **Sessions cleared as collateral damage — only on an unscoped reset.** `resetServerState` without `only` drops sessions and active users in addition to limiter buckets. The runner's own resets pass `only: ['rateLimits']` and leave sessions alone; a direct call without `only` still logs out every logged-in test.
 - **`?include=hooks` is opt-in for a reason.** Do not call `resetServerState` with this flag in a long-running test process — framework-internal hooks register only at boot, and dropping them mid-run will break presence, post-logout cleanup, and similar side-effects until restart.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@luckystack/test-runner",
-  "version": "0.10.0",
+  "version": "0.10.2",
   "private": false,
   "publishConfig": {
     "access": "public",
@@ -54,11 +54,11 @@
     "test": "vitest run"
   },
   "dependencies": {
-    "@luckystack/core": "^0.10.0"
+    "@luckystack/core": "^0.10.2"
   },
   "peerDependencies": {
-    "@luckystack/login": "^0.10.0",
-    "@luckystack/secret-manager": "^0.10.0",
+    "@luckystack/login": "^0.10.2",
+    "@luckystack/secret-manager": "^0.10.2",
     "socket.io-client": "^4.8.0",
     "zod": "^4.0.0"
   },