@classytic/arc-testkit 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +30 -12
- package/dist/conformance.d.mts +380 -0
- package/dist/conformance.mjs +831 -0
- package/dist/index.d.mts +0 -2
- package/dist/index.mjs +1 -2
- package/dist/mongo-commands.d.mts +0 -2
- package/dist/mongo-commands.mjs +1 -2
- package/dist/mongo-explain.d.mts +0 -2
- package/dist/mongo-explain.mjs +1 -2
- package/dist/mongo.d.mts +0 -2
- package/dist/mongo.mjs +1 -2
- package/dist/pool.d.mts +0 -2
- package/dist/pool.mjs +3 -4
- package/dist/story.d.mts +202 -0
- package/dist/story.mjs +330 -0
- package/llms.txt +33 -0
- package/package.json +23 -3
- package/skills/arc-testkit/SKILL.md +113 -0
- package/skills/arc-testkit/references/conformance.md +104 -0
- package/skills/arc-testkit/references/fixtures.md +92 -0
- package/skills/arc-testkit/references/list-filters.md +3 -0
- package/skills/arc-testkit/references/story.md +80 -0
package/llms.txt
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# @classytic/arc-testkit
|
|
2
|
+
|
|
3
|
+
> Test harness for arc apps: boot modules on in-memory MongoDB, pool servers, cost and plan asserts,
|
|
4
|
+
> conformance guards and a story kit. Needs @classytic/arc >=2.43.0; the guards and stories need vitest >=3.
|
|
5
|
+
|
|
6
|
+
Hand-rolling any row below is a bug. Skill: skills/arc-testkit/SKILL.md (+ references/).
|
|
7
|
+
|
|
8
|
+
| Need | Use | Import |
|
|
9
|
+
|---|---|---|
|
|
10
|
+
| Boot module(s) on a real DB | bootModuleApp | @classytic/arc-testkit |
|
|
11
|
+
| One mongod per run, DB per file | createTestDatabasePool, mongoServerProvider | /pool, /mongo |
|
|
12
|
+
| Worker cap by cores and RAM | safeWorkerCount | /mongo |
|
|
13
|
+
| Wait for index builds | awaitIndexBuilds | /mongo |
|
|
14
|
+
| N+1 detection | countCommands, assertConstantCommands | /mongo-commands |
|
|
15
|
+
| Index plan proof | assertIndexServed, summarizeExplain | /mongo-explain |
|
|
16
|
+
| List routes answer the repo-core envelope | checkListEnvelopes, registerListEnvelopeGuard | /conformance |
|
|
17
|
+
| Actions and write routes have schemas | checkRouteSchemas, registerRouteSchemaGuard | /conformance |
|
|
18
|
+
| Verb and route-family naming | checkNaming, registerNamingGuard | /conformance |
|
|
19
|
+
| OpenAPI shape, security, secrets | checkOpenApiShape, checkSecurity, checkRoutesDocumented, checkAnonymousRefused, checkNoSecrets, registerOpenApiGuard | /conformance |
|
|
20
|
+
| List filters match the route | checkFilterDeclarations | /conformance |
|
|
21
|
+
| Client routes exist in the spec | checkClientDrift, registerClientDriftGuard, dumpOpenApi, loadOpenApiDump | /conformance |
|
|
22
|
+
| Error bodies are ErrorContract | checkErrorContracts, expectErrorContract, registerErrorContractGuard | /conformance |
|
|
23
|
+
| Bare collection paths | checkCollectionRoutes, registerCollectionRoutesGuard | /conformance |
|
|
24
|
+
| Refusal leaves no side effect | registerRefusalDoesNoWorkGuard | /conformance |
|
|
25
|
+
| App as caller, spec, route table | injectCaller, fetchOpenApi, routeTable, expectClean | /conformance |
|
|
26
|
+
| Numbered, clause-tagged steps | describeStory, need, clauseMessage | /story |
|
|
27
|
+
| Users with roles | createCast, bearer | /story |
|
|
28
|
+
| Time travel, civil dates | createStoryClock, civilDateOf, addCivilDays, addCivilMonths, daysBetween, instantOn | /story |
|
|
29
|
+
| Outbox, events, feed, mail, audit asserts | createStoryAssertions, createEventLog | /story |
|
|
30
|
+
| Sweep twice changes nothing | expectIdempotent, expectSweepsIdempotent | /story |
|
|
31
|
+
|
|
32
|
+
Rules: every guard is check* (returns problems) + register* (vitest tests); exception lists fail when stale;
|
|
33
|
+
fixture-based hosts call check* then expectClean inside the test; pass getters, not values, for an app booted in beforeAll.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@classytic/arc-testkit",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.6.0",
|
|
4
4
|
"description": "Boot a real @classytic/arc app around the module(s) under test on in-memory MongoDB — the ecosystem test harness. Thin Mongo-typed wrapper over @classytic/arc/testing (arc 2.22+). If your arc module boots green here, it composes into any arc host.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"exports": {
|
|
@@ -8,6 +8,14 @@
|
|
|
8
8
|
"types": "./dist/index.d.mts",
|
|
9
9
|
"default": "./dist/index.mjs"
|
|
10
10
|
},
|
|
11
|
+
"./conformance": {
|
|
12
|
+
"types": "./dist/conformance.d.mts",
|
|
13
|
+
"default": "./dist/conformance.mjs"
|
|
14
|
+
},
|
|
15
|
+
"./story": {
|
|
16
|
+
"types": "./dist/story.d.mts",
|
|
17
|
+
"default": "./dist/story.mjs"
|
|
18
|
+
},
|
|
11
19
|
"./pool": {
|
|
12
20
|
"types": "./dist/pool.d.mts",
|
|
13
21
|
"default": "./dist/pool.mjs"
|
|
@@ -29,6 +37,8 @@
|
|
|
29
37
|
"types": "./dist/index.d.mts",
|
|
30
38
|
"sideEffects": false,
|
|
31
39
|
"files": [
|
|
40
|
+
"skills",
|
|
41
|
+
"llms.txt",
|
|
32
42
|
"dist",
|
|
33
43
|
"README.md"
|
|
34
44
|
],
|
|
@@ -41,6 +51,7 @@
|
|
|
41
51
|
"scripts": {
|
|
42
52
|
"build": "tsdown",
|
|
43
53
|
"typecheck": "tsc --noEmit",
|
|
54
|
+
"check:skills": "vitest run --root ../.. packages/arc-testkit/tests/skills.test.ts",
|
|
44
55
|
"test": "vitest run --root ../.. packages/arc-testkit/tests",
|
|
45
56
|
"lint": "biome check src/",
|
|
46
57
|
"prepublishOnly": "node ../../scripts/prepublish-gate.mjs"
|
|
@@ -49,8 +60,17 @@
|
|
|
49
60
|
"mongodb-memory-server": ">=10.0.0"
|
|
50
61
|
},
|
|
51
62
|
"peerDependencies": {
|
|
52
|
-
"
|
|
53
|
-
"
|
|
63
|
+
"vitest": ">=5.0.5",
|
|
64
|
+
"@classytic/arc": ">=2.46.0",
|
|
65
|
+
"mongoose": ">=9.11.4"
|
|
66
|
+
},
|
|
67
|
+
"peerDependenciesMeta": {
|
|
68
|
+
"mongoose": {
|
|
69
|
+
"optional": true
|
|
70
|
+
},
|
|
71
|
+
"vitest": {
|
|
72
|
+
"optional": true
|
|
73
|
+
}
|
|
54
74
|
},
|
|
55
75
|
"keywords": [
|
|
56
76
|
"arc",
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: arc-testkit
|
|
3
|
+
description: |
|
|
4
|
+
@classytic/arc-testkit - test harness for arc apps: boot modules on in-memory MongoDB, pool servers, count wire commands, assert index plans, conformance guards (list envelope, route schemas, naming, OpenAPI, client drift, error contract), story kit (cast, clock, steps, evidence asserts). Triggers: arc-testkit, bootModuleApp, conformance, story test.
|
|
5
|
+
license: MIT
|
|
6
|
+
metadata:
|
|
7
|
+
author: Classytic
|
|
8
|
+
minVersion: "0.6.0"
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# @classytic/arc-testkit
|
|
12
|
+
|
|
13
|
+
Needs `@classytic/arc >=2.43.0`; `mongoose >=9.4.1` for root, `/pool`, `/mongo*`; `/conformance` and `/story` need `vitest >=3` only. Guards are `check*` (returns problems) plus `register*` (vitest tests).
|
|
14
|
+
|
|
15
|
+
## Do not hand-roll
|
|
16
|
+
|
|
17
|
+
| Need | Use | Import |
|
|
18
|
+
|---|---|---|
|
|
19
|
+
| Boot module(s) on a real DB | `bootModuleApp(modules, options)` | `@classytic/arc-testkit` |
|
|
20
|
+
| One mongod per run, one DB per file | `createTestDatabasePool`, `mongoServerProvider` | `/pool`, `/mongo` |
|
|
21
|
+
| Worker cap by cores and RAM | `safeWorkerCount(envVar)` | `/mongo` |
|
|
22
|
+
| Wait for index builds | `awaitIndexBuilds` | `/mongo` |
|
|
23
|
+
| Count driver commands (N+1) | `countCommands`, `assertConstantCommands` | `/mongo-commands` |
|
|
24
|
+
| Prove a query uses an index | `assertIndexServed`, `summarizeExplain` | `/mongo-explain` |
|
|
25
|
+
| Every list answers the repo-core envelope | `checkListEnvelopes`, `registerListEnvelopeGuard` | `/conformance` |
|
|
26
|
+
| Every action/write route has a schema | `checkRouteSchemas`, `registerRouteSchemaGuard` | `/conformance` |
|
|
27
|
+
| Verb and route-family naming | `checkNaming`, `registerNamingGuard` | `/conformance` |
|
|
28
|
+
| OpenAPI shape, security, secrets | `checkOpenApiShape`, `checkSecurity`, `checkRoutesDocumented`, `checkAnonymousRefused`, `checkNoSecrets`, `registerOpenApiGuard` | `/conformance` |
|
|
29
|
+
| Client routes exist in the spec | `checkClientDrift`, `registerClientDriftGuard`, `dumpOpenApi`, `loadOpenApiDump` | `/conformance` |
|
|
30
|
+
| List filters match the route | `checkFilterDeclarations` | `/conformance` |
|
|
31
|
+
| Error bodies are `ErrorContract` | `checkErrorContracts`, `expectErrorContract`, `registerErrorContractGuard` | `/conformance` |
|
|
32
|
+
| Bare collection paths | `checkCollectionRoutes`, `registerCollectionRoutesGuard` | `/conformance` |
|
|
33
|
+
| Refusal leaves no side effect | `registerRefusalDoesNoWorkGuard` | `/conformance` |
|
|
34
|
+
| App as caller, spec, routes | `injectCaller(app, headers)`, `fetchOpenApi(caller)`, `routeTable(app)`, `expectClean(what, problems)` | `/conformance` |
|
|
35
|
+
| Numbered, clause-tagged story steps | `describeStory`, `need`, `clauseMessage` | `/story` |
|
|
36
|
+
| Named users with roles | `createCast`, `bearer` | `/story` |
|
|
37
|
+
| Time travel, civil dates | `createStoryClock`, `civilDateOf`, `addCivilDays`, `addCivilMonths`, `daysBetween`, `instantOn` | `/story` |
|
|
38
|
+
| Outbox, events, feed, mail, audit asserts | `createStoryAssertions`, `createEventLog` | `/story` |
|
|
39
|
+
| Sweep run twice changes nothing | `expectIdempotent`, `expectSweepsIdempotent` | `/story` |
|
|
40
|
+
|
|
41
|
+
## Golden path
|
|
42
|
+
|
|
43
|
+
Boot (`replset: true` only for transactions):
|
|
44
|
+
|
|
45
|
+
```ts
|
|
46
|
+
import { bootModuleApp } from "@classytic/arc-testkit";
|
|
47
|
+
|
|
48
|
+
const t = await bootModuleApp(({ connection }) => [createThingModule({ connection, permissions })], { replset: true });
|
|
49
|
+
const res = await t.app.inject({ method: "GET", url: "/things" });
|
|
50
|
+
await t.close();
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Pool (vitest `globalSetup` exports `setup`/`teardown`; suites boot through `withPool`):
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
import { mongoServerProvider } from "@classytic/arc-testkit/mongo";
|
|
57
|
+
import { createTestDatabasePool } from "@classytic/arc-testkit/pool";
|
|
58
|
+
import mongoose from "mongoose";
|
|
59
|
+
|
|
60
|
+
export const dbPool = createTestDatabasePool(mongoServerProvider({ mongoose }), { envPrefix: "APP_TEST", preboot: ["transactional"] });
|
|
61
|
+
export const setup = () => dbPool.globalSetup.setup();
|
|
62
|
+
export const teardown = () => dbPool.globalSetup.teardown();
|
|
63
|
+
// suite: const boot = dbPool.withPool(bootModuleApp); vitest config: maxWorkers: safeWorkerCount("TEST_MAX_WORKERS")
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Guards inside a test with the app and an admin caller:
|
|
67
|
+
|
|
68
|
+
```ts
|
|
69
|
+
import { checkListEnvelopes, checkSecurity, expectClean, fetchOpenApi } from "@classytic/arc-testkit/conformance";
|
|
70
|
+
|
|
71
|
+
test("contract", async () => {
|
|
72
|
+
expectClean("list envelope", (await checkListEnvelopes({ app: t.app, caller: admin, prefix: "/api", notLists: { "/api/me": "one document" } })).problems);
|
|
73
|
+
expectClean("security", await checkSecurity({ spec: await fetchOpenApi(admin), publicOperations: [{ path: "/api/health", why: "probe" }] }));
|
|
74
|
+
});
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Story:
|
|
78
|
+
|
|
79
|
+
```ts
|
|
80
|
+
import { createCast, describeStory } from "@classytic/arc-testkit/story";
|
|
81
|
+
|
|
82
|
+
describeStory("checkout", (s) => {
|
|
83
|
+
s.step("customer places an order", { clauses: ["ORD-1"] }, async (t) => {
|
|
84
|
+
const res = await cast.get("ann").post("/orders", {});
|
|
85
|
+
t.expect(res.status, "the order is accepted").toBe(201);
|
|
86
|
+
});
|
|
87
|
+
});
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## Gotchas
|
|
91
|
+
|
|
92
|
+
- Exception lists (`notLists`, `customShapes`, `actionExceptions`, `routeExceptions`, `exemptVerbs`, `publicOperations`) fail when an entry matches nothing: delete the entry, never keep it "just in case".
|
|
93
|
+
- `register*` runs at describe time with getters (`app: () => t.app`, app booted in `beforeAll`). Fixture-based hosts (`test.extend`) use `check*` + `expectClean` inside the test, or pass `test` and `fixtures` to `describeStory`.
|
|
94
|
+
- `checkListEnvelopes` skips non-200 GETs and paths with params; set `minRoutes` and `expectLists` so an empty probe cannot pass.
|
|
95
|
+
- `checkErrorContracts` needs a probe per class (`require` narrows them); its unknown-route probe also fails when the query string is echoed.
|
|
96
|
+
- Fastify's own 404 body fails the contract: set a not-found handler.
|
|
97
|
+
- `checkRouteSchemas` reads `app.arc.registry`: the app must be booted by arc.
|
|
98
|
+
- A pool needs `globalSetup` to preboot, else every worker boots its own mongod. `countCommands` needs `monitorCommands: true`.
|
|
99
|
+
- `loadOpenApiDump` throws when the dump is older than the `.ts` under `sources`: regenerate it.
|
|
100
|
+
|
|
101
|
+
## Never
|
|
102
|
+
|
|
103
|
+
- Hand-roll a route-table parser, OpenAPI walker or per-file `MongoMemoryServer`.
|
|
104
|
+
- Import from `dist/` or `src/` paths; use the subpaths above only.
|
|
105
|
+
- Assert `ErrorContract` bodies field by field; use `expectErrorContract`.
|
|
106
|
+
- Put domain rules in the kit.
|
|
107
|
+
|
|
108
|
+
## References
|
|
109
|
+
|
|
110
|
+
- `references/fixtures.md` - read when booting apps, pooling servers, capping workers, counting commands or asserting plans.
|
|
111
|
+
- `references/conformance.md` - read when adding a guard: what it enforces, every parameter, how to read its failure.
|
|
112
|
+
- `references/story.md` - read when writing a story test: cast, clock, steps, ports, idempotency.
|
|
113
|
+
- `references/list-filters.md` - read when a UI sends list filters, search or sorts.
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
# Conformance guards (`@classytic/arc-testkit/conformance`)
|
|
2
|
+
|
|
3
|
+
Common shapes. `Source<T>` is a value or `() => T | Promise<T>`. `app` is the booted Fastify instance (anything with
|
|
4
|
+
`inject` and `printRoutes`). `Caller` is `{ get(path) }` returning `{ status, body }`; a host test client fits, or build
|
|
5
|
+
one with `injectCaller(app, { authorization: "Bearer ..." })` (adds `post`, `put`, `patch`, `delete`, `request`).
|
|
6
|
+
`Exceptions` is `string[]`, a `Set`, or `Record<key, reason>`. Each `check*` returns `string[]` (problems);
|
|
7
|
+
`register*` wraps it in `it(...)`; `expectClean(what, problems)` fails with one problem per line.
|
|
8
|
+
|
|
9
|
+
## List envelope
|
|
10
|
+
|
|
11
|
+
`checkListEnvelopes(opts)` returns `{ problems, lists, probed }`. Calls every static GET (no `:` or `*`) as `caller`; a
|
|
12
|
+
200 body must be an offset page (`method: "offset"`, numeric `page limit total pages`, boolean `hasNext hasPrev`), a
|
|
13
|
+
keyset page (`limit`, boolean `hasMore`, `next` string or null) or exactly `{ data: [...] }`.
|
|
14
|
+
|
|
15
|
+
| Option | Meaning |
|
|
16
|
+
|---|---|
|
|
17
|
+
| `app`, `caller` | required; caller must be allowed to read every list |
|
|
18
|
+
| `prefix` | only GET paths starting with it |
|
|
19
|
+
| `notLists` | GETs that are not lists (single documents, tables), with reason |
|
|
20
|
+
| `customShapes` | GETs that answer their own shape (package routes) |
|
|
21
|
+
| `skip` | RegExp of paths never probed; default `/(stream\|export\|content\|download)(/\|$)` |
|
|
22
|
+
| `forbiddenKeys` | keys a list never carries; default `docs rows items pagination totalPages` |
|
|
23
|
+
| `methods` | accepted `method` values; default offset and keyset |
|
|
24
|
+
| `minRoutes`, `expectLists` | probe size floor; paths that must have answered a list |
|
|
25
|
+
|
|
26
|
+
Failures: `PATH: a bare array`; `PATH: not a list and not in notLists (keys)`; `PATH: carries docs`; `PATH: offset page:
|
|
27
|
+
\`total\` is not a number`; `stale exception PATH: no such GET route is registered`. Non-200 answers are skipped, so
|
|
28
|
+
a route needing a filter is never reported: assert it separately.
|
|
29
|
+
|
|
30
|
+
## Route schemas
|
|
31
|
+
|
|
32
|
+
`checkRouteSchemas({ app, actionExceptions, routeExceptions, bodyMethods, minActions })` returns
|
|
33
|
+
`{ problems, actions, routes }`. Reads `app.arc.registry.getAll()`. Every action needs `schema`; every custom route with
|
|
34
|
+
method in `bodyMethods` (default POST PUT PATCH) needs `schema.body`. Keys: actions `resource.action`, routes
|
|
35
|
+
`resource METHOD /path` (path as declared on the resource, e.g. `thing POST /import`). Failures: `action thing.archive
|
|
36
|
+
declares no schema (add it to actionSchemas)`, `route thing POST /recompute declares no body schema`, `stale action
|
|
37
|
+
exception KEY`, `stale route exception KEY`. Use an exception only for package-mounted routes and routes without a
|
|
38
|
+
JSON body (multipart), with the reason as the record value.
|
|
39
|
+
|
|
40
|
+
## Naming
|
|
41
|
+
|
|
42
|
+
`checkNaming({ app, verbPattern, exemptVerbs, requiredVerbs, minVerbs, pathRules, forbiddenPaths })` returns problems.
|
|
43
|
+
`verbPattern` default `/^[a-z][a-zA-Z0-9]*$/` (camelCase action verbs). `pathRules`: `[{ name, match: RegExp, allowed:
|
|
44
|
+
["POST /api/x/sweep"] }]`; every route whose path matches belongs to the family and the set must equal `allowed`
|
|
45
|
+
(HEAD and OPTIONS ignored). `forbiddenPaths`: RegExps no route may match. Failures: `action r.set_due does not match
|
|
46
|
+
...`, `sweeps: unexpected route POST ...`, `sweeps: missing route ...`, `stale exempt verb v`.
|
|
47
|
+
|
|
48
|
+
## OpenAPI
|
|
49
|
+
|
|
50
|
+
Options (`OpenApiOptions`): `spec` (document or getter; `fetchOpenApi(caller, path = "/openapi.json")`), `app`,
|
|
51
|
+
`anonymous` (unauthenticated caller), `prefix`, `publicOperations: [{ path: string | RegExp | fn, why }]`,
|
|
52
|
+
`undocumented` (RegExp of route-table paths not required in the spec), `skipAnonymous` (default `/\{|\/stream$/`),
|
|
53
|
+
`minOperations`, `secretValues` (strings of length 6 or more), `secretPatterns` (default mongodb and redis URLs).
|
|
54
|
+
|
|
55
|
+
| Function | Enforces | Failure text |
|
|
56
|
+
|---|---|---|
|
|
57
|
+
| `checkOpenApiShape` | `openapi` 3.x, paths under `prefix` (public ones exempt), no trailing slash, `minOperations` | `PATH is outside the /api prefix` |
|
|
58
|
+
| `checkRoutesDocumented` | needs `app`; every route-table method+path (shape-normalised) is in the spec | `GET /x is not in the document` |
|
|
59
|
+
| `checkSecurity` | each operation has `security` (own, else document-level) unless a public rule matches; unused rule fails | `GET /x has no security requirement`, `stale public operation (why): ...` |
|
|
60
|
+
| `checkAnonymousRefused(opts, 401)` | needs `anonymous`; GET of each protected, param-free path answers 401 | `GET /x -> 200, expected 401` |
|
|
61
|
+
| `checkNoSecrets` | no `secretValues` entry or pattern in the serialised spec | `a configuration value is in the document` |
|
|
62
|
+
|
|
63
|
+
`registerOpenApiGuard(opts)` registers all five (the two with optional input only when `app` / `anonymous` is given).
|
|
64
|
+
|
|
65
|
+
## Client drift
|
|
66
|
+
|
|
67
|
+
`checkClientDrift({ spec, routes, bodyFields, minRoutes })`. `routes`: `[{ method, path, source? }]` the client can emit;
|
|
68
|
+
a path may be concrete (`/api/x/507f...`), a template (`/api/x/:id`, `/api/x/{id}`), with a query or trailing slash.
|
|
69
|
+
`bodyFields`: `[{ method, path: "/api/x/{id}/action", fields: [...], in: "request" | "response", status: "200", name? }]`
|
|
70
|
+
(path spelled as the spec does; fields resolved through `$ref`, `oneOf`, `allOf`, `anyOf`). Failures: `client route
|
|
71
|
+
missing from the document: DELETE /api/x/:id/archive (source)`, `NAME: field "f" is not in the schema`, `NAME: the
|
|
72
|
+
document declares no request schema`. Carry the spec between processes: `dumpOpenApi(spec, file)` on the server,
|
|
73
|
+
`loadOpenApiDump(file, { sources: [dirs], regenerate: "cd apps/server && npm run openapi:dump" })` on the client; it throws
|
|
74
|
+
when the dump is missing or older than a `.ts` under `sources`. Exported helpers: `specMatcher`, `missingClientRoutes`,
|
|
75
|
+
`bodyFieldProblems`, `propertyNames`, `resolveRef`. Needs the node environment (reads files).
|
|
76
|
+
|
|
77
|
+
## Error contract
|
|
78
|
+
|
|
79
|
+
`checkErrorContracts({ probes, caller, require, allowedKeys })`. Classes: `validation unauthorized forbidden notFound
|
|
80
|
+
unknownRoute conflict duplicateKey`. A probe is `{ provoke: () => Promise<Reply>, status?, code?: string | RegExp,
|
|
81
|
+
details?: "some" | "none", detailCode? }` or an array of them. Defaults: status 400/401/403/404/404/409/409, `details:
|
|
82
|
+
"some"` for validation and duplicateKey, `detailCode: "duplicate_key"` for duplicateKey, `code` any non-empty string.
|
|
83
|
+
`require` (default all seven) lists classes that must have a probe; `unknownRoute` has a built-in probe when `caller`
|
|
84
|
+
is given. The body may carry only `code message status details correlationId meta` (details only `path code message
|
|
85
|
+
meta`). Failures are prefixed with the class: `unknownRoute: foreign keys on the body: error, statusCode`, `validation:
|
|
86
|
+
code "x", expected "y"`, `duplicateKey: a detail has code "z", expected "duplicate_key"`. Single replies:
|
|
87
|
+
`errorContractProblems(reply, expected)` and `expectErrorContract(reply, { status, code, details })` (returns the body).
|
|
88
|
+
|
|
89
|
+
## Collection routes
|
|
90
|
+
|
|
91
|
+
`checkCollectionRoutes({ app, collections: [{ path: "/api/x", methods: ["GET", "POST"] }], served: { caller, paths } })`.
|
|
92
|
+
The bare path must be registered with each verb; `served` GETs must answer 200. Failures: `POST /api/x is not
|
|
93
|
+
registered`, `/api/x/: a collection is addressed by its bare path`, `GET /api/x answered 401`.
|
|
94
|
+
|
|
95
|
+
## Refusal does no work
|
|
96
|
+
|
|
97
|
+
`checkRefusalDoesNoWork({ app, anonymous, unprivileged, snapshot, routes, prefix, exclude, paramValue, refusedStatuses, settleMs, minRoutes })`.
|
|
98
|
+
Calls every mutating route (default: POST PUT PATCH DELETE of the route table, `:param` filled with `paramValue`; or
|
|
99
|
+
`routes: [{ method, path, body? }]`) as `anonymous` and as `unprivileged` (`InjectCaller`s). Each call must answer
|
|
100
|
+
401/403 and leave `snapshot()` (host state: row counts, a phase, an outbox length) deep-equal before and after. A gate
|
|
101
|
+
that only calls `reply.send` still answers 403 when 2+ async `preSerialization`/`onSend` hooks defer the write, yet the
|
|
102
|
+
handler runs: only the snapshot shows it. `exclude`: `METHOD /path` keys that accept the caller, with the reason.
|
|
103
|
+
Failures: `anonymous POST /x: answered 403 but the snapshot changed: the handler ran after the refusal`, `unprivileged
|
|
104
|
+
PATCH /x/1: answered 200, expected a refusal (401/403)`, `stale exclusion KEY`. Fix: `return reply.send(...)` or throw.
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
# Fixtures: boot, pool, workers, cost and plan asserts
|
|
2
|
+
|
|
3
|
+
## bootModuleApp (`@classytic/arc-testkit`)
|
|
4
|
+
|
|
5
|
+
`bootModuleApp(modules, options?)` boots arc (`preset: "testing"`, `auth: false`, `sensible: false`) around module(s) on
|
|
6
|
+
in-memory MongoDB and returns `{ app, uri, connection, exports(name), close() }` (`close` is idempotent).
|
|
7
|
+
|
|
8
|
+
`modules` is a module, an array, or a factory `({ connection, mongoUri }) => module | module[]`.
|
|
9
|
+
`options`: `replset` (single-node replica set, needed for transactions), `database` (a `TestDatabaseFactory`, e.g.
|
|
10
|
+
`pool.factory`), `app` (extra `createApp` options merged over the defaults), `actor` (default actor for requests without
|
|
11
|
+
an `x-test-actor` header), `ports` (host ports for the modules).
|
|
12
|
+
|
|
13
|
+
```ts
|
|
14
|
+
const t = await bootModuleApp(({ connection }) => [createThingModule({ connection, permissions })], { replset: true });
|
|
15
|
+
const engine = t.exports<ThingEngine>("thing"); // typed module export
|
|
16
|
+
await t.close();
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Types: `TestkitContext`, `ModulesInput`, `BootModuleAppOptions`, `ModuleTestApp`.
|
|
20
|
+
|
|
21
|
+
## Pool (`/pool`, `/mongo`)
|
|
22
|
+
|
|
23
|
+
One server per run, one isolated database per suite. `createTestDatabasePool(provider, options?)` returns
|
|
24
|
+
`{ factory, withPool, leaseNamespace, globalSetup, stats }`.
|
|
25
|
+
|
|
26
|
+
```ts
|
|
27
|
+
// tests/support/db-pool.ts
|
|
28
|
+
import { mongoServerProvider } from "@classytic/arc-testkit/mongo";
|
|
29
|
+
import { createTestDatabasePool } from "@classytic/arc-testkit/pool";
|
|
30
|
+
import mongoose from "mongoose";
|
|
31
|
+
|
|
32
|
+
export const dbPool = createTestDatabasePool(mongoServerProvider({ mongoose, awaitIndexes: false }), {
|
|
33
|
+
envPrefix: "APP_TEST",
|
|
34
|
+
preboot: ["transactional"],
|
|
35
|
+
});
|
|
36
|
+
// tests/support/global-setup.ts
|
|
37
|
+
export const setup = () => dbPool.globalSetup.setup();
|
|
38
|
+
export const teardown = () => dbPool.globalSetup.teardown();
|
|
39
|
+
// a suite
|
|
40
|
+
const boot = dbPool.withPool(bootModuleApp);
|
|
41
|
+
const lease = await dbPool.leaseNamespace("transactional"); // { uri, namespace, release() }, for a host that connects itself
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
| Option | Default | Meaning |
|
|
45
|
+
|---|---|---|
|
|
46
|
+
| `envPrefix` | `ARC_TESTKIT` | env prefix of the cross-worker handshake; use one per run kind |
|
|
47
|
+
| `preboot` | `["transactional"]` | kinds `globalSetup.setup()` starts and publishes; `[]` only for a single-process run |
|
|
48
|
+
| `shareTransactionalServer` | `true` | one replica set answers `standalone` and `transactional` |
|
|
49
|
+
| `poolSize` | `1` | servers per kind; size with `safePoolSize(n)` (bounded by free RAM) |
|
|
50
|
+
| `slotOf` | vitest worker id | which server a worker uses |
|
|
51
|
+
| `stopWhenIdle` | `false` | stop servers after the last release (watch mode) |
|
|
52
|
+
| `disabled` | `false` | per-suite private servers |
|
|
53
|
+
|
|
54
|
+
`leaseNamespace(kind, { exact, forcePrivate, createOptions })`: `forcePrivate` gives a suite its own server (use it for
|
|
55
|
+
fail points and `setParameter`); `exact` stops a `standalone` request resolving to the shared replica set.
|
|
56
|
+
`mongoServerProvider({ args, awaitIndexes = true, mongoose })` builds the Mongo provider; it waits for index builds after
|
|
57
|
+
the app boots (`awaitIndexBuilds(connection?)`), which is what makes unique-index assertions deterministic. Inject the
|
|
58
|
+
`mongoose` instance your models register on. `publishPoolServers(servers, envPrefix)` publishes servers a custom
|
|
59
|
+
`globalSetup` started.
|
|
60
|
+
|
|
61
|
+
## Workers and memory (`/mongo`)
|
|
62
|
+
|
|
63
|
+
- `safeWorkerCount(envVar = "ARC_TEST_WORKERS")`: explicit env value, else min(cores - 2, RAM / 6 GB), at least 2. Use in
|
|
64
|
+
`defineConfig({ test: { maxWorkers: safeWorkerCount("TEST_MAX_WORKERS") } })`.
|
|
65
|
+
- `safePoolSize(requested)`: servers affordable by free RAM, at least 1.
|
|
66
|
+
- `cacheSizeGB()` (env `TEST_MONGO_CACHE_GB`, floor 0.25), `mongodMemoryArgs()`, `withDbName(uri, dbName)`.
|
|
67
|
+
- One heavy command at a time on a shared machine; cap workers before running a whole suite.
|
|
68
|
+
|
|
69
|
+
## Wire-command counts (`/mongo-commands`)
|
|
70
|
+
|
|
71
|
+
The mongoose connection must be created with `monitorCommands: true`, else `countCommands` throws.
|
|
72
|
+
|
|
73
|
+
```ts
|
|
74
|
+
import { assertConstantCommands, countCommands } from "@classytic/arc-testkit/mongo-commands";
|
|
75
|
+
|
|
76
|
+
const { total, byCollection, byCommand, commands } = await countCommands(conn, () => placeOrder(), { collections: ["orders"], commands: ["find"], ignore: ["createIndexes"] });
|
|
77
|
+
await assertConstantCommands(conn, (n) => reader.page(n), [5, 50]); // same commands at every size, at least one: a per-item read fails
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
## Plan asserts (`/mongo-explain`)
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
import { assertIndexServed, summarizeExplain } from "@classytic/arc-testkit/mongo-explain";
|
|
84
|
+
|
|
85
|
+
const explain = await Model.find(filter).explain("executionStats");
|
|
86
|
+
assertIndexServed(explain, { index: "org_state", stages: ["OR"], maxDocsExamined: 40 });
|
|
87
|
+
const { stages, indexes, docsExamined } = summarizeExplain(explain);
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
`assertIndexServed` fails on COLLSCAN, on a plan scanning no index (empty or missing collection), on an index in `index`
|
|
91
|
+
that is not scanned, a stage in `stages` that is absent, and on `maxDocsExamined` exceeded (needs `executionStats`).
|
|
92
|
+
Rejected plans and query predicates are never read.
|
|
@@ -0,0 +1,3 @@
|
|
|
1
|
+
# List filters against the route
|
|
2
|
+
|
|
3
|
+
`checkFilterDeclarations({ spec, declarations })`; `declarations`: `[{ route, filters: [{ queryField, op? }], search?, sorts? }]` (op default `eq`, `between` = `gte` + `lte`). A filter must be a `field[op]` query parameter of the route's GET, or in arc's `x-arc-query` (`filterFields`, `operators`, `sortFields`); `search` needs the `search` parameter. Failure: `/api/x: filter status[contains] is not declared by the route`.
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# Story kit (`@classytic/arc-testkit/story`)
|
|
2
|
+
|
|
3
|
+
One continuous scenario through a booted app. Steps are numbered vitest tests in one file (vitest runs them in order);
|
|
4
|
+
state passes through the host's own object. The kit knows no domain: roles, routes and clause ids are yours.
|
|
5
|
+
|
|
6
|
+
## Steps
|
|
7
|
+
|
|
8
|
+
`describeStory(title, define, options?)`; `define` receives `s` with `s.step(title, [{ clauses, timeout }], body)`.
|
|
9
|
+
`body(t, ctx)`. Test names are `N. title`.
|
|
10
|
+
|
|
11
|
+
| Option | Default | Meaning |
|
|
12
|
+
|---|---|---|
|
|
13
|
+
| `firstStep` | `1` | number of the first step |
|
|
14
|
+
| `bail` | `true` | after a hard failure the remaining steps are skipped with "step N failed" |
|
|
15
|
+
| `clauses` | none | clause tags added to every step |
|
|
16
|
+
| `test`, `fixtures` | vitest `it` | an extended `test` and the fixture names steps use; they arrive as `ctx` |
|
|
17
|
+
|
|
18
|
+
`t.expect(actual, what)` is `expect` with the message `[CLAUSE-1 | CLAUSE-2] what`. `t.soft(...)` is `expect.soft`: the
|
|
19
|
+
failure is reported, the step continues. `t.known("DEFECT-7", actual, what)` is a soft assertion tagged
|
|
20
|
+
`KNOWN DEFECT DEFECT-7`, for an open finding that must stay visible without hiding later asserts. `t.msg(what)` builds the
|
|
21
|
+
tagged text for other matchers. `clauseMessage(clauses, what)` is the pure form. `need(bag, key)` reads a value an
|
|
22
|
+
earlier step produced and throws "the story needs KEY from an earlier step" instead of a bare `undefined`.
|
|
23
|
+
|
|
24
|
+
Reading a failure: the first line names the clauses and what was expected; the step number is in the test name; steps
|
|
25
|
+
after the first hard failure show as skipped (look at the first red step, not the skipped ones).
|
|
26
|
+
|
|
27
|
+
```ts
|
|
28
|
+
describeStory("onboarding", (s) => {
|
|
29
|
+
s.step("a hire is captured", { clauses: ["HR-1"] }, async (t) => {
|
|
30
|
+
const res = await cast.get("hr").post("/api/people", payload);
|
|
31
|
+
t.expect(res.status, "the person is created").toBe(201);
|
|
32
|
+
t.known("D-3", res.body.tasks.length, "four onboarding tasks").toBe(4);
|
|
33
|
+
});
|
|
34
|
+
}, { firstStep: 0 });
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Cast
|
|
38
|
+
|
|
39
|
+
`createCast((roles, opts, name) => provisionedActor)`; the provisioner signs the user up on the app and returns what the
|
|
40
|
+
story talks through (a client, a token). `await cast.build({ ann: ["admin"], bob: { roles: ["viewer"], opts: { mfa: false } } })`
|
|
41
|
+
returns the actors by name; `cast.get(name)`, `cast.add(name, roles, opts?)`, `cast.names()`. `bearer(token)` returns
|
|
42
|
+
`{ authorization: "Bearer ..." }`. Pair with `injectCaller` from `/conformance` for `get`/`post`.
|
|
43
|
+
|
|
44
|
+
## Clock and civil dates
|
|
45
|
+
|
|
46
|
+
`createStoryClock({ start, zone })` returns `{ now(), zone, set, advanceMs, advanceDays, travelTo(day, hour = 12), today(),
|
|
47
|
+
onChange(listener) }`; `now()` returns a fresh `Date`, so it fits any `{ now(): Date }` clock port. `travelTo` jumps to
|
|
48
|
+
`hour:00` of a civil date in the zone (DST-safe). Pure helpers: `civilDateOf(at, zone)`, `addCivilDays(day, n)`,
|
|
49
|
+
`addCivilMonths(day, n)`, `daysBetween(a, b)`, `instantOn(day, zone, hour)`. Dates are `YYYY-MM-DD` in the operator zone.
|
|
50
|
+
|
|
51
|
+
## Evidence ports
|
|
52
|
+
|
|
53
|
+
`createStoryAssertions(ports)` where every port is optional and a missing one throws "no `mail` port was supplied" at the
|
|
54
|
+
call: `drain` (outbox pass), `events` (rows), `feed(actor)` (items with `title`, `message`), `mail` (rows), `audit(filter)`
|
|
55
|
+
(rows), `eventTypeKey` (default `type`).
|
|
56
|
+
|
|
57
|
+
| Method | Behaviour |
|
|
58
|
+
|---|---|
|
|
59
|
+
| `drain(rounds = 3)` | drain the outbox repeatedly (a handler may emit more) |
|
|
60
|
+
| `relayed(assertion, { timeout = 30000, interval = 50 })` | drain, then run the assertion, retrying until it holds |
|
|
61
|
+
| `expectEvent(match, { count, min, since, clauses })` | events matching; default at least one |
|
|
62
|
+
| `expectNoEvent(match, { since, clauses })` | none match |
|
|
63
|
+
| `expectFeed(actor, match, spec)`, `expectNoFeed(actor, match, spec)` | feed items whose title or message match |
|
|
64
|
+
| `expectMail(match, spec)` | mail rows; string matches `subject` |
|
|
65
|
+
| `expectAudit(filter, match, spec)` | audit rows; string matches `action` |
|
|
66
|
+
| `eventsOf(match, since)`, `feedOf(actor)` | the matching rows, for custom checks |
|
|
67
|
+
|
|
68
|
+
`match` is a string or RegExp (tested on the type, title, subject or action field), a predicate, or an object matched as a
|
|
69
|
+
deep subset (RegExp and function leaves are tested, arrays match when each expected element is found).
|
|
70
|
+
`createEventLog()` returns `{ record, all, mark, clear, tap(subscribe, patterns) }`; `tap` subscribes your bus
|
|
71
|
+
(`(pattern, handler) => ...`) and records every event; `mark()` returns a count to pass as `since` for "what happened
|
|
72
|
+
next". Feed `events: () => log.all()` into the ports.
|
|
73
|
+
|
|
74
|
+
## Idempotency
|
|
75
|
+
|
|
76
|
+
`expectIdempotent({ run, snapshot, settle?, clauses?, label? })` runs the sweep, settles, snapshots, runs again, settles,
|
|
77
|
+
snapshots, and fails unless the two snapshots are equal (`LABEL run twice on one day changes nothing the second time`).
|
|
78
|
+
Returns `{ first, second, afterFirst, afterSecond }`. `expectSweepsIdempotent({ name: run, ... }, { snapshot, settle, clauses })`
|
|
79
|
+
does it for several sweeps. Snapshot what must not grow: row counts, ids, mail rows, feed items. Do not advance the
|
|
80
|
+
clock between the runs.
|