@mindbricks/genesis-set 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +34 -0
- package/LICENSE +202 -0
- package/README.md +147 -2
- package/bin/genesis-set.mjs +153 -0
- package/corpus/README.md +25 -0
- package/corpus/genesis/compile-results.md +246 -0
- package/corpus/genesis/compiler-api.md +265 -0
- package/corpus/genesis/custom-code.md +519 -0
- package/corpus/genesis/deploy.md +190 -0
- package/corpus/genesis/pitfalls.md +231 -0
- package/corpus/genesis/procedures/README.md +17 -0
- package/corpus/genesis/procedures/agent-hub-blueprint.md +13 -0
- package/corpus/genesis/procedures/auth-blueprint.md +14 -0
- package/corpus/genesis/procedures/bff-blueprint.md +16 -0
- package/corpus/genesis/procedures/blueprint-authoring.md +16 -0
- package/corpus/genesis/procedures/howto-mail-to-tasks.md +26 -0
- package/corpus/genesis/procedures/howto-payment-flow.md +112 -0
- package/corpus/genesis/procedures/howto-subscription.md +33 -0
- package/corpus/genesis/procedures/notifications-blueprint.md +13 -0
- package/corpus/genesis/procedures/patterns-catalog.md +12 -0
- package/corpus/genesis/procedures/project-lifecycle.md +27 -0
- package/corpus/genesis/run-locally.md +310 -0
- package/corpus/genesis/testing.md +532 -0
- package/lib/config-cli.mjs +179 -0
- package/lib/install.mjs +639 -0
- package/lib/mcp-config.mjs +220 -0
- package/lib/semver.mjs +94 -0
- package/mcp/blueprint.mjs +64 -0
- package/mcp/client.mjs +141 -0
- package/mcp/config.mjs +166 -0
- package/mcp/key-page-html.mjs +275 -0
- package/mcp/key-page.mjs +217 -0
- package/mcp/key-store.mjs +204 -0
- package/mcp/key-tools.mjs +225 -0
- package/mcp/server.mjs +164 -0
- package/mcp/tools.mjs +270 -0
- package/package.json +43 -4
- package/skills/genesis-compiler-api/SKILL.md +56 -0
- package/skills/genesis-deploy/SKILL.md +36 -0
- package/skills/genesis-generated-code/SKILL.md +48 -0
- package/skills/genesis-run-and-test/SKILL.md +43 -0
|
@@ -0,0 +1,532 @@
|
|
|
1
|
+
# Testing the generated services and seeding sample data
|
|
2
|
+
|
|
3
|
+
How to prove that the services Genesis generated do what the blueprint says, and how to fill
|
|
4
|
+
a running stack with a small, believable dataset. Genesis runs nothing for you: the suite and
|
|
5
|
+
the seed are code in your own repo, run against your own stack (see [run-locally.md](run-locally.md)),
|
|
6
|
+
and they read your own logs and database when a case goes red.
|
|
7
|
+
|
|
8
|
+
For the plan and the report documents that mind-set writes, see
|
|
9
|
+
`procedures/design/writers/test-plan.md` and `procedures/design/writers/test-report.md`
|
|
10
|
+
(schema: `procedures/design/schemas/test-report.v1.json`). This page does not repeat them; it
|
|
11
|
+
covers the suite itself.
|
|
12
|
+
|
|
13
|
+
## 1. What the suite is
|
|
14
|
+
|
|
15
|
+
- Plain Node (>= 20) ES modules under `tests/` in a repo you own (for example
|
|
16
|
+
`<codename>-tests`). No test framework, no dependencies.
|
|
17
|
+
- It drives the public HTTP surface the way a client does: real registration and login, real
|
|
18
|
+
fixtures built through the APIs, one observable assertion per designer API.
|
|
19
|
+
- It is not a unit test of generated code, and it never writes rows straight into the database.
|
|
20
|
+
A row the API could not have created proves nothing and pollutes later cases.
|
|
21
|
+
- It proves: every designer API has a case that passed, or a skip with a stated reason, and the
|
|
22
|
+
number of APIs with neither is zero. Completeness comes from a script, never from your own claim.
|
|
23
|
+
|
|
24
|
+
Layout:
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
tests/
|
|
28
|
+
helpers/config.mjs base URLs per service, User-Agent, random suffix, credential resolution
|
|
29
|
+
helpers/http.mjs JSON in/out, bearer token, brief() for evidence
|
|
30
|
+
helpers/asserts.mjs status checks, envelope unwrap, list/get envelope checks
|
|
31
|
+
helpers/runner.mjs Suite(service).case(api, title, fn) / .skip(api, reason)
|
|
32
|
+
helpers/session.mjs identity bootstrap (section 5), SkipError
|
|
33
|
+
helpers/fixtures.mjs cross-service fixtures, built once per process (section 6)
|
|
34
|
+
<service>.flow.mjs ONE per service, runnable alone
|
|
35
|
+
run-all.mjs every flow in one process, in dependency order -> report.json
|
|
36
|
+
coverage-gate.mjs API list vs report.json; exit 0 only at 0 MISSING
|
|
37
|
+
inventory/<service>.json the committed route lists the gate reads
|
|
38
|
+
seed/seed-sample-data.mjs the sample-data seed (section 10)
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Commit `inventory/` so a revision's gate can be reproduced. Git-ignore `report.json`.
|
|
42
|
+
|
|
43
|
+
## 2. Conventions that make the gate computable
|
|
44
|
+
|
|
45
|
+
- **Case key = `(service, apiName)`**, with `apiName` copied verbatim from the blueprint (and the
|
|
46
|
+
service spec's Business APIs section). One case per API is the minimum; extra cases are fine
|
|
47
|
+
and count as "extra".
|
|
48
|
+
- **The population** is every designer-authored API of every service, including APIs authored on
|
|
49
|
+
`auth`. A framework built-in of a system service (auth's `login`, `register`, `currentUser`,
|
|
50
|
+
`mintSessionForUser`, the api-key and machine-session routes, the notification, agent-hub and
|
|
51
|
+
bff surfaces) counts only when the test plan lists that flow; a listed built-in the product
|
|
52
|
+
does not use gets a `BUILT-IN-UNUSED` skip (section 8). An unlisted built-in is out of scope,
|
|
53
|
+
never MISSING.
|
|
54
|
+
- **Copy routes, never compose them.** The method and path of a case come from the route list the
|
|
55
|
+
generated service ships, not from the API name or a guess:
|
|
56
|
+
- `_api.json` at the root of the service repo: one JSON file listing every HTTP, socket and SSE
|
|
57
|
+
endpoint (`operationId`, `method`, `path`, auth, path/query parameters, request and response
|
|
58
|
+
shapes, the tenant header where the API reads the session tenant). Read it without booting the
|
|
59
|
+
service.
|
|
60
|
+
- `GET /openapi.json` (and `/swagger` for a browser) on the running service: the same surface
|
|
61
|
+
as OpenAPI 3.
|
|
62
|
+
Framework auth routes have their own shapes (`/login`, `/register`, ...); custom business APIs
|
|
63
|
+
mount under `/v1/`. `serviceBase(service)` in `config.mjs` is the only place that knows how a
|
|
64
|
+
service is addressed on your stack (local ports, a gateway, or the container URLs your deployer
|
|
65
|
+
set).
|
|
66
|
+
- **Send a `User-Agent` header on every request.** Many front doors refuse the default
|
|
67
|
+
`curl` / `python` / `node` agents (Cloudflare error 1010); without one the suite fails before
|
|
68
|
+
the first assertion and looks like an outage.
|
|
69
|
+
- **Anchor dates to the real today** (`procedures/pitfalls.md`). Put a random per-process suffix in
|
|
70
|
+
every id, SKU, slug and email so re-runs never hit a unique index.
|
|
71
|
+
- **Accept the 200 family** where the API answers 201 (register, login, most creates):
|
|
72
|
+
`expectStatus(res, [200, 201])`. The session and list envelopes are in `procedures/pitfalls.md`.
|
|
73
|
+
- **No secret in a tracked file.** `brief()` truncates bodies and never prints headers. No token,
|
|
74
|
+
password or provider key is logged, reported or committed. Fake tokens in negative auth cases
|
|
75
|
+
are fine.
|
|
76
|
+
|
|
77
|
+
## 3. Scaffold, then fill
|
|
78
|
+
|
|
79
|
+
Never author the whole suite in one output. The order follows what each piece reads:
|
|
80
|
+
|
|
81
|
+
1. **Route list.** For every service (system services too), copy the route rows from `_api.json`
|
|
82
|
+
into `inventory/<service>.json` next to the blueprint's API names. The dump is stale after any
|
|
83
|
+
rebuild of that service: refresh it before you gate again.
|
|
84
|
+
2. **Helpers once, then `auth.flow.mjs`.** Every other flow gets its sessions from it. Run it
|
|
85
|
+
alone (`node tests/auth.flow.mjs`) until the identity bootstrap of section 5 is green.
|
|
86
|
+
3. **One flow per service in dependency order** (catalog before order before review; bff and
|
|
87
|
+
notification last). Per API: read the route row, write the case (fixtures, one call, one
|
|
88
|
+
observable assertion), run that flow, fix, next API. Kafka-only, cron-only, internal-only and
|
|
89
|
+
machine-only rows are driven through the test transport of section 7, not skipped.
|
|
90
|
+
4. **`node tests/run-all.mjs`**, then **`node tests/coverage-gate.mjs`**. The gate's MISSING list
|
|
91
|
+
is the fill queue; the loop ends at `GATE: PASS`.
|
|
92
|
+
|
|
93
|
+
Every case has one shape: get a session (`ctx.customer()`, `ctx.staff()`, `ctx.superAdmin()`),
|
|
94
|
+
get the fixture it needs, make ONE call, assert the status and the one thing the API promises
|
|
95
|
+
(an `id` in the list rows, a transition's resulting state, a 403 for the wrong role on a
|
|
96
|
+
role-gated API).
|
|
97
|
+
|
|
98
|
+
### A 2xx is not the whole verdict on a write
|
|
99
|
+
|
|
100
|
+
A create, update or delete whose row committed but whose post-write actions did not all
|
|
101
|
+
complete answers `200` / `201` WITH an `actionFailures: [{ action, error, retryable?, errorCode?, outbox? }]`
|
|
102
|
+
member. So:
|
|
103
|
+
|
|
104
|
+
- Every write case asserts `body.actionFailures === undefined` next to the status; do it once in
|
|
105
|
+
the helper that reads a single-item body.
|
|
106
|
+
- When it is present, **the row exists. Never re-run a create to "retry" it**: that makes a second
|
|
107
|
+
row (one pilot stranded 17 that way). The case fails, naming each `action` and `error`; triage
|
|
108
|
+
reads the named action, not the status.
|
|
109
|
+
- `outbox: "queued"` (a dropped event the service republishes itself) is the one entry a case may
|
|
110
|
+
tolerate, and only when it asserts nothing downstream of that event.
|
|
111
|
+
- `errorCode: "ACTION_NOT_RUN"` entries are consequences: fix the first entry and they go away.
|
|
112
|
+
|
|
113
|
+
### An approval-gated API answers `403 APPROVAL_REQUIRED` first
|
|
114
|
+
|
|
115
|
+
An API whose blueprint sets `authOptions.requireApproval` answers its first call with `403` and
|
|
116
|
+
`errorCode: "APPROVAL_REQUIRED"`, sends a one-time code to the caller's verified email or mobile,
|
|
117
|
+
and runs only when the SAME request (same parameters, same user) is repeated with the headers
|
|
118
|
+
`api-approval-id` and `api-approval-code`. That is a step, never a red.
|
|
119
|
+
|
|
120
|
+
The challenge is in the problem body's `errors` member: `{ approvalId, channel, codeIndex,
|
|
121
|
+
expireTime, expiresAt, actionName, actionDescription }`, plus `secretCode` only where the service
|
|
122
|
+
runs in a dev-like environment (the dev slot). On stage and prod the code is never echoed.
|
|
123
|
+
|
|
124
|
+
Keep one helper, `approvedCall(method, url, body, token)`:
|
|
125
|
+
|
|
126
|
+
1. Make the call. Anything other than `403` + `APPROVAL_REQUIRED` is returned as is.
|
|
127
|
+
2. On the challenge, read `errors.approvalId` and `errors.secretCode`. With no `secretCode`, assert
|
|
128
|
+
the challenge shape and throw `SkipError('ENVIRONMENT: approval code not echoed on <env> - the challenge is the evidence')`.
|
|
129
|
+
Never guess a code.
|
|
130
|
+
3. Repeat the identical call with the two headers and return that response, plus the challenge
|
|
131
|
+
facts for the case to assert on.
|
|
132
|
+
|
|
133
|
+
A case on such an API asserts both legs: the first answer was the challenge, and the retry is the
|
|
134
|
+
API's normal success. Never widen a status set to `[200, 403]`; never retry with changed
|
|
135
|
+
parameters (that is `403 APPROVAL_INVALID` by design, worth one negative case). The other answers:
|
|
136
|
+
|
|
137
|
+
| Answer | Meaning |
|
|
138
|
+
|---|---|
|
|
139
|
+
| `403 APPROVAL_INVALID` | wrong, used or expired code, or changed input; the approval is gone after 5 attempts |
|
|
140
|
+
| `429 APPROVAL_RESEND_WINDOW` | a code for a different input went out under a minute ago: wait, or finish the first approval |
|
|
141
|
+
| `403 APPROVAL_NOT_AVAILABLE` | the actor has no verified email or mobile: an identity finding, like `BLOCKED-IDENTITY` |
|
|
142
|
+
|
|
143
|
+
Send an idempotency key unchanged on both legs.
|
|
144
|
+
|
|
145
|
+
## 4. Running and reporting
|
|
146
|
+
|
|
147
|
+
- `node tests/<service>.flow.mjs` runs one flow: the authoring unit.
|
|
148
|
+
- `node tests/run-all.mjs` runs all flows in ONE process, so sessions and fixtures are shared,
|
|
149
|
+
and writes `tests/report.json`. Exit code 1 on any fail.
|
|
150
|
+
- `node tests/coverage-gate.mjs` takes the best verdict per `(service, api)` (pass > fail > skip),
|
|
151
|
+
prints per service covered / failing / skipped (with reasons) / MISSING / extra, and exits 0 only
|
|
152
|
+
at 0 MISSING. A gate PASS with fails is still a red suite: the gate measures coverage, the run
|
|
153
|
+
measures health. Report both numbers.
|
|
154
|
+
- Run one suite at a time. Two concurrent runs race each other's fixtures (unique slugs, cart
|
|
155
|
+
state, order transitions). Do not run the suite while a rebuild or a redeploy is in flight, and
|
|
156
|
+
make sure the stack runs the build you think it runs (check `/health` after a restart): a run
|
|
157
|
+
against the previous build "proves" a fix that is not live.
|
|
158
|
+
|
|
159
|
+
If you want the run recorded as a test report, make `report.json` match the schema of
|
|
160
|
+
`procedures/design/schemas/test-report.v1.json`. The shape the schema expects:
|
|
161
|
+
|
|
162
|
+
```json
|
|
163
|
+
{ "project": "<codename>", "env": "dev", "base": "<base url>",
|
|
164
|
+
"ranAt": "<iso>", "durationMs": 86498,
|
|
165
|
+
"summary": { "pass": 84, "fail": 0, "skip": 40 },
|
|
166
|
+
"services": { "auth": { "pass": 44, "fail": 0, "skip": 4 } },
|
|
167
|
+
"results": [ { "service": "auth", "api": "login", "title": "fresh user can log in",
|
|
168
|
+
"status": "pass", "ms": 1287 },
|
|
169
|
+
{ "service": "order", "api": "shipOrder", "title": "...",
|
|
170
|
+
"status": "skip", "reason": "KNOWN-DEFECT <ref>: ..." } ] }
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
`summary` and every `services.<service>` must equal the counts of `results[]`; `api` is the API
|
|
174
|
+
name verbatim; `reason` carries the class prefix of section 8. Keep headers, bodies and tokens out
|
|
175
|
+
of it.
|
|
176
|
+
|
|
177
|
+
## 5. Identity bootstrap (`session.mjs`)
|
|
178
|
+
|
|
179
|
+
Three actors, all through the PUBLIC auth surface:
|
|
180
|
+
|
|
181
|
+
- **customer.** `POST /register` (the user shape from the auth spec: custom user properties are
|
|
182
|
+
required when the blueprint says so). On `emailVerificationNeeded`, run the two-call verify chain
|
|
183
|
+
(`POST /api/auth/verify/email-verification/start`, then `complete` with `secretCode` as a
|
|
184
|
+
STRING; `procedures/pitfalls.md`). On the dev slot the start call echoes `secretCode`; on stage
|
|
185
|
+
and prod it never does (a real mailbox, or a sink you read yourself). Then `POST /login`.
|
|
186
|
+
Register and login answer 201 with the flat session envelope: `accessToken` at the top level,
|
|
187
|
+
`userId` beside it. Read the token from the body first, `set-cookie` only as a fallback.
|
|
188
|
+
- **staff** (any custom role). A customer, then the super-admin grants the role with
|
|
189
|
+
`PATCH /userrole/:userId { roleId: "<role>" }`, then a FRESH login: a session minted before the
|
|
190
|
+
grant does not carry it. Under `usersHaveMultipleRoles` the stored `roleId` is an array: send and
|
|
191
|
+
assert the array form. A role-gated API answers 403 to the wrong actor; that 403 is a valid
|
|
192
|
+
assertion, not a reason to add the role to `absoluteRoles`.
|
|
193
|
+
- **super-admin.** The blueprint's bootstrap credential
|
|
194
|
+
(`authentication` -> `loginDefinition.userSettings.superAdminIdentifier` / `superAdminPassword`),
|
|
195
|
+
inserted once when absent. In a Genesis project that value should be an `$ENV:` reference
|
|
196
|
+
(`compiler-api.md`). The suite resolves it at runtime from environment variables you name (the
|
|
197
|
+
same variables the blueprint references) and nowhere else: not a literal in `config.mjs`, not
|
|
198
|
+
a line in a report, not a committed file. Never print it, `cat` it or paste it into a prompt.
|
|
199
|
+
|
|
200
|
+
Sessions are memoised per process, and a bootstrap failure degrades instead of aborting:
|
|
201
|
+
`superAdmin()` that cannot log in throws `SkipError('BLOCKED-IDENTITY: ...')`, `staff()` inherits
|
|
202
|
+
it, and every case that needs either actor records a reasoned skip. A blocked identity is a
|
|
203
|
+
FINDING to report with the exact 403 body, never a reason to reset a password in the database.
|
|
204
|
+
|
|
205
|
+
`registerUser(tag, extra)` is the one-call composite (register, verify if needed, login)
|
|
206
|
+
every fixture uses for a throwaway actor. Give it a `tag` so the data is traceable to the case.
|
|
207
|
+
|
|
208
|
+
## 6. Fixtures across services
|
|
209
|
+
|
|
210
|
+
Fixtures are built THROUGH the APIs, once per process, as a chain that mirrors the product's own
|
|
211
|
+
flow, for example: staff creates a category and a published product with stock, then the customer
|
|
212
|
+
checks out and pays, then staff ships and delivers, then the customer writes the review.
|
|
213
|
+
|
|
214
|
+
- **`once()` memoises the rejection too, classified.** A memo helper that caches only success
|
|
215
|
+
re-runs a permanently blocked fixture for every case that asks for it (one pilot lane burned
|
|
216
|
+
16 cases x 2 throwaway users x ~15 s on one blocked checkout, another left 17 stranded
|
|
217
|
+
rows). Cache the ERROR OBJECT of a permanent failure, so the second caller sees the same
|
|
218
|
+
evidence; never cache a transient one.
|
|
219
|
+
|
|
220
|
+
| Class | Tell | Memo |
|
|
221
|
+
|---|---|---|
|
|
222
|
+
| permanent | a `SkipError`; a 4xx about the request or the actor (401, 403, 404 on an advertised route, a validation 422); a precondition the suite cannot create | cache the rejection |
|
|
223
|
+
| transient | a 5xx, a network error, a 429, a not-ready state the fixture polls for | do not memoise; the next caller retries |
|
|
224
|
+
|
|
225
|
+
- **A fixture failure is a real failure.** It throws with `brief(res)` and the cases that need it
|
|
226
|
+
fail; only a classified skip (section 8) turns it into a skip.
|
|
227
|
+
- **Keep parents open until the children exist** (`procedures/pitfalls.md`). Destructive cases
|
|
228
|
+
(cancel, refund, reject) get a fresh fixture, never the shared one. A state-shaped 422 means
|
|
229
|
+
"pick another open parent", not "reopen".
|
|
230
|
+
- **Fixtures carry their evidence** (for a paid order, each payment call's status and body), so a
|
|
231
|
+
red downstream can be read without re-running.
|
|
232
|
+
- **Restate the precondition before reusing a fixture** in another service: the same-shape failure
|
|
233
|
+
in review (no delivered fact for THIS user and product) has a different root than in order.
|
|
234
|
+
|
|
235
|
+
### Mock payment discipline
|
|
236
|
+
|
|
237
|
+
The suite never sends card data, not a real card, not a "test card number": the mock adapter needs
|
|
238
|
+
none, and a payment-method API that stores or echoes such fields is a security finding.
|
|
239
|
+
The payment ladders are in `genesis/procedures/howto-payment-flow.md` and
|
|
240
|
+
`genesis/procedures/howto-subscription.md`; the test
|
|
241
|
+
discipline on top of them:
|
|
242
|
+
|
|
243
|
+
1. **Probe first, once per run:** `GET /<order-service>-api/v1/payments/mock-status` (or the same
|
|
244
|
+
path on your service address). For your provider's row, `mock: true` with `transport: "present"`
|
|
245
|
+
is the expected answer when the provider keys are blank. **`mock: false` means the service is
|
|
246
|
+
bound to live credentials: stop and do not run the paid chain.** `transport: "absent"` (the row
|
|
247
|
+
is also in the top-level `blocking[]`) is the one case where the paid chain is unrunnable.
|
|
248
|
+
2. **Path A** (the customer's product path): `PATCH /v1/startorderpayment/:orderId`, then
|
|
249
|
+
`POST /v1/ordermockauthenticate/:orderId` (no body). Copy both routes from your own route list;
|
|
250
|
+
the names here belong to one pilot's blueprint.
|
|
251
|
+
3. **Path B** (operator fallback): `POST /v1/payments/admin/simulate-capture/order/:orderId` as
|
|
252
|
+
super-admin. A staff session answers `403 OPERATOR_REQUIRED`; do not grant the capability to
|
|
253
|
+
the test role to make it pass. It needs the payment ticket Path A creates (else 404 "no payment
|
|
254
|
+
ticket").
|
|
255
|
+
4. **Poll the order** (a handful of 1.5 s reads) until its status leaves the pending family, and
|
|
256
|
+
assert on the ORDER's state, not on the payment call's 200.
|
|
257
|
+
5. **Read `transport`, not the brand.** `mock: true` says the credentials are blank; `transport`
|
|
258
|
+
says whether the mock path answers. Only a brand whose row says `absent` makes the fixture
|
|
259
|
+
throw a `SkipError` at the payment step, so every dependent case records the same reason.
|
|
260
|
+
|
|
261
|
+
## 7. The test transport: driving what HTTP cannot reach
|
|
262
|
+
|
|
263
|
+
Four API kinds have no public caller: **kafka-only** (runs when a message lands on a topic),
|
|
264
|
+
**cron-only** (runs on a schedule; `0 4 1 * *` is a month), **internal-only**
|
|
265
|
+
(`controllers: ["internal"]`, reachable only from inside the service) and **machine-only**
|
|
266
|
+
(`requiredCallerKind: "machine"` answers `403 M2M_CALLER_REQUIRED` to every session, super-admin
|
|
267
|
+
included). Every generated nodejs service mounts a drive surface for them under its own prefix,
|
|
268
|
+
for example `/<service>-api/admin/test-transport`.
|
|
269
|
+
|
|
270
|
+
**When it is open.** Only on an explicit non-production slot: `CONFIG_ENV` and `SERVICE_CONFIG`
|
|
271
|
+
both set, both non-production and agreeing (every generated `Dockerfile.<slot>` sets both), and
|
|
272
|
+
`MB_TEST_TRANSPORT` unset. Anything else answers `TEST_TRANSPORT_DISABLED`: production, or a plain
|
|
273
|
+
`npm start` with no `SERVICE_CONFIG` (the loader then falls back to `.test.env`), so set it yourself
|
|
274
|
+
for a local run. A non-empty `MB_TEST_TRANSPORT` switches the surface off outright.
|
|
275
|
+
That means a stage deployment is open too: set `MB_TEST_TRANSPORT=off` there if the surface must
|
|
276
|
+
not be reachable. Every caller that is not super-admin gets `TEST_TRANSPORT_FORBIDDEN`, so the
|
|
277
|
+
suite calls it with `ctx.superAdmin()`.
|
|
278
|
+
|
|
279
|
+
| Route | What it does |
|
|
280
|
+
|---|---|
|
|
281
|
+
| `GET /admin/test-transport` | census: declared and subscribed kafka topics, every cron job (with `registrationFailed`), the internal-call API names. Read it instead of guessing a name |
|
|
282
|
+
| `POST /admin/test-transport/kafka/publish` `{ topic, data, type?, old?, tenantCodename?, tenantId? }` | publishes through the service's own publisher (topic scope, envelope, scrub), so the consumer sees what production hands it. Call it on the service that OWNS the consumer: a topic outside its census is `400 TEST_TRANSPORT_TOPIC_NOT_CONSUMED`; a dropped publish is `503 TEST_TRANSPORT_PUBLISH_DROPPED`. `subscribed: false` means delivered but nobody listens. On a multi-tenant project the event goes out in the root tenant unless you name the tenant; unknown codename is `404 TEST_TRANSPORT_TENANT_NOT_FOUND` |
|
|
283
|
+
| `POST /admin/test-transport/crons/:name/run-now` `{ inputData? }` | runs the job once and waits. `ran: true` is the transport's verdict; assert on `outcome.httpStatus` / `outcome.output`, because a sweep that refused (400) also "ran". A handler that threw reports `outcome.httpStatus: 500` |
|
|
284
|
+
| `POST /admin/test-transport/internal/:apiName` `{ params? }` | invokes the internal-call API through the same registry a designer script uses. `invoked: true` is the transport's verdict; the API's own is `outcome.failed` / `outcome.httpStatus` / `outcome.errorCode` |
|
|
285
|
+
| `POST /admin/test-transport/machine-token` `{ targetService?, ttlSeconds? }` | mints a service-signed token (default 120 s, ceiling 300 s; a longer one is refused with `TEST_TRANSPORT_TTL_TOO_LONG`, never clamped). Send it as `x-m2m-token` to the machine-only API. It verifies only on the slot that minted it, so mint and call on the same deployment. A target in a language whose verifier does not check that binding answers `422 TEST_TRANSPORT_TARGET_UNVERIFIED` |
|
|
286
|
+
|
|
287
|
+
Other refusals: `TEST_TRANSPORT_LOGIN_REQUIRED` (no session), `TEST_TRANSPORT_CRON_UNKNOWN`,
|
|
288
|
+
`TEST_TRANSPORT_NO_SCHEDULER`, `TEST_TRANSPORT_API_UNKNOWN`. The minting service is the issuer, so
|
|
289
|
+
minting on a service the callee's `allowedM2MClients` does not list is how `M2M_CLIENT_NOT_ALLOWED`
|
|
290
|
+
is produced.
|
|
291
|
+
|
|
292
|
+
A kafka- or cron-driven case asserts on the EFFECT (the row the consumer wrote, the state the sweep
|
|
293
|
+
moved), polled a handful of times like the order in section 6. The publish or run-now answer only
|
|
294
|
+
says the drive happened.
|
|
295
|
+
|
|
296
|
+
**Auth's `mintSessionForUser`.** It is a trusted proxy-login route: it answers `403
|
|
297
|
+
errMsg_UntrustedProxyLoginCaller` to every user session, super-admin included, because it wants a
|
|
298
|
+
verified `x-m2m-token`. That is not a reason to skip it: mint a token on auth itself
|
|
299
|
+
(`POST <auth base>/admin/test-transport/machine-token` as super-admin, body `{}`), then call
|
|
300
|
+
`POST <auth base>/mintsessionforuser` with `x-m2m-token` and `{ userId, reason }`. Cases to write:
|
|
301
|
+
the `200` whose `body.userId` is the requested user, plus a `/currentuser` read on the returned
|
|
302
|
+
`accessToken` naming the same user; a `404` for an unknown `userId`; the `403` for a plain user
|
|
303
|
+
session (that negative is the contract); a `401` with no credential. If the deployment sets
|
|
304
|
+
`PROXY_SESSION_TRUSTED_SERVICES` to a list without `auth`, or auth is built in a language the mint
|
|
305
|
+
refuses, throw `SkipError('ENVIRONMENT: ...')` from the live response.
|
|
306
|
+
|
|
307
|
+
**Steering the decision mock.** A decision provider with no key (or the `mock` brand) answers from
|
|
308
|
+
a hash; `GET /<service>-api/v1/decisions/mock-status` says which providers are on it. To drive every
|
|
309
|
+
branch of an `objectModeration` flow (one `choice` question named `verdict` over `accept`, `reject`,
|
|
310
|
+
`review`), put a directive in any text field of the content the decision reads:
|
|
311
|
+
|
|
312
|
+
[[mock-decision verdict=reject]]
|
|
313
|
+
[[mock-decision verdict=accept@0.6]] (low confidence, below the threshold: routes to review)
|
|
314
|
+
[[mock-decision *=no]] (every question; other decision kinds)
|
|
315
|
+
|
|
316
|
+
Question names and legal values come from the blueprint's decision questions. A value the question
|
|
317
|
+
cannot take is refused with `MOCK_DECISION_DIRECTIVE_INVALID`, never silently replaced. Directives
|
|
318
|
+
are inert on production and on the real providers.
|
|
319
|
+
|
|
320
|
+
**A failed fiscal document.** The fiscal `mock` provider takes `configuration.behavior` in the
|
|
321
|
+
blueprint (a suite cannot pick it per document): `accept` (default), `reject`, `pending`, `fail`
|
|
322
|
+
(the first submit the service process serves throws, every later one accepts; a restart re-arms
|
|
323
|
+
it) and `failAlways`. A `failed` document is reachable no other way.
|
|
324
|
+
|
|
325
|
+
## 8. Reasoned skips: the taxonomy
|
|
326
|
+
|
|
327
|
+
A skip is a covered-with-reason verdict. The reason is a string with a class prefix, so the gate
|
|
328
|
+
counts it as covered and a human can audit every one. Two mechanics:
|
|
329
|
+
|
|
330
|
+
- **static** `suite.skip(api, reason)`: decided from the route row before any call;
|
|
331
|
+
- **thrown** `throw new SkipError(reason)` inside a case or fixture: decided from live evidence, so
|
|
332
|
+
when the cause disappears the case PASSES with no code change.
|
|
333
|
+
|
|
334
|
+
| Class (prefix) | When | Mechanic | Must carry |
|
|
335
|
+
|---|---|---|---|
|
|
336
|
+
| `STATIC-TRANSPORT` / `cron-only` / `internal-call-only` | no external transport AND the test transport cannot reach it either (a service built before the transport existed: the census route answers 404) | static | the transport the row declares, and why section 7 does not reach it |
|
|
337
|
+
| `KNOWN-DEFECT <ref>: <symptom>` | a filed product defect blocks the case | thrown | the issue ref AND `brief(res)` evidence (status, body head, requestId) |
|
|
338
|
+
| `ENVIRONMENT: <what>` | your stack cannot support it (a capacity cap, an unbound provider, no echoed approval code) | static or thrown | the evidence and what would unblock it |
|
|
339
|
+
| `BLOCKED-IDENTITY: <why>` | a section 5 actor could not be bootstrapped | thrown by `session.mjs`, cascades | the login failure `brief` |
|
|
340
|
+
| `BUILT-IN-UNUSED: <what the product does instead>` | a built-in row the plan looked at and the product does not use. Never on a designer API: an unused designer API is a design question, not a skip | static | the one-line product reason |
|
|
341
|
+
| precondition (`no rows yet`, `provider boundary: ...`) | the case's data precondition did not hold on this run | thrown | the response that showed it |
|
|
342
|
+
|
|
343
|
+
Watch the precondition class. It is legitimate when the product allows the empty state, and a
|
|
344
|
+
hidden fail when the precondition SHOULD have held (a fixture that silently did not create the
|
|
345
|
+
row). A precondition skip on a fixture-backed case is re-triaged as a fail.
|
|
346
|
+
|
|
347
|
+
**A gate carries its true root's class, never a bare `precondition`.** The fixture holds the
|
|
348
|
+
response that caused its failure; no downstream case does. So at the point a fixture gives up it
|
|
349
|
+
classifies its own first failure (section 9 ladder) and throws that class with evidence. Every
|
|
350
|
+
dependent case inherits the same string, so the skip ledger counts one classified family, not N
|
|
351
|
+
unexplained preconditions. A `precondition` skip on a case whose fixture ran and failed is a
|
|
352
|
+
mislabelled gate: fix the gate's class, do not re-triage the case.
|
|
353
|
+
|
|
354
|
+
**Never:** weaken an assertion, widen a status set to turn a red green, delete a case, mark a fail
|
|
355
|
+
as a skip without a class and evidence, or change the product to fit the test. 95 % passing with
|
|
356
|
+
real defects accurately flagged beats 100 % with failures skipped.
|
|
357
|
+
|
|
358
|
+
## 9. Triage: the ladder and the fix policy
|
|
359
|
+
|
|
360
|
+
After every run, classify EVERY fail before touching anything. In the pilot, 27 of 32 first-round
|
|
361
|
+
reds were one root (a `whereClause` that named `userId` where the pattern's column is `ownerId`),
|
|
362
|
+
and the second round's 27 were one root again (an inter-service 404 on an unmounted route family).
|
|
363
|
+
|
|
364
|
+
1. **Group by first-failing fixture.** A red fixture takes every dependent case with it. Read the
|
|
365
|
+
first failure's `brief`, fix that, re-run, then look at the survivors. Never fix a cascade case
|
|
366
|
+
by case.
|
|
367
|
+
2. **Read the evidence you have:** `brief(res)` (status, body, `requestId`, `errMsg_*`), the
|
|
368
|
+
fixture's trace, then the service's own log around the request time. The log line with the same
|
|
369
|
+
`requestId` names the stage that threw. If the log does not show it, reproduce the request
|
|
370
|
+
against your local stack with the same case. Query the database read-only to compare a row with
|
|
371
|
+
the API's answer; never write to it.
|
|
372
|
+
3. **Classify.** The class decides the remedy:
|
|
373
|
+
|
|
374
|
+
| Class | Tell | Remedy |
|
|
375
|
+
|---|---|---|
|
|
376
|
+
| Test fault | wrong parameter name, wrong envelope key, a precondition the case never set up, a validator message that contradicts the doc (the live error outranks the doc) | fix the test; the generated service's source or `_api.json` settles a shape question in one read |
|
|
377
|
+
| Design fault (blueprint) | an authored slot is wrong: a `whereClause` column, an MScript expression, a missing role, a validation the spec never asked for | fix the BLUEPRINT, validate, rebuild, redeploy, refresh the route list, re-run. Cap 3 attempts per failure class, each changing something; then surface |
|
|
378
|
+
| Codegen fault | generated framework code fails: a 500 with a stack in generated code, a route the list advertises but the app does not mount, a sibling call that 404s on a generated path | collect the evidence (route, `brief`, `requestId`, log excerpt, route row, blueprint path), report it to the Genesis team, and make the case a `KNOWN-DEFECT <ref>` skip thrown from the live response |
|
|
379
|
+
| Environment | your stack cannot do it (capacity, an unbound provider, a missing external surface) | `ENVIRONMENT` skip with evidence and the unblock condition |
|
|
380
|
+
| Partial write | a `200` / `201` carrying `actionFailures` | classify by the FIRST entry's `action` + `error` like a red of that action; never by the status, never by re-sending a create |
|
|
381
|
+
|
|
382
|
+
Restate a fix's precondition before applying it to a second case: same-shape reds have different
|
|
383
|
+
roots.
|
|
384
|
+
|
|
385
|
+
**Fix policy, non-negotiable.**
|
|
386
|
+
|
|
387
|
+
- A failing test is not license to change the product. A design fault is fixed in the blueprint
|
|
388
|
+
and rebuilt (`procedures/blueprint-authoring.md`); the suite re-runs against the new build.
|
|
389
|
+
Never hand-edit generated files to pass a test (they are overwritten on the next build; see
|
|
390
|
+
[custom-code.md](custom-code.md) for where your own code lives), and never write rows into the
|
|
391
|
+
database to set up state the API could not create.
|
|
392
|
+
- A workaround you ship (a fixture that routes around a broken generated path, a widened status
|
|
393
|
+
set "because the service answers 500 sometimes") is a defect report's trigger: report first,
|
|
394
|
+
then decide whether it stays as a tripwire skip.
|
|
395
|
+
- Cap and surface: three attempts per failure class, then the user decides, with the reds quoted,
|
|
396
|
+
the classification and the evidence trail, never summarised away.
|
|
397
|
+
|
|
398
|
+
## 10. Sample data: the committed seed
|
|
399
|
+
|
|
400
|
+
After testing, a stack is full of `test1`, `aaa` and half-finished flows. The seed is a written,
|
|
401
|
+
committed script that produces the opposite, a small image-carrying dataset that makes the product
|
|
402
|
+
legible. When tests dirty the data again, the cycle is "reset, seed", not a person doing it by
|
|
403
|
+
hand. It doubles as the cheapest end-to-end smoke test.
|
|
404
|
+
|
|
405
|
+
The seed is NOT the blueprint's `SeedData` pattern. `SeedData` upserts declared rows at service
|
|
406
|
+
startup, at database level, per data object, for reference and config rows. The sample-data seed is
|
|
407
|
+
post-test, cosmetic, cross-service, and goes through the public API so that validation, ownership,
|
|
408
|
+
derived fields, events and the search index all really turn.
|
|
409
|
+
|
|
410
|
+
### The five rules
|
|
411
|
+
|
|
412
|
+
1. **Public API only.** Every record is created with `fetch` against the service's API, never
|
|
413
|
+
through the database. A database-level seed leaves search empty and events unfired.
|
|
414
|
+
2. **Idempotent and deterministic.** Every record carries a stable key in a natural field (slug,
|
|
415
|
+
sku, email, name), and the script looks the record up before creating it (`ensure(kind, key,
|
|
416
|
+
{ find, create })`). A second run reports `created: 0`. Note the contrast with the suite, whose
|
|
417
|
+
ids carry random suffixes so re-runs never collide: the seed uses stable keys so re-runs
|
|
418
|
+
converge. If an object has no natural field for the key, put a recognisably stable value in a
|
|
419
|
+
unique field and match on it; "create blindly" is how a seed stops being re-runnable.
|
|
420
|
+
3. **Seeded stock images, never drawn or generated.**
|
|
421
|
+
- An image URL field gets `https://picsum.photos/seed/<stable-key>/<w>/<h>`: the seeded path
|
|
422
|
+
keeps the same picture per record on every run. Never the unseeded `picsum.photos/<w>/<h>`.
|
|
423
|
+
- An image stored as a bucket key (the create call takes a storage key, not a URL) gets the same
|
|
424
|
+
seeded photo downloaded once, cached under `seed/images/` (commit the cache: a re-run needs no
|
|
425
|
+
network), and uploaded through the product's own bucket upload route inside the record's
|
|
426
|
+
`create()`.
|
|
427
|
+
- Never draw images with code and never generate them with an image model: both cost more than
|
|
428
|
+
a download and look worse.
|
|
429
|
+
- For topical photos use a stock library whose licence allows commercial use without
|
|
430
|
+
attribution (Unsplash, Pexels, Pixabay). Their search APIs need a free key: keep it in the
|
|
431
|
+
environment only, never in the seed, and cache the downloads the same way. Avoid Wikimedia
|
|
432
|
+
Commons unless every file's attribution travels with the picture.
|
|
433
|
+
4. **A real login.** The seed authenticates through the same public login the frontend uses,
|
|
434
|
+
registers its demo users through the real register, verify, login chain and creates the tenant(s)
|
|
435
|
+
it needs. On a multi-tenant project it sends the tenant codename header; copy the header name
|
|
436
|
+
from the route list (`_api.json` carries it per endpoint) and the auth spec, never guess.
|
|
437
|
+
5. **Fail loud.** Any non-2xx stops the seed and names the API, the call and the payload. Nothing
|
|
438
|
+
is swallowed or blind-retried. A red seed is a FINDING: triage it with the ladder of section 9
|
|
439
|
+
before touching the seed.
|
|
440
|
+
|
|
441
|
+
### Where it lives, and what it reads first
|
|
442
|
+
|
|
443
|
+
- It is ONE self-contained script, `seed/seed-sample-data.mjs` in the same tests repo (`npm run
|
|
444
|
+
seed`), with one README line on what it seeds, how to run it and the idempotence check.
|
|
445
|
+
- Self-contained: Node >= 18, built-ins only, no imports from `helpers/`, base URL from an
|
|
446
|
+
environment variable, because it runs alone, in its own phase, long after the suite.
|
|
447
|
+
- Before filling it, read: the route lists of every service (routes are copied, never composed),
|
|
448
|
+
the auth spec (roles, required register fields, tenant shape), the data model (which objects have
|
|
449
|
+
states, which fields are images) and `procedures/pitfalls.md` (the verify chain with `secretCode`
|
|
450
|
+
as a string, real-today dates, keep-parents-open).
|
|
451
|
+
|
|
452
|
+
### Filling the story
|
|
453
|
+
|
|
454
|
+
The data is cosmetic: a small story the UI can walk through, not a fixture dump.
|
|
455
|
+
|
|
456
|
+
- **Per service, per state.** Every meaningful state of every object the UI shows gets at least one
|
|
457
|
+
record (a draft AND a published product; an open, a shipped AND a delivered order). Reach states
|
|
458
|
+
through the PUBLIC transitions in a second pass: create everything open first, seed children
|
|
459
|
+
while parents are open, then transition a few parents.
|
|
460
|
+
- **Plausible and coherent.** Real-looking names, sensible relations (the review is written by the
|
|
461
|
+
customer whose order was delivered), prices and dates that could be true, anchored to the real
|
|
462
|
+
today.
|
|
463
|
+
- **Enough volume:** roughly 5 to 15 records per primary object, at least one page where the
|
|
464
|
+
frontend paginates, at least two distinct values for every field the UI filters on.
|
|
465
|
+
- **Images everywhere the UI shows one**, sized for the slot (card, hero, avatar).
|
|
466
|
+
- **A small, reusable cast:** one owner or staff actor and one or two customers, with stable
|
|
467
|
+
seed-key emails and a deterministic demo password from the environment, so the second run logs in
|
|
468
|
+
instead of registering and a person can log in AS a demo user. The super-admin credential is
|
|
469
|
+
resolved at runtime only (section 5) and never appears in the committed file.
|
|
470
|
+
- Use reserved non-deliverable addresses for demo users (`.invalid`, `.test`, `.example`,
|
|
471
|
+
`example.com`) so no real provider ever receives a demo message.
|
|
472
|
+
|
|
473
|
+
### Running and verifying
|
|
474
|
+
|
|
475
|
+
```
|
|
476
|
+
cd tests
|
|
477
|
+
BASE=<your stack url> node seed/seed-sample-data.mjs # seed (idempotent)
|
|
478
|
+
BASE=<your stack url> node seed/seed-sample-data.mjs --expect-existing # the twice-run check
|
|
479
|
+
```
|
|
480
|
+
|
|
481
|
+
(Name the base-URL variable whatever your `config.mjs` uses; keep it the same in both files.)
|
|
482
|
+
|
|
483
|
+
- The stack must be up and healthy first (`/health`; [run-locally.md](run-locally.md)).
|
|
484
|
+
- First run: expect `created: N, existing: 0` and a dataset a person can navigate.
|
|
485
|
+
- Second run straight after: `--expect-existing` exits non-zero if ANYTHING was created. That is
|
|
486
|
+
the acceptance check for idempotence; run it every time the seed changes.
|
|
487
|
+
- Seeding never deletes. To start clean, drop your local stack's data volumes and bring the stack
|
|
488
|
+
up again (the generated services re-seed the bootstrap super-admin and every blueprint `seedData`
|
|
489
|
+
row at boot), check you can log in as super-admin, then seed. A seed on top of test data without
|
|
490
|
+
a reset is allowed but said out loud in the run report: the dataset is additive.
|
|
491
|
+
- Never delete rows by hand to "help": the database and the search index must keep agreeing. To
|
|
492
|
+
remove one row, use the product's own delete or archive API; if the object has none, that is a
|
|
493
|
+
blueprint gap to fix there.
|
|
494
|
+
- After a payment-bearing seed step, provision plans and webhooks first as described in
|
|
495
|
+
`genesis/procedures/howto-subscription.md`.
|
|
496
|
+
- Verify by read-back, not by the exit code: count rows per service in the database and compare
|
|
497
|
+
with what the API lists, and check the search index moved with the database (database up and
|
|
498
|
+
index flat means the seed wrote past the search layer: a finding, never something to seed over).
|
|
499
|
+
Put the counts in the report as numbers.
|
|
500
|
+
|
|
501
|
+
### Seeding a shared stage
|
|
502
|
+
|
|
503
|
+
Only when the owner asks, after a stage deploy, and never as an automatic step:
|
|
504
|
+
|
|
505
|
+
- Confirm the base URL with the owner before the first call. No reset, ever; the seed stays
|
|
506
|
+
additive and idempotent.
|
|
507
|
+
- Catalog only: definition data (categories, products, brands, settings). Anything bound to a user
|
|
508
|
+
(orders, carts, reviews, profiles) is skipped and named in the report; no demo end users are
|
|
509
|
+
registered, because stage runs live verification and codes never ride the answer.
|
|
510
|
+
- A multi-tenant catalog lives inside a tenant, so write it as an owner of ONE example tenant,
|
|
511
|
+
reusing it when it exists. Payment steps run only when the stage payment provider holds
|
|
512
|
+
test-mode keys; otherwise skip them with the reason.
|
|
513
|
+
- Read back every created record through the public API and fail loud on one that is missing.
|
|
514
|
+
|
|
515
|
+
## 11. Checklist
|
|
516
|
+
|
|
517
|
+
- [ ] Test plan written (`procedures/design/writers/test-plan.md`) before the first script.
|
|
518
|
+
- [ ] `inventory/<service>.json` fresh for every service (re-copied after each rebuild).
|
|
519
|
+
- [ ] Helpers and `auth.flow.mjs` green; three actors bootstrap, or a `BLOCKED-IDENTITY` finding
|
|
520
|
+
is reported with the 403 body.
|
|
521
|
+
- [ ] One flow per service; every designer API has a case or a class-prefixed skip; built-ins only
|
|
522
|
+
where the plan lists them.
|
|
523
|
+
- [ ] Routes copied from `_api.json` / `/openapi.json`, a `User-Agent` on every call.
|
|
524
|
+
- [ ] Every write case asserts no `actionFailures`; every approval-gated API goes through
|
|
525
|
+
`approvedCall` and asserts both legs.
|
|
526
|
+
- [ ] `mock-status` probed (`mock: true`, `transport: "present"`); no card data anywhere.
|
|
527
|
+
- [ ] Kafka, cron, internal and machine-only rows driven through the test transport.
|
|
528
|
+
- [ ] `run-all` then `coverage-gate`: `GATE: PASS`, and fails are 0 or each classified and capped.
|
|
529
|
+
- [ ] Every skip carries its class and evidence; fixture gates throw their true root's class.
|
|
530
|
+
- [ ] Design faults fixed in the blueprint and rebuilt; codegen faults reported with evidence.
|
|
531
|
+
- [ ] Seed: public API only, stable keys, seeded stock images, real login, fail loud, run twice
|
|
532
|
+
with `created: 0` on the second run, counts verified by read-back.
|