@pikku/skills 0.12.54 → 0.12.55
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/dist/skills.gen.js +2 -2
- package/dist/snippets.js +1 -1
- package/package.json +1 -1
- package/skills/pikku-architect/SKILL.md +1 -1
- package/skills/pikku-architect/references/plan-defects.md +7 -7
- package/skills/pikku-auth/references/better-auth.md +92 -96
- package/skills/pikku-build/SKILL.md +9 -9
- package/skills/pikku-build/references/app.md +56 -61
- package/skills/pikku-build/references/design.md +14 -14
- package/skills/pikku-build/references/feature.md +3 -3
- package/skills/pikku-build/references/mocks.md +6 -0
- package/skills/pikku-build/references/multi-app.md +16 -16
- package/skills/pikku-build/references/openapi.md +34 -34
- package/skills/pikku-build/references/platform.md +4 -5
- package/skills/pikku-build/references/post-clone.md +1 -1
- package/skills/pikku-build/references/quick.md +63 -36
- package/skills/pikku-build/references/scenarios.md +17 -16
- package/skills/pikku-build/references/ship.md +14 -16
- package/skills/pikku-build/references/theming.md +2 -2
- package/skills/pikku-changes/SKILL.md +2 -5
- package/skills/pikku-concepts/SKILL.md +26 -27
- package/skills/pikku-concepts/references/language.md +6 -6
- package/skills/pikku-deploy/references/nextjs.md +1 -1
- package/skills/pikku-guide/SKILL.md +18 -17
- package/skills/pikku-i18n/references/messages.md +5 -6
- package/skills/pikku-knowledge/SKILL.md +3 -3
- package/skills/pikku-meta/references/audit.md +1 -1
- package/skills/pikku-permissions/SKILL.md +1 -1
- package/skills/pikku-react/SKILL.md +18 -20
- package/skills/pikku-react/references/client.md +2 -37
- package/skills/pikku-report/SKILL.md +10 -10
- package/skills/pikku-scenario/SKILL.md +63 -66
- package/skills/pikku-scenario/references/browser.md +5 -5
- package/skills/pikku-scenario/references/coverage.md +4 -4
- package/skills/pikku-scenario/references/steps.md +31 -30
- package/skills/pikku-scenario/references/{persona-run.md → virtual-user-run.md} +38 -43
- package/skills/pikku-scenario/references/virtual-users.md +114 -0
- package/skills/pikku-seo/SKILL.md +1 -1
- package/skills/pikku-services/references/audit-wire-service.md +2 -2
- package/skills/pikku-services/references/audit.md +1 -2
- package/skills/pikku-services/references/services.md +1 -1
- package/skills/pikku-software-archaeology/SKILL.md +13 -13
- package/skills/pikku-software-archaeology/references/blueprint.schema.json +9 -9
- package/skills/pikku-theme/SKILL.md +1 -1
- package/skills/pikku-workflow/SKILL.md +1 -1
- package/skills/pikku-admin-to-fabric/SKILL.md +0 -198
- package/skills/pikku-blueprint-to-fabric/SKILL.md +0 -362
- package/skills/pikku-blueprint-to-fabric/scripts/inventory.mjs +0 -367
- package/skills/pikku-fabric/SKILL.md +0 -331
- package/skills/pikku-fabric/references/database.md +0 -128
- package/skills/pikku-fabric/references/debugging.md +0 -114
- package/skills/pikku-fabric/references/deploy.md +0 -179
- package/skills/pikku-scenario/references/personas.md +0 -123
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: pikku-scenario
|
|
3
3
|
description: >-
|
|
4
|
-
Use when writing or running Pikku scenarios, running a
|
|
4
|
+
Use when writing or running Pikku scenarios, running a virtual user against a stage, or when
|
|
5
5
|
asked to test Pikku functions or improve coverage. A scenario (pikkuScenario) drives the app
|
|
6
|
-
the way users do — steps run as
|
|
6
|
+
the way users do — steps run as virtual users over the real transport against a running server — so a
|
|
7
7
|
flow doubles as an e2e test and a staged/production health check. Covers scenario.do /
|
|
8
8
|
expectEventually / expectError / expectService / expectScore, declared steps via
|
|
9
9
|
pikkuScenarioStep (browser steps driven by @pikku/playwright) written as intent rather than
|
|
10
|
-
clicks,
|
|
11
|
-
`pikku scenario` and `pikku
|
|
10
|
+
clicks, virtual users / virtual users / environments in pikku.config.json, VIRTUAL_USER_SIGNING_KEY, the
|
|
11
|
+
`pikku scenario` and `pikku virtual-user` commands, and live coverage
|
|
12
12
|
via `pikku dev --coverage`. TRIGGER when: user asks about scenarios, testing a Pikku function,
|
|
13
|
-
coverage, e2e flows, browser/UI e2e, health checks,
|
|
13
|
+
coverage, e2e flows, browser/UI e2e, health checks, virtual users, or adversarial
|
|
14
14
|
runs against a stage. DO NOT TRIGGER when: user asks about running an existing suite (use
|
|
15
15
|
Bash) or CI config.
|
|
16
16
|
installGroups: [core]
|
|
@@ -31,23 +31,23 @@ installGroups: [core]
|
|
|
31
31
|
This skill covers writing and running scenarios end to end. Five topics are one level down, and
|
|
32
32
|
each says when to open it:
|
|
33
33
|
|
|
34
|
-
| Read
|
|
35
|
-
|
|
|
36
|
-
| `references/steps.md`
|
|
37
|
-
| `references/browser.md`
|
|
38
|
-
| `references/
|
|
39
|
-
| `references/coverage.md`
|
|
40
|
-
| `references/
|
|
34
|
+
| Read | For |
|
|
35
|
+
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
|
36
|
+
| `references/steps.md` | Authoring a `pikkuScenarioStep` — intent, witnesses, what a step is given |
|
|
37
|
+
| `references/browser.md` | Browser bindings, and locating by message key in a translated app |
|
|
38
|
+
| `references/virtual-users.md` | Virtual users vs virtual users, `defineVirtualUsers`, upstream credentials for virtual users, and human sign-in as a virtual user |
|
|
39
|
+
| `references/coverage.md` | Live coverage, filling it, and unit tests for pure logic |
|
|
40
|
+
| `references/virtual-user-run.md` | Running a model-driven virtual user against a stage |
|
|
41
41
|
|
|
42
42
|
## What a scenario is
|
|
43
43
|
|
|
44
|
-
A scenario is a `pikkuScenario` export that drives the app **as real
|
|
44
|
+
A scenario is a `pikkuScenario` export that drives the app **as real virtual users over the real transport**, against a running server. That is what lets one artifact serve as both an e2e test and a staged/production health check.
|
|
45
45
|
|
|
46
46
|
Consequences that matter, and bite if ignored:
|
|
47
47
|
|
|
48
48
|
- **There is no state reset.** A scenario runs against a live server. Scope what you create (unique ids, your own rows) and never assume a clean database.
|
|
49
|
-
- **Every effect runs as somebody, or as a declared step.** `scenario.do(...)` without `{
|
|
50
|
-
- **
|
|
49
|
+
- **Every effect runs as somebody, or as a declared step.** `scenario.do(...)` without `{ virtualUser }` throws `Scenario tried to run '<rpc>' as an internal step…` — there is no bare internal-RPC step. The other way to do work is `scenario.given/when/then`, which runs a `pikkuScenarioStep`; its virtual user is optional (setup steps have none) unless it declares `browser: true`.
|
|
50
|
+
- **Virtual users must be configured and signed in**, or the scenario cannot run.
|
|
51
51
|
|
|
52
52
|
Scenarios live in `srcDirectories` like any other function — by convention `*.scenario.ts`.
|
|
53
53
|
|
|
@@ -64,10 +64,10 @@ export const orderSupportScenario = pikkuScenario<
|
|
|
64
64
|
>({
|
|
65
65
|
title: 'Order support (scenario)',
|
|
66
66
|
tags: ['scenario'],
|
|
67
|
-
func: async ({ logger }, data, { scenario,
|
|
68
|
-
if (!
|
|
67
|
+
func: async ({ logger }, data, { scenario, virtualUsers }) => {
|
|
68
|
+
if (!virtualUsers?.shopper || !virtualUsers?.support) {
|
|
69
69
|
throw new Error(
|
|
70
|
-
'orderSupportScenario needs run
|
|
70
|
+
'orderSupportScenario needs run virtualUsers (shopper + support) — run via `pikku scenario run <environment>`'
|
|
71
71
|
)
|
|
72
72
|
}
|
|
73
73
|
|
|
@@ -75,7 +75,7 @@ export const orderSupportScenario = pikkuScenario<
|
|
|
75
75
|
'shopper doubles their order',
|
|
76
76
|
'doubleValue',
|
|
77
77
|
{ value: data?.value ?? 21 },
|
|
78
|
-
{
|
|
78
|
+
{ virtualUser: virtualUsers.shopper }
|
|
79
79
|
)
|
|
80
80
|
|
|
81
81
|
const settled = await scenario.expectEventually(
|
|
@@ -83,7 +83,7 @@ export const orderSupportScenario = pikkuScenario<
|
|
|
83
83
|
'formatMessage',
|
|
84
84
|
{ greeting: 'Hello', name: 'Support' },
|
|
85
85
|
(out: { message: string }) => out.message.length > 0,
|
|
86
|
-
{
|
|
86
|
+
{ virtualUser: virtualUsers.support, within: '5s', interval: 50 }
|
|
87
87
|
)
|
|
88
88
|
|
|
89
89
|
return { doubled: doubled.result, message: settled.message }
|
|
@@ -91,19 +91,19 @@ export const orderSupportScenario = pikkuScenario<
|
|
|
91
91
|
})
|
|
92
92
|
```
|
|
93
93
|
|
|
94
|
-
A scenario takes the same config fields as a workflow (`title`, `description`, `tags`, `input`/`output`, `auth`, `permissions`, `middleware`, `version`, …). The third argument is the scenario context: `{ scenario,
|
|
94
|
+
A scenario takes the same config fields as a workflow (`title`, `description`, `tags`, `input`/`output`, `auth`, `permissions`, `middleware`, `version`, …). The third argument is the scenario context: `{ scenario, virtualUsers }`.
|
|
95
95
|
|
|
96
96
|
### The scenario API
|
|
97
97
|
|
|
98
|
-
| Call
|
|
99
|
-
|
|
|
100
|
-
| `scenario.do(step, rpc, data, {
|
|
101
|
-
| `scenario.expectEventually(step, rpc, data, predicate, {
|
|
102
|
-
| `scenario.expectError(step, rpc, data, {
|
|
103
|
-
| `scenario.expectService(step, 'service.method', {
|
|
104
|
-
| `scenario.expectScore(step, runId, scorer, { atLeast, atMost, reference })`
|
|
105
|
-
| `scenario.given(stepName, step, data, {
|
|
106
|
-
| `scenario.runScheduledTask(name)`
|
|
98
|
+
| Call | Purpose |
|
|
99
|
+
| ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
|
|
100
|
+
| `scenario.do(step, rpc, data, { virtualUser })` | Run an RPC as that virtual user. The step name is what appears in the run output. |
|
|
101
|
+
| `scenario.expectEventually(step, rpc, data, predicate, { virtualUser, within, interval })` | Poll until `predicate(out)` passes or `within` elapses. For anything asynchronous — queues, workers, eventual state. |
|
|
102
|
+
| `scenario.expectError(step, rpc, data, { virtualUser, matches })` | Assert the call **fails**. For fault injection and negative paths. |
|
|
103
|
+
| `scenario.expectService(step, 'service.method', { virtualUser, calledWith })` | Assert a stubbed service was called. Requires the server to run with `--test`. |
|
|
104
|
+
| `scenario.expectScore(step, runId, scorer, { atLeast, atMost, reference })` | Grade a finished agent run with a declared scorer and assert the score. See below. |
|
|
105
|
+
| `scenario.given(stepName, step, data, { virtualUser })` | Run a declared `pikkuScenarioStep` as setup. `when` is the same call; `then` also makes the step's bindings witnesses. |
|
|
106
|
+
| `scenario.runScheduledTask(name)` | Fire a wired scheduler on the target now, rather than waiting for its cron. |
|
|
107
107
|
|
|
108
108
|
`expectEventually` is **scenario-only**. Calling it from a `pikkuWorkflowFunc` is a critical inspector error (`PKU675`) pointing you at `pikkuScenario`.
|
|
109
109
|
|
|
@@ -123,7 +123,7 @@ const { runId } = await scenario.when(
|
|
|
123
123
|
{
|
|
124
124
|
prompt: data.prompt,
|
|
125
125
|
},
|
|
126
|
-
{
|
|
126
|
+
{ virtualUser: virtualUsers.user }
|
|
127
127
|
)
|
|
128
128
|
|
|
129
129
|
await scenario.expectScore('answered briefly', runId, 'brevity', {
|
|
@@ -152,8 +152,8 @@ the default suite excludes it.
|
|
|
152
152
|
A scenario config takes `before` and `after`. Both have the **same signature as `func`** — `(services, data, wire)` — with the return value discarded:
|
|
153
153
|
|
|
154
154
|
```typescript
|
|
155
|
-
const resetsCredentials = async (_services, _data, {
|
|
156
|
-
await
|
|
155
|
+
const resetsCredentials = async (_services, _data, { virtualUsers }) => {
|
|
156
|
+
await virtualUsers!.admin!.invoke('resetCredentials', {})
|
|
157
157
|
}
|
|
158
158
|
|
|
159
159
|
export const credentialScenario = pikkuScenario({
|
|
@@ -161,7 +161,7 @@ export const credentialScenario = pikkuScenario({
|
|
|
161
161
|
tags: ['scenario', 'credential'],
|
|
162
162
|
before: resetsCredentials,
|
|
163
163
|
after: removesInstalledAddon,
|
|
164
|
-
func: async (services, data, { scenario,
|
|
164
|
+
func: async (services, data, { scenario, virtualUsers }) => {
|
|
165
165
|
/* … */
|
|
166
166
|
},
|
|
167
167
|
})
|
|
@@ -176,7 +176,7 @@ export const credentialScenario = pikkuScenario({
|
|
|
176
176
|
| Neither runs when the run is suspended or waiting — teardown only fires at a terminal outcome. |
|
|
177
177
|
| Hooks are **not** ladder rows. The runner records nothing for them; a failure is labelled by phase. |
|
|
178
178
|
|
|
179
|
-
A hook reaches the app the same way the body does: through `wire.
|
|
179
|
+
A hook reaches the app the same way the body does: through `wire.virtualUsers`. If you want cleanup to be _visible_ on the ladder, make it an ordinary `scenario.then(...)` instead.
|
|
180
180
|
|
|
181
181
|
Hooks are scenario-only. A `before`/`after` on a `pikkuWorkflowFunc` never runs — a workflow is durable and resumable, so a callback that reran on every replay would have no honest meaning.
|
|
182
182
|
|
|
@@ -229,7 +229,7 @@ read. Mark plumbing features `document: false`.
|
|
|
229
229
|
|
|
230
230
|
`scenario.do` can only name an RPC. A **step** — `pikkuScenarioStep` — is a named, typed unit of
|
|
231
231
|
scenario behaviour whose body is an ordinary pikku function, so it can call several RPCs as its
|
|
232
|
-
|
|
232
|
+
virtual user, assert, or drive a browser. It is referenced by its typed string name (checked against the
|
|
233
233
|
generated step map), its body lives under a surface binding (`default` / `browser` / `cli`, never a
|
|
234
234
|
`func`), and one row of the report is one step.
|
|
235
235
|
|
|
@@ -238,7 +238,7 @@ suite survives its first redesign:
|
|
|
238
238
|
|
|
239
239
|
- **What a step is given** — a step runs in the CLI process, not where the app runs. There is no
|
|
240
240
|
`kysely`, no `secrets`, and `rpc` is present but throws on purpose. Everything reaches the app
|
|
241
|
-
through the
|
|
241
|
+
through the virtual user.
|
|
242
242
|
- **Steps describe intent, not actions** — `buys the £5 strawberry milkshake`, never
|
|
243
243
|
`clicks [data-testid=add]`. The clicking lives in plain utilities the step composes.
|
|
244
244
|
- **`then` bindings are witnesses, not alternatives** — a `then` runs _every_ binding it declares
|
|
@@ -256,11 +256,11 @@ score a perfect run while checking nothing.
|
|
|
256
256
|
|
|
257
257
|
## Configuration
|
|
258
258
|
|
|
259
|
-
|
|
260
|
-
whole project — codegen builds the `
|
|
261
|
-
scenario
|
|
259
|
+
Virtual users are declared in TypeScript. One `defineVirtualUsers({ … })` call for the
|
|
260
|
+
whole project — codegen builds the `VirtualUserId` union from it, materialises one
|
|
261
|
+
scenario virtual user per person, and seeds a user row each:
|
|
262
262
|
|
|
263
|
-
```ts snippet:
|
|
263
|
+
```ts snippet:defineVirtualUsers
|
|
264
264
|
|
|
265
265
|
```
|
|
266
266
|
|
|
@@ -269,41 +269,40 @@ scenario actor per person, and seeds a user row each:
|
|
|
269
269
|
```json
|
|
270
270
|
{
|
|
271
271
|
"scenarios": {
|
|
272
|
-
"emailDomain": "
|
|
272
|
+
"emailDomain": "virtualuser.example.com",
|
|
273
273
|
"browserDriver": "@pikku/playwright",
|
|
274
274
|
"model": "claude-sonnet-4-5"
|
|
275
275
|
},
|
|
276
276
|
"environments": {
|
|
277
277
|
"local": {
|
|
278
278
|
"apiUrl": "http://localhost:4077",
|
|
279
|
-
"signInPath": "/api/auth/sign-in/
|
|
279
|
+
"signInPath": "/api/auth/sign-in/virtual-user"
|
|
280
280
|
}
|
|
281
281
|
}
|
|
282
282
|
}
|
|
283
283
|
```
|
|
284
284
|
|
|
285
|
-
- `scenarios.emailDomain` is the mail domain
|
|
285
|
+
- `scenarios.emailDomain` is the mail domain virtual user addresses are built on.
|
|
286
286
|
- `scenarios.browserDriver` is the package driving `browser` bindings.
|
|
287
|
-
- `scenarios.model` is the model a
|
|
288
|
-
- `environments.<name>` are the targets a run can point at. The key is the required positional of `pikku scenario run`; `apiUrl` is required; `signInPath` defaults to `/auth/sign-in/
|
|
287
|
+
- `scenarios.model` is the model a virtual user thinks with (`virtualUser.converse`, `pikku virtual-user run`).
|
|
288
|
+
- `environments.<name>` are the targets a run can point at. The key is the required positional of `pikku scenario run`; `apiUrl` is required; `signInPath` defaults to `/auth/sign-in/virtual-user`, `rpcPath` to `/rpc`, and `sessionPath` has a default too.
|
|
289
289
|
|
|
290
|
-
**`references/
|
|
291
|
-
person and one
|
|
292
|
-
for someone only acted upon, `
|
|
293
|
-
than evaluated, and the same
|
|
294
|
-
switcher.
|
|
290
|
+
**`references/virtual-users.md`** covers the parts that bite: one entry is one
|
|
291
|
+
person and one virtual user, emails are derived and never written, `runnable: false`
|
|
292
|
+
for someone only acted upon, `defineVirtualUsers` being read from source rather
|
|
293
|
+
than evaluated, and the same virtual user list powering a human sign-in.
|
|
295
294
|
|
|
296
|
-
-
|
|
295
|
+
- **The signing key never goes in `pikku.config.json`.** `pikku dev` creates one under `.pikku-runtime/virtual-user/`; a deployed stage's deployer holds `VIRTUAL_USER_SIGNING_KEY`. Each run signs a short-lived statement with it. `pikku scenario run` throws without a key.
|
|
297
296
|
|
|
298
297
|
## Running
|
|
299
298
|
|
|
300
299
|
```bash
|
|
301
300
|
pikku scenario list # features with their scenarios indented, then ungrouped scenarios
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
301
|
+
pikku scenario run local
|
|
302
|
+
pikku scenario run local --flows orderSupportScenario
|
|
303
|
+
pikku scenario run local --features credentialFeature
|
|
304
|
+
pikku scenario run local --tags smoke,scenario
|
|
305
|
+
pikku scenario run local --spawn --exclude-tags ai-live
|
|
307
306
|
```
|
|
308
307
|
|
|
309
308
|
`run` takes the environment as a **required positional** — the key from `environments`. Every filter narrows the same plan, so narrowing a feature to two of its five scenarios still runs the feature's hooks exactly once around those two.
|
|
@@ -325,7 +324,7 @@ SCENARIO_ACTOR_SECRET=… pikku scenario run local --spawn --exclude-tags ai-liv
|
|
|
325
324
|
|
|
326
325
|
Output is `PASS <name> (<ms>) → <output>` / `FAIL <name> (<ms>): <error>`, then `N/M scenarios passed against '<env>'`. A scenario inside a feature is named `<Feature> › <scenario> <data>`.
|
|
327
326
|
|
|
328
|
-
**Exit code is 1** if any scenario fails _or_ if no scenario matched the filter — a typo'd `--flows` is a hard error, not a silent zero-run pass. It throws outright on an unknown environment, an unknown flow name, or a missing
|
|
327
|
+
**Exit code is 1** if any scenario fails _or_ if no scenario matched the filter — a typo'd `--flows` is a hard error, not a silent zero-run pass. It throws outright on an unknown environment, an unknown flow name, or a missing signing key.
|
|
329
328
|
|
|
330
329
|
## Coverage
|
|
331
330
|
|
|
@@ -335,7 +334,7 @@ derived from unit tests:
|
|
|
335
334
|
```bash
|
|
336
335
|
pikku enable scenarios # sets scaffold.scenarios = true
|
|
337
336
|
pikku dev --coverage # V8 precise coverage, in-process
|
|
338
|
-
|
|
337
|
+
pikku scenario run local --coverage
|
|
339
338
|
```
|
|
340
339
|
|
|
341
340
|
That writes `<outDir>/coverage/scenario-coverage.json`. **`references/coverage.md`** has the
|
|
@@ -346,20 +345,18 @@ and where a plain unit test is still the right tool.
|
|
|
346
345
|
|
|
347
346
|
| Smell | Why it's wrong |
|
|
348
347
|
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
349
|
-
| `.feature` files / Gherkin for function tests | Scenarios are TypeScript, not Gherkin.
|
|
350
|
-
| `scenario.do(...)` with no `{
|
|
348
|
+
| `.feature` files / Gherkin for function tests | Scenarios are TypeScript, not Gherkin. There is no in-process function world. |
|
|
349
|
+
| `scenario.do(...)` with no `{ virtualUser }` | Throws. Every step runs as somebody. |
|
|
351
350
|
| A scenario per function | Scenarios are user flows. One flow covers many functions; that is the point. |
|
|
352
351
|
| Assuming a clean database | There is no state reset — it may be a staging server. Scope what you create. |
|
|
353
352
|
| `sleep()` before asserting | Use `expectEventually`. |
|
|
354
|
-
| A step named `clicksAddToBasket` / `opensThePage` | That is an action, not an intent. Name the step for what the
|
|
353
|
+
| A step named `clicksAddToBasket` / `opensThePage` | That is an action, not an intent. Name the step for what the virtual user wanted; put the clicking in a utility. |
|
|
355
354
|
| A step named `kauftEinenApfel` / a `vorgang` table | Identifiers are English in every project. The German belongs in `description` / `template`, and only when `pikku.config.json` sets `metaLocale`. |
|
|
356
355
|
| A browser step that assumes it is already on a page | It can then only run mid-flow. Arrive first — check the URL, navigate if needed. |
|
|
357
356
|
| `getByLabel('Full Name')` in a translated app | Passes only in the base locale, and a copy edit breaks it as an unexplained timeout. Locate by message key. |
|
|
358
|
-
| A `browser` binding guarding `if (!browser)` | The binding guarantees it. The guard hides the real error, which is a missing
|
|
357
|
+
| A `browser` binding guarding `if (!browser)` | The binding guarantees it. The guard hides the real error, which is a missing virtual user (`PKU677`). |
|
|
359
358
|
| A step with a `func:` instead of a surface binding | There is no `func` on a step. Bodies live under `default` / `browser` / `cli`; a step with none throws at load. |
|
|
360
359
|
| `expectEventually` in a `pikkuWorkflowFunc` | `PKU675` — scenario-only. |
|
|
361
|
-
| Coverage silently 0 | Server not run with `--coverage`, verbose functions meta not deployed, `scaffold.scenarios` unset, or no
|
|
362
|
-
|
|
363
|
-
`@pikku/cucumber` is a **browser/e2e** harness (`Actor`, `BrowserWorld`, `PersonaData`, `DbUtils`) — out of scope here.
|
|
360
|
+
| Coverage silently 0 | Server not run with `--coverage`, verbose functions meta not deployed, `scaffold.scenarios` unset, or no virtual users configured. |
|
|
364
361
|
|
|
365
362
|
See `pikku-concepts` for the core mental model.
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Declaring a `browser` binding is the whole switch: inside that binding `wire.browser` is guaranteed present and non-optional, and a step without one never sees a browser at all. There is nothing to null-check.
|
|
4
4
|
|
|
5
|
-
A `browser` binding gets a session bound to **its
|
|
5
|
+
A `browser` binding gets a session bound to **its virtual user**, signed in through the same signed-statement path the HTTP virtual users use, so the browser and the RPC calls are one identity. Calling such a step without a virtual user is a critical error (`PKU677`).
|
|
6
6
|
|
|
7
7
|
Browser steps are where **intent, not actions** earns its keep: the step is one intent, the clicking lives in shared utilities, and the step arrives before it acts. Write the mechanics below into utilities and keep the step body to three or four calls that read as a sentence.
|
|
8
8
|
|
|
@@ -17,8 +17,8 @@ export const opensTheCart = pikkuScenarioStep<
|
|
|
17
17
|
await browser.goto(path)
|
|
18
18
|
return { url: browser.page.url() }
|
|
19
19
|
},
|
|
20
|
-
default: async (_services, _data, {
|
|
21
|
-
url: (await
|
|
20
|
+
default: async (_services, _data, { virtualUser }) => ({
|
|
21
|
+
url: (await virtualUser.invoke('getCart', {})).url,
|
|
22
22
|
}),
|
|
23
23
|
})
|
|
24
24
|
```
|
|
@@ -54,6 +54,6 @@ await page
|
|
|
54
54
|
|
|
55
55
|
- Type off `messages/<baseLocale>.json`, **not** the generated Paraglide output — `i18n/paraglide/` is build output, so typing against it makes the tests unbuildable until the app has been built. The JSON is the tracked source.
|
|
56
56
|
- Fall back to the base locale for a key a locale has not translated. That is what Paraglide does at run time, so a helper that throws instead would disagree with the screen the test is looking at.
|
|
57
|
-
- This is not only about locators. A copy literal passed to a **project helper** (`pick('Where would you like to work?', …)`) reaches the DOM the same way, and so does a pane name quoted back in a failure message. `pikku
|
|
57
|
+
- This is not only about locators. A copy literal passed to a **project helper** (`pick('Where would you like to work?', …)`) reaches the DOM the same way, and so does a pane name quoted back in a failure message. `pikku validate` scans every string in a `*.steps.ts` / `*.scenario.ts` against the base catalogue and errors on any verbatim match, wherever it sits — except comments, and the `name` / `description` / `template` declared directly on a `pikkuFeature`, `pikkuScenario` or `pikkuScenarioStep`, which are Console meta written in the project's `locale` rather than app copy.
|
|
58
58
|
- A regex locator (`{ name: /^Next$/i }`) hides the literal but not the problem. `{ name: t('key'), exact: true }` is both stricter and locale-correct.
|
|
59
|
-
- Strings the catalogue does not own — a test id, a fixture filename, a seeded value — stay literal. The catalogue is the test for whether something is copy.
|
|
59
|
+
- Strings the catalogue does not own — a test id, a fixture filename, a seeded value — stay literal. The catalogue is the test for whether something is copy.
|
|
@@ -18,7 +18,7 @@ one as a flag would be worse than failing.
|
|
|
18
18
|
```bash
|
|
19
19
|
pikku dev --coverage # V8 precise coverage, in-process
|
|
20
20
|
pikku dev --coverage --test # also enable stubs (needed for expectService)
|
|
21
|
-
|
|
21
|
+
pikku scenario run local --coverage
|
|
22
22
|
```
|
|
23
23
|
|
|
24
24
|
The run resets coverage before each scenario and snapshots after, writing **`<outDir>/coverage/scenario-coverage.json`**:
|
|
@@ -33,7 +33,7 @@ The run resets coverage before each scenario and snapshots after, writing **`<ou
|
|
|
33
33
|
}
|
|
34
34
|
```
|
|
35
35
|
|
|
36
|
-
Coverage is best-effort: it disables itself with a warning if the server is not collecting or the first
|
|
36
|
+
Coverage is best-effort: it disables itself with a warning if the server is not collecting or the first virtual user cannot invoke, and it needs at least one configured virtual user. If you get no coverage, check those first.
|
|
37
37
|
|
|
38
38
|
**There is no AI-prompt output.** The old `--ai-out` flag died with `pikku tests`; nothing replaced it. To find what needs work, read `scenario-coverage.json` yourself and cross-reference `pikku meta functions list` for input/output schemas.
|
|
39
39
|
|
|
@@ -41,7 +41,7 @@ Coverage is best-effort: it disables itself with a warning if the server is not
|
|
|
41
41
|
|
|
42
42
|
1. `pikku scenario run <env> --coverage`, then read `<outDir>/coverage/scenario-coverage.json` to see what is unexercised.
|
|
43
43
|
2. `pikku meta functions list` for those functions' schemas.
|
|
44
|
-
3. Write a `pikkuScenario` that reaches them **through a real user flow** with
|
|
44
|
+
3. Write a `pikkuScenario` that reaches them **through a real user flow** with a virtual user — not a scenario per function. Scenarios are flows; coverage is a consequence.
|
|
45
45
|
4. Re-run to confirm.
|
|
46
46
|
|
|
47
47
|
## Unit tests for pure logic
|
|
@@ -67,4 +67,4 @@ describe('createTodo', () => {
|
|
|
67
67
|
node --import tsx --test src/**/*.test.ts
|
|
68
68
|
```
|
|
69
69
|
|
|
70
|
-
Services are plain objects — a Pikku function is pure business logic, so a mock is just the shape the function destructures. Build real services via the `pikkuServices` / `pikkuWireServices` factories when a test needs them.
|
|
70
|
+
Services are plain objects — a Pikku function is pure business logic, so a mock is just the shape the function destructures. Build real services via the `pikkuServices` / `pikkuWireServices` factories when a test needs them.
|
|
@@ -16,7 +16,7 @@ Browser bindings have their own file: **`references/browser.md`**.
|
|
|
16
16
|
|
|
17
17
|
## What a step is
|
|
18
18
|
|
|
19
|
-
`scenario.do` can only name an RPC. A **step** is a named, typed unit of scenario behaviour whose body is an ordinary pikku function — so it can call several RPCs as its
|
|
19
|
+
`scenario.do` can only name an RPC. A **step** is a named, typed unit of scenario behaviour whose body is an ordinary pikku function — so it can call several RPCs as its virtual user, assert, or drive a browser.
|
|
20
20
|
|
|
21
21
|
```typescript
|
|
22
22
|
import { pikkuScenarioStep } from '#pikku/scenarios'
|
|
@@ -28,9 +28,9 @@ export const buysAnApple = pikkuScenarioStep<
|
|
|
28
28
|
name: 'buysAnApple',
|
|
29
29
|
description: 'buys an apple',
|
|
30
30
|
template: 'buys {qty} apples',
|
|
31
|
-
|
|
32
|
-
default: async (_services, { qty }, {
|
|
33
|
-
return await
|
|
31
|
+
virtualUser: true,
|
|
32
|
+
default: async (_services, { qty }, { virtualUser }) => {
|
|
33
|
+
return await virtualUser.invoke('placeOrder', { qty })
|
|
34
34
|
},
|
|
35
35
|
})
|
|
36
36
|
```
|
|
@@ -44,7 +44,7 @@ await scenario.given(
|
|
|
44
44
|
'buys an apple',
|
|
45
45
|
'buysAnApple',
|
|
46
46
|
{ qty: 1 },
|
|
47
|
-
{
|
|
47
|
+
{ virtualUser: virtualUsers.shopper }
|
|
48
48
|
)
|
|
49
49
|
// reporter renders: Given shopper buys 1 apples ✓ 412ms
|
|
50
50
|
```
|
|
@@ -53,14 +53,14 @@ Rules that bite:
|
|
|
53
53
|
|
|
54
54
|
- **The step is referenced by its typed string name, not by importing the const** — exactly like `workflow.do`. The name is the step's `pikkuFuncId` and is checked against the generated step map. A non-literal target is a critical error (`PKU678`).
|
|
55
55
|
- **Steps are not RPCs.** They are deliberately never network-callable — a browser-driving step must not be.
|
|
56
|
-
- **`
|
|
57
|
-
- **A step that runs as somebody declares `
|
|
56
|
+
- **`virtualUser.invoke` is typed over the exposed RPC map**, so the name and the payload are checked and the result comes back narrowed — no cast. `virtualUser.invokeRaw(name, data, { headers })` is the same call reporting `{ status, ok, body }` instead of throwing; use it whenever the refusal _is_ the assertion.
|
|
57
|
+
- **A step that runs as somebody declares `virtualUser: true`**, and the runner injects `wire.virtualUser` — non-optional inside every binding, with no guard to write and nothing to unwrap. A `browser` binding implies it, because a window is opened as somebody. Leave it off for a step with no virtual user to be: an assertion over what an earlier step returned, or one that posts credentials precisely because it must not reuse a virtual user's session. Dispatching a step that declared it without `{ virtualUser: virtualUsers.x }` fails before the body runs (`ScenarioVirtualUserRequired`); a step that did not declare it has no `virtualUser` on its wire at all.
|
|
58
58
|
- **`env` is optional on the wire**, because most steps need nothing from it. Narrow it with `requireScenarioEnv(scenarioStep)` from `#pikku/scenario` rather than a local guard — it names the step and says what to pass. `env` is `{ apiUrl, appUrl? }` from the environment the run targets, and is how a raw-HTTP step learns the target's URL: a step runs in the CLI process, where there is no `variables` service and `process.env` is not the answer.
|
|
59
59
|
- **Steps default to `retries: 0`**, unlike ordinary workflow steps. Retrying a failed assertion is wrong; pass `retries` explicitly if a step is genuinely flaky-by-nature.
|
|
60
60
|
- **Step results are persisted**, so return JSON-serialisable data — never a `Locator` or a client object.
|
|
61
61
|
- **`description` documents the step; `template` is what the report renders.** `template`'s `{placeholders}` are filled from the input the step was called with, so one step reads differently for each call — `sees {state} addon {packageName}` reports as "sees available addon @pikku/addon-stripe". Reflect every input field in the template, and type the values so they read as words (`state?: 'installed' | 'available'`, not `installed?: boolean`). A placeholder with no value renders as nothing and the whitespace collapses.
|
|
62
|
-
- **Never write the
|
|
63
|
-
- Prose precedence is `options.description` → the step's `template` → the step's own `description` → the positional step name. Repeated names get `#1`, `#2` ordinals, so a `for` loop over a data set is how you write a Scenario Outline. A loop-generated step name is not statically known, so it is matched back to its declaration by step function instead — which works as long as that function's call sites agree on their phase,
|
|
62
|
+
- **Never write the virtual user into the prose.** The reporter renders the virtual user as the sentence's subject, so a step authored as `` `'sam' creates the client` `` run as `{ virtualUser: virtualUsers.sam }` reads "Given sam 'sam' creates the client" — and the hardcoded name desyncs the moment the call site changes virtual user. Write a bare third-person predicate (`creates the {name} client`) and let the virtual user supply the subject. Prose that opens with its own virtual user's key — quoted, capitalised or possessive — is `PKU681`; naming someone **else** mid-sentence ("sends nadia an invite") is ordinary prose and is left alone, as is a virtual user keyed after a role noun used as a noun ("creates the admin client" as `virtualUsers.admin`).
|
|
63
|
+
- Prose precedence is `options.description` → the step's `template` → the step's own `description` → the positional step name. Repeated names get `#1`, `#2` ordinals, so a `for` loop over a data set is how you write a Scenario Outline. A loop-generated step name is not statically known, so it is matched back to its declaration by step function instead — which works as long as that function's call sites agree on their phase, virtual user and prose. Two call sites that disagree make the loop step report under its bare runtime name.
|
|
64
64
|
|
|
65
65
|
## What a step is given
|
|
66
66
|
|
|
@@ -82,7 +82,7 @@ not.
|
|
|
82
82
|
`guardRpc` whose every member refuses:
|
|
83
83
|
|
|
84
84
|
> Scenario tried to run 'getOrder' as an internal step. Every workflow.do in a
|
|
85
|
-
> scenario must carry {
|
|
85
|
+
> scenario must carry { virtualUser: virtualUsers.x } so it executes against 'local'.
|
|
86
86
|
|
|
87
87
|
The same guard covers `rpc.agent.run/stream/resume/approve/interrupt` and
|
|
88
88
|
`startWorkflow`.
|
|
@@ -90,10 +90,10 @@ The same guard covers `rpc.agent.run/stream/resume/approve/interrupt` and
|
|
|
90
90
|
This is the design, not a gap: **everything a step touches of the application
|
|
91
91
|
goes over the wire as somebody.** A test that could reach into the database
|
|
92
92
|
would be testing a different program from the one a person uses. So there are
|
|
93
|
-
exactly three ways in, and they are all through the
|
|
93
|
+
exactly three ways in, and they are all through the virtual user:
|
|
94
94
|
|
|
95
|
-
- `
|
|
96
|
-
|
|
95
|
+
- `virtualUser.invoke(name, data)` — typed over the exposed RPC map, carrying the
|
|
96
|
+
virtual user's session. Declare `virtualUser: true` and destructure it off the wire.
|
|
97
97
|
- `.invokeRaw(name, data, { headers })` — same call, reporting
|
|
98
98
|
`{ status, ok, body }`, for when the refusal is the assertion.
|
|
99
99
|
- a plain `fetch` against `requireScenarioEnv(scenarioStep).apiUrl`, for
|
|
@@ -108,9 +108,10 @@ Two consequences follow, and both shape how steps get written:
|
|
|
108
108
|
- **`agentRunner` is conditional.** It is built only when the project declares
|
|
109
109
|
agents, and `createDevAgentRunner` needs a base URL _and_ a key together
|
|
110
110
|
(`OPENAI_BASE_URL` + `OPENAI_API_KEY`, or the LiteLLM pair). With a key alone
|
|
111
|
-
it returns nothing and `agentRunner` is `undefined`, so `
|
|
112
|
-
fails before the
|
|
111
|
+
it returns nothing and `agentRunner` is `undefined`, so `virtualUser.converse`
|
|
112
|
+
fails before the virtual user says anything. A suite that would rather own its own
|
|
113
113
|
model can pass an `llm` to `runConversation` instead of relying on this one.
|
|
114
|
+
|
|
114
115
|
---
|
|
115
116
|
|
|
116
117
|
## Steps describe intent, not actions
|
|
@@ -193,7 +194,7 @@ export const buysTheItem = pikkuScenarioStep<
|
|
|
193
194
|
name: 'buysTheItem',
|
|
194
195
|
description: 'finds one item in the shop and puts it in the basket',
|
|
195
196
|
template: 'buys the {name}',
|
|
196
|
-
// One intent, one implementation per surface
|
|
197
|
+
// One intent, one implementation per surface a virtual user can drive it through.
|
|
197
198
|
browser: async (_services, { name }, { browser }) => {
|
|
198
199
|
await ensureOnShop(browser)
|
|
199
200
|
await searchFor(browser, name)
|
|
@@ -215,7 +216,7 @@ await scenario.when(
|
|
|
215
216
|
'buys a milkshake',
|
|
216
217
|
'buysTheItem',
|
|
217
218
|
{ name: '£5 strawberry milkshake' },
|
|
218
|
-
{
|
|
219
|
+
{ virtualUser: virtualUsers.shopper }
|
|
219
220
|
)
|
|
220
221
|
// reporter renders: When shopper buys the £5 strawberry milkshake ✓ 1.2s
|
|
221
222
|
```
|
|
@@ -254,7 +255,7 @@ const order = await scenario.do(
|
|
|
254
255
|
'Shopper checks out',
|
|
255
256
|
'createOrder',
|
|
256
257
|
{ basketId, shippingAddress },
|
|
257
|
-
{
|
|
258
|
+
{ virtualUser: virtualUsers.shopper }
|
|
258
259
|
)
|
|
259
260
|
```
|
|
260
261
|
|
|
@@ -282,9 +283,9 @@ export const seesTheOrderConfirmed = pikkuScenarioStep<
|
|
|
282
283
|
{ status: string }
|
|
283
284
|
>({
|
|
284
285
|
name: 'seesTheOrderConfirmed',
|
|
285
|
-
// Both bindings run as the
|
|
286
|
-
// injects `wire.
|
|
287
|
-
|
|
286
|
+
// Both bindings run as the virtual user, so the step declares one and the runner
|
|
287
|
+
// injects `wire.virtualUser` — non-optional in every binding.
|
|
288
|
+
virtualUser: true,
|
|
288
289
|
template: 'sees order {orderId} confirmed',
|
|
289
290
|
// Both run on `--run browser`. Each returns what it observed, and the runner
|
|
290
291
|
// compares them — so this fails when the page disagrees with the database.
|
|
@@ -293,9 +294,9 @@ export const seesTheOrderConfirmed = pikkuScenarioStep<
|
|
|
293
294
|
.locate({ testId: 'order-status', where: { 'data-order': orderId } })
|
|
294
295
|
.getAttribute('data-status'),
|
|
295
296
|
}),
|
|
296
|
-
// Through the
|
|
297
|
-
default: async (_services, { orderId }, {
|
|
298
|
-
status: (await
|
|
297
|
+
// Through the virtualUser, not through a `rpc` service — see "What a step is given".
|
|
298
|
+
default: async (_services, { orderId }, { virtualUser }) => ({
|
|
299
|
+
status: (await virtualUser.invoke('getOrder', { orderId })).status,
|
|
299
300
|
}),
|
|
300
301
|
})
|
|
301
302
|
```
|
|
@@ -304,7 +305,7 @@ Three rules follow, and they are the ones that get broken:
|
|
|
304
305
|
|
|
305
306
|
- **A browser witness must observe on the page.** One that quietly calls an RPC to check the result is worse than no binding at all — it reports a tick for a surface it never looked at.
|
|
306
307
|
- **Return what you observed, don't just assert.** A witness returning a value lets the runner diff the two. A witness that only throws still works, but it can never disagree with anything, so it proves less. Read structured state with `where` on the test-id selector rather than parsing translated copy.
|
|
307
|
-
- **A step with no binding for the run's surface is counted, not excused.** `--run browser` prints `n/m steps ran on browser` over _every_ step, so an action that quietly fell back to the server lowers the number just as an assertion does. A `then` that fell back is additionally named — `--strict` fails on those, because a sentence saying the
|
|
308
|
+
- **A step with no binding for the run's surface is counted, not excused.** `--run browser` prints `n/m steps ran on browser` over _every_ step, so an action that quietly fell back to the server lowers the number just as an assertion does. A `then` that fell back is additionally named — `--strict` fails on those, because a sentence saying the virtual user saw something nobody looked at is a different problem from a shortcut. Not being in the UI _is_ the finding: do not add a browser binding that fakes it.
|
|
308
309
|
|
|
309
310
|
**Always give a `then` a `default` witness.** It is the floor every run can fall back to, and an assertion with no witness the run can execute is fatal (`ScenarioNoWitness`) — not a coverage gap. The distinction is the point: a `then` checked server-side under `--run browser` did happen, it just wasn't seen where the prose claims; one checked nowhere never happened at all, and without the error it would return `undefined` and render as a tick. A browser-only `then` is therefore a step that fails the fast suite, which is rarely what you want.
|
|
310
311
|
|
|
@@ -343,9 +344,9 @@ export const buysAnApple = pikkuScenarioStep<
|
|
|
343
344
|
name: 'buysAnApple', // identifier — English, always
|
|
344
345
|
description: 'kauft einen Apfel', // prose — follows locale
|
|
345
346
|
template: 'kauft {qty} Äpfel', // prose — follows locale
|
|
346
|
-
|
|
347
|
-
default: async (_services, { qty }, {
|
|
348
|
-
await
|
|
347
|
+
virtualUser: true,
|
|
348
|
+
default: async (_services, { qty }, { virtualUser }) =>
|
|
349
|
+
await virtualUser.invoke('placeOrder', { qty }),
|
|
349
350
|
})
|
|
350
351
|
```
|
|
351
352
|
|
|
@@ -356,11 +357,11 @@ A product with a non-English UI is not on its own a reason to set `metaLocale`
|
|
|
356
357
|
is the app's language, not the team's. Ask, or leave it `en`.
|
|
357
358
|
|
|
358
359
|
**Where a non-`en` `metaLocale` still shows English, today.** The reporter composes a
|
|
359
|
-
sentence as `<Keyword> <
|
|
360
|
+
sentence as `<Keyword> <virtual user> <template>` (`composeStepProse`), and the keyword is
|
|
360
361
|
an English literal. The Console translates the Given/When/Then keywords into its own
|
|
361
362
|
UI language; the CLI reporter does not, so `metaLocale: "de"` gives you German step
|
|
362
363
|
prose inside an English frame — `Given shopper kauft 1 Äpfel`. Write templates that read
|
|
363
364
|
acceptably in that frame rather than trying to defeat it. A second gap: where a
|
|
364
365
|
function or scenario declares no `title`, the Console falls back to splitting the
|
|
365
366
|
**identifier** into an English-looking label (`toEnglishName`), so under a
|
|
366
|
-
non-`en` `metaLocale` meta is worth authoring rather than leaving to the fallback.
|
|
367
|
+
non-`en` `metaLocale` meta is worth authoring rather than leaving to the fallback.
|