@arjunkhera/atlas 0.3.6 → 0.3.8

package/skills/tests/SKILL.md

@@ -0,0 +1,234 @@
+---
+name: tests
+description: >
+  Scenario tests for an area of a repo. Trigger when the owner asks to put tests on an area,
+  to review the words of a scenario, to write or convert a test from a scenario, when a test
+  is red, or to prove that tests catch faults. It holds five procedures: onboard an area,
+  review words, convert, sort a red test, and plant faults.
+---
+
+# Tests
+
+A scenario is a Markdown file. It says what must be true. A test is code
+that checks it. Atlas ships the code that joins the two: the contract
+library, the drivers, the stand-in helper and the link check. This skill
+holds the steps to use them.
+
+## The files of an area
+
+| File | What it holds | Who changes it |
+|---|---|---|
+| `atlas/tests.yaml` | How to start the product, the ways in, the stand-ins, the personas, the guards | You, at onboarding. The owner approves its guard parts |
+| `scenarios/<id>.md` | The words: start, steps, waits, case tables, assertions | The design. The builder copies the approved words |
+| `scenarios/map.md` | How to reach each thing: calls, page parts, error forms, gaps | The builder, as it learns facts |
+| `test/atlas/` | The Atlas kit, copied byte for byte | Only `atlas tests write` |
+| `test/scenarios/` | One test file for each scenario | The builder |
+| `test/support/` | The stand-ins, the actions, the makers, the adapter | The builder |
+| `test/evidence/` | The evidence of each run | The library. Git ignores it |
+
+In this skill, "you" is the agent that runs the skill. It is never the owner.
+
+Never edit a file in `test/atlas/`. A hand edit stops the next
+`atlas tests write`, and `atlas tests check` reports it. A person merges
+every file that command writes.
+
+## The three commands
+
+1. `atlas tests write --root <repo>` puts the kit in `test/atlas/`. It
+   refuses to replace a file that you edited, and it refuses to write
+   through a link. It writes `test/evidence/.gitignore`. It adds a `.env.test`
+   line to the `.gitignore` at the repo root. It says what it added.
+2. `atlas tests check --root <repo>` runs the link check. It prints one line
+   for each finding, as `file:line  code  words`. Run it before you ask for a merge.
+3. `atlas tests proof --root <repo>` prints the proof table of one run.
+
+The link check has three halves. Run the first two before the tests and the
+third after them:
+
+1. `--halves static,lint` reads the scenarios, `tests.yaml` and the code. It
+   also checks each file in `test/atlas/` against `manifest.json`. The manifest
+   must match a kit that a release of Atlas shipped.
+2. `--halves run --run <id>` reads the evidence of that one run. With no
+   `--run`, it reads the newest run of each scenario, and `atlas tests proof`
+   does the same. It fails on an assertion that the run did not check. It fails
+   on evidence of an older version of a scenario file.
+3. `--guards-from <file>` takes `tests.yaml` from the main branch. A guard
+   part that differs is a finding. CI must run the check from the main branch.
+
+The environment variable `ATLAS_GUARDS_FROM` names that file for both the
+check and the scenario run. A scenario run whose guard parts differ from
+that file ends `blocked` and starts nothing. A local run without it uses the branch file.
+
+## Run the tests in CI
+
+CI runs the tests from the main branch, so a pull request cannot change its own judge.
+
+1. Fetch the main branch.
+2. Run the copy of `<area>/test/atlas/` that main holds. If main has none
+   yet, run the copy of the branch, and print a warning that says so.
+3. Set `ATLAS_GUARDS_FROM` to `atlas/tests.yaml` of the main branch.
+4. Run the scenarios with the runner command of `tests.yaml`. Set `ATLAS_RUN_ID`
+   or let CI make the run id.
+5. Run the link check with `--run <that run id>`.
+6. Upload `<area>/test/evidence/` for 14 days.
+
+## Onboard an area
+
+Path 1 is for an area that has integration tests which call its routes,
+and which people keep using. Old unit tests that call functions do not make path 1.
+Use path 2: the area has no integration test that calls its routes. If it has
+route tests that people keep, stop. Tell the owner that path 1 comes later.
+
+1. Send the `atlas:code-explorer` crew to map the area. Ask for routes,
+   tools, pages, storage, calls to outside services and existing tests.
+2. Run `atlas tests write --root <repo>`.
+3. Draft `atlas/tests.yaml`. Give it `local` and `ci`, each way in, a persona
+   for each caller, and `guards.hosts` with local addresses only.
+4. Give the product no real secret. Write `{run.secret}` for each secret
+   of the product. The library makes it new for each run.
+5. Write a stand-in for each outside service. Use `serveStandIn` from
+   `test/atlas/stand-in.mjs`. Give the product a fake key for the real service.
+6. Write the contract scenario of each stand-in (kind `contract`). It proves
+   that the stand-in answers like the real service.
+7. Put two personas on one secret only with `same-identity` on both.
+8. Draft `scenarios/map.md` from the code digest and the docs of the repo.
+9. Write two or three first scenarios. Start with the smallest.
+10. Review the words (next section). Run `atlas ste` on each scenario.
+11. Convert each scenario into a test.
+12. Run `atlas tests check --root <repo> --halves static,lint`, then the tests,
+    then `atlas tests check --root <repo> --halves run`.
+13. Ask the `atlas:verifier` crew to start each environment once and run every scenario.
+14. Plant a fault for each assertion (last section).
+15. Open one pull request. Show the owner the first scenarios, and the guard
+    parts of `tests.yaml` as a short list in plain words.
+
+Worked run, on a meal planner with a web page, an MCP server and storage in
+a note service:
+
+1. Input: the code digest names the routes, the MCP server and the page. It
+   finds 11 tests that call functions and none that call a route.
+2. Process: path 2. Draft `tests.yaml` with two environments, three ways in,
+   a stand-in for the note service and four personas. Write the stand-in
+   and two scenarios.
+3. Output: one pull request. The owner reads two scenarios and three lines of guards.
+
+## Write assertions
+
+Use these steps when you write or change an assertion. They come from
+sections 9.2 and 6.10 of the design.
+
+1. Write the definition of done as assertions. A feature gets full scenarios.
+   A small fix gets a short card with its assertions.
+2. Name each assertion that you add, change or remove by its full id in the design.
+3. Write one outcome in each assertion. Then one planted fault points at one assertion.
+4. Mark it `exact` when code checks one right answer. Mark it `judged` only
+   when it is about meaning, and say why in the design.
+5. Never let a judged assertion decide an access rule alone.
+6. Mark `gate` on an assertion that later steps need.
+7. Start a "when" assertion with the step that makes it true: "step 3", or a kept result.
+8. Name what it reads in backticks.
+9. Write a "not" assertion so that a blank answer fails it.
+10. Add `repeat` only when the subject varies by design, such as a model. Say why.
+
+## Review the words
+
+A separate agent reads each new or changed scenario before any code exists.
+It proposes words. It never edits. It asks ten questions of each assertion:
+
+| Problem | The question |
+|---|---|
+| two-readings | Can two careful agents write different tests from it? |
+| not-exact | Does an "exact" assertion have one right answer? |
+| many-claims | Does it hold more than one outcome? |
+| duplicate | Does another assertion check the same thing? |
+| weak | How could the product pass it while the claim is false? |
+| hidden-precondition | Does it need a state that `Start with` does not name? |
+| missing-data | Does it need a value that the scenario does not give? |
+| shared-data | Can another run change the data it reads? |
+| empty-when | Does a "when" assertion name a step that makes it true? |
+| loose-not | Does a "not" assertion pass on a blank or broken answer? |
+
+Then run `atlas ste` on the scenario. The owner reads these pages.
+
+Rules for the words:
+
+1. One outcome for each assertion. Mark `exact` or `judged`.
+2. Mark `gate` on an assertion that later steps need.
+3. A wait ends with "Give up after <time>". The link check refuses it otherwise.
+4. A "when" assertion names its step: "step 3", or a result that a step keeps.
+5. Put each name in backticks. A name is a persona, way in, data rule, setup, fixture, table or kept result.
+6. An access rule is always exact. A judge never decides it alone.
+
+## Convert a scenario into a test
+
+1. Read the scenario, the map, the actions and one worked example. Do not
+   read the product source if the map covers the scenario.
+2. Write one test file for each scenario, with `scenario(path, body, { adapter })`
+   from `test/atlas/contract.mjs`. The library runs the body once for each way in.
+3. Name each assertion by its full id, as a plain string:
+   `run.check('<scenario>/e1#<fingerprint>', () => assert...)`.
+   Use `run.gate` for a `gate` assertion. Get the fingerprints from the failure message of `atlas tests check`.
+4. Use an action, or add one. An action for a new call also adds its row to the map.
+5. Make data with `run.fresh(rule)`. Read a case table with `run.table`, `run.cells` or `run.cases`.
+6. Wait with `run.wait` and a limit. Never sleep for a fixed time.
+7. Use `run.notExercised(id, why)` for a "when" assertion that the run never made true.
+8. Mark each guess in a comment. List the guesses in the pull request.
+9. Never change the words to make a test pass. Only the design changes the words.
+10. Run the test before the change. A test for new behaviour must fail.
+
+The adapter binds the library to the product. All its parts are optional:
+`standIns`, `warmUp`, `drivers`, `functions`, `loadModule`, `makers`, `signIns`,
+`data` and `actions`. The header of `test/atlas/contract.mjs` describes each one.
+
+A result is `pass`, `fail`, `not checked`, `not exercised`, `blocked` or
+`not here`. A test is red when any assertion is not `pass`. The one
+exception is `not here`: the scenario does not run in that environment, it is
+not counted, and the test passes.
+
+After the product is ready, a lost connection, a server exit or a time-out is
+`fail`. `blocked` is for a guard refusal, a start-up fault and the first-call check.
+
+## Sort a red test
+
+Do not edit a red test before you know why it is red. There are four causes:
+
+| Cause | Sign | What happens | What may change |
+|---|---|---|---|
+| The product broke | The words still hold. The product does not | Fix the code | Product code only |
+| The behaviour changed on purpose | The approved design names the assertion | Change the assertion and its test | The assertions that the design names, and their tests |
+| The test is out of date | A page part moved, or the form of an answer changed. The verifier's own check passes | Repair the actions, the map or the finders | Actions, map and finders. No assertion and no assert statement |
+| The environment failed | The result is `blocked`: the product did not start, or a stand-in is down | Fix the setup | `tests.yaml` without its guard parts, and the support code |
+
+Two checks follow:
+
+1. A repair that changes an assertion or an assert statement is not a repair. Refuse it unless the design names the assertion.
+2. A repair must keep the verifier's own check green. The verifier does not use the actions.
+
+A `blocked` result because a guard refused, or because the stand-in saw no
+first call from the product, is a finding, not noise.
+Find out why the product reached for something else.
+
+## Plant faults
+
+A fault proves that an assertion can go red. Plant one for each new or
+changed assertion, and for each assertion at onboarding.
+
+1. Commit first. A clone holds committed files only, so a change that is not
+   committed is not in the clone. Then clone the worktree to a throwaway
+   folder outside it, with `git clone --no-hardlinks`.
+2. Remove every remote from the clone. Run `git remote` there. It must print nothing.
+3. Plant faults in product code only. Never touch `tests.yaml`, the support code or the kit.
+4. Use the cheap model first: Haiku. After two failed tries on one assertion, use Sonnet.
+5. Give the planter one assertion and the product code. Ask for one fault that makes only that claim false.
+6. Run the scenario tests from the real worktree. Set `ATLAS_PRODUCT_ROOT` to
+   the area folder in the clone, and `ATLAS_FAULT_RUN` to a label. The product,
+   the MCP server and the module of the `function` driver start in the clone.
+   The guards, the kit and `tests.yaml` load from the worktree.
+7. The test for that assertion must go red.
+8. Ask the `atlas:verifier` crew for its own check of that assertion, on the same clone. It uses a raw driver, not the actions.
+9. A fault is valid only when the verifier's own check also fails. Discard any other fault and try again.
+10. Report each assertion as caught, missed, or "no fault of its own" after Sonnet also fails.
+11. Delete the clone. Run `git status` in the worktree. It must show no change from the faults.
+
+Evidence from a fault run carries the label. The link check ignores it, so it never counts as proof.
+Never push from the clone.

package/tests/contract.mjs

@@ -0,0 +1,398 @@
+// The contract library (design section 8.1). It sits between the words of a
+// scenario and the test code, on `node:test`.
+//
+//   // in test/scenarios/week-conflict.test.mjs, with the kit copy in test/atlas/
+//   scenario('scenarios/week-conflict.md', async (run, way) => {
+//     const first = await run.persona('planner-1', way);
+//     // ... steps, through the actions of the area ...
+//     run.gate('week-conflict/e1#be9cf0', () => assert.equal(firstAnswer.revision, 1));
+//     run.check('week-conflict/e2#7f1d74', () => assert.equal(secondAnswer.revision, 2));
+//   }, { adapter });
+//
+// What `scenario(path, body, options)` does
+//   1. Reads the scenario and atlas/tests.yaml, and refuses to run when either
+//      has a fault.
+//   2. Registers one node:test test for each way in of `through:`. It runs the
+//      body with (run, way).
+//   3. Starts the environment once for the scenario (environment.mjs): the
+//      stand-ins, then the product. It stops them when the last way is done.
+//   4. Writes the evidence file (evidence.mjs): at the start, and after each
+//      assertion.
+//   5. A test fails (node:test red) when any assertion is not `pass`. That
+//      holds for `fail`, `not checked`, `not exercised` and `blocked`.
+//      When the scenario's `runs-in` excludes the environment, the result is
+//      `not here` and the test passes.
+//
+// Guard parts at run time: with ATLAS_GUARDS_FROM=<tests.yaml of the main
+// branch>, a branch whose guard parts differ ends `blocked` and starts nothing.
+// After the body and the results are settled, the run is closed: a late
+// run.check throws and changes nothing. After the product is ready, a lost
+// connection, a server exit or a time-out is `fail`, not `blocked`.
+//
+// The adapter binds the library to one product. Every part is optional:
+//   adapter.standIns  { name: async ({ name, runId, hosts }) => ({ url, stop, calls }) }
+//                     a stand-in made in this process, instead of a start command
+//   adapter.warmUp    async (env) => {}   make the product call its stand-ins once
+//   adapter.drivers   { wayOrDriver: async (ctx) => driver }  a driver the kit does not ship
+//   adapter.functions the module (or { way: module }) for the function driver
+//   adapter.loadModule async (file) => module   loads the `module` of a function way in; the
+//                     kit gives the path under the product root, so a fault run loads the copy
+//   adapter.makers    { name: async (ctx) => ({ credential, headers, cleanup }) } for `make:`
+//   adapter.signIns   { name: async (ctx) => ({ credential, headers, cleanup }) } for `sign-in:`
+//   adapter.data      { rule: ({ runId, count }) => value }  a data rule of the repo
+//   adapter.actions   an object, or (run) => object: the verbs of the area, on run.actions
+import { describe, it, after } from 'node:test';
+import { existsSync, mkdirSync, readFileSync } from 'node:fs';
+import { dirname, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { Blocked, GateFailed } from './errors.mjs';
+import { Redactor } from './redact.mjs';
+import { readTestsYaml, checkTestsYaml, compareGuardParts } from './tests-yaml.mjs';
+import { readScenario, addFingerprints, combinations, pairs } from './scenario.mjs';
+import { Evidence, clip } from './evidence.mjs';
+import { makeFresh, makeRunId } from './fresh.mjs';
+import { startEnvironment, provideSecrets } from './environment.mjs';
+import { makeDriver } from './drivers/index.mjs';
+import { parseLimit, waitUntil } from './wait.mjs';
+
+export { waitUntil };
+export const defineAdapter = (adapter) => adapter;
+
+const FULL_ID = /^(.+)\/(e\d+)#([0-9a-f]{6})$/;
+const isThenable = (x) => x && typeof x.then === 'function';
+
+export const RUN_ID = makeRunId();
+
+function locate(path) {
+  if (path instanceof URL) return fileURLToPath(path);
+  if (typeof path === 'string' && path.startsWith('file:')) return fileURLToPath(path);
+  return resolve(String(path));
+}
+
+function findRoot(file) {
+  let dir = dirname(file);
+  for (;;) {
+    if (existsSync(join(dir, 'atlas', 'tests.yaml'))) return dir;
+    const up = dirname(dir);
+    if (up === dir) throw new Error(`no atlas/tests.yaml above ${file}`);
+    dir = up;
+  }
+}
+
+function load(path, options) {
+  const file = locate(path);
+  const root = findRoot(file);
+  const yaml = readTestsYaml(join(root, 'atlas', 'tests.yaml'));
+  const findings = checkTestsYaml(yaml);
+  if (findings.length) throw new Error(`atlas/tests.yaml: ${findings.map((f) => `line ${f.line}: ${f.message}`).join('; ')}`);
+  const sc = readScenario(file);
+  if (sc.problems.length) throw new Error(`${path}: ${sc.problems.map((p) => `line ${p.line}: ${p.message}`).join('; ')}`);
+  const tests = yaml.value;
+  addFingerprints(sc, tests);
+  // ATLAS_GUARDS_FROM names tests.yaml of the main branch. The guard parts of this
+  // run come from that file, and a branch whose guard parts differ is refused
+  // (`blocked`). CI sets it. A local run without it uses the branch file.
+  let guardRefusal = null;
+  const guardsFrom = options.guardsFrom ?? process.env.ATLAS_GUARDS_FROM;
+  if (guardsFrom) {
+    try {
+      const changes = compareGuardParts(readFileSync(guardsFrom, 'utf8'), yaml.value);
+      if (changes.length) guardRefusal = `the guard parts of atlas/tests.yaml differ from the main branch copy (${changes.map((c) => c.part).join(', ')}); the run is refused. A change to a guard part comes to the owner`;
+    } catch (error) {
+      guardRefusal = `ATLAS_GUARDS_FROM names a file that cannot be read as tests.yaml (${error.code ?? error.message}); the run is refused`;
+    }
+  }
+  const environment = options.environment ?? process.env.ATLAS_ENV ?? (process.env.CI ? 'ci' : 'local');
+  const evidenceDir = options.evidenceDir ?? process.env.ATLAS_EVIDENCE_DIR ?? join(root, 'test', 'evidence');
+  return { file, root, tests, sc, environment, evidenceDir, guardRefusal };
+}
+
+const listOf = (x) => (Array.isArray(x) ? x.map(String) : x === undefined ? [] : [String(x)]);
+
+export function scenario(path, body, options = {}) {
+  const ctx = load(path, options);
+  const { sc, tests, root, environment } = ctx;
+  const adapter = options.adapter ?? {};
+  const runsHere = listOf(sc.fields['runs-in']).includes(environment);
+  const limitMs = parseLimit(sc.fields.limit);
+
+  describe(`${sc.id}: ${sc.title}`, () => {
+    const state = {
+      redactor: new Redactor(), evidence: null, envPromise: null, env: null,
+      drivers: new Map(), personas: new Map(), cleanups: [], fresh: null,
+    };
+    const evidence = () => {
+      state.evidence ??= new Evidence({
+        evidenceDir: ctx.evidenceDir, runId: RUN_ID, environment, scenario: sc, root,
+        redactor: state.redactor, providedSecrets: [],
+      });
+      return state.evidence;
+    };
+    const getEnv = () => {
+      state.envPromise ??= (async () => {
+        if (ctx.guardRefusal) throw new Blocked(ctx.guardRefusal);
+        const named = [...new Set([...sc.start, ...sc.steps, ...sc.assertions].flatMap((x) => x.names))]
+          .filter((n) => Object.hasOwn(tests.secrets ?? {}, n));
+        const provided = provideSecrets(named, { root, tests });
+        const env = await startEnvironment({ root, tests, name: environment, runId: RUN_ID, redactor: state.redactor, adapter, provided });
+        evidence().record.provided_secrets = named;
+        state.env = env;
+        return env;
+      })();
+      return state.envPromise;
+    };
+
+    if (!runsHere) {
+      it(`${sc.id} [not here: runs-in excludes ${environment}]`, () => {
+        const ev = evidence();
+        for (const way of sc.through) {
+          for (const a of sc.assertions) ev.setResult(a.fullId, way, 'not here', { why: `runs-in does not list ${environment}` });
+          ev.setWay(way, { verdict: 'not here', started: new Date().toISOString(), ended: new Date().toISOString() });
+        }
+      });
+      return;
+    }
+
+    for (const way of sc.through) {
+      it(`${sc.id} [${way}]`, async () => {
+        const ev = evidence();
+        ev.setWay(way, { started: new Date().toISOString() });
+        let run = null;
+        let error = null;
+        try {
+          const env = await getEnv();
+          run = makeRun({ ctx, state, way, env, ev, adapter });
+          await withLimit(Promise.resolve().then(() => body(run, way)), limitMs, sc.fields.limit);
+        } catch (thrown) {
+          error = thrown;
+        }
+        const outcome = settle({ sc, way, ev, error });
+        // The run is closed: a late call (after a time limit, say) changes nothing.
+        run?.close();
+        await cleanUp(state, run, ev);
+        // The problem list goes to node:test and then to the CI log: redact it first.
+        if (outcome.problems.length) throw new Error(state.redactor.text(outcome.problems.join('\n')));
+      });
+    }
+
+    after(async () => {
+      for (const driver of state.drivers.values()) { try { await (await driver).close(); } catch { /* the stop goes on */ } }
+      try { await (await state.envPromise)?.stop(); } catch { /* it did not start */ }
+    });
+  });
+}
+
+function withLimit(promise, ms, text) {
+  let timer;
+  const limit = new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(`the scenario ran out of time: its limit is ${text}`)), ms); timer.unref?.(); });
+  return Promise.race([promise, limit]).finally(() => clearTimeout(timer));
+}
+
+async function cleanUp(state, run, ev) {
+  const list = run ? run.cleanups.splice(0).reverse() : [];
+  for (const fn of list) { try { await fn(); } catch (error) { ev.note(`clean-up failed: ${error.message}`); } }
+}
+
+// Turns the end of a body into results, a verdict and a list of problems.
+function settle({ sc, way, ev, error }) {
+  const mine = ev.record.assertions.filter((a) => a.way === way);
+  const problems = [];
+  let verdict = 'pass';
+  let reason = null;
+  // `blocked` is for a guard refusal, a start-up fault and the first-call check. After
+  // the product is ready, a lost connection, a server exit or a timeout is a `fail`.
+  if (error instanceof Blocked) {
+    verdict = 'blocked';
+    reason = error.message;
+    for (const a of mine) if (a.result === 'not checked') { a.result = 'blocked'; a.proof = { why: reason }; }
+  } else if (error && !(error instanceof GateFailed)) {
+    verdict = 'fail';
+    reason = error.message;
+  } else if (error instanceof GateFailed) {
+    verdict = 'fail';
+    reason = 'a gate assertion failed, so the run stopped';
+  }
+  for (const a of mine) {
+    if (a.result === 'pass' || a.result === 'not here') continue;
+    if (verdict === 'pass') verdict = 'fail';
+    problems.push(`[${a.id}] ${a.result}${a.proof?.error ? `: ${a.proof.error}` : a.proof?.why ? `: ${a.proof.why}` : ''}`);
+  }
+  if (verdict === 'blocked') problems.unshift(`BLOCKED: ${reason}`);
+  else if (reason && !(error instanceof GateFailed)) problems.push(`the run stopped: ${reason}`);
+  ev.setWay(way, { verdict, reason, ended: new Date().toISOString() });
+  return { verdict, problems };
+}
+
+// ---------------------------------------------------------------- the run object
+
+function makeRun({ ctx, state, way, env, ev, adapter }) {
+  const { sc, tests, root } = ctx;
+  state.fresh ??= makeFresh(RUN_ID, tests.data, adapter.data);
+  const cleanups = [];
+  let mark = ev.record.calls.length;
+  const callsSince = () => { const from = mark; mark = ev.record.calls.length; return ev.record.calls.slice(from).map((c) => c.n); };
+
+  let closed = false;
+  const open = (what) => { if (closed) throw new Error(`the run is closed: ${what} came after the scenario ended, so it changed nothing`); };
+  const record = (call) => (closed ? undefined : ev.addCall(call));
+
+  const driver = (wayName = way) => {
+    if (!state.drivers.has(wayName)) {
+      const spec = tests['ways-in'][wayName];
+      if (!spec) throw new Error(`there is no way in "${wayName}" in tests.yaml`);
+      state.drivers.set(wayName, makeDriver({ way: wayName, spec, env, root, hosts: env.hosts, redactor: state.redactor, record, adapter, runId: RUN_ID }));
+    }
+    return state.drivers.get(wayName);
+  };
+
+  const find = (full) => {
+    const m = FULL_ID.exec(String(full));
+    if (!m) throw new Error(`"${full}" is not a full assertion id like ${sc.id}/e1#abc123`);
+    if (m[1] !== sc.id) throw new Error(`"${full}" names the scenario "${m[1]}"; this is "${sc.id}"`);
+    const a = sc.assertions.find((x) => x.id === m[2]);
+    if (!a) throw new Error(`the scenario has no assertion ${m[2]}`);
+    return { a, fp: m[3] };
+  };
+
+  const settleOne = (full, fn, opts = {}, mode) => {
+    open(`run.${mode}`);
+    const { a, fp } = find(full);
+    const target = opts.way ?? way;
+    if (!sc.through.includes(target)) throw new Error(`"${target}" is not a way in of this scenario`);
+    const entry = ev.entry(a.fullId, target);
+    const calls = callsSince();
+    let failure = null;
+    let value;
+    if (fp !== a.fp) failure = `the fingerprint ${fp} is old: the words of ${a.id} now print ${a.fp}; convert the test again`;
+    else if (mode === 'exercise') {
+      if (entry.result === 'pass' || entry.result === 'fail') return a;
+      ev.setResult(a.fullId, target, 'not exercised', { why: String(fn), calls });
+      return a;
+    } else {
+      try {
+        value = fn();
+        if (isThenable(value)) { failure = 'a check must not be async: wait first, then check'; value = undefined; }
+      } catch (e) { failure = e?.message ?? String(e); }
+    }
+    if (failure) ev.setResult(a.fullId, target, 'fail', { error: failure, calls });
+    else if (entry.result !== 'fail') ev.setResult(a.fullId, target, 'pass', { value: value === undefined ? undefined : clip(value, state.redactor), calls });
+    if (failure && mode === 'gate') throw new GateFailed(`[${a.id}] ${failure}`);
+    return a;
+  };
+
+  const tableOf = (name) => {
+    const t = sc.tables.get(name);
+    if (!t) throw new Error(`the scenario has no case table ${name}`);
+    return t;
+  };
+  const checkTable = (t, expected) => {
+    ev.record.tables[t.name] = t.fingerprint;
+    if (expected && expected !== t.fingerprint) throw new Error(`the table ${t.name} now prints ${t.fingerprint}, not ${expected}; its rows changed`);
+  };
+
+  const run = {
+    id: sc.id, title: sc.title, runId: RUN_ID, way, env, scenario: sc, tests, root, cleanups,
+    redact: (text) => state.redactor.text(text),
+    driver,
+
+    // One assertion. `fn` runs at once, must not be async, and must hold an
+    // assert statement. A failure is recorded and the run goes on.
+    check: (full, fn, opts) => { settleOne(full, fn, opts, 'check'); },
+    // The same, and a failure stops the run. Later assertions read `not checked`.
+    gate: (full, fn, opts) => { settleOne(full, fn, opts, 'gate'); },
+    // A "when" assertion whose situation never happened in this run.
+    notExercised: (full, why) => { settleOne(full, why, {}, 'exercise'); },
+    // The test could not run. Never use it for a product result.
+    blocked: (why) => { throw new Blocked(why); },
+
+    // The rows of a rule table, as objects keyed by the header. A test that
+    // reads a table gets every row of the file at run time.
+    table(name, { fingerprint } = {}) {
+      const t = tableOf(name);
+      checkTable(t, fingerprint);
+      return t.rows.map((row) => ({ ...row }));
+    },
+    // The cells of a rule table: one case each, with a label that names the cell.
+    cells(name, { fingerprint } = {}) {
+      const t = tableOf(name);
+      if (t.kind !== 'rule') throw new Error(`${name} is a combination table; use run.cases`);
+      checkTable(t, fingerprint);
+      const [first, ...rest] = t.header;
+      return t.rows.flatMap((row, rowIndex) => rest.map((column) => ({ row: row[first], column, value: row[column], rowIndex, label: `${name}: ${row[first]} / ${column}` })));
+    },
+    // Every case of a combination table: every combination, or a set that holds every pair.
+    cases(name, { fingerprint } = {}) {
+      const t = tableOf(name);
+      if (t.kind !== 'combination') throw new Error(`${name} is a rule table; use run.table or run.cells`);
+      checkTable(t, fingerprint);
+      return (t.cover === 'pair' ? pairs(t.dimensions) : combinations(t.dimensions)).map((one, i) => ({ ...one, label: `${name} case ${i + 1}` }));
+    },
+
+    // A test user from tests.yaml, made or loaded for this run.
+    async persona(name, wayName = way) {
+      const key = `${name}\u0000${wayName}`;
+      if (state.personas.has(key)) return state.personas.get(key);
+      const def = tests.personas?.[name];
+      if (!def) throw new Error(`there is no persona "${name}" in tests.yaml`);
+      const made = (async () => {
+        const drv = await driver(wayName);
+        let extra = {};
+        const maker = def.make ? adapter.makers?.[def.make] : def['sign-in'] ? adapter.signIns?.[def['sign-in']] : null;
+        if ((def.make || def['sign-in']) && !maker) throw new Blocked(`persona "${name}" needs the ${def.make ? 'maker' : 'sign-in'} "${def.make ?? def['sign-in']}" and the adapter does not give it`);
+        if (maker) {
+          extra = (await maker({ run, name, def, way: wayName, env, driver: drv, fresh: run.fresh, persona: (n) => run.persona(n, wayName) })) ?? {};
+          if (extra.cleanup) cleanups.push(extra.cleanup);
+        }
+        const credential = extra.credential ?? (def.secret ? (env.vars[def.secret] ?? null) : null);
+        state.redactor.add(credential);
+        for (const value of Object.values(extra.headers ?? {})) state.redactor.add(String(value));
+        const persona = {
+          ...extra, name, role: def.role ?? null, way: wayName, credential,
+          call: (request, opts = {}) => drv.call(request, { credential, headers: extra.headers ?? {}, persona: name, ...opts }),
+        };
+        return persona;
+      })();
+      state.personas.set(key, made);
+      return made;
+    },
+
+    // A fresh value from a data rule. It carries the run id; a date is picked from it.
+    fresh: (rule) => {
+      open('run.fresh');
+      const value = state.fresh(rule);
+      ev.record.fresh = state.fresh.made.slice();
+      ev.write();
+      return value;
+    },
+
+    // The wait of a scenario step: wait for a state, never for a fixed time.
+    wait: waitUntil,
+    // The same, and a wait that fails or runs out marks the assertion it
+    // serves as failed, with the last state as proof. The run stops.
+    async waitFor(full, read, opts) {
+      open('run.waitFor');
+      const { a } = find(full);
+      try { return await waitUntil(read, opts); } catch (e) {
+        if (e.name !== 'WaitFailed' && e.name !== 'WaitTimeout') throw e;
+        ev.setResult(a.fullId, way, 'fail', { error: e.message, last: clip(e.last?.body ?? e.last, state.redactor) });
+        throw new GateFailed(`[${a.id}] ${e.message}`);
+      }
+    },
+
+    note: (text) => { open('run.note'); ev.note(state.redactor.text(String(text))); },
+    // Keep a file, such as a screenshot, as evidence. Returns the path to write to.
+    attach(fileName) {
+      open('run.attach');
+      const dir = join(ev.dir, sc.id);
+      mkdirSync(dir, { recursive: true });
+      const path = join(dir, fileName);
+      ev.addFile(path);
+      return path;
+    },
+    onCleanup: (fn) => { cleanups.push(fn); },
+    // Called by the library after the body and the results are settled.
+    close: () => { closed = true; },
+  };
+  run.actions = typeof adapter.actions === 'function' ? adapter.actions(run) : (adapter.actions ?? {});
+  return run;
+}

package/tests/drivers/function.mjs

@@ -0,0 +1,30 @@
+// The function driver. It calls a function or a module directly, for a rule
+// that holds below every way in. It has no address and no credential, so
+// duties 1 and 2 of the driver contract do not apply. It records the inputs
+// and the return or the error, and it opens nothing.
+//
+// The shipped kit cannot load a module by name. The repo loads the module
+// itself and gives it to the driver through the adapter, as the value of
+// adapter.functions (one module, or one module for each way in).
+export function createFunctionDriver({ way, target, record: rawRecord, redactor }) {
+  const record = rawRecord && ((call) => rawRecord(redactor ? redactor.deep(call) : call));
+  if (!target) throw new Error(`the way in "${way}" has no target: give the module in adapter.functions`);
+
+  // request: { fn, args } - or { args } when the target is itself a function.
+  async function call(request = {}, { persona = null } = {}) {
+    const args = request.args ?? [];
+    const fn = request.fn === undefined ? target : target[request.fn];
+    if (typeof fn !== 'function') throw new Error(`the target of "${way}" has no function "${request.fn}"`);
+    let answer;
+    try {
+      const value = await fn.apply(request.fn === undefined ? undefined : target, args);
+      answer = { status: 'ok', body: value, isError: false, raw: value };
+    } catch (error) {
+      answer = { status: 'threw', body: { name: error?.name ?? 'Error', message: String(error?.message ?? error) }, isError: true, raw: error };
+    }
+    record?.({ way, persona, request: { fn: request.fn ?? '(target)', args }, response: { status: answer.status, body: answer.body } });
+    return answer;
+  }
+
+  return { name: 'function', way, call, async close() {} };
+}