@ultimat3/testing 30.6.0 → 30.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +3 -3
- package/package.json +17 -17
- package/src/define-exercise.ts +19 -7
- package/src/errors.ts +6 -6
- package/src/exercise-actor-user.ts +63 -0
- package/src/exercise-actor.ts +36 -7
- package/src/exercise-crash.ts +7 -3
- package/src/exercise-drift-hint.ts +28 -0
- package/src/exercise-input.ts +3 -0
- package/src/exercise-job-tenant.ts +10 -5
- package/src/exercise-override-input.ts +26 -5
- package/src/exercise-report.ts +38 -2
- package/src/exercise-sweeps.ts +49 -7
- package/src/exercise.ts +41 -11
- package/src/index.ts +4 -0
- package/src/job-crash-points.ts +36 -0
- package/src/job-crash-proof.ts +23 -6
- package/src/job-sim-errors.ts +24 -0
- package/src/plan-conditions.ts +106 -0
- package/src/plan-oracle.ts +65 -49
- package/src/plan-sort-key.ts +65 -0
- package/src/plan-sweep-fix.ts +87 -21
- package/src/plan-sweep-report.ts +48 -4
- package/src/seed-graph-values.ts +20 -11
- package/src/seed-graph.ts +30 -1
- package/src/tenant-proof-report.ts +12 -3
- package/src/tenant-proof.ts +32 -4
- package/src/seed-pattern-sample.ts +0 -216
package/CLAUDE.md
CHANGED
|
@@ -45,19 +45,19 @@ import is `registry-isolation.ts`, own entry point, off the barrel. Full rule: `
|
|
|
45
45
|
| `LOG_LEVEL` above `info` filters the TERMINAL | `quiet-logs.ts` pins the logger at `info` while core loads, so a collecting `setLogSink` still sees `warn` under `LOG_LEVEL=error`. It stays the preload's FIRST import |
|
|
46
46
|
| Injection | `SqlRunner` and `connect` are parameters, so unit tests need no server |
|
|
47
47
|
| Fixtures | the preload registers the framework bag — an app registers only what it alone knows (`seed`, `actorFor`) |
|
|
48
|
-
| Every `e2eTest` is audited | `e2e-a11y.ts` (rules), `e2e-a11y-run.ts` (per-test run, `allow`, dedupe), `ax-node.ts` (the one AX reader);
|
|
48
|
+
| Every `e2eTest` is audited | `e2e-a11y.ts` (rules), `e2e-a11y-run.ts` (per-test run, `allow`, dedupe), `ax-node.ts` (the one AX reader); full tree only in `cdp-e2e-page.ts`. No switch; `allow` needs a `because`. Raw tabs: `e2e-a11y-tabs.ts` ([history](../../docs/history/testing.md#a11y-on-raw-tabs)) |
|
|
49
49
|
| The exercise pass runs what is registered | `exercise.ts` (`runExercisePass`): every action, query and job once, its own tenant pair off one `seedGraph`, an actor `exercise-actor.ts` picks, 39's `runExercise` as the verdict. Rows COMMITTED. Overrides only via `defineExercise` (blank `off` refused). A job runs ALONE (`exercise-job-run.ts`): a fresh queue, one `tick()`, what it queued left on the caller's queue unrun — never a `drain()` |
|
|
50
50
|
| An override `input` is awaited, as the call's actor | `exercise-override-input.ts`: run inside the call's own request, so `useContext().actor` is the seated actor; the one way to a signed input. `async` ones are left out of `x drill`'s load |
|
|
51
51
|
| A passkey test signs for real | `softAuthenticator` is `@ultimat3/auth/soft-authenticator`'s, never re-exported here (one value, one path: `flight-copies`) and never a second one (auth's own suites need it, and auth cannot import testing) |
|
|
52
52
|
| A reached job is crash-proven | `exercise-crash.ts`: `crashProof` with the control's input and actor, after the tenant proof; only `diverged` is red (`X_JOB_CRASH_DIVERGED`), its place `crashEffectPlace` (`inside` a step via jobs' `currentStep()`, or `after` one). Its world is `alone`: only the job's own row is claimed, a child is an `enqueue:` effect, never run. Committed: job bodies escape transactions |
|
|
53
53
|
| The tenant proof is taint only | `tenant-proof.ts`: B, A, B; B is A's role minus `tenancy:cross`. Leak = seeded canary or unhanded id (`tenant-canary.ts`); write = moved digest. Only `crossed` is red |
|
|
54
|
-
| Every statement the pass sent is planned | `plan-sweep.ts` (`sweepPlans`): first relation an app table, `PREPARE` in a READ-ONLY transaction with seq/bitmap scans and sorts off, judged by `plan-oracle.ts
|
|
54
|
+
| Every statement the pass sent is planned | `plan-sweep.ts` (`sweepPlans`): first relation an app table, `PREPARE` in a READ-ONLY transaction with seq/bitmap scans and sorts off, judged by `plan-oracle.ts` (keys: `plan-sort-key.ts`, conditions: `plan-conditions.ts`); red = `X_QUERY_UNINDEXED` / `X_QUERY_SORT_UNSERVED` with the `indexes:` entry (`plan-sweep-fix.ts`). `scan:` or `expectedSeqScan` is exempt, listed; credited: a partial unique whose predicate the statement states (one row), a declared equality-then-range index under a limit, every `or` arm a declared index leads; no generic plan is `X_QUERY_PLAN_UNAVAILABLE`, never a pass |
|
|
55
55
|
| A shape that grows with the rows is red | `scale-sweep-run.ts`: two more tenants at `SCALE_SIZES` (2, 8 rows a table), every reached primitive once in each; `judgeScale` (pure) reds `X_QUERY_SCALES_WITH_ROWS`. Under `expectedQueryLoop` it is listed with its reason |
|
|
56
56
|
| An erase is PROVED, not asserted | `subject-proof.ts` (`subjectProof`, run by cli's `exercise-subject.ts`): canaries in every table, one person erased, every stored row searched; a survivor outside `retain` is `X_SUBJECT_ERASE_INCOMPLETE`, naming table and column, never the value |
|
|
57
57
|
| A captured failure replays as a test | `repro.ts`: `reproduce(bundle, { driver })` seeds the rows it read, freezes the clock at its instant, invokes as its actor; masked fields are stood in (`repro-stand-in.ts`). An unregistered name ANSWERS as the failure. `reproJob(runJobs, failure)` for a dead job. What `x repro --write` emits |
|
|
58
58
|
| A vendor's answers are a cassette | `cassette.ts`: replay = one `mockFetch` route per recorded URL, seal kept (`X_CASSETTE_MISSING`, `X_CASSETTE_UNUSED`). Record = `ULTIMATE_CASSETTE=record` + `record.hosts`, refused under `CI` (`X_CASSETTE_RECORD_REFUSED`), headers allow-listed, credential keys masked, bytes scanned before disk (`X_CASSETTE_SECRET`). The request side is never written |
|
|
59
59
|
| One seed writer | `entitySeedWriter({ entities, client \| driver })` (`seed-writer.ts`) writes through each entity's own repository; `snapshot` is required; `entity` loads on the first write, so the barrel stays entity-free |
|
|
60
|
-
| Factory values are DECLARED | `factoriesFor` and `seedGraph` share `seed-graph-values.ts`: default, enum, kind, a `matches()` pattern (`
|
|
60
|
+
| Factory values are DECLARED | `factoriesFor` and `seedGraph` share `seed-graph-values.ts`: default, enum, kind, a `matches()` pattern (schema's `patternSamples`; CHECK read by `seed-graph-patterns.ts`), `$meta.schema` (sampler `require`d on first use), through `$parse`, never the name. `seed-graph*.ts` type-import `entity` only |
|
|
61
61
|
| e2e without a driver | `e2eTest` becomes `test.skip`; the gate reports the step SKIPPED (or `X_VERIFY_SUITE_VANISHED` when `x.verify.json` requires it), never passed |
|
|
62
62
|
| The seam has an inverse | `resetE2eDriver()`: `useE2eDriver` writes MODULE scope, so a file that installs a browser undoes it |
|
|
63
63
|
| Built vs declared | `clock` `mail` `network` `runJobs` `statements` `subscribe` are built in-process; `page` `budget` `signIn` `deploy` are declared and wait for a driver (`X_TEST_FIXTURE_UNAVAILABLE`). |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/testing",
|
|
3
|
-
"version": "30.
|
|
3
|
+
"version": "30.7.0",
|
|
4
4
|
"description": "Test harness: cloned template DBs per worker, frozen clock, sealed network, 6 test types",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ultimate",
|
|
@@ -46,21 +46,21 @@
|
|
|
46
46
|
"test": "bun test"
|
|
47
47
|
},
|
|
48
48
|
"dependencies": {
|
|
49
|
-
"@ultimat3/action": "30.
|
|
50
|
-
"@ultimat3/cache": "30.
|
|
51
|
-
"@ultimat3/core": "30.
|
|
52
|
-
"@ultimat3/db": "30.
|
|
53
|
-
"@ultimat3/entity": "30.
|
|
54
|
-
"@ultimat3/http": "30.
|
|
55
|
-
"@ultimat3/i18n": "30.
|
|
56
|
-
"@ultimat3/jobs": "30.
|
|
57
|
-
"@ultimat3/mail": "30.
|
|
58
|
-
"@ultimat3/policy": "30.
|
|
59
|
-
"@ultimat3/pwa": "30.
|
|
60
|
-
"@ultimat3/query": "30.
|
|
61
|
-
"@ultimat3/realtime": "30.
|
|
62
|
-
"@ultimat3/render": "30.
|
|
63
|
-
"@ultimat3/schema": "30.
|
|
64
|
-
"@ultimat3/time": "30.
|
|
49
|
+
"@ultimat3/action": "30.7.0",
|
|
50
|
+
"@ultimat3/cache": "30.7.0",
|
|
51
|
+
"@ultimat3/core": "30.7.0",
|
|
52
|
+
"@ultimat3/db": "30.7.0",
|
|
53
|
+
"@ultimat3/entity": "30.7.0",
|
|
54
|
+
"@ultimat3/http": "30.7.0",
|
|
55
|
+
"@ultimat3/i18n": "30.7.0",
|
|
56
|
+
"@ultimat3/jobs": "30.7.0",
|
|
57
|
+
"@ultimat3/mail": "30.7.0",
|
|
58
|
+
"@ultimat3/policy": "30.7.0",
|
|
59
|
+
"@ultimat3/pwa": "30.7.0",
|
|
60
|
+
"@ultimat3/query": "30.7.0",
|
|
61
|
+
"@ultimat3/realtime": "30.7.0",
|
|
62
|
+
"@ultimat3/render": "30.7.0",
|
|
63
|
+
"@ultimat3/schema": "30.7.0",
|
|
64
|
+
"@ultimat3/time": "30.7.0"
|
|
65
65
|
}
|
|
66
66
|
}
|
package/src/define-exercise.ts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
// `defineExercise({ publishPost: { input, actor }, sendInvoice: { off: 'why' } })`: an app's
|
|
2
|
-
//
|
|
3
|
-
//
|
|
1
|
+
// `defineExercise({ publishPost: { input, actor }, sendInvoice: { off: 'why' } })`: an app's ONE
|
|
2
|
+
// declaration of each primitive's exercise. `x exercise` reads it, and so does every contract
|
|
3
|
+
// test — the whole value handed to `.contract({ exercise })` (action's `ExerciseDeclarations`).
|
|
4
4
|
// A blank `off` is refused here, at declaration; an unknown name when the pass runs.
|
|
5
5
|
|
|
6
6
|
import type { ExerciseOptions } from '@ultimat3/action';
|
|
@@ -11,12 +11,17 @@ import type { SeedFactory } from './seed-graph-plan';
|
|
|
11
11
|
/** 39's option type, exactly: `{ actor, input?, runs? }` or `{ off: '<reason>' }`. */
|
|
12
12
|
export type ExerciseOverride = ExerciseOptions;
|
|
13
13
|
|
|
14
|
-
/**
|
|
14
|
+
/**
|
|
15
|
+
* What `defineExercise` returns: the overrides, by primitive name. Read by `runExercisePass` and,
|
|
16
|
+
* structurally, by `.contract({ exercise })` (`define-exercise.test.ts` pins the assignability).
|
|
17
|
+
*/
|
|
15
18
|
export interface ExerciseOverrides {
|
|
16
19
|
readonly kind: 'exercise-overrides';
|
|
17
20
|
readonly entries: ReadonlyMap<string, ExerciseOverride>;
|
|
18
21
|
/** The entity an actor is a row of, by name: its id is bound to a seeded row. */
|
|
19
22
|
readonly actorEntity?: string;
|
|
23
|
+
/** The column of that entity whose value IS the actor's id (`memberships.userId`); else its key. */
|
|
24
|
+
readonly actorColumn?: string;
|
|
20
25
|
/** By entity name: the pass's graph builds those rows with them (`seedGraph`'s `factories`). */
|
|
21
26
|
readonly factories?: Readonly<Record<string, SeedFactory>>;
|
|
22
27
|
/** The app's hook, run once inside the pass's sealed room (`exercise-setup.ts`). */
|
|
@@ -28,9 +33,12 @@ export interface ExerciseAppOptions {
|
|
|
28
33
|
/**
|
|
29
34
|
* The entity whose rows ARE the app's actors (`members`, `users`): each actor's id becomes a
|
|
30
35
|
* seeded row's key in its own tenant, so a handler writing `ctx.actor.id` into a foreign key
|
|
31
|
-
* writes a row that exists.
|
|
36
|
+
* writes a row that exists. `{ entity, column }` names the column that holds the user instead
|
|
37
|
+
* (`{ entity: membership, column: 'userId' }`). Omitted, a deterministic uuid per role and tenant.
|
|
32
38
|
*/
|
|
33
|
-
readonly actor?:
|
|
39
|
+
readonly actor?:
|
|
40
|
+
| { readonly $name: string }
|
|
41
|
+
| { readonly entity: { readonly $name: string }; readonly column: string };
|
|
34
42
|
/**
|
|
35
43
|
* By entity name, `seedGraph`'s own option: a table whose rows a generated value cannot satisfy
|
|
36
44
|
* (an `invariant` across columns) is built by the app's `defineFactory`. An unknown name is
|
|
@@ -67,7 +75,11 @@ export function defineExercise(
|
|
|
67
75
|
return Object.freeze({
|
|
68
76
|
kind: 'exercise-overrides',
|
|
69
77
|
entries,
|
|
70
|
-
...(app.actor === undefined
|
|
78
|
+
...(app.actor === undefined
|
|
79
|
+
? {}
|
|
80
|
+
: 'entity' in app.actor
|
|
81
|
+
? { actorEntity: app.actor.entity.$name, actorColumn: app.actor.column }
|
|
82
|
+
: { actorEntity: app.actor.$name }),
|
|
71
83
|
...(app.factories === undefined ? {} : { factories: app.factories }),
|
|
72
84
|
...(app.setup === undefined ? {} : { setup: app.setup }),
|
|
73
85
|
});
|
package/src/errors.ts
CHANGED
|
@@ -70,6 +70,8 @@ export const TESTING_ERROR_CODES = [
|
|
|
70
70
|
'X_CASSETTE_RECORD_REFUSED',
|
|
71
71
|
'X_TEST_MODULE_MOCK_UNRESTORABLE',
|
|
72
72
|
'X_EXERCISE_SETUP_UNSEALED',
|
|
73
|
+
'X_JOB_CRASH_UNBOUNDED',
|
|
74
|
+
'X_EXERCISE_ACTOR_USER_UNSEATED',
|
|
73
75
|
] as const;
|
|
74
76
|
|
|
75
77
|
export type TestingErrorCode = (typeof TESTING_ERROR_CODES)[number];
|
|
@@ -135,6 +137,8 @@ export const TESTING_ERROR_TITLES: Readonly<Record<TestingErrorCode, string>> =
|
|
|
135
137
|
X_CASSETTE_RECORD_REFUSED: 'a cassette recording was refused',
|
|
136
138
|
X_TEST_MODULE_MOCK_UNRESTORABLE: 'a mocked module could not be put back at the file boundary',
|
|
137
139
|
X_EXERCISE_SETUP_UNSEALED: 'an exercise setup let a host through the seal',
|
|
140
|
+
X_JOB_CRASH_UNBOUNDED: 'a crash proof needs more kill points than its bound',
|
|
141
|
+
X_EXERCISE_ACTOR_USER_UNSEATED: "the exercise pass could not seat its actor's x_users row",
|
|
138
142
|
};
|
|
139
143
|
|
|
140
144
|
// Titles must be registered for `format()` to render the contract's first line. Every code above is
|
|
@@ -146,12 +150,8 @@ registerErrorCodes(
|
|
|
146
150
|
),
|
|
147
151
|
);
|
|
148
152
|
|
|
149
|
-
// No `docs:`
|
|
150
|
-
//
|
|
151
|
-
// `wiki/` is the framework's only public documentation surface and a code lives there in a TABLE ROW,
|
|
152
|
-
// which has no anchor. The `https://ultimate.dev/errors/<code>` links this file built until 9.x
|
|
153
|
-
// answered 404, host included, on every error it has ever thrown; restating the replacement here
|
|
154
|
-
// would be the same constant in eight places waiting to drift again.
|
|
153
|
+
// No `docs:` below: `UltimateError` fills core's `ERROR_DOCS_URL`, one page per code set (a wiki
|
|
154
|
+
// TABLE ROW has no anchor; per-code links answered 404 until 9.x). Restating it here would drift.
|
|
155
155
|
|
|
156
156
|
/**
|
|
157
157
|
* A test reached the network without a mock or an allowlist entry. Always a bug, never a flake.
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
// The user an exercise actor IS: one `x_users` row per actor the pass acts as, written before the
|
|
2
|
+
// call, so a handler reading its caller through `@ultimat3/auth` (`findUserById(ctx.actor.id)`)
|
|
3
|
+
// finds the row production always has. Only where the table exists — the framework schema every
|
|
4
|
+
// `x exercise` database is built with — and only for an actor whose id has a user's shape.
|
|
5
|
+
|
|
6
|
+
import type { Actor } from '@ultimat3/core';
|
|
7
|
+
import { isUltimateError, renderThrowable, UltimateError } from '@ultimat3/core';
|
|
8
|
+
import type { DbClient } from '@ultimat3/db';
|
|
9
|
+
|
|
10
|
+
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
|
11
|
+
|
|
12
|
+
/** Whether `x_users` exists on this client, asked once per client: the schema does not move. */
|
|
13
|
+
const hasUsers = new WeakMap<DbClient, Promise<boolean>>();
|
|
14
|
+
|
|
15
|
+
/** The insert was refused: the pass cannot act as a user it could not create. */
|
|
16
|
+
export class ExerciseActorUserUnseatedError extends UltimateError {
|
|
17
|
+
constructor(error: unknown) {
|
|
18
|
+
super({
|
|
19
|
+
code: 'X_EXERCISE_ACTOR_USER_UNSEATED',
|
|
20
|
+
cause: `x exercise writes one x_users row per actor it acts as, and the insert was refused: ${isUltimateError(error) ? error.cause : renderThrowable(error)}`,
|
|
21
|
+
fix: 'x exercise --json # after a migration under packages/db/migrations/ gives the column the app added to x_users a default, or makes it nullable',
|
|
22
|
+
sourceError: error,
|
|
23
|
+
});
|
|
24
|
+
}
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
/**
|
|
28
|
+
* Seat every actor in `actors` as an `x_users` row: id, a reserved `.invalid` address unique per
|
|
29
|
+
* id, its tenant, roles, permissions and scopes. Idempotent (`on conflict (id) do nothing`), so
|
|
30
|
+
* the same role in the same tenant — one id, `exerciseActorId` — is written once however often the
|
|
31
|
+
* pass acts as it. No installed client, no table, no uuid: nothing to seat, nothing written.
|
|
32
|
+
*/
|
|
33
|
+
export async function seatActorUsers(actors: readonly (Actor | null)[]): Promise<void> {
|
|
34
|
+
const users = [
|
|
35
|
+
...new Map(
|
|
36
|
+
actors.filter((a): a is Actor => a !== null && UUID.test(a.id)).map((a) => [a.id, a]),
|
|
37
|
+
).values(),
|
|
38
|
+
];
|
|
39
|
+
if (users.length === 0) return;
|
|
40
|
+
const { db, dbClientInstalled, sql } = await import('@ultimat3/db');
|
|
41
|
+
if (!dbClientInstalled()) return;
|
|
42
|
+
const client = db();
|
|
43
|
+
let known = hasUsers.get(client);
|
|
44
|
+
if (known === undefined) {
|
|
45
|
+
known = client
|
|
46
|
+
.one<{ found: boolean }>(sql`select to_regclass('x_users') is not null as found`)
|
|
47
|
+
.then((row) => row?.found === true);
|
|
48
|
+
hasUsers.set(client, known);
|
|
49
|
+
}
|
|
50
|
+
if (!(await known)) return;
|
|
51
|
+
for (const user of users) {
|
|
52
|
+
const org = typeof user.orgId === 'string' && UUID.test(user.orgId) ? user.orgId : null;
|
|
53
|
+
try {
|
|
54
|
+
await client.execute(sql`
|
|
55
|
+
insert into x_users (id, email, org_id, roles, permissions, scopes)
|
|
56
|
+
values (${user.id}, ${`${user.id}@exercise.invalid`}, ${org}, ${[...(user.roles ?? [])]},
|
|
57
|
+
${[...(user.permissions ?? [])]}, ${[...(user.scopes ?? [])]})
|
|
58
|
+
on conflict (id) do nothing`);
|
|
59
|
+
} catch (error) {
|
|
60
|
+
throw new ExerciseActorUserUnseatedError(error);
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
}
|
package/src/exercise-actor.ts
CHANGED
|
@@ -26,6 +26,21 @@ export type ActorChoice =
|
|
|
26
26
|
export const inTenant = (actor: Actor | null, tenant: string): Actor | null =>
|
|
27
27
|
actor === null ? null : Object.freeze({ ...actor, orgId: tenant });
|
|
28
28
|
|
|
29
|
+
/**
|
|
30
|
+
* A role's actor id in one tenant: a uuid, as `ctx.actor.id` always is in production (an
|
|
31
|
+
* `x_users` key), and the same one every time for that role in that tenant — so a rerun on a seed
|
|
32
|
+
* replays byte for byte, and A's actor is never B's. Name-based (SHA-1, version 5 bits), never
|
|
33
|
+
* random: nothing about it may depend on the order the pass asks.
|
|
34
|
+
*/
|
|
35
|
+
export function exerciseActorId(role: string, tenant: string): string {
|
|
36
|
+
const digest = new Bun.CryptoHasher('sha1').update(`x-exercise-actor\0${role}\0${tenant}`);
|
|
37
|
+
const bytes = digest.digest().subarray(0, 16);
|
|
38
|
+
bytes[6] = ((bytes[6] ?? 0) & 0x0f) | 0x50;
|
|
39
|
+
bytes[8] = ((bytes[8] ?? 0) & 0x3f) | 0x80;
|
|
40
|
+
const hex = bytes.toHex();
|
|
41
|
+
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`;
|
|
42
|
+
}
|
|
43
|
+
|
|
29
44
|
type Verdict = 'allowed' | 'denied' | 'undecidable';
|
|
30
45
|
|
|
31
46
|
/**
|
|
@@ -64,7 +79,8 @@ export function chooseActor(
|
|
|
64
79
|
tenant: string,
|
|
65
80
|
override: ExerciseOverride | undefined,
|
|
66
81
|
): ActorChoice {
|
|
67
|
-
|
|
82
|
+
// An input-only entry (`{ input }`, no `actor`) keeps the pick below: only a DECLARED actor wins.
|
|
83
|
+
if (override !== undefined && !('off' in override) && override.actor !== undefined) {
|
|
68
84
|
return { kind: 'actor', actor: inTenant(override.actor, tenant), name: 'override' };
|
|
69
85
|
}
|
|
70
86
|
const candidates = grants(tenant);
|
|
@@ -92,12 +108,23 @@ export function chooseActor(
|
|
|
92
108
|
: `none of ${ordered.map((c) => c.name).join(', ')} is allowed by ${policy.label} or holds ${permissions.join(', ') || 'its grant'}`,
|
|
93
109
|
};
|
|
94
110
|
}
|
|
95
|
-
|
|
111
|
+
const [{ actor, name }] = pick;
|
|
112
|
+
return { kind: 'actor', actor: asRoleUser(actor, name, tenant), name };
|
|
96
113
|
}
|
|
97
114
|
|
|
115
|
+
/** A role's actor with the id a production actor has: `exerciseActorId(role, tenant)`. */
|
|
116
|
+
const asRoleUser = (actor: Actor | null, name: string, tenant: string): Actor | null =>
|
|
117
|
+
actor === null ? null : Object.freeze({ ...actor, id: exerciseActorId(name, tenant) });
|
|
118
|
+
|
|
98
119
|
/** The seeded rows an actor can be one of: the pass's graph, read for one entity. */
|
|
99
120
|
export interface ActorRows {
|
|
100
121
|
readonly entity: { readonly $name: string; readonly $primaryKey: readonly string[] };
|
|
122
|
+
/**
|
|
123
|
+
* The column whose value IS the user (`memberships.userId`); omitted, the row's own key. A
|
|
124
|
+
* membership's id is not a user's id, and a handler that reads `ctx.actor.id` as `x_users`'s
|
|
125
|
+
* must find the membership by the column that holds it.
|
|
126
|
+
*/
|
|
127
|
+
readonly column?: string;
|
|
101
128
|
readonly rows: readonly {
|
|
102
129
|
readonly tenant: string | null;
|
|
103
130
|
readonly row: Readonly<Record<string, unknown>>;
|
|
@@ -106,13 +133,15 @@ export interface ActorRows {
|
|
|
106
133
|
|
|
107
134
|
/**
|
|
108
135
|
* The actor as a ROW of the entity the app named (`defineExercise(…, { actor: members })`): its
|
|
109
|
-
* id becomes the key of a seeded row in its tenant, the one carrying
|
|
110
|
-
* does. Without it an actor's id is
|
|
111
|
-
*
|
|
136
|
+
* id becomes the key — or the named `column` — of a seeded row in its tenant, the one carrying
|
|
137
|
+
* one of its roles when any does. Without it an actor's id is `exerciseActorId(role, tenant)`, a
|
|
138
|
+
* user no table of the app's holds, so a handler writing `ctx.actor.id` into a foreign key —
|
|
139
|
+
* `authorId` — is refused by the database.
|
|
112
140
|
*/
|
|
113
141
|
export function asSeededRow(actor: Actor | null, tenant: string, rows: ActorRows): Actor | null {
|
|
114
|
-
const key =
|
|
115
|
-
|
|
142
|
+
const key =
|
|
143
|
+
rows.column ?? (rows.entity.$primaryKey.length === 1 ? rows.entity.$primaryKey[0] : undefined);
|
|
144
|
+
if (actor === null || key === undefined) return actor;
|
|
116
145
|
const own = rows.rows.filter((row) => row.tenant === tenant || row.tenant === null);
|
|
117
146
|
const roles = actor.roles ?? [];
|
|
118
147
|
const row =
|
package/src/exercise-crash.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
|
-
// The crash proof, as the exercise pass runs it: a job its control reached is killed at every
|
|
2
|
-
// boundary with the control's own input and actor (`crashProof`), and the answer is a plain record
|
|
1
|
+
// The crash proof, as the exercise pass runs it: a job its control reached is killed at every
|
|
2
|
+
// distinct step boundary with the control's own input and actor (`crashProof`), and the answer is a plain record
|
|
3
3
|
// `.x/exercise.json` can hold. Run after the control, so its runs meet the rows the control left.
|
|
4
4
|
|
|
5
5
|
import type { Actor } from '@ultimat3/core';
|
|
@@ -34,7 +34,11 @@ export async function proveCrash(
|
|
|
34
34
|
const worker = ctxOf(actor === null ? { role: 'worker' } : { role: 'worker', actor });
|
|
35
35
|
try {
|
|
36
36
|
const report = await crashProof(handle, input, { context: () => worker });
|
|
37
|
-
const counted = {
|
|
37
|
+
const counted = {
|
|
38
|
+
boundaries: report.boundaries.length,
|
|
39
|
+
crossed: report.crossed,
|
|
40
|
+
reruns: report.reruns.length,
|
|
41
|
+
};
|
|
38
42
|
return report.divergences.length === 0
|
|
39
43
|
? { verdict: 'held', ...counted }
|
|
40
44
|
: { verdict: 'diverged', ...counted, findings: report.divergences.map(findingOf) };
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
// The override line a drifted primitive's record carries: `name: { input: (drawn) => ({ ...drawn,
|
|
2
|
+
// digest: "<must match ^[0-9a-f]{64}$>" }) },` — the field no draw can fill, named and bound,
|
|
3
|
+
// rendered by core's one `renderDrawnInput`. No `actor:`: the pass picks one when the entry has none.
|
|
4
|
+
|
|
5
|
+
import { renderDrawnInput, renderExerciseKey, sampleGapBinding } from '@ultimat3/core';
|
|
6
|
+
import type { StandardSchemaV1 } from '@ultimat3/schema';
|
|
7
|
+
import type { Sampler } from './exercise-input';
|
|
8
|
+
|
|
9
|
+
/** `idFields` are the input's id-shaped keys: each wants a real row's id, as the other hints say. */
|
|
10
|
+
export function driftOverrideLine(
|
|
11
|
+
name: string,
|
|
12
|
+
schema: StandardSchemaV1,
|
|
13
|
+
sampler: Sampler,
|
|
14
|
+
idFields: readonly string[],
|
|
15
|
+
): string {
|
|
16
|
+
const gaps = sampler.sampleGaps(schema);
|
|
17
|
+
const bindings = [
|
|
18
|
+
...gaps.map((path) => sampleGapBinding(path, sampler.describeSampleGap(schema, path))),
|
|
19
|
+
...idFields
|
|
20
|
+
.filter((field) => !gaps.includes(field))
|
|
21
|
+
.map((field) => ({ path: field, placeholder: '<a real id>' })),
|
|
22
|
+
];
|
|
23
|
+
const input =
|
|
24
|
+
bindings.length === 0
|
|
25
|
+
? "(drawn) => ({ ...drawn, field: '<a value the refinement accepts>' })"
|
|
26
|
+
: renderDrawnInput(bindings);
|
|
27
|
+
return `${renderExerciseKey(name)}: { input: ${input} },`;
|
|
28
|
+
}
|
package/src/exercise-input.ts
CHANGED
|
@@ -10,6 +10,9 @@ import type { SeedEntity } from './seed-graph-shape';
|
|
|
10
10
|
export interface Sampler {
|
|
11
11
|
sampleInput(schema: StandardSchemaV1): unknown;
|
|
12
12
|
arbitrary(schema: StandardSchemaV1, options: { seed: number; run: number }): unknown;
|
|
13
|
+
/** The fields the minimal draw cannot fill, and what each needs: a drift's hint names them. */
|
|
14
|
+
sampleGaps(schema: StandardSchemaV1): readonly string[];
|
|
15
|
+
describeSampleGap(schema: StandardSchemaV1, path: string): string;
|
|
13
16
|
}
|
|
14
17
|
|
|
15
18
|
/** On demand: a `packages/core` test that never exercises anything must not load the walker. */
|
|
@@ -9,9 +9,13 @@ export interface TenantOf {
|
|
|
9
9
|
tenantFor(input: unknown): string | undefined;
|
|
10
10
|
}
|
|
11
11
|
|
|
12
|
-
/**
|
|
12
|
+
/**
|
|
13
|
+
* `seated`: the input names the tenant. `fixed`: no field it reads moves the org it answers.
|
|
14
|
+
* `none`: `tenant: 'none'` — no org reaches the run, so whoever enqueues it the run is the same.
|
|
15
|
+
*/
|
|
13
16
|
export type JobTenantBinding =
|
|
14
17
|
| { readonly kind: 'seated'; readonly input: unknown }
|
|
18
|
+
| { readonly kind: 'none'; readonly input: unknown }
|
|
15
19
|
| { readonly kind: 'fixed'; readonly declared: string | undefined };
|
|
16
20
|
|
|
17
21
|
type Path = readonly string[];
|
|
@@ -85,16 +89,17 @@ function setAt(input: unknown, path: Path, value: unknown): unknown {
|
|
|
85
89
|
}
|
|
86
90
|
|
|
87
91
|
/**
|
|
88
|
-
* `input` naming `tenant` wherever `handle.tenantFor` reads it.
|
|
89
|
-
*
|
|
90
|
-
* `tenant`
|
|
92
|
+
* `input` naming `tenant` wherever `handle.tenantFor` reads it. One already naming it comes back as
|
|
93
|
+
* given; a `'none'` job comes back as given, as `none` — the tenant proof's write half reads it
|
|
94
|
+
* (`tenant-proof.ts`, `TenantAttack.tenantless`). A read path is kept only if setting it moves the
|
|
95
|
+
* answer to `tenant` — a field read and then ignored is left as drawn.
|
|
91
96
|
*/
|
|
92
97
|
export function bindJobTenant(handle: TenantOf, input: unknown, tenant: string): JobTenantBinding {
|
|
93
98
|
const declared = answer(handle, input);
|
|
94
99
|
if (declared === tenant) return { kind: 'seated', input };
|
|
95
100
|
const reads = pathsRead(handle, input);
|
|
96
101
|
// `tenant: 'none'` is never called: no org named, none read, so none to disagree with.
|
|
97
|
-
if (declared === undefined && reads.length === 0) return { kind: '
|
|
102
|
+
if (declared === undefined && reads.length === 0) return { kind: 'none', input };
|
|
98
103
|
const moving = reads.filter((path) => answer(handle, setAt(input, path, tenant)) === tenant);
|
|
99
104
|
const bound = moving.reduce<unknown>((value, path) => setAt(value, path, tenant), input);
|
|
100
105
|
return moving.length > 0 && answer(handle, bound) === tenant
|
|
@@ -15,13 +15,34 @@ export function overrideInput(
|
|
|
15
15
|
name: string,
|
|
16
16
|
override: ExerciseOverride | undefined,
|
|
17
17
|
): OverrideInput | undefined {
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
);
|
|
18
|
+
// Any entry but `off` may bind: `{ input }` alone keeps the pass's own actor and still binds.
|
|
19
|
+
const bind = bindDrawn(override === undefined || 'off' in override ? undefined : override.input);
|
|
21
20
|
if (bind === undefined) return undefined;
|
|
22
|
-
|
|
21
|
+
const declared = override === undefined || 'off' in override ? undefined : override.actor;
|
|
22
|
+
return async (drawn, actor) =>
|
|
23
|
+
rehomed(
|
|
24
|
+
await withExerciseRequest(name, actor, async () => bind(drawn)),
|
|
25
|
+
declared?.orgId,
|
|
26
|
+
actor?.orgId,
|
|
27
|
+
);
|
|
23
28
|
}
|
|
24
29
|
|
|
30
|
+
/**
|
|
31
|
+
* The bound input, re-homed with its actor: the pass seats a declared actor in each tenant it
|
|
32
|
+
* seeds, so a top-level value equal to the org the entry DECLARED (`orgId: '<org>'`, the same
|
|
33
|
+
* literal the contract test runs with) is that tenant's. Without it A's own call names an org its
|
|
34
|
+
* actor is not in, and a policy comparing the two denies every run.
|
|
35
|
+
*/
|
|
36
|
+
function rehomed(input: unknown, from: string | undefined, to: string | undefined): unknown {
|
|
37
|
+
if (from === undefined || to === undefined || from === to || !isPlainObject(input)) return input;
|
|
38
|
+
return Object.fromEntries(
|
|
39
|
+
Object.entries(input).map(([key, value]) => [key, value === from ? to : value]),
|
|
40
|
+
);
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
const isPlainObject = (value: unknown): value is Readonly<Record<string, unknown>> =>
|
|
44
|
+
typeof value === 'object' && value !== null && !Array.isArray(value);
|
|
45
|
+
|
|
25
46
|
/**
|
|
26
47
|
* Whether an override's input is declared `async` — state the pass computes, which a load that
|
|
27
48
|
* draws with no pass behind it (`x drill`) cannot reproduce. The function's own tag, never a call:
|
|
@@ -29,6 +50,6 @@ export function overrideInput(
|
|
|
29
50
|
*/
|
|
30
51
|
export const isAsyncInput = (override: ExerciseOverride | undefined): boolean =>
|
|
31
52
|
override !== undefined &&
|
|
32
|
-
'
|
|
53
|
+
!('off' in override) &&
|
|
33
54
|
override.input !== undefined &&
|
|
34
55
|
Object.prototype.toString.call(override.input) === '[object AsyncFunction]';
|
package/src/exercise-report.ts
CHANGED
|
@@ -79,12 +79,21 @@ export interface CrashFinding {
|
|
|
79
79
|
readonly inside?: string;
|
|
80
80
|
}
|
|
81
81
|
|
|
82
|
-
/**
|
|
82
|
+
/**
|
|
83
|
+
* The crash proof's verdict on a reached job: killed at every distinct step boundary, a loop's rows
|
|
84
|
+
* as one (`crashProof`). `boundaries` is the kills, `crossed` the boundaries they stand for.
|
|
85
|
+
*/
|
|
83
86
|
export type CrashProof =
|
|
84
|
-
| {
|
|
87
|
+
| {
|
|
88
|
+
readonly verdict: 'held';
|
|
89
|
+
readonly boundaries: number;
|
|
90
|
+
readonly crossed?: number;
|
|
91
|
+
readonly reruns: number;
|
|
92
|
+
}
|
|
85
93
|
| {
|
|
86
94
|
readonly verdict: 'diverged';
|
|
87
95
|
readonly boundaries: number;
|
|
96
|
+
readonly crossed?: number;
|
|
88
97
|
/** Lost answers that reran a step: at-least-once, counted, never red. */
|
|
89
98
|
readonly reruns: number;
|
|
90
99
|
readonly findings: readonly CrashFinding[];
|
|
@@ -118,6 +127,33 @@ export interface ExerciseRecord {
|
|
|
118
127
|
readonly plans?: PlanProof;
|
|
119
128
|
/** On a reached primitive: its statement shapes at 2 and at 8 rows a table (`scale-sweep.ts`). */
|
|
120
129
|
readonly scale?: ScaleProof;
|
|
130
|
+
/** Wall time of each proof that ran on it; `ms` above is the control's own call. */
|
|
131
|
+
readonly proofMs?: ProofMs;
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
/**
|
|
135
|
+
* Milliseconds per proof of one record. No `plans`: one statement is planned once for every
|
|
136
|
+
* primitive that sent it, so its time is the pass's (`ExercisePhaseMs.plans`), never one record's.
|
|
137
|
+
*/
|
|
138
|
+
export interface ProofMs {
|
|
139
|
+
/** B's two legs and the snapshots around them, the control's own call excepted. */
|
|
140
|
+
readonly tenant?: number;
|
|
141
|
+
/** Every run the crash proof made. */
|
|
142
|
+
readonly crash?: number;
|
|
143
|
+
/** Its calls at both sizes; seeding the sized graphs is the pass's (`ExercisePhaseMs.scale`). */
|
|
144
|
+
readonly scale?: number;
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
/** Milliseconds per phase of one pass, so a slow pass names its slow part. */
|
|
148
|
+
export interface ExercisePhaseMs {
|
|
149
|
+
/** The pass's one seed graph, written. */
|
|
150
|
+
readonly seed: number;
|
|
151
|
+
/** Every primitive's control and its tenant and crash proofs. */
|
|
152
|
+
readonly primitives: number;
|
|
153
|
+
/** The scale sweep: its two sized graphs written, every reached primitive called in each. */
|
|
154
|
+
readonly scale: number;
|
|
155
|
+
/** The plan sweep: every distinct statement the pass sent, planned. */
|
|
156
|
+
readonly plans: number;
|
|
121
157
|
}
|
|
122
158
|
|
|
123
159
|
/** What an effect carried, for the tenant proof's read half: searched, never reported as is. */
|
package/src/exercise-sweeps.ts
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
// What runs after every primitive's own call: the scale sweep (each reached one again, at 2 and at 8
|
|
2
2
|
// rows a table) and the plan sweep (every statement the pass sent, planned on the migrated
|
|
3
|
-
// database). Both verdicts land on the record of the primitive that sent the statements
|
|
3
|
+
// database). Both verdicts land on the record of the primitive that sent the statements, and each
|
|
4
|
+
// sweep's time on the pass.
|
|
4
5
|
|
|
5
6
|
import type { Actor } from '@ultimat3/core';
|
|
6
7
|
import type { ExerciseOverrides } from './define-exercise';
|
|
@@ -9,11 +10,11 @@ import { asSeededRow, chooseActor } from './exercise-actor';
|
|
|
9
10
|
import type { Sampler } from './exercise-input';
|
|
10
11
|
import { bindIds, drawInput, seededIds } from './exercise-input';
|
|
11
12
|
import { overrideInput } from './exercise-override-input';
|
|
12
|
-
import type { ExerciseRecord } from './exercise-report';
|
|
13
|
+
import type { ExercisePhaseMs, ExerciseRecord } from './exercise-report';
|
|
13
14
|
import type { Subject } from './exercise-subjects';
|
|
14
15
|
import { seatIn } from './exercise-subjects';
|
|
15
16
|
import { sweepPlans } from './plan-sweep';
|
|
16
|
-
import type { ScaleSweepOptions } from './scale-sweep-run';
|
|
17
|
+
import type { ScaleSubject, ScaleSweepOptions } from './scale-sweep-run';
|
|
17
18
|
import { scaleSweep } from './scale-sweep-run';
|
|
18
19
|
import type { SeedGraph } from './seed-graph-plan';
|
|
19
20
|
import type { SeedEntity } from './seed-graph-shape';
|
|
@@ -30,7 +31,12 @@ export const actorRowsOf = (options: CallOptions, graph: SeedGraph): ActorRows |
|
|
|
30
31
|
const name = options.overrides?.actorEntity;
|
|
31
32
|
const entity = options.entities.find((one) => one.$name === name);
|
|
32
33
|
if (entity === undefined) return undefined;
|
|
33
|
-
|
|
34
|
+
const column = options.overrides?.actorColumn;
|
|
35
|
+
return {
|
|
36
|
+
entity,
|
|
37
|
+
rows: graph.inserts.filter((row) => row.entity === entity.$name),
|
|
38
|
+
...(column === undefined ? {} : { column }),
|
|
39
|
+
};
|
|
34
40
|
};
|
|
35
41
|
|
|
36
42
|
/** The actor of `chosen`, seated as a row of the app's actor entity in `tenant` when it names one. */
|
|
@@ -66,6 +72,25 @@ export async function sizedCall(
|
|
|
66
72
|
return { actor, input: seatIn(subject, given, tenant) };
|
|
67
73
|
}
|
|
68
74
|
|
|
75
|
+
/** The records with both sweeps' verdicts and scale time on them, and each sweep's whole time. */
|
|
76
|
+
export interface Swept {
|
|
77
|
+
readonly records: readonly ExerciseRecord[];
|
|
78
|
+
readonly ms: Pick<ExercisePhaseMs, 'scale' | 'plans'>;
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
/** `subject`'s calls, each timed into `spent` under its name: the sweep calls it once per size. */
|
|
82
|
+
const timed = (subject: ScaleSubject, spent: Map<string, number>): ScaleSubject => ({
|
|
83
|
+
...subject,
|
|
84
|
+
call: async (...args) => {
|
|
85
|
+
const started = performance.now();
|
|
86
|
+
try {
|
|
87
|
+
return await subject.call(...args);
|
|
88
|
+
} finally {
|
|
89
|
+
spent.set(subject.name, (spent.get(subject.name) ?? 0) + performance.now() - started);
|
|
90
|
+
}
|
|
91
|
+
},
|
|
92
|
+
});
|
|
93
|
+
|
|
69
94
|
/**
|
|
70
95
|
* The scale sweep inside the room, the room closed, then the plan sweep — on the installed client
|
|
71
96
|
* only: with none, the pass ran on a driver that sends no SQL and there is no plan to read.
|
|
@@ -73,24 +98,41 @@ export async function sizedCall(
|
|
|
73
98
|
export async function attachSweeps(
|
|
74
99
|
records: readonly ExerciseRecord[],
|
|
75
100
|
options: ScaleSweepOptions,
|
|
76
|
-
): Promise<
|
|
101
|
+
): Promise<Swept> {
|
|
102
|
+
const spent = new Map<string, number>();
|
|
77
103
|
let scale: Awaited<ReturnType<typeof scaleSweep>>;
|
|
104
|
+
const scaleStarted = performance.now();
|
|
78
105
|
try {
|
|
79
|
-
scale = await scaleSweep(
|
|
106
|
+
scale = await scaleSweep({
|
|
107
|
+
...options,
|
|
108
|
+
subjects: options.subjects.map((subject) => timed(subject, spent)),
|
|
109
|
+
});
|
|
80
110
|
} finally {
|
|
81
111
|
await options.room[Symbol.asyncDispose]();
|
|
82
112
|
}
|
|
113
|
+
const plansStarted = performance.now();
|
|
83
114
|
const db = await import('@ultimat3/db');
|
|
84
115
|
const plans = db.dbClientInstalled()
|
|
85
116
|
? await sweepPlans(db.db(), options.room.sent(), options.entities)
|
|
86
117
|
: new Map();
|
|
87
|
-
|
|
118
|
+
const ms = {
|
|
119
|
+
scale: Math.round(plansStarted - scaleStarted),
|
|
120
|
+
plans: Math.round(performance.now() - plansStarted),
|
|
121
|
+
};
|
|
122
|
+
const swept = records.map((record) => {
|
|
88
123
|
const proof = plans.get(record.name);
|
|
89
124
|
const sized = scale.get(record.name);
|
|
125
|
+
const scaleMs = spent.get(record.name);
|
|
126
|
+
const proofMs = {
|
|
127
|
+
...record.proofMs,
|
|
128
|
+
...(scaleMs === undefined ? {} : { scale: Math.round(scaleMs) }),
|
|
129
|
+
};
|
|
90
130
|
return {
|
|
91
131
|
...record,
|
|
92
132
|
...(proof === undefined ? {} : { plans: proof }),
|
|
93
133
|
...(sized === undefined ? {} : { scale: sized }),
|
|
134
|
+
...(Object.keys(proofMs).length === 0 ? {} : { proofMs }),
|
|
94
135
|
};
|
|
95
136
|
});
|
|
137
|
+
return { records: swept, ms };
|
|
96
138
|
}
|