@voltro/cli 0.1.0

package/templates/agent-docs/testing.md

@@ -0,0 +1,376 @@
+# Testing
+
+> Voltro's test story — the @voltro/testing package, the test pyramid (unit handlers/tools → workflow runner → e2e), and the voltro test / voltro e2e CLI commands.
+
+
+
+---
+
+<!-- source: en/testing/overview.md -->
+## Overview
+
+_Voltro's test story — the @voltro/testing package, the test pyramid (unit handlers/tools → workflow runner → e2e), and the voltro test / voltro e2e CLI commands._
+
+## The package
+
+Test utilities live in **`@voltro/testing`** — a separate, public package. It is **not** bundled into a scaffolded project, so the moment you write your first test, add it (and `vitest`) as a **devDependency** of the app you're testing:
+
+```bash
+pnpm --filter @my-app/api add -D @voltro/testing vitest
+```
+
+Everything in it is in-memory and synchronous-friendly — no docker, no running server, no real model calls. A handler test boots in milliseconds.
+
+| Export | Use |
+|---|---|
+| `makeTestContext(options?)` | The request `ctx` a handler/tool sees at runtime — subject-scoped `ctx.store`, `clock`, `email`, optional `ai`. → [Unit testing](/docs/testing/unit-testing) |
+| `mockStore(seed)` | Seed rows: `makeTestContext({ store: mockStore({ todos: [...] }) })`. |
+| `MockClock` / `MockEmail` / `MockLLM` | Deterministic clock, captured email, queued LLM responses. |
+| `makeWorkflowRunner({ ctx, workflows })` | Drive a workflow in-process and assert on its steps. → [Workflows](/docs/testing/workflows) |
+| `runDialectParity(fixture)` | Run one fixture across a SQL dialect for portability — from the `@voltro/testing/dialect` subpath. → [Dialect parity](/docs/testing/dialect-parity) |
+
+## The test pyramid
+
+Voltro's primitives are designed so the cheapest test covers the most surface. Reach for the lowest layer that proves what you need:
+
+1. **Unit — handlers, tools, mutations, queries.** Call the executor directly with a `makeTestContext()` ctx. The store is the real mixin-wrapped store, so tenant scoping, soft-delete, and audit auto-fill all behave as in production — without a database. This is 90% of your tests. See [Unit testing](/docs/testing/unit-testing).
+2. **Workflow — durable, multi-step logic.** `makeWorkflowRunner` drives a workflow end-to-end in-process and hands back the same step log the dashboard shows, including real retry counts. See [Workflows](/docs/testing/workflows).
+3. **Parity — hand-written SQL that must stay portable.** `runDialectParity` runs a fixed scenario suite against a dialect you stand up. Mostly for dialect-package authors and apps with `unsafe()` SQL. See [Dialect parity](/docs/testing/dialect-parity).
+4. **End-to-end — the running stack.** `voltro e2e` boots the api + web siblings, runs your tests against the live processes, and tears them down.
+
+Agents are integration tests; tools are unit tests. Don't reach for an e2e harness to assert logic a `makeTestContext` test can prove.
+
+## The CLI
+
+```bash
+voltro test    # vitest against the current app
+voltro e2e     # boot api + web siblings, run tests, tear down
+```
+
+`voltro test` is a thin wrapper over vitest — it runs the app's test files with the framework's config. Everything `@voltro/testing` exports is plain TypeScript you import inside those files; there's no special test runner.
+
+## What NOT to do
+
+- **Don't spin up Postgres for a handler test.** `makeTestContext`'s in-memory store enforces the same mixin behaviour — tenant scoping, soft-delete, audit — so a unit test catches the same class of bug far faster. Save a real database for [dialect parity](/docs/testing/dialect-parity) and e2e.
+- **Don't call a real model in agent tests.** Queue deterministic responses with `mockAi` / `useMockAi` (see [Unit testing → agents](/docs/testing/unit-testing#testing-agents-and-the-model-loop)). A flaky test that hits the network on every run is worse than no test.
+- **Don't assert on wall-clock time.** Inject `MockClock` and `advance(...)` deterministically — never `setTimeout` + real sleeps.
+
+
+
+---
+
+<!-- source: en/testing/unit-testing.md -->
+## Unit testing
+
+_makeTestContext — the in-memory request ctx for handler and tool tests. The real mixin-wrapped store, mockStore seeding, subject/tenant re-scoping, and the deterministic clock / email / LLM mocks._
+
+## `makeTestContext`
+
+`makeTestContext(options?)` returns an **`AppContext`** — the exact `ctx` a mutation / query / action / tool executor receives at runtime — with the deterministic test doubles added on top. Because it *is* an `AppContext`, you pass it straight into a handler: `await myHandler(input, makeTestContext({ … }))`. The acting subject is at `ctx.request.subject`; `ctx.store` is the **real** mixin-wrapped store backed by an in-memory data store (so tenant auto-scoping, soft-delete filtering, audit auto-fill, and the fluent `select` / `update` / `delete` builders behave exactly as in production); `ctx.cache` is a real in-memory cache whose TTLs honour `ctx.clock`.
+
+Tables are auto-wired from whatever the test imported (the globally-registered tables), so you never re-declare schema. Seed rows with `mockStore`.
+
+```ts
+import { makeTestContext, mockStore } from '@voltro/testing'
+import createTodo from '../mutations/todos.create.mutation.server'
+
+test('createTodo inserts a row for the caller’s tenant', async () => {
+  const ctx = makeTestContext({
+    subject: { type: 'user', id: 'user_1', tenantId: 't1' },
+    store:   mockStore({ todos: [] }),
+  })
+
+  const row = await createTodo({ tenantId: 't1', title: 'buy milk' }, ctx)
+
+  expect(row.title).toBe('buy milk')
+  // The real store middleware auto-stamped audit + tenant columns —
+  // no different from a live mutation.
+})
+```
+
+The same call tests a tool — a tool's `execute` / handler takes `(input, ctx)`:
+
+```ts
+import { makeTestContext, mockStore } from '@voltro/testing'
+import searchDocs from '../tools/searchDocs.tool'
+
+test('searchDocs returns matching rows', async () => {
+  const ctx = makeTestContext({ store: mockStore({ docs: [{ id: 'd1', body: 'hello world' }] }) })
+  const out = await searchDocs({ query: 'hello' }, ctx)
+  expect(out).toHaveLength(1)
+})
+```
+
+### Options
+
+`MakeTestContextOptions`:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `subject` | anonymous, no tenant | The acting `Subject`. |
+| `tables` | all registered | Tables to wire into the store's schema. |
+| `store` | `{}` | Seed rows keyed by table name — use `mockStore({...})`. |
+| `clockStart` | `2026-01-01T00:00:00Z` | Frozen start instant for `ctx.clock`. |
+| `ai` | — | Injected AI mock (a `mockAi({...})` value). |
+| `llmResponses` | `[]` | Queued responses for the bundled `ctx.llm` (`MockLLM`). |
+| `env` | ambient `process.env` | Env values sealed into the boot snapshot so handler code reading `getSecret('X')` / `serverEnv.X` resolves under test. Merged over `process.env` (these win). |
+
+The returned `TestContext` carries `{ clock, email, llm, ai?, request, cache, store, withSubject, withTenant }` — read the acting subject at `ctx.request.subject` and the in-memory cache at `ctx.cache`.
+
+## `invoke` — run a handler through its input Schema
+
+Calling an executor directly skips the input-Schema **decode** the real dispatcher runs first, so a test can feed the handler a value the wire would reject. `invoke(descriptor, executor, rawInput, ctx)` decodes `rawInput` through `descriptor.input` (throwing a `ParseError` on invalid input — the same validation that would 400 in production) and only then calls the executor with the decoded value.
+
+```ts
+import { invoke, makeTestContext } from '@voltro/testing'
+import { createNote } from './notes.mutation'          // the descriptor
+import { createNoteHandler } from './notes.mutation.server' // the executor
+
+const ctx = makeTestContext({ subject: { type: 'user', id: 'u1', tenantId: 't1' } })
+
+// valid input decodes, then the handler runs
+const note = await invoke(createNote, createNoteHandler, { title: 'hi' }, ctx)
+
+// a bad shape rejects at the decode — the handler never runs
+await expect(invoke(createNote, createNoteHandler, { title: 42 }, ctx)).rejects.toThrow()
+```
+
+`invoke` reproduces the **input-Schema validation** hop only. RBAC / tenant / auth **guards** are rpc middleware wired at dispatch time — not carried on the descriptor — so to assert a guard, run the real handler under a scoped `ctx.request.subject` (via `makeTestContext({ subject })` / `ctx.withSubject`) and let the handler's own `requireScope(...)` execute.
+
+## Subject + tenant re-scoping
+
+`withSubject` and `withTenant` re-scope to a different principal for one block, sharing the **same** underlying data — so cross-subject reads exercise real tenant scoping, not a closure stub. This is how you prove isolation: write as one tenant, then assert another tenant can't see the row.
+
+```ts
+const ctx = makeTestContext({
+  subject: { type: 'user', id: 'u1', tenantId: 't1' },
+  store:   mockStore({ todos: [] }),
+})
+
+await ctx.store.insert('todos', { title: 'private to t1' })
+
+await ctx.withTenant('t2', async (ctx2) => {
+  const rows = await ctx2.store.select('todos').all()
+  expect(rows).toHaveLength(0)   // t2 sees none of t1's rows
+})
+```
+
+`withSubject(subject, fn)` swaps the full identity; `withTenant(tenantId, fn)` keeps the current subject and only changes the tenant. Both return a `Promise` of whatever `fn` returns.
+
+## Asserting on hidden rows
+
+`ctx.store` applies the same scoping a handler gets — the tenant filter plus `deletedAt IS NULL`. To assert on rows the scope hides (that a mutation soft-deleted a row, or wrote into another tenant), read **past** the scope with `.unscoped()` (drops the tenant filter) and `.withDeleted()` (includes soft-deleted rows) — both typed, no cast:
+
+```ts
+await deleteNote({ id: 'n1' }, ctx)                 // soft-delete
+expect(await ctx.store.select('notes').all()).toHaveLength(0)   // hidden from normal reads
+
+const raw = await ctx.store.select('notes').unscoped().withDeleted().all()
+expect(raw[0]?.deletedAt).not.toBeNull()            // …but still there, tombstoned
+```
+
+## The deterministic mocks
+
+### `ctx.clock` — `MockClock`
+
+A frozen clock you advance by hand. No real sleeping.
+
+```ts
+ctx.clock.now()            // current instant in ms
+ctx.clock.date()           // current instant as a Date
+ctx.clock.advance('15m')   // jump forward — ms | s | m | h | d, or a number of ms
+ctx.clock.advance(900_000) // same, in milliseconds
+```
+
+### `ctx.email` — `MockEmail`
+
+Captures outgoing email for assertion.
+
+```ts
+ctx.email.sent                 // SentEmail[] — every send, in order
+ctx.email.lastTo('a@b.com')    // the most recent email to that address, or undefined
+ctx.email.clear()              // reset between cases
+```
+
+### `ctx.llm` — `MockLLM`
+
+A queue of canned model responses plus a record of every call. Each `MockResponse` is one of `{ text }`, `{ toolCall: { name, input } }`, or `{ error: { code, message? } }`.
+
+```ts
+const ctx = makeTestContext({
+  llmResponses: [{ text: 'first' }, { toolCall: { name: 'search', input: { q: 'x' } } }],
+})
+
+// inside the handler under test, the model consumes the queue in order;
+// afterwards:
+expect(ctx.llm.calls).toHaveLength(2)   // each prompt that triggered a call
+expect(ctx.llm.remaining()).toBe(0)     // queue drained
+```
+
+`MockLLM` throws `MockLLM: no more responses queued` if the code under test asks for one more response than you queued — a useful assertion that the loop ran exactly as many turns as expected.
+
+## Testing agents and the model loop
+
+`ctx.llm` is the low-level queue. For a full agent / model-loop test, install a deterministic provider with `mockAi` / `useMockAi` from `@voltro/ai/test` — the AI package owns the full mock; `@voltro/testing` only types the injected `MockAi` slot on `makeTestContext({ ai })`.
+
+```ts
+import { useMockAi } from '@voltro/ai/test'
+
+const mock = useMockAi({
+  stream: ['Hello, ', 'world.'],                  // token deltas
+  turns: [                                        // scripted tool loop
+    { toolCalls: [{ name: 'searchDocs', input: { query: 'x' } }] },
+    { text: 'Based on the docs.' },
+  ],
+})
+// … run the agent …
+mock.reset()
+```
+
+`mockAi({...})` returns a `MockAi` value you pass as `makeTestContext({ ai: mockAi({...}) })`; `throwNTimes(n, value)` exercises retry paths. See [AI → Providers](/docs/ai/providers#mock-for-tests) for the full surface.
+
+
+
+---
+
+<!-- source: en/testing/workflows.md -->
+## Testing workflows
+
+_makeWorkflowRunner — drive a workflow end-to-end in-process and assert on the same step log the dashboard reads, including real retry counts._
+
+## `makeWorkflowRunner`
+
+A workflow's logic — steps, retries, branching on a signal — deserves a test that doesn't boot the cluster engine. `makeWorkflowRunner({ ctx, workflows })` runs a workflow end-to-end in-process on an in-memory engine and hands back the same step log the [workflow dashboard](/docs/workflows/debugging) reads, so a test can assert on attempts, durations, and intermediate values.
+
+Discovery is explicit: pass each workflow under test as a `{ workflow, execute }` entry. The `workflow` is the value from `workflow({ name, ... })` (the descriptor); the `execute` is the same `(payload, executionId) => Effect` the framework hands to `workflow.toLayer(...)` at registration time — i.e. `buildExecute(ctx)` from your `*.workflow.server.tsx`. The runner matches `start(name, …)` against `entry.workflow.name`.
+
+```ts
+import { makeTestContext, mockStore, makeWorkflowRunner } from '@voltro/testing'
+import { SummarizeTodos } from '../workflows/todos.summarize.workflow'
+import buildExecute from '../workflows/todos.summarize.workflow.server'
+
+test('summarize walks fetch → summarize and succeeds', async () => {
+  const ctx = makeTestContext({ store: mockStore({ todos: [{ id: 't1', title: 'x' }] }) })
+
+  const runner = makeWorkflowRunner({
+    ctx,
+    workflows: [
+      { workflow: SummarizeTodos as never, execute: buildExecute(ctx) as never },
+    ],
+  })
+
+  const result = await runner.start('todos.summarize', { tenantId: 't1' })
+
+  expect(result.status).toBe('succeeded')
+  expect(result.steps.map((s) => s.name)).toEqual(['fetch-todos', 'summarize'])
+})
+```
+
+The `as never` casts keep the call-site ergonomic — the runner is type-erased over each workflow's payload / success / error shapes (it provides the engine layer itself), and `never` assigns to any field type without an `any` leaking into the public type.
+
+## The result
+
+`start(name, payload)` blocks until the run is terminal and resolves a `WorkflowRunResult`:
+
+| Field | Meaning |
+|---|---|
+| `status` | `'succeeded'` or `'failed'`. |
+| `output` | The workflow's success value (`null` on failure). |
+| `error` | `{ tag, message }` on failure (`null` on success). `tag` is the typed error's `_tag` when there is one. |
+| `steps` | One `InspectedStep` per step name — `{ name, attempt, attempts, status, input, output, error, durationMs }`. |
+| `runId` | The synthetic run id, for `inspect(...)`. |
+
+### Asserting on retries
+
+`steps[i].attempts` reflects the **real** retry count: a step that throws twice then succeeds records three attempt rows, so `attempts === 3`. This is the cleanest way to prove a step's retry policy actually fired.
+
+```ts
+const charge = result.steps.find((s) => s.name === 'charge-card')
+expect(charge?.status).toBe('completed')
+expect(charge?.attempts).toBe(3)   // failed twice, succeeded on the third
+```
+
+### Inspecting a finished run
+
+`inspect(runId)` returns the recorded `InspectedWorkflow` for a previously-started run **without re-running it**, or `null` for an unknown id — the same shape the dashboard renders (`{ id, name, status, input, output, subject, source: 'test-runner', steps, startedAt, finishedAt, … }`).
+
+```ts
+const run = await runner.inspect(result.runId)
+expect(run?.source).toBe('test-runner')
+```
+
+## Notes
+
+- The runner uses the in-memory workflow engine and an in-memory recorder — nothing persists, nothing schedules on the real clock. Pair it with [the mock clock](/docs/testing/unit-testing#the-deterministic-mocks) for any time-dependent step.
+- Starting a workflow the runner doesn't know about throws a clear error naming the missing workflow — register it in the `workflows` array.
+- This tests the workflow's **logic**. To exercise the durable engine itself (crash/resume, cluster handoff), that's an integration concern beyond the in-process runner.
+
+
+
+---
+
+<!-- source: en/testing/dialect-parity.md -->
+## Dialect parity
+
+_runDialectParity — a fixed scenario suite that proves a SQL backend behaves identically to the others. For dialect-package authors and apps with hand-written portable SQL._
+
+## When you need this
+
+Most apps never touch this page. The framework's DSL already hides every cross-dialect difference, and a [unit test](/docs/testing/unit-testing) on the in-memory store proves your handler logic. Reach for `runDialectParity` only when you have code that talks to a real database in a dialect-specific way:
+
+- You're authoring a new `@voltro/sql-*` dialect package and need to prove it before plugging it into `loadDialect()`.
+- Your app drops to `unsafe()` SQL or custom store logic that must stay portable across the backends you ship on.
+
+## `runDialectParity`
+
+`runDialectParity(fixture)` registers a vitest suite (`describe('dialect parity: <name>')`) that runs a fixed scenario set against the database your fixture stands up — and asserts identical behaviour on each. It's exported from the **`@voltro/testing/dialect`** subpath — the only part of `@voltro/testing` that imports `vitest`, so `makeTestContext` consumers carry no `vitest` peer. The scenarios cover the contract the DSL relies on:
+
+- DDL idempotency (re-applying the schema is a no-op)
+- insert / query round-trip, update post-image, delete
+- `onChange` fires insert + update + delete events
+- `transactional` rolls back on throw (and drops queued change events) / commits and drains events in order
+- `upsert` (insert then update-on-conflict, no duplicate)
+- `updateMany` with a predicate, and the atomic compare-and-set (the wakeup-claim pattern)
+- `retryFilter` returns a stable `'retry' | 'noRetry'` decision
+
+The harness intentionally does **not** bundle docker setup — that lives next to the dialect, so the parity packages' devDeps stay minimal. Your fixture owns the lifecycle (scratch docker Postgres, temp-file SQLite, …).
+
+```ts
+// sql-postgres/__tests__/parity.test.ts
+import { runDialectParity, applySchema } from '@voltro/testing/dialect'
+import { postgresDialect } from '../src/index'
+
+runDialectParity({
+  dialect: postgresDialect,
+  name: 'postgres (scratch docker)',
+  setup:    async () => { /* stand up a clean scratch database */ },
+  teardown: async () => { /* drop it */ },
+  make:     async () => ({
+    store:   /* the live DataStore for this scenario */,
+    migrate: (tables) => /* applySchema(tables, dialect.id) against the same runtime */,
+    dispose: async () => { /* close store, drop scratch state */ },
+  }),
+})
+```
+
+## The fixture contract
+
+`DialectFixture`:
+
+| Field | Meaning |
+|---|---|
+| `dialect` | The `SqlDialect` under test (its `id`, `retryFilter`, etc.). |
+| `name?` | Label for the test output (defaults to `dialect.id`). |
+| `setup` | Runs before each scenario — give it a clean slate. |
+| `teardown` | Runs after each scenario. |
+| `make` | Returns a `DialectFixtureSession` for the scenario. |
+
+`DialectFixtureSession`:
+
+| Field | Meaning |
+|---|---|
+| `store` | The live `DataStore` the scenarios drive. |
+| `migrate(tables)` | Apply `applySchema(tables, dialect.id)` against the same runtime the store uses, so DDL and DML hit the same database. |
+| `dispose()` | Tear the session down (close store, drop scratch DB). |
+
+`applySchema` is re-exported from `@voltro/testing/dialect` (the same subpath as `runDialectParity`, not the root entry) so a fixture can build its `migrate` without pulling `@voltro/database/sql` separately. Once a dialect green-checks the harness, it's ready for `loadDialect()`.