@davidrydberg/proofy 0.0.0-stage → 0.1.0

package/templates/e2e/AGENTS.md

@@ -0,0 +1,90 @@
+# End-to-end tests with proofy
+
+This folder holds end-to-end tests that check what the people who run this app asked for, through the real app in a real browser.
+You, the agent, write the requirements, the tests and the fake outside services.
+The human approves them.
+`proofy run` then checks the app in a sealed Docker sandbox with no AI, and every result must be one the human can trust.
+
+Run `proofy` at any time: it shows what is set up, every flow's state and the next step.
+
+## Rules
+
+These are not optional.
+A test suite that breaks them produces green reports nobody can trust, which is worse than no tests.
+
+1. Never run `proofy approve`. Only the human approves.
+2. Never weaken a check to make a run green. A red result caused by the app is a finding: report it to the human with the evidence.
+3. Never edit an approved requirements file without asking the human first.
+4. Write requirements before reading the app's code, from what the human asked for. Read the code only after they are written, to find buttons and fields.
+5. Write fakes for outside services from the provider's public documentation, never from what this app sends. Link the documentation at the top of the fake.
+6. Every claim compares a real value: an amount, a status, a count, a stored record, a request a fake received or an email in Mailpit. Text on the screen alone does not prove a requirement, unless the requirement is about what the screen shows.
+7. Every test creates its own data with `tag()`. Never rely on another test's data or on order.
+
+## Files
+
+| Path | What |
+|---|---|
+| `proofy.json` | Config. Every field is described in its `$schema` |
+| `proofy.lock.json` | Approvals, written only by `proofy approve`. Commit it |
+| `specs/<flow>.md` | Requirements for one flow, numbered R1, R2 ... |
+| `tests/<flow>.spec.ts` | The test for that flow, same name |
+| `standins/<host>.mjs` | A fake for one outside host |
+| `templates/` | Examples to copy |
+| `out/` | Runs, ignored by git |
+
+## Setup
+
+Do this once, when the human asks you to set up proofy.
+`proofy` lists what is missing.
+
+1. **The app in Docker.** `compose` in `proofy.json` lists the app's compose files. If the repository has none, draft one from `templates/compose.yml`, show it to the human and wait for their OK. The app must start from an empty database, and everything it needs must be installed when the image is built: the sandbox has no internet once it runs. Give the database a `healthcheck`, so the app starts only when it is ready.
+2. **Where the browser starts.** `app` is the compose service and the port inside the container, not a published port.
+3. **Settings.** `env` holds the app's settings per service. `proofy init` filled it from `.env.example` and replaced secret-looking values with fakes. The repository's `.env` is never used. Check every value makes sense for an empty sandbox.
+4. **The seed.** `seed` is one command that prepares the empty database, such as migrations and the reference data every test needs. Test-specific data does not belong here. Leave it out if the app needs nothing.
+5. **Commands for tests.** `exec` names the commands tests may run inside a service, such as reading a stored order or running a scheduled job. A test calls `exec("name", [extra, args])` and gets stdout. Prefer the app's own CLI over raw SQL.
+6. **Outside services.** Find every host the app calls: search the code and config for `http://` and `https://`, SDK clients and mail settings. List each in `hosts`:
+   - `fake` when the app needs a real answer, such as a payment or a shipping quote. Write `standins/<host>.mjs` from `templates/standin.mjs` and the provider's public docs.
+   - `record` when the app ignores the answer, such as analytics or tracking.
+   - Leave a host out on purpose only if the app copes with it being down. Every refused call is listed in the report.
+7. **HTTPS.** Fakes answer HTTPS with a certificate authority proofy creates. Make the app trust it in the compose file; `templates/compose.yml` shows how for PHP, Node and Python.
+8. **Mail.** Point the app's SMTP settings at host `mailpit`, port `1025`, no TLS.
+9. **Check it.** Run `proofy dev up`. It builds the images, starts the sealed sandbox and seeds it. Fix what fails, using `e2e/out/sandbox.log` and `e2e/out/seed.log`. Run `proofy` until setup shows no gaps, then tell the human what you set up and what you could not.
+
+## Adding a flow
+
+1. **Interview.** Ask until every wish is checkable. Always cover:
+   - Who does it: a guest, a customer, an admin? Starting from which page?
+   - With what: which product, quantity, country, currency, payment method?
+   - What counts as success, as something you can read: the order's status and total, an email to whom, a stock count, a request to which outside service?
+   - What must never happen: charging twice, an order without payment, an email to the wrong person?
+   Ask one round of questions at a time. Do not guess an answer the human has not given.
+2. **Requirements.** Write `specs/<flow>.md` from `templates/spec.md`. Short lowercase name with dashes, such as `guest-checkout`. Show it to the human and wait for their OK.
+3. **Walk it.** Start `proofy dev up`, open the app address it prints and go through the flow once by hand to learn the buttons and fields. chrome-devtools-axi or Playwright's own tools are fine here. If the app redirects to its in-sandbox name, learn the page from its templates instead, or from a first draft of the test with `shot()` after each step.
+4. **Test.** Write `tests/<flow>.spec.ts` from `templates/test.spec.ts`. Import only from `@davidrydberg/proofy/test` and Node's built-in modules; nothing else is installed in the runner. From that module:
+   - `test`, `expect`: Playwright's, with proofy's bookkeeping. The browser starts at the `app` address.
+   - `claim({ spec, did, expected, actual, ok, evidence })`: one per requirement, citing its number. `ok` comes from real values.
+   - `notCovered(spec, why)`: for what the sandbox cannot show, such as a payment inside a provider's own card fields.
+   - `shot(label)`: a screenshot after every meaningful step.
+   - `tag()`: a value unique to the test for names, emails and phone numbers.
+   - `exec(name, args)`: a command from `exec` in `proofy.json`; returns stdout.
+   - `mail.for(address, { atLeast, settleMs })`: that person's email from Mailpit, waiting until `atLeast` arrived. `settleMs` keeps reading until nothing new came for that long, so "exactly one" also catches a late second email. Use `atLeast: 0` to check nothing was sent.
+   - `mail.search(query, { atLeast, settleMs })`: every email matching a Mailpit search, whoever it went to. Search for something only this test's data has, such as a `tag()` value. Each email has `to`, `cc` and `bcc`.
+   - `standin.requests(host, { atLeast })`: what the app sent to a fake or recorded host.
+5. **Prove it.** With `proofy dev up` running, run `proofy dev test <flow>` until three runs in a row give identical results per requirement. `proofy` shows the count. A requirement that fails because the app is wrong stays red: that is a finding, not a broken test. A flow whose runs differ is flaky: fix the test, never add retries.
+6. **Second opinion.** Ask a separate agent, given only `specs/<flow>.md` and `tests/<flow>.spec.ts` and no app code, to answer for each requirement: "what wrong behaviour of the app would still pass this check?" Tighten the checks it finds, then give the human its answers.
+7. **Hand over.** Tell the human the flow is ready, what each requirement checks, any findings, and that they can run `proofy approve <flow>`.
+
+## Changing a flow
+
+- When a change to the app touches a covered flow, propose the requirement changes to the human before writing code.
+- A changed requirements file, test or fake makes the flow out of date until the human approves it again. Other flows keep running.
+- When the app changed on purpose, update the requirements first, then the test, then prove it again.
+
+## When a run is red
+
+1. Read `out/<run>/report.md`: per requirement what was expected, what happened, the evidence and the screenshots. Red tests also have a Playwright trace.
+2. Decide which it is and say which, with the evidence:
+   - **The app is wrong.** Report it as a bug. Do not touch the test.
+   - **The test is wrong.** For example it typed into a field that was already filled in. Fix it, prove it again, and tell the human; it needs approval again.
+   - **A fake is wrong.** Check it against the provider's docs, fix it, and tell the human.
+   - **The requirement is unclear.** Ask the human.

package/templates/e2e/templates/compose.yml

@@ -0,0 +1,33 @@
+# Example compose file for an app proofy can test.
+# proofy adds its own network, fakes, Mailpit and the browser around it. Keep this file about the app only.
+#
+# Rules for a sandbox-ready compose file:
+# - No named volumes for data: every run starts from an empty database.
+# - No published ports needed: the browser runs inside the sandbox. A published port is fine for your own use.
+# - Settings come from e2e/proofy.json "env", not from an env_file pointing at .env.
+# - The app trusts proofy's certificate authority, mounted at /proofy/ca.crt, so it accepts the fakes' HTTPS.
+
+services:
+  app:
+    build: .
+    depends_on: [db]
+    expose: ["8080"]
+    environment:
+      DB_HOST: db
+      MAIL_HOST: mailpit
+      MAIL_PORT: "1025"
+      # Trust proofy's certificate authority. Keep the lines for your stack:
+      # Node:
+      NODE_EXTRA_CA_CERTS: /proofy/ca.crt
+      # Python (requests, httpx):
+      # REQUESTS_CA_BUNDLE: /proofy/ca-bundle.crt
+      # SSL_CERT_FILE: /proofy/ca-bundle.crt
+      # PHP (curl, Guzzle): add to the image or entrypoint:
+      #   cp /proofy/ca.crt /usr/local/share/ca-certificates/proofy.crt && update-ca-certificates
+
+  db:
+    image: mysql:8.4
+    environment:
+      MYSQL_DATABASE: app
+      MYSQL_ROOT_PASSWORD: sandbox
+    tmpfs: [/var/lib/mysql]

package/templates/e2e/templates/spec.md

@@ -0,0 +1,20 @@
+# Guest checkout
+
+Asked by: the shop owner, 2026-10-20.
+A guest buys one product and pays with an offline payment method.
+
+## Requirements
+
+R1. After paying, the guest sees a thank-you page with an order number.
+R2. The order with that number has status "processing" and a total of 249.00 SEK including shipping.
+R3. The guest receives one confirmation email with the order number.
+R4. Stock for the product goes down by one.
+
+## Must never happen
+
+R5. Leaving the payment step without paying creates no order.
+R6. The confirmation email goes to anyone but the guest.
+
+## Not covered
+
+- Paying by card inside the provider's own card fields: the sandbox is sealed, so the provider's browser code cannot load.

package/templates/e2e/templates/standin.mjs

@@ -0,0 +1,29 @@
+// Fake for api.example-payments.com.
+// Written from the provider's public docs: https://docs.example-payments.com/api/payments
+// Never from what this app sends: a fake that copies the app agrees with every bug in it.
+// Save as e2e/standins/<host>.mjs and list the host as "fake" in e2e/proofy.json.
+
+/**
+ * Called for every request to this host. proofy records every request before calling it.
+ * @param {{ method: string, path: string, query: Record<string, string>, headers: Record<string, string>, body: string, json: unknown }} req
+ * @param {{ state: Map<string, unknown> }} ctx state lives for one run, shared by every request to this host
+ * @returns {{ status: number, json?: unknown, body?: string, headers?: Record<string, string> }}
+ */
+export default function handle(req, ctx) {
+  if (req.method === "POST" && req.path === "/v1/payments") {
+    const amount = req.json?.amount;
+    // Reject what the real service rejects, as its docs describe, so a wrong request from the app shows up as red.
+    if (!Number.isInteger(amount) || amount <= 0) {
+      return { status: 400, json: { error: { code: "amount_invalid", message: "amount must be a positive integer in minor units" } } };
+    }
+    const id = `pay_${ctx.state.size + 1}`;
+    const payment = { id, amount, currency: req.json.currency, status: "succeeded" };
+    ctx.state.set(id, payment);
+    return { status: 201, json: payment };
+  }
+  if (req.method === "GET" && req.path.startsWith("/v1/payments/")) {
+    const payment = ctx.state.get(req.path.slice("/v1/payments/".length));
+    return payment ? { status: 200, json: payment } : { status: 404, json: { error: { code: "not_found" } } };
+  }
+  return { status: 404, json: { error: { code: "not_faked", message: `${req.method} ${req.path} is not in this fake` } } };
+}

package/templates/e2e/templates/test.spec.ts

@@ -0,0 +1,28 @@
+// Requirements: e2e/specs/guest-checkout.md
+import { claim, exec, mail, notCovered, shot, tag, test } from "@davidrydberg/proofy/test";
+
+test("guest checkout", async ({ page }) => {
+  const guest = { name: `Guest ${tag()}`, email: `guest-${tag()}@example.test` };
+
+  await page.goto("/product/coffee/");
+  await page.getByRole("button", { name: "Add to basket" }).click();
+  await shot("coffee in the basket");
+
+  await page.goto("/checkout/");
+  await page.getByLabel("Email").fill(guest.email);
+  await page.getByLabel("First name").fill(guest.name);
+  await page.getByRole("button", { name: "Place order" }).click();
+  await shot("thank-you page");
+
+  const orderNumber = (await page.getByTestId("order-number").textContent())?.trim() ?? "";
+  claim({ spec: "R1", did: "placed the order as a guest", expected: "an order number on the thank-you page", actual: orderNumber, ok: /^\d+$/.test(orderNumber) });
+
+  // Read the stored order, not the screen: the page can say "thank you" while nothing was saved.
+  const order = JSON.parse(await exec("order", [orderNumber])) as { status: string; total: string };
+  claim({ spec: "R2", did: `read order ${orderNumber}`, expected: "processing, 249.00", actual: `${order.status}, ${order.total}`, ok: order.status === "processing" && order.total === "249.00", evidence: order });
+
+  const messages = await mail.for(guest.email);
+  claim({ spec: "R3", did: `read ${guest.email}'s mail`, expected: "one email with the order number", actual: `${messages.length} emails`, ok: messages.length === 1 && messages[0]!.text.includes(orderNumber) });
+
+  notCovered("R4", "example only: read the stock count with exec() before and after");
+});