playthrough 0.3.1 → 0.4.1

package/INSTRUCTIONS.md

@@ -3,7 +3,7 @@ to its end; each command then says what's next.
 
   playthrough start <review> "<title>" "<what was asked>"
                                      start a review: a title for the change, and what was asked, in the person's words
-  playthrough start <review>             once its flows are written, record Before: the app as it is
+  playthrough start <review>             once its flows are written, try the setup and keep Before: the app as it is
   playthrough record <review>            record the round being worked on, every flow, the way a person would use it
   playthrough wait <review>              hand the round over, then wait for the notes to answer
   playthrough walk <review>              walk the person's steps in the app again, on the app as it is now
@@ -14,13 +14,15 @@ THE LOOP
 1. Start a review with a title, as you'd title its pull request, and what was asked, word for word: it's what the person
    judges each round against. Before you change anything, write its flows, one for each thing a person does that the
    task touches, from the task, not from the code.
-2. Run playthrough start <review> again. It records Before: each flow on the app as it is, as far as it can go. There,
-   what a flow expects of the finished work only waits a moment; the flow stops only at a step the app can't do yet.
+2. Run playthrough start <review> again. It tries the app's setup, then keeps the app as it is as Before, for the person
+   to compare each round with. Nothing's recorded yet: the flows are written for the work.
 3. Change the app. Keep each flow's name, and each step's while it does the same job: they're matched across rounds by
    name. Once the work is done, add a flow for each choice you made that the person can see and didn't ask for.
-4. playthrough record <review> records the round and compares it with every round before it. In the round's round.json,
+4. playthrough record <review> records the round, and each flow new to the review on Before too, doing what the round
+   does as far as the app as it was can, then compares the round with every round before it. In the round's round.json,
    say why each change since the round before is there, under "accounts" by its id: { "line": "<why, in your words>" },
-   { "note": "<the id of the note it was made for>" }, or { "notReal": "<why nothing really changed>" }.
+   { "note": "<the id of the note it was made for>" }, or { "notReal": "<why nothing really changed>" }. What only the
+   pictures show is the person's to look at, and asks nothing of you, unless you made it for a note: then say so.
 5. playthrough wait <review> hands the round over and waits for the notes, which can take hours: run it in the
    background, so you're told when it ends. Each note comes with a picture of what it's on, or what the person did in
    the app. Change the app, record the next round, and answer each note in its round.json, under "replies", in its
@@ -33,16 +35,18 @@ The person can only judge what the flows show, so keep them true to how a person
 
 THE APP
 
+The first time, add playthrough to the app's devDependencies with the app's own package manager, and run
+npx playwright install chromium: flows import from it, and record in that browser.
+
 playthrough/app.json says how to start the app, and is committed so every review uses it:
 
   { "start": "npm run dev", "url": "http://localhost:3000" }
 
 Flows run several at once. Where a page shows what every run made, as a blog's index lists every post, add
 "oneAtATime": true. Every round is recorded on whatever serves that url, and Playthrough starts the app only when nothing
-does: leave a server you didn't start running, and let it follow the working copy, as a dev server does. A flow recorded
-on Before later sets your work aside with git stash for a minute, puts the app as it was in its place, with playthrough/
-as it is now, and puts your work back after, even when Playthrough is stopped. Until round 1 is handed over, Before
-follows what you fix: a flow whose file changes is recorded on it again, and so is every flow once playthrough/ changes.
+does: leave a server you didn't start running, and let it follow the working copy, as a dev server does. Recording on
+Before sets your work aside with git stash for a minute, puts the app as it was in its place, with playthrough/ as it is
+now, and puts your work back after, even when Playthrough is stopped.
 
 FLOWS
 
@@ -64,7 +68,7 @@ name is the flow's:
 Act on the page as a person would, by role and label. Playthrough glides the pointer, types a key at a time, and reads
 the page once it rests after each step. A dialog the browser shows is recorded, and answered No unless the flow says Yes
 first: page.once("dialog", (dialog) => dialog.accept()). A flow that fails stops the round: record names it and the
-step, and keeps the page it failed on in .playthrough/<review>/failed/, as Before does where each flow stops.
+step, and keeps the page it failed on in .playthrough/<review>/failed/.
 
 While the person watches, Playthrough runs the flow again in the background, so taking it over is quick. If its steps do
 something outside the app that mustn't happen again, like sending an email, say so, and it runs only when taken over:

package/README.md

@@ -1,4 +1,4 @@
-# Playthrough
+# <img src="https://raw.githubusercontent.com/itmostlyworks/playthrough/main/packages/viewer/public/logo.svg" alt="" height="28"> Playthrough
 
 The agent says it's done. You decide if it's good.
 
@@ -7,37 +7,28 @@ Playthrough records your coding agent's change the way a person would use it, be
 ## How a review goes
 
 1. You ask your agent for a change, and to show it to you with Playthrough.
-2. Before it touches the code, the agent writes a flow for each thing a person does that the change touches, and Playthrough records them on the app as it is: Before.
-3. The agent makes the change, and Playthrough records every flow again. It finds what changed itself, including what the agent didn't mention, and the agent says why each change is there.
+2. Before it touches the code, the agent writes a flow for each thing a person does that the change touches, and Playthrough keeps the app as it is: Before.
+3. The agent makes the change, and Playthrough records every flow, on the app as it is now and on Before, as far as the app went before. It finds what changed itself, including what the agent didn't mention, and the agent says why each change is there.
 4. You watch. Leave a note on what's wrong, where it's wrong; compare with Before; take over and try it yourself. Then send.
 5. The agent answers each note in the next round, on the spot the note is on. You check its answers, and accept once every flow looks good.
 
 ## Try it
 
-Playthrough needs Node 24 or later. In your app's repo:
+Ask your agent for a change in your app, as you would anyway, and add:
 
-```bash
-npm install --save-dev playthrough
-npx playwright install chromium
-```
-
-Then ask your agent for a change, and to show it to you:
+> Show me this change with `npx playthrough`.
 
-> Show only the overdue invoices. Use Playthrough to show me the change: run `npx playthrough` and follow what it says.
-
-The agent starts a review, records it, and hands it to you with `playthrough wait`. Open it:
+The agent does the rest. The first time, it adds Playthrough to your app and writes down how to start it and sign in, to commit with your app. Each time, it records the change and hands it to you. Open it:
 
 ```bash
 npx playthrough open
 ```
 
-The first time, the agent writes `playthrough/app.json`, which says how to start your app, and `playthrough/setup.ts` if flows need something first: a signed-in user, an account with data in it, or the people who use the app. Both are committed with your app, so every review after uses them.
-
-Reviews are kept in `.playthrough/` in your app's repo, recordings and all. Nothing is uploaded anywhere.
+Reviews stay in `.playthrough/` in your app's repo, recordings and all; nothing is uploaded. Playthrough needs Node 24 or later.
 
 ## On the example
 
-`example/` in this repo is a small invoices app to try it on before your own. Clone the repo, run `npm install`, `npx playwright install chromium` and `npm run build`, and ask your agent for the same change from `example/`.
+`example/` in this repo is a small invoices app to try it on before your own. Clone the repo, run `npm install`, `npx playwright install chromium` and `npm run build`, and from `example/`, ask your agent to show only the overdue invoices, and to show you the change with `npx playthrough`.
 
 ## In the viewer
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "playthrough",
-  "version": "0.3.1",
+  "version": "0.4.1",
   "description": "Review an AI agent's work on a web app by watching recordings of it.",
   "type": "module",
   "license": "Apache-2.0",
@@ -13,7 +13,7 @@
     "node": ">=24"
   },
   "bin": {
-    "playthrough": "./src/playthrough.js"
+    "playthrough": "src/playthrough.js"
   },
   "exports": {
     "./test": {

package/src/{flows-BcISV1M-.js → flows-BkG0TgHA.js}

@@ -1,13 +1,13 @@
-import { C as Refused, M as list, N as quote, h as glimpsesOf, m as firstLine, w as Reviews, y as readingsOf } from "./reporter-BEodHYRI.js";
+import { A as list, b as Refused, g as readingsOf, j as quote, m as glimpsesOf, p as firstLine, x as Reviews } from "./reporter-DE8aueAO.js";
 import * as modules from "node:module";
 import { createRequire } from "node:module";
 import { existsSync, watch } from "node:fs";
 import { spawn } from "node:child_process";
 import { mkdir, mkdtemp, readFile, readdir, rename, rm, stat, writeFile } from "node:fs/promises";
+import { createHash, randomUUID } from "node:crypto";
 import { constants, tmpdir } from "node:os";
 import { dirname, join, relative } from "node:path";
 import { chromium, request } from "@playwright/test";
-import { createHash, randomUUID } from "node:crypto";
 import { fileURLToPath, pathToFileURL } from "node:url";
 // ../review/src/steps.ts
 function inStep(at, from, to, clicks) {
@@ -1827,16 +1827,20 @@ function unsteadyIn(flows) {
 async function comparing(page, earlier, round, unsteady = () => /* @__PURE__ */ new Set()) {
 	const changes = /* @__PURE__ */ new Map();
 	const pieces = /* @__PURE__ */ new Map();
-	const see = ({ id, words, place, of = [id] }) => {
+	const see = ({ id, words, place, of = [id], pictureOnly }) => {
 		if (unsteady(place.flow).has(id)) return;
 		const change = changes.get(id);
 		for (const piece of of) (pieces.get(id) ?? pieces.set(id, /* @__PURE__ */ new Set()).get(id)).add(piece);
 		if (!change) changes.set(id, {
 			id,
 			words,
-			seen: [place]
+			seen: [place],
+			...pictureOnly && { pictureOnly }
 		});
-		else if (!change.seen.some((seen) => seen.flow === place.flow)) change.seen.push(place);
+		else {
+			if (!change.seen.some((seen) => seen.flow === place.flow)) change.seen.push(place);
+			if (!pictureOnly) delete change.pictureOnly;
+		}
 	};
 	for (const flow of round.flows) {
 		const before = earlier.flows.find((kept) => kept.name === flow.name);
@@ -1874,7 +1878,8 @@ async function compareFlow(page, earlier, before, round, flow, unsteady) {
 			then: around(glimpsed.then, before, was.step),
 			now: around(glimpsed.now, flow, reading.step)
 		};
-		for (const found of [...changesAt(was, reading, unsteady, shown), ...await picturesAt(page, was, reading, pictures)]) {
+		const pictured = await picturesAt(page, was, reading, pictures);
+		for (const found of [...changesAt(was, reading, unsteady, shown), ...pictured]) {
 			const on = found.element ? {
 				element: found.element,
 				...found.alike && { alike: found.alike }
@@ -1887,7 +1892,8 @@ async function compareFlow(page, earlier, before, round, flow, unsteady) {
 					at: reading.at,
 					...on
 				},
-				of: found.pieces
+				of: found.pieces,
+				...pictured.includes(found) && { pictureOnly: true }
 			});
 		}
 	}
@@ -1985,7 +1991,7 @@ async function trySetup(repo, again) {
 	console.log(`\n${line("The app", startedIn === void 0 ? `running already at ${app.url}` : `started with ${quote(app.start)}, answering at ${app.url} in ${seconds(startedIn)}`)}`);
 	for (const { who, did, took, at, refused, picture } of tried) console.log(line(who, `${did}${did === "nothing made" ? "" : ` in ${seconds(took)}`}, then at ${at}${refused ? `, which the app answered with ${refused}` : ""}: ${repo.show(picture)}`));
 	const missing = [];
-	if (!setup.after && tried.some(({ made }) => made)) missing.push("playthrough/setup.ts has no after, so what each run makes stays in the app: one for each flow recorded, two for a new one, and one each time the person pauses a recording, as it's made ready to take over.");
+	if (!setup.after && tried.some(({ made }) => made)) missing.push("playthrough/setup.ts has no after, so what each run makes stays in the app: one for each flow recorded, three for a new one, and one each time the person pauses a recording, as it's made ready to take over.");
 	const slowest = Math.max(...tried.map(({ took }) => took));
 	if (slowest > SLOW) missing.push(`Setup took ${seconds(slowest)}: each flow waits that long before its first step, and so does each moment taken over.`);
 	console.log(`
@@ -2100,16 +2106,20 @@ function howToWriteFlows(repo, id) {
 The test's name is the flow's, and steps are matched across rounds by their names. What every flow starts from, like
 the account it opens, is made in playthrough/setup.ts${missing}`;
 }
-async function runFlows(repo, id, round, { later, first }) {
+async function flowsOf(repo, id) {
 	const flows = repo.flows(id);
 	if (!(await readdir(flows).catch(() => [])).some(isFlow)) throw new Refused([`${id} has no flows yet. ${howToWriteFlows(repo, id)}`]);
 	await makeFlowsFolder(repo, id);
+	return flows;
+}
+async function runFlows(repo, id, round, { before, first = () => false }) {
+	const flows = await flowsOf(repo, id);
 	const app = await repo.app();
-	if (!later) await trySetup(repo, round === 0 ? `playthrough start ${id}` : `playthrough record ${id}`);
+	if (!before) await trySetup(repo, `playthrough record ${id}`);
 	const { started, rounds } = await repo.reviews.get(id);
 	const recorder = join(dirname(createRequire(import.meta.url).resolve("@rrweb/record")), "record.umd.min.cjs");
 	const longest = (rounds.filter((kept) => kept.number < round).at(-1)?.flows ?? []).toSorted((a, b) => b.length - a.length).map((flow) => flow.name);
-	const run = (only, results, stops) => runOnce({
+	const run = (only, results) => runOnce({
 		flows,
 		only,
 		longest,
@@ -2117,16 +2127,20 @@ async function runFlows(repo, id, round, { later, first }) {
 		app,
 		root: repo.root,
 		recorder,
-		before: round === 0,
-		taken: later?.taken ?? [],
+		before: !!before,
+		taken: before?.taken ?? [],
 		started,
-		stops
+		...before && { doneInRound: before.doneInRound }
 	});
-	const last = round > 0 ? await failedLastTime(repo, id) : [];
+	if (before) {
+		console.log(`Recording ${called(before.flows)} on Before, the app as it was…`);
+		return run(before.flows);
+	}
+	const last = await failedLastTime(repo, id);
 	if (last.length) {
 		console.log(`Running ${called(last)} first: ${last.length > 1 ? "they" : "it"} failed last time…`);
 		const { recorded, failed } = await run(last, repo.failed(id));
-		await keepFailed(repo, id, round, failed);
+		await keepFailed(repo, id, failed);
 		const [one] = failed;
 		if (failed.length > 1) throw new Refused([
 			`\nThese flows failed again, so the other flows weren't run:`,
@@ -2137,20 +2151,15 @@ async function runFlows(repo, id, round, { later, first }) {
 		const gone = !recorded.length;
 		console.log(`\n${last.length > 1 ? gone ? "They aren't flows any more" : "They pass now" : gone ? "It isn't a flow any more" : "It passes now"}, so every flow is recorded.\n`);
 	}
-	console.log(later ? `Recording ${called(later.flows)} on Before, the app as it was…` : round === 0 ? `Recording Before: each flow of ${id} on the app as it is…` : `Recording round ${round} of ${id}…`);
-	const once = await run(later?.flows, repo.failed(id));
-	await keepFailed(repo, id, round, once.failed);
+	console.log(`Recording round ${round} of ${id}…`);
+	const once = await run(void 0, repo.failed(id));
+	await keepFailed(repo, id, once.failed);
 	const again = once.recorded.filter(({ flow }) => first(flow.name));
-	if (!again.length || round > 0 && once.failed.length) return once;
+	if (!again.length || once.failed.length) return once;
 	console.log(`\nRecording ${called(again.map(({ flow }) => flow.name))} again, to see what changes by itself…`);
-	const stops = Object.fromEntries(once.failed.flatMap(({ flow, at }) => at ? [[flow, at]] : []));
-	const results = once.failed.length ? void 0 : repo.failed(id);
-	const twice = await run(again.map(({ flow }) => flow.name), results, stops);
-	if (results) await keepFailed(repo, id, round, twice.failed);
-	const unlike = again.filter(({ flow }) => {
-		const failed = (ran) => ran.failed.find((failure) => failure.flow === flow.name)?.at;
-		return !twice.recorded.some((kept) => kept.flow.name === flow.name) || failed(once) !== failed(twice);
-	});
+	const twice = await run(again.map(({ flow }) => flow.name), repo.failed(id));
+	await keepFailed(repo, id, twice.failed);
+	const unlike = again.filter(({ flow }) => !twice.recorded.some((kept) => kept.flow.name === flow.name));
 	if (unlike.length) throw new Refused([
 		`\n${list(unlike.map(({ flow }) => quote(flow.name)))} didn't go the same way twice on the same app:`,
 		...unlike.map(({ flow }) => `  ${quote(flow.name)}: the first time, ${went(once, flow.name)}; the second, ${went(twice, flow.name)}.`),
@@ -2176,7 +2185,7 @@ function went(ran, name) {
 	const because = failure.why ? `, where Playwright says “${failure.why}”` : "";
 	return failure.at ? `it stopped at ${quote(failure.at)}${because}` : `it stopped before its first step${because}`;
 }
-async function keepFailed(repo, id, round, failed) {
+async function keepFailed(repo, id, failed) {
 	const folder = repo.failed(id);
 	if (!failed.length) return rm(folder, {
 		recursive: true,
@@ -2186,7 +2195,6 @@ async function keepFailed(repo, id, round, failed) {
 		const path = join(folder, entry);
 		if (entry === ".last-run.json" || !(await readdir(path)).length) await rm(path, { recursive: true });
 	}
-	if (round === 0) return;
 	await mkdir(folder, { recursive: true });
 	await writeFile(join(folder, FAILED), JSON.stringify(failed.map(({ flow }) => flow)));
 }
@@ -2222,13 +2230,12 @@ async function runOnce(run) {
 			written: {}
 		};
 		if (!ran && !failed.length) throw new Refused(["\nThe flows didn't run, for the reason Playwright gives above. Fix that and run this again."]);
-		const withFiles = ({ flow, files, wrote }) => Promise.all(files.map(async (name) => ({
+		const withFiles = ({ files, ...recorded }) => Promise.all(files.map(async (name) => ({
 			name,
 			bytes: await readFile(join(folder, name))
 		}))).then((files) => ({
-			flow,
-			files,
-			...wrote && { wrote }
+			...recorded,
+			files
 		}));
 		return {
 			recorded: await Promise.all(recorded.map(withFiles)),
@@ -2311,4 +2318,4 @@ async function playwright(run) {
 		});
 	});
 }
-export { make as C, pausedAt as D, repo as E, giveBackLeft as S, setupIn as T, keepRunning as _, failedAt as a, changes as b, makeFlowsFolder as c, walkFork as d, whereItStopped as f, shownAt as g, placeIn as h, called as i, runFlows as l, compareWithEarlier as m, REACHES_OUTSIDE as n, howToWriteFlows as o, trySetup as p, WALKED as r, isFlow as s, ALONGSIDE as t, waitsFor as u, serves as v, personIn as w, firstOf$1 as x, stopApp as y };
+export { firstOf$1 as C, setupIn as D, personIn as E, repo as O, changes as S, make as T, placeIn as _, failedAt as a, serves as b, isFlow as c, waitsFor as d, walkFork as f, compareWithEarlier as g, writtenNow as h, called as i, pausedAt as k, makeFlowsFolder as l, trySetup as m, REACHES_OUTSIDE as n, flowsOf as o, whereItStopped as p, WALKED as r, howToWriteFlows as s, ALONGSIDE as t, runFlows as u, shownAt as v, giveBackLeft as w, stopApp as x, keepRunning as y };