@impeccable/reviews 0.1.0

package/README.md

@@ -0,0 +1,68 @@
+# impeccable-reviews
+
+Sets a repository up for Impeccable for PRs from your terminal. Run it in a checkout of the repository: it signs you in through the browser, then a setup agent reads the checkout, runs what it needs, saves the settings through Impeccable's setup API, asks you only what it cannot find out, opens a test drive pull request for Impeccable to review (and a setup pull request to merge, when the repository needs one) and writes a setup report. A web page mirrors the run as it goes and follows the review.
+
+You need Node 22.19 or newer, git, and the `gh` command signed in to GitHub as someone who can push branches to the repository (the agent opens the pull request as you).
+
+## Run it
+
+In a checkout of the repository you want set up:
+
+```sh
+cd ~/code/your-repo
+npx @impeccable/reviews
+```
+
+`npm install -g @impeccable/reviews` puts the `impeccable-reviews` command on your PATH instead.
+
+Your browser opens Impeccable's consent page; after Approve it goes on to the run's web page. Keep that tab open: it shows the task list, a question waiting for you, the pull requests and, at the end, the report and the review. The terminal takes over its window while it works, one screen at a time in the same frame (the start; the sign-in, its link copied to the clipboard; understanding your app; a question when it needs you; the tasks; the test drive, then the setup pull request, each opened in your browser and copied to the clipboard as its screen appears), and gives it back with a short summary: the outcome, the test drive, the setup pull request, the web page and what is left for you.
+
+Keys: Enter goes on, Esc cancels the intro or the sign-in and skips a question (every question can be skipped); on the run screen, Esc twice (or Ctrl+C) stops the run. A screen taller than the terminal scrolls with ↑↓ and PgUp PgDn (↑↓ still move among a question's choices). The screens are black and white, with emphasis by intensity: greys on a terminal that takes true colour and reports its background, bold and dim elsewhere, and under `NO_COLOR`. The mark at the top is the logo's own image in a terminal that shows images (iTerm2, kitty, Ghostty, WezTerm; never under tmux), a square with its slash beside the letter-spaced wordmark elsewhere, and plain text on a dumb terminal.
+
+## Options
+
+| Flag | Default | What it does |
+| --- | --- | --- |
+| `--host <origin>` | `https://impeccable.reviews` | Impeccable's address. Plain `http` only for localhost. |
+| `--model <id>` | the model the service hands out (Sonnet 5.5 today) | The model the setup agent runs on, through Impeccable's gateway. |
+| `--effort <level>` | `high` | `off`, `minimal`, `low`, `medium`, `high`, `xhigh` or `max`. |
+| `--yes` | off | Runs the allowlisted commands without asking, which they already do; anything off the list is still refused. |
+| `--debug` | off | Writes a debug log and a transcript of the run (`transcript-<run>.md`, with its raw events in `transcript-<run>.jsonl`) to the temp folder, tokens removed, and prints where they are. |
+| `--transcript <path>` | the temp folder | Writes the transcript there instead: a folder, or a `.md` file with the `.jsonl` beside it. |
+| `--frames <dir>` | off | Writes each screen into the folder as it changes (`NNN-<state>.ansi` with its styling, `NNN-<state>.txt`, and `index.json`), for the eval. |
+
+## What happens
+
+1. **Sign in.** The consent page (pick the organization and this repository) answers a small server on the first free port of 8239, 8238, 8240, 8237, 8236 and 8235, with PKCE and a state check, then sends the browser on to the run's web page. On a remote shell, press P and paste the address the browser ends on.
+2. **A web front end?** Impeccable reviews web front ends, mobile web included. A checkout with none (a React Native, Expo or Flutter app without a web target, or a service with no page, template, stylesheet or web framework) ends here: the terminal and the web page say "Impeccable reviews web front ends, and this repository does not have one. Nothing was set up.", and it exits 2. When the checkout cannot tell, Impeccable's own scan decides. A checkout with a web front end that the scan skipped stops the same way, with the setup page, where the person chooses Add anyway before running it again.
+3. **Set up.** On a run with its own capped model token (model calls go through Impeccable's gateway, so this machine holds no model key), the agent reads the checkout, runs the build, saves settings through the setup API, and asks you in a dialog only what it cannot find out. It never asks you to run a command: what it cannot run goes into the report, with the command. When Vercel or Netlify already deploys previews of your pull requests, it opens the newest one (Netlify's it finds on GitHub, and through the Netlify CLI when you are signed in to it on this machine, read only), and once that page is your app it takes the host's previews as they are, with no question and no setup pull request. When the repository is set up for Vercel or Netlify but that host has never previewed it, it asks you for one thing, to sign in to the host, and only when its CLI on this machine is signed out: one screen, Sign in or Skip, the login in your browser on Enter, and a retry when it did not complete (two failed retries count as a skip). Then it does the rest itself. On Vercel it creates the project with `vercel link` in the team the CLI's current scope names (asking which, only when you have several and none is current), connects the repository, pushes the test drive and waits up to five minutes for Vercel's build of it (the task in progress carries the timer, "1:05 of 5:00"), proves that preview, takes Vercel's previews as they are and only then opens the test drive's pull request, so its review renders; a new project starts with Vercel Authentication on, which the run gets past with a bypass secret on your yes. On Netlify it creates the site in your Netlify team (asking which, only when you have several), connects the repository through Netlify's site settings and Netlify's GitHub app, so pull requests get deploy previews, pushes the test drive and opens it as a draft, which Impeccable leaves unreviewed, waits for its deploy preview, proves it, takes Netlify's previews as they are and marks the draft ready. When the host's GitHub app cannot reach the repository, one more screen opens the app's page on GitHub for you to give it access, and the run waits. On skip it builds Impeccable's own previews as usual.
+4. **Two pull requests,** opened as you with `gh`, even when something is left for a person, which goes into the report, both into the default branch and in this order:
+   - the test drive, safe to close (`impeccable/test-drive`, "Impeccable test drive (safe to close)"), always: a few real style or layout changes on a page the preview renders, on top of the setup commit when there is one, so it carries the workflow, whose job builds the preview on the pull request while Impeccable's review waits for it. Its pull request opens only once its preview is proven: the host's build of the pushed branch, or the local proof of Impeccable's own preview. No preview, no test drive: the run then says why in the report and what you can do, and still opens the setup pull request when the repository needs one;
+   - the setup pull request, to merge (`impeccable/setup`, "Set up Impeccable previews"), only when the repository needs a change to keep previews working: the preview workflow, any fix the build needed.
+5. **The end.** The run ends once the pull requests are open and recorded. The report (what it found, configured, skipped and could not verify, and what to do before merging) goes to the web page, which follows Impeccable's review of the test drive; the terminal ends on the test drive's screen (when there is a setup pull request too), then "You're all set up" with the setup pull request, or the test drive when there is none, and what is left, and its summary keeps both pull requests and the web page; exit 0. If something stops it, it says why in a sentence and exits 1.
+
+The sign-in's setup token lasts an hour. A run that goes longer asks you to sign in again, in the same terminal, and carries on.
+
+## What the agent may run
+
+The fence keeps the setup to its own steps and out of `.env` files; it is not a sandbox: the repository's scripts and builds run as you, with your files and your `gh` and git sign-ins, as they would for a developer, but without the tokens, keys and passwords in your environment.
+
+Without asking: `git status|diff|log|show|add|commit|checkout -b|push -u origin <branch>`, `gh pr create|view` and `gh pr ready` for the test drive's branch by name (a Netlify draft, marked ready once its preview is proven), `npm|pnpm|yarn|bun install|ci`, the scripts of the package.json where the command runs (`npm run`, `pnpm run`, `bun run`, `yarn <script>`), `npx`, `pnpm exec`, `yarn` and `bunx` for `tsc`, `eslint`, `prettier`, `next`, `vite`, `astro`, `nuxt`, `prisma`, `playwright`, `drizzle-kit`, `sequelize` and `knex`, and `ls`, `cat`, `head`, `grep` and `find` inside the checkout (by relative or absolute path) and the run's scratch folder. A package manager or tool runner may have `KEY=value` settings in front; any of these may start with a `cd` into a folder of the checkout and be chained with `&&`, `||`, `;` or `|` to others on the list.
+
+`vercel whoami` and `vercel --version`. Once you are signed in to Vercel for the setup: `vercel link --yes`, `vercel project add|ls`, `vercel git connect` and `vercel deploy` for a preview, on the project named after the folder and in the one team the sign-in settled on (the CLI's current scope, your only team, or the one you pick when it asks): no other `--scope`, `--team` or `--project`, and a global install of the Vercel CLI itself when it is missing (`npm install -g vercel`, or your package manager's global add). The sign-in (`vercel login`) is the run's own tool, never the agent's command. Never a production deploy, a project's removal, its environment variables or protection settings (`vercel project protection`, `api` and `curl` print a project's bypass secrets, so the agent's shell never runs them), or a token on the command line.
+
+`netlify --version` and `netlify status`. Once you are signed in to Netlify for the setup: `netlify link`, `netlify sites:create` and `netlify deploy` for a deploy preview (`--build --context deploy-preview`), in your own current Netlify team, or with `--account-slug` for the one team you picked when the run asked, and a global install of the Netlify CLI itself when it is missing (`npm install -g netlify-cli`, or your package manager's global add). The sign-in (`netlify login`) and the repository's connection (the site's settings, through Netlify's API) are the run's own tools, never the agent's commands; `netlify init` never runs. Never a production deploy, a site's removal, its environment variables, or a token on the command line.
+
+Scratch output, such as the log of a server the agent starts, goes to a folder in the OS temp directory, never into the checkout; a `>` redirect into that run's folder is the one write outside the checkout that runs, and the agent reads what it wrote there back (the path with its symlinks resolved).
+
+Refused, never asked, with the reason going back to the agent, which finds another way or leaves the step to you in the report: anything else (`node -e` too), a read outside the checkout (symlinks followed), a read of a `.env` file, a recursive `grep` without `--exclude=.env*`, a push to the default branch, a global install, publishing, a script whose name says it publishes, releases or deploys, a setting such as `PATH`, `EDITOR` or `NODE_OPTIONS` that changes what runs, a `cd` that may not run (after `&&` or `||`, or in a pipe) or goes into a folder that is not there, git patterns that can match a `.env` file, a `git add` of every change or of a folder without `':(exclude,glob)**/.env*'`, a commit message or template read from a `.env` file or from outside the checkout and the scratch folder, package scripts and the tools on the list once the agent has changed a `package.json` or a file they load (a `*.config.*` or `.*rc` file, `tsconfig*.json`, anything under `node_modules`…; after `vite.config.*`, whose preview hosts the setup may correct, the local preview check still runs), and, past 400 tool calls in a run, any but the progress update and the report. `rm -rf`, force pushes, `git reset --hard`, `sudo`, database commands that throw data away (`prisma migrate reset`, `db push --force-reset`, `sequelize db:drop`…), other writes outside the checkout and writes into `.git` never run.
+
+## Tokens
+
+The setup token and the run token stay in memory. The agent never sees them: the tools add them to their own requests. The debug log and the transcript remove them.
+
+A test account's password, typed hidden on the account's screen (after a screen that says why the run asks), goes from there to Impeccable's test-accounts setting: the agent only holds a reference to it, and the debug log and the transcript remove it.
+
+Vercel's bypass secret, which lets reviews open previews behind Vercel's login, is handled the same way. When Vercel's login stands in front of a preview, the command asks you once, on one screen, whether to create a bypass secret for Impeccable, and on your yes creates it as you through the Vercel CLI, labelled "Impeccable reviews" (`vercel api` for Vercel's protection-bypass route; an older CLI's `vercel project protection enable --protection-bypass`), then reads it back from the project's settings. A project that already holds a secret with that label gives it without a question. When the CLI is signed out, this folder is not linked, or your role cannot make one, a screen names the settings page and takes the secret you type, hidden. It opens the preview with the secret first and saves it, with the setting that sends it in the `x-vercel-protection-bypass` header, only once the page shows your app. The agent never holds it: the secret goes from the command's own process to Vercel and to Impeccable's setting, and the debug log and the transcript remove it. Vercel documents this as Protection Bypass for Automation, which every plan includes; you revoke it in the project's Settings, Deployment Protection.
+
+Working on the CLI itself (a local build, the smoke test, the evals): [docs/development.md](docs/development.md).

package/dist/agent.js

@@ -0,0 +1,183 @@
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createAgentSessionFromServices, createAgentSessionRuntime, createAgentSessionServices, createBashToolDefinition, defineTool, ModelRuntime, SessionManager, SettingsManager, } from '@earendil-works/pi-coding-agent';
+import { shellEnvironment } from './fence.js';
+import { createTranscript } from './transcript.js';
+import { reportStatus } from './ui/bridge.js';
+import { statusFromEvent } from './ui/run-state.js';
+import { atExit, showRun } from './ui/screens.js';
+import { onSensitiveAnswer } from './ui/sensitive.js';
+/** The provider pi sends every model call to: our gateway, with the run token as its key. */
+export const PROVIDER = 'impeccable';
+const BUILT_IN_TOOLS = ['read', 'bash', 'edit', 'write', 'grep', 'find', 'ls'];
+/** Sent when the model ends its turn without the report: nobody reads its prose, so it carries on or finishes. */
+const CARRY_ON = 'Only your [STATUS] lines reach the person; the rest of your message does not, and they cannot reply to it. Carry on with the setup: ask the person through impeccable_ask when you need them, and call impeccable_report once everything that can be done is done.';
+const MAX_CARRY_ONS = 2;
+/** A value pi reads literally: in its config values `$` interpolates the environment and a leading `!` runs a command. */
+const literal = (value) => value.replace(/\$/g, '$$$$').replace(/^!/, '$!');
+/**
+ * The model as our gateway serves it, with pi's catalog limits for that id when pi knows it. One newer than pi's catalog
+ * is a current Claude model: adaptive thinking at the effort asked for (a thinking budget is refused), never off.
+ */
+function gatewayModel(modelRuntime, id) {
+    const known = modelRuntime.getModel('anthropic', id);
+    if (!known) {
+        return { id, name: id, reasoning: true, input: ['text', 'image'], contextWindow: 200_000, maxTokens: 32_000, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+            thinkingLevelMap: { off: null, minimal: null, xhigh: 'xhigh', max: 'max' }, compat: { forceAdaptiveThinking: true } };
+    }
+    const { contextWindow, maxTokens, reasoning, input, cost, thinkingLevelMap, compat, promptCache, inputLimits } = known;
+    return { id, name: id, contextWindow, maxTokens, reasoning, input, cost, thinkingLevelMap, compat, promptCache, inputLimits };
+}
+/**
+ * Runs the setup agent on pi, headless: our own screen shows the run (or its
+ * status lines are printed), and nothing ever prompts but the agent's own
+ * questions. It ends with runtime.dispose(), whose session_shutdown hands over
+ * to the command's finish.
+ */
+export async function runAgent(opts) {
+    // Inside our command pi makes no calls of its own: no version check, no catalog refresh, no install telemetry.
+    process.env.PI_OFFLINE ??= '1';
+    // pi's own configuration stays out of this: no settings, credentials, extensions or sessions from ~/.pi.
+    const agentDir = mkdtempSync(join(tmpdir(), 'impeccable-reviews-pi-'));
+    process.on('exit', () => rmSync(agentDir, { recursive: true, force: true }));
+    const settingsManager = SettingsManager.inMemory({
+        quietStartup: true,
+        collapseChangelog: true,
+        enableInstallTelemetry: false,
+        enableAnalytics: false,
+        cacheWarming: 'off',
+        defaultProjectTrust: 'never',
+    });
+    const modelRuntime = await ModelRuntime.create({ authPath: join(agentDir, 'auth.json'), modelsPath: null });
+    const provider = (pi) => {
+        pi.registerProvider(PROVIDER, {
+            name: 'Impeccable',
+            baseUrl: opts.gatewayUrl,
+            apiKey: literal(opts.runToken),
+            api: 'anthropic-messages',
+            models: [gatewayModel(modelRuntime, opts.model)],
+        });
+        // The thinking comes back as a readable summary, for the transcript.
+        pi.on('before_provider_request', (event) => {
+            const body = event.payload;
+            return body.thinking && body.thinking.type !== 'disabled' ? { ...body, thinking: { ...body.thinking, display: 'summarized' } } : undefined;
+        });
+    };
+    const firstContext = (pi) => {
+        let sent = false;
+        pi.on('before_agent_start', () => {
+            if (sent)
+                return undefined;
+            sent = true;
+            return { message: { customType: 'impeccable-setup', content: opts.firstContext, display: false } };
+        });
+    };
+    // The debug log's view of the run: each model turn's text, outcome and usage, and each tool that ran.
+    const trace = (pi) => {
+        pi.on('message_end', (event) => {
+            const m = event.message;
+            if (m.role !== 'assistant')
+                return;
+            const text = (m.content ?? []).filter((c) => c.type === 'text').map((c) => c.text ?? '').join('\n').trim();
+            opts.log.write('model turn', { model: m.model, stopReason: m.stopReason, usage: m.usage, error: m.errorMessage, text: text || undefined });
+        });
+        pi.on('tool_execution_end', (event) => opts.log.write(`tool ${event.toolName}${event.isError ? ' failed' : ''}`));
+    };
+    const createRuntime = async ({ cwd, agentDir: dir, sessionManager, sessionStartEvent }) => {
+        const services = await createAgentSessionServices({
+            cwd,
+            agentDir: dir,
+            settingsManager,
+            modelRuntime,
+            resourceLoaderOptions: {
+                noExtensions: true,
+                noSkills: true,
+                noPromptTemplates: true,
+                noThemes: true,
+                noContextFiles: true,
+                systemPrompt: opts.systemPrompt,
+                extensionFactories: [provider, firstContext, ...opts.extensions, ...(opts.log.path ? [trace] : [])],
+            },
+        });
+        const model = services.modelRuntime.getModel(PROVIDER, opts.model);
+        if (!model)
+            throw new Error(`the model ${opts.model} did not register`);
+        const created = await createAgentSessionFromServices({
+            services,
+            sessionManager,
+            sessionStartEvent,
+            model,
+            thinkingLevel: opts.effort,
+            tools: [...BUILT_IN_TOOLS, ...opts.tools.map((t) => t.name)],
+            // pi's own bash, whose commands run without the person's credentials in their environment.
+            customTools: [defineTool(createBashToolDefinition(cwd, { spawnHook: (spawn) => ({ ...spawn, env: shellEnvironment(spawn.env) }) })), ...opts.tools],
+        });
+        return { ...created, services, diagnostics: services.diagnostics };
+    };
+    const runtime = await createAgentSessionRuntime(createRuntime, { cwd: opts.cwd, agentDir, sessionManager: SessionManager.inMemory(opts.cwd) });
+    const problems = runtime.diagnostics.filter((d) => d.type === 'error');
+    if (problems.length)
+        throw new Error(problems.map((d) => d.message).join('; '));
+    const transcript = opts.log.path || opts.transcript
+        ? createTranscript({
+            runId: opts.runId ?? new Date().toISOString().replace(/[:.]/g, '-'),
+            location: opts.transcript,
+            heading: `Run ${opts.runId ?? '(no id)'} · ${opts.model} · effort ${opts.effort}`,
+            secrets: [opts.runToken],
+        })
+        : undefined;
+    if (transcript)
+        atExit(() => transcript.report());
+    // However short: a short one also blanks the same letters inside other words, which is the lesser harm.
+    const forgetSecrets = onSensitiveAnswer((value) => {
+        opts.log.secret(value);
+        transcript?.secret(value);
+    });
+    const session = runtime.session;
+    /** The run asked to end: the report, a model error pi cannot get past, or the person pressing Ctrl+C. */
+    let ending = false;
+    let lastStop;
+    const screen = opts.interactive
+        ? showRun({
+            onCancel: () => {
+                ending = true;
+                opts.onCancel?.();
+                void session.abort();
+            },
+        })
+        : null;
+    try {
+        await session.bindExtensions({
+            mode: 'print',
+            shutdownHandler: () => void (ending = true),
+            onError: (e) => opts.log.write(`extension ${e.extensionPath} failed`, e.error),
+        });
+        const unsubscribe = session.subscribe((event) => {
+            transcript?.record(event);
+            if (event.type === 'message_end' && event.message.role === 'assistant')
+                lastStop = event.message.stopReason;
+            const status = statusFromEvent(event);
+            if (!status)
+                return;
+            if (screen)
+                reportStatus(status);
+            else
+                opts.print(status);
+        });
+        let message = opts.firstMessage;
+        for (let carryOns = 0;; carryOns++) {
+            await session.prompt(message);
+            if (ending || lastStop === 'error' || lastStop === 'aborted' || carryOns === MAX_CARRY_ONS)
+                break;
+            opts.log.write('the model ended its turn without the report; telling it to carry on');
+            message = CARRY_ON;
+        }
+        unsubscribe();
+    }
+    finally {
+        forgetSecrets();
+        await screen?.close();
+    }
+    await runtime.dispose();
+}

package/dist/bin.js

@@ -0,0 +1,11 @@
+#!/usr/bin/env node
+// The only file that runs before the Node check, so it imports nothing that
+// needs a newer Node than the one it is checking for.
+const [major = 0, minor = 0] = process.versions.node.split('.').map(Number);
+if (major < 22 || (major === 22 && minor < 19)) {
+    console.error(`impeccable-reviews needs Node 22.19 or newer, and this is Node ${process.versions.node}. Install a newer Node from https://nodejs.org and run it again.`);
+    process.exit(1);
+}
+const { main } = await import('./main.js');
+await main(process.argv.slice(2));
+export {};