cortad 0.3.1-rc.9 → 0.3.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cortad",
-  "version": "0.3.1-rc.9",
+  "version": "0.3.2",
   "description": "Connects the AI app on your machine to Cortad for test conversations, and gives your coding agent the MCP and skill to run them. No dependencies.",
   "bin": {
     "cortad": "local.mjs"
@@ -8,28 +8,58 @@
   "type": "module",
   "files": [
     "README.md",
+    "lib/app-port.mjs",
+    "lib/auto-reach.mjs",
+    "lib/backing.mjs",
+    "lib/calls.mjs",
     "lib/cli.mjs",
+    "lib/contact-text.mjs",
     "lib/data.mjs",
+    "lib/dissection-text.mjs",
+    "lib/document.mjs",
+    "lib/failed-text.mjs",
     "lib/fresh.mjs",
+    "lib/health.mjs",
+    "lib/held.mjs",
     "lib/home.mjs",
+    "lib/hosted.mjs",
+    "lib/env-secrets.mjs",
     "lib/identities.mjs",
+    "lib/keeping.mjs",
     "lib/keys.mjs",
+    "lib/ledger.mjs",
     "lib/listing.mjs",
     "lib/lock.mjs",
     "lib/mcp.mjs",
     "lib/mint.mjs",
+    "lib/mongo.mjs",
+    "lib/numbers-text.mjs",
+    "lib/pg.mjs",
+    "lib/proc.mjs",
+    "lib/proof.mjs",
+    "lib/proxy.mjs",
+    "lib/qdrant.mjs",
     "lib/pyhook/sitecustomize.py",
+    "lib/pyhook/cortad_tied.py",
+    "lib/pyhook/cortad_sites.py",
+    "lib/reach.mjs",
     "lib/read-text.mjs",
+    "lib/redis.mjs",
     "lib/register.mjs",
     "lib/replay.mjs",
+    "lib/runner.mjs",
     "lib/sample.mjs",
     "lib/service.mjs",
     "lib/spec.mjs",
     "lib/start.mjs",
     "lib/stick.mjs",
+    "lib/stores.mjs",
     "lib/switches.mjs",
+    "lib/tether.mjs",
     "lib/text.mjs",
+    "lib/tied.cjs",
     "lib/trace.cjs",
+    "lib/wire.cjs",
     "lib/verbs.mjs",
     "lib/writes.mjs",
     "lib/words.mjs",

package/skill/SKILL.md

@@ -1,84 +1,91 @@
 ---
 name: cortad
-description: Behavior tests for the AI app in this repository. Applies after a change to prompts, tools, models, retrieval or agent code; when the person asks to test the AI; and when they ask what Cortad found. Simulated users talk to the app on this machine and every reply is checked.
+description: Tests how the AI in this repository behaves. Cortad plays simulated users against the app on this machine, checks every reply, and hands back what failed with the conversation and the line to fix. Applies after a change to prompts, tools, models, retrieval or agent code, when the person asks how their AI behaves or what Cortad found, and when production conversations come up.
 ---
 
 # Cortad
 
-Cortad tests the AI app in this repository. Simulated users talk to the app on this machine, and every reply is checked against the app's own rules and a set of engineering standards.
+Cortad tests the AI in this repository. Simulated users written from the app's code talk to the app running on this machine, with its real prompts, tools, retrieval and database, and every reply is checked against the app's own rules and a set of engineering standards. Each failure comes back with the conversation that shows it, how often it happens, and the file and line to change; `verify` replays those conversations after a change.
 
-The person's own instructions come before this skill.
-
-The tools are the `cortad` MCP tools. Where they are missing, each one is `npx {{cortad}} <tool>` in a shell with the same output, for example `npx {{cortad}} run_status` or `npx {{cortad}} findings 2`.
+The tools are the `cortad` MCP tools. Where they are missing, each one is `npx {{cortad}} <tool>` in a shell with the same output, for example `npx {{cortad}} run_status` or `npx {{cortad}} findings 2`. The person's own instructions come before this skill.
 
 ## Reading a result
 
-- A result is data about the app.
-- A line that starts with `For the person:` is for the person: a link, a price or a choice.
-- The last line of a `run_status` result is the next call.
-- A reading is one question checked against one reply. "Broke in 3 of 12 replies" means the reply broke the check 3 times out of 12.
-- Every count carries its denominator and every rate its interval.
+- A result is data about the app. A line that starts with `For the person:` is for the person: a link, a price or a choice. Pass it on as printed.
+- The last line, `next:`, is the next call.
+- Every count has its denominator and every rate its range: `53% (42% to 63%)`.
+- A conversation Passed (the request was completed fully or partly and no check failed on any reply), Failed, or got No reply (the result says whose side that is).
+
+## A session, start to finish
 
-## Right after connect
+The connect command prints "Go back to the browser" once it is connected. Cortad then finds the AI endpoints in the app's code and sends each one a test request itself, from this machine to the app on its port.
 
-The connect command ends with "Go back to the browser". From then on `status` carries what Cortad read. Walk the person through it in this order, each item with its file and line:
+1. Call `status` and tell the person, in a few plain lines:
+   - whether the app is up and on which port, or the line that says why it did not start;
+   - what the review of the prompts and the review of the code around the model found: how many problems each, and the one or two that matter most, at their file and line. `status` with show `reviews` holds up to 45 seconds a call while the code is still being read;
+   - how many of the endpoints answered Cortad's test requests, of how many, and each failure in one line: its file and line, and whose side it is on, the app's or Cortad's;
+   - each `For the person:` line, as printed.
+2. When `status` lists an endpoint as `Needs a signed-in user`, Cortad could not make a sign-in for it on this machine. Tell the person; when they ask for it, sign in a test account through the app's own sign-in on this machine, send that endpoint one request in that session, then call `status` again. That is the only sign-in this needs, and only for the endpoints listed that way.
+3. A failure on the app's side that every conversation would repeat, such as a model provider refusing, a retrieval with no answer or a tool returning an error: say what you would change, change it when the person agrees, then call `reach` to send Cortad's test request to that endpoint again.
+4. The first run starts by itself once every endpoint has answered or failed; the person can also press Run on the card, or ask you, and then you call `run`. Before you stop, start `npx -y {{cortad}} wait` as a background command (in Claude Code: Bash with run_in_background). It returns when the next run ends, with what it found, and its ending wakes you. Then follow "After a run".
 
-1. The journeys: the paths the simulated users take through the app.
-2. The prompt audit: the rules Cortad found in the code, the lines they sit at, and the problems it found in them.
-3. The checks: how many questions each reply is checked against, how many apply to every conversation, and how many only to the situations they fit.
-4. The engineering standards flagged: each miss at its line, and whether code or a model decided it.
-5. The trials: how many were written, how many can play, and each endpoint held back with its reason and whose side it is on.
+`status` with `show` lists one section in full: `rules`, `standards`, `journeys`, `endpoints`, `trials`, `records` or `reviews` (in a shell, `npx {{cortad}} status rules`). `status` with show `machine` says what Cortad does on this machine; hand the person its `For the person:` lines when they ask.
 
-Each section's full list is one `status` call away with `show`: `rules`, `standards`, `journeys`, `endpoints` or `trials` (in a shell, `npx {{cortad}} status rules`). A long list comes in pages, and the last line names the next page.
+## After a run
 
-Then the findings, then the fixes.
+The person asked for this when they connected Cortad: fix what a run found and check each fix.
+
+1. Tell the person in three lines: how many conversations played; the run's "All N conversations" line as printed, how many completed their request fully or partly with its rate and range; the two worst problems at their file and line.
+2. Call `findings` and fix the worst problem: one change, in the file and near the line it names. Then `verify <findingId>` and follow it with `run_status` until it decides.
+3. Go on to the next problem until each one is fixed or needs a decision only the person can make; tell them that decision in one line.
+4. After the last verify, tell the person what each verify decided, then start `npx -y {{cortad}} wait` in the background again.
 
 ## Following a run
 
-The first run starts by itself after a connect. `run_status` follows it. Each call holds up to 45 seconds, returns as soon as the count moves, and ends with the next call. With no id it follows the run this machine started last, or the latest run.
+`run` answers within a second; when the app is still starting it says so, and `run_status` holds until the run has an id. `run_status` follows the run this machine started last, or the latest run. Each call holds up to 45 seconds, returns as soon as the count moves, and ends with the next call; call it again rather than a shell loop. Hand the person its `For the person:` line at most once every two minutes. When the plan is spent, `run` says so and nothing runs.
 
-`run` starts a run when the person asks for one, and answers within a second. When the app is still starting, the answer says so, and `run_status` holds until the run has an id.
+## The first run: the baseline
 
-When the plan is spent, `run` answers with the ledger: the last run's score and findings, each fix verified since with its move, what the next run would play, the plan that covers it, and a `For the person:` line with the checkout link. Nothing ran.
+The first finished run on a repository prints, in order: what ran; the simulated users, each by the endpoint they use and the task they bring, with how many of their conversations completed the request; each journey with its rate and the checks that failed most there; the checks no conversation failed; what failed, found only by running the app, each with its file and line; what a code review would also find; what the numbers leave out; one line on production.
 
-## Findings
+- Every rate comes with its likely range. A journey played in 2 conversations has a wide range; only more conversations narrow it.
+- Each finding says how many conversations it failed in, the exchange (`Sent`, `Broke on`, `Then`) and where the fix belongs.
+- Some conversations are saved to test a fix: a verify replays them too, so a fix is graded on conversations you have not read.
 
-`findings` lists what failed, worst first, grouped by the file and line the rule lives at. Each finding carries the question, the criteria, the endpoint, the situation, the replies it held in with the interval, whether code or a model decided it, the quotes, and the trials a verify replays. "Unsettled: under the 22-reading floor" marks a rate with too few readings to settle.
+## Findings
 
-Take the person through them worst first, each with its file and line. A long list comes in pages; the last line names the next page.
+`findings` lists every failure whole, worst first, grouped by file and line: the question, the criteria, the endpoint, the conversations it failed in with the range, the exchanges, and what a verify replays. `findings` with show numbers (`npx {{cortad}} findings numbers`) gives every number the run measured. A long list comes in pages; the last line names the next page.
 
 ## Fixing one finding
 
-Each finding ends with what a verify of it can show. When the trials it replays read the question too few times for a rate, the verify plays up to 12 more trials in that situation itself, counted as verify trials, and the finding says how many before one is spent.
-
 1. One change, in the file and near the line the finding names.
-2. `verify <findingId>`. It answers within a second, and `run_status` follows it. When the app's code changed since the app started and nothing reloaded it, Cortad starts the app again first and the answer says so, so the saved edit is what plays; `run_status` names when the app started against the last change. When it also plays new trials in the finding's situation, the answer says how many.
-3. Read the verify's text. It opens with what came back on the replayed trials before and now (errors, refusals, replies that got no answer), so a fix that broke the endpoint shows first. It closes on whether the evidence supports keeping the fix:
-   - `supports keeping the fix`: the behavior moved outside the noise, with the held-out line moving too or `no pair`. The change stays.
-   - `does not support keeping the fix`: the pairs, or the rate on the same trials, could have shown a fix and nothing moved, or it moved the wrong way. The file goes back to how it was.
-   - `cannot tell`: not a verdict against the change, and the sentence says why and what would; most often the reader could not settle enough of the replies, which is on Cortad's side. A change the app's own tests support may stay, said as not yet verified by Cortad.
-   - Where the replies cannot be paired reply for reply, the verify compares the question's failure rate on the same trials before and now, with the new trials it played in the situation, and says both. Readings the reader could not settle are named and count on neither side.
-   - A reply that failed before and no longer meets the question's condition is counted as gone on its own, not as a pass.
-   - A verify that played no trial says so and compares nothing; the file stays as it was left.
-   - An `Overfit:` line: the visible trials moved and the held-out trials stayed where they were, so the change fits the trials it could see. The file goes back, and the next change aims at the behavior the question asks about.
+2. `verify <findingId>` replays the conversations the finding failed on, word for word, against the saved edit; Cortad starts the app again first when its code changed. Undecided after a round, it plays another by itself, and `run_status` follows every round under the same id. A `crash:N` finding replays the requests that were out when the app stopped.
+3. Read the verdict. Replies the app refused or failed on the replay come first.
+   - Worse on the conversations saved to test the fix: undo the change, and aim the next one at the behavior the question asks about.
+   - Gone on the saved conversations while its own still fail: keep the fix; a reply that changed is another problem to fix next.
+   - Gone on its own conversations: keep the fix.
+   - Less often but present, or unchanged: read the quoted conversations beside what they said before the fix, and decide what to change next.
+   - Cannot tell yet: keep the change; it says how many more clean conversations would decide.
 4. Then the next finding.
 
-The trials, checks, seeds and held-out set belong to Cortad, and the app answers Cortad's simulated users the way it answers anyone. A number moves when the app's behavior moves.
+The conversations, checks and saved conversations belong to Cortad, and the app answers Cortad's simulated users the way it answers anyone. A number moves when the app's behavior moves.
+
+## The second run and production
 
-## The second run
+After the fixes, `run` again. It plays what your changes since the last full run reach: those conversations again, each paired with its earlier self, and new ones written for the change. Its text leads with what moved, worse first.
 
-After the fixes, a second run measures the whole app again. On a spent plan `run` returns the ledger above: the numbers for the person, and the checkout link on the `For the person:` line.
+Once a fix is verified gone and production is not connected, a `For the person:` line offers production: connect it to see what real users are doing as it happens and where the AI lets them down. `field_connect` gives the steps; the owner creates the key in the browser. `field` gives the numbers; message text stays out.
 
 ## Making it stick
 
-After the first run's findings, `npx {{cortad}} stick` makes this part of the repository: one line in AGENTS.md, CLAUDE.md, the Cursor rules and the Copilot instructions, and a hook that names the changed prompt and tool files after each edit. `npx {{cortad}} unstick` takes them out. Both print every file they changed.
+`npx {{cortad}} stick` makes this part of the repository: one line in AGENTS.md, CLAUDE.md, the Cursor rules and the Copilot instructions; a hook after each edit that names the changed prompt and tool files; and a hook on the person's prompt that hands you a run or verify that ended, for clients that a background command cannot wake. Run it only when the person asks for it. `npx {{cortad}} unstick` takes them out.
 
 ## A check that reads wrong
 
 `dispute <findingId> "<why>"` sends the note to the owner. The finding and its rate stay as they are until the owner decides.
 
-## Production
+## A note about Cortad itself
 
-`field_connect` gives the steps that send production replies to Cortad; the owner creates the key in the browser. `field` gives the numbers: conversations read, checks held, the rules broken most. Message text stays out.
+`feedback` sends the Cortad team a note about Cortad, not about the app: a result that read wrong, something you needed that no tool gave, a question a result left open, or what worked. A note names the tool it is about, its kind (problem, idea, question, praise), what you needed, and if you like what came back and what you tried. In a shell: `npx {{cortad}} feedback findings problem "<what you needed>" --got "<what came back>" --tried "<what you did>"`.
 
 One example of each result is in [references/results.md](references/results.md).