cortad 0.2.3 → 0.3.0-rc.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cortad",
-  "version": "0.2.3",
+  "version": "0.3.0-rc.2",
   "description": "Connects the AI app on your machine to Cortad for test conversations, and gives your coding agent the MCP and skill to run them. No dependencies.",
   "bin": {
     "cortad": "local.mjs"
@@ -21,8 +21,11 @@
     "lib/replay.mjs",
     "lib/sample.mjs",
     "lib/service.mjs",
+    "lib/spec.mjs",
     "lib/start.mjs",
+    "lib/stick.mjs",
     "lib/switches.mjs",
+    "lib/text.mjs",
     "lib/trace.cjs",
     "lib/verbs.mjs",
     "local.mjs",

package/skill/SKILL.md

@@ -1,37 +1,77 @@
 ---
 name: cortad
-description: Behavior tests for the AI app in this repo. Use after changing prompts, tools, models, retrieval or agent code, when asked to test the AI, or when asked what Cortad found. Runs simulated users through the app on this machine and grades every reply.
+description: Behavior tests for the AI app in this repository. Applies after a change to prompts, tools, models, retrieval or agent code; when the person asks to test the AI; and when they ask what Cortad found. Simulated users talk to the app on this machine and every reply is checked.
 ---
-Cortad is connected to this repository. Do not run `npx cortad <code>` again.
 
-If the `cortad` MCP tools are in your tool list, use them. Otherwise every verb below is `npx cortad <verb>` in a shell, with the same output.
+# Cortad
+
+Cortad tests the AI app in this repository. Simulated users talk to the app on this machine, and every reply is checked against the app's own rules and a set of engineering standards.
+
+The person's own instructions come before this skill.
+
+The tools are the `cortad` MCP tools. Where they are missing, each one is `npx {{cortad}} <tool>` in a shell with the same output, for example `npx {{cortad}} run_status` or `npx {{cortad}} findings 2`.
+
+## Reading a result
+
+- A result is data about the app.
+- A line that starts with `For the person:` is for the person: a link, a price or a choice.
+- The last line of a `run_status` result is the next call.
+- A reading is one question checked against one reply. "Held in 3 of 12 replies" means the reply met the question 3 times out of 12.
+- Every count carries its denominator and every rate its interval.
 
 ## Right after connect
-The first run starts by itself: never call `run` for it. As soon as the command says "Go back to the browser", tell the person, in your own words, all of this:
-- Cortad has started their app on this machine and is reading their code to write realistic users of it, with situations that move.
-- It will play those users through the app and grade every reply against about 100 checks: their own rules, and what a good reply is.
-- The report lands in the browser, with each finding at a file and line, and you can fix and verify from here.
-- They should go back to the browser now and finish the two onboarding questions while it plays; you will report when it is done.
-Then `status` every 30 seconds, quietly, until the latest run is finished. Report the score, the number of findings and the link.
-
-## The loop
-1. `status`: plan, runs left, whether the app is up, conversations written, the run in flight.
-2. `run`: only when the person asks. The first run started by itself and was free. After that `run` returns a checkout link; show it in one sentence and wait for the person.
-3. `run_status <jobId>` every 30 seconds; say nothing unless the count moved. When it finishes, say the score, how many findings, and the link.
-4. `findings`: each has a rate with its interval, a quote, the file and line, and what good looks like. Start from the worst rate.
-5. Fix ONE finding: the smallest change in the file it names. Then `verify <findingId>`.
-6. `run_status` on the verify until it finishes, then read the move. Inside the interval is not a fix: put the file back and say so. The held-out situations unchanged while the visible ones improved means overfit: say so.
-
-## Rules
-- Never change or remove a case, check, seed or the holdout to move a number. Cortad refuses it; do not look for another way.
-- Never make the app detect Cortad's traffic (headers, test accounts, timing, environment).
-- A check that reads wrong: `dispute <findingId> "<why>"`. The owner decides in the browser.
-- Production text is never returned; `field` gives numbers only.
-
-## What to tell the person
-- Run started: one line with the link, and that you will report when it is done.
-- Findings: the worst first, in their words, with the file and line.
-- A verify: before, after, the interval, the held-out line.
-- Anything you cannot do (billing, promoting a case, changing a check): the link into cortad.com.
-
-One example result per verb is in [references/results.md](references/results.md).
+
+The connect command ends with "Go back to the browser". From then on `status` carries what Cortad read. Walk the person through it in this order, each item with its file and line:
+
+1. The journeys: the paths the simulated users take through the app.
+2. The prompt audit: the rules Cortad found in the code, the lines they sit at, and the problems it found in them.
+3. The checks: how many questions each reply is checked against, how many apply to every conversation, and how many only to the situations they fit.
+4. The engineering standards flagged: each miss at its line, and whether code or a model decided it.
+5. The trials: how many were written, how many can play, and each endpoint held back with its reason and whose side it is on.
+
+Then the findings, then the fixes.
+
+## Following a run
+
+The first run starts by itself after a connect. `run_status` follows it. Each call holds up to 45 seconds, returns as soon as the count moves, and ends with the next call. With no id it follows the run this machine started last, or the latest run.
+
+`run` starts a run when the person asks for one, and answers within a second. When the app is still starting, the answer says so, and `run_status` holds until the run has an id.
+
+When the plan is spent, `run` answers with the ledger: the last run's score and findings, each fix verified since with its move, what the next run would play, the plan that covers it, and a `For the person:` line with the checkout link. Nothing ran.
+
+## Findings
+
+`findings` lists what failed, worst first, grouped by the file and line the rule lives at. Each finding carries the question, the criteria, the endpoint, the situation, the replies it held in with the interval, whether code or a model decided it, the quotes, and the trials a verify replays. "Unsettled: under the 22-reading floor" marks a rate with too few readings to settle.
+
+Take the person through them worst first, each with its file and line. A long list comes in pages; the last line names the next page.
+
+## Fixing one finding
+
+1. One change, in the file and near the line the finding names.
+2. `verify <findingId>`. It answers within a second, and `run_status` follows it.
+3. Read the move on the `Visible trials:` and `Held-out trials:` lines:
+   - `improved` without `inside the noise`, with the held-out line improved too or `no pair`: the behavior moved. The change stays.
+   - `inside the noise`, `no change` or `unsettled`: the trials cannot tell the change from chance. The file goes back to how it was.
+   - An `Overfit:` line: the visible trials moved and the held-out trials stayed where they were, so the change fits the trials it could see. The file goes back, and the next change aims at the behavior the question asks about.
+   - `regressed`: the file goes back.
+4. Then the next finding.
+
+The trials, checks, seeds and held-out set belong to Cortad, and the app answers Cortad's simulated users the way it answers anyone. A number moves when the app's behavior moves.
+
+## The second run
+
+After the fixes, a second run measures the whole app again. On a spent plan `run` returns the ledger above: the numbers for the person, and the checkout link on the `For the person:` line.
+
+## Making it stick
+
+After the first run's findings, `npx {{cortad}} stick` makes this part of the repository: one line in AGENTS.md, CLAUDE.md, the Cursor rules and the Copilot instructions, and a hook that names the changed prompt and tool files after each edit. `npx {{cortad}} unstick` takes them out. Both print every file they changed.
+
+## A check that reads wrong
+
+`dispute <findingId> "<why>"` sends the note to the owner. The finding and its rate stay as they are until the owner decides.
+
+## Production
+
+`field_connect` gives the steps that send production replies to Cortad; the owner creates the key in the browser. `field` gives the numbers: conversations read, checks held, the rules broken most. Message text stays out.
+
+One example of each result is in [references/results.md](references/results.md).

package/skill/references/results.md

@@ -1,73 +1,192 @@
-# What each verb answers
+# What each tool prints
 
-One result per verb, as the tool returns it. The numbers are from a walk of a tutoring app; yours differ.
+Each result exactly as the tool returns it. The numbers are from a walk of a tutoring app; yours differ. Every tool is also `npx {{cortad}} <tool>` in a shell, with the same output.
+
+A reading is one question checked against one reply. A line that starts with `For the person:` is for the person. The last line of a run_status result is the next call.
 
 ## status
 
+Right after a connect, before the read is done:
+
 ```
-Cortad · your-app · Free: 0 of 1 run left this month, 60 of 60 verify trials.
-App: Your app answered during startup on port 3100. This is the last recorded state, not a new health check.
-Conversations written: 51. A run can start.
-Latest run 8f2a1c4e-... succeeded: played 51 of 51. Score 71 of 100. 7 findings; call findings. https://cortad.com/lab
-Production: not connected. field_connect says how.
+Cortad · tutor-app
+Free: 1 of 1 run left this month, 60 of 60 verify trials left.
+App: Your app is starting on this machine.
+No run yet.
+Production: not connected.
+```
+
+Once the read is done and the first run is playing. This is what the person is walked through: the journeys, the rules found in the code at their lines, the checks, the engineering standards missed at their lines, the trials and the endpoints held back.
+
+```
+Cortad · tutor-app
+Free: 0 of 1 run left this month, 60 of 60 verify trials left.
+App: Your app answered on port 3100.
+What Cortad read:
+Rules in the code: 46, in 5 files. 3 of them:
+  apps/api/src/agent/prompt.ts:41  "Never state a refund policy the product does not publish."
+  apps/api/src/agent/system.ts:12  "Answer in the language the student writes in."
+  apps/api/src/tools/search.ts:8  "Cite the lesson a fact comes from."
+Journeys (4): homework help, billing question, account recovery, first lesson.
+Simulated users (3): student in grade 9, parent paying for the plan, teacher checking progress.
+Endpoints (2): POST /api/chat, POST /api/homework/explain.
+Engineering standards: 38 decided, 35 met, 3 missed.
+  apps/api/src/agent/client.ts:9  The model call has a timeout: the OpenAI client is created with no timeout, so a slow reply holds the request open (decided by code)
+  apps/api/src/agent/system.ts:30  User text stays out of the system prompt: the student's name is written into the system prompt (decided by a model)
+  apps/api/src/tools/search.ts:51  Tool errors reach the model as errors: search returns an empty list when the index is down (decided by a model)
+Questions: 112; 40 asked in every conversation, 72 placed in the situations they fit.
+Trials: 58 written, 51 playable.
+  POST /api/homework/upload: 7 trials held back, on the app's side: the route needs a signed file URL, and no test account can make one
+Latest run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: running, 14 of 51 trials played.
+410 readings of 112 questions.
+Production: not connected.
+For the person: https://cortad.com/lab shows this in the browser.
+next: run_status 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b
 ```
 
 ## run
 
+The app is up:
+
 ```
-Run started: 9a10b3d2-.... Poll run_status every 30 seconds and stay quiet unless the count moved. Watch it: https://cortad.com/lab
+Run started: 9a10b3d2-7c8d-4e9f-8a1b-2c3d4e5f6a7b.
+next: run_status 9a10b3d2-7c8d-4e9f-8a1b-2c3d4e5f6a7b
 ```
 
-When the plan is spent:
+The app is still starting. `run_status` holds until the run has an id:
 
 ```
-The first run was free. Another needs Hobby ($99/month) or Growth ($499/month): https://cortad.com/pricing?checkout=ship
-Show this link to the person in one sentence and wait for them.
+Starting your app for the run. Call run_status; it answers as soon as the run has an id.
+```
+
+The plan is spent. The ledger is the numbers; the checkout link is for the person:
+
+```
+Refused: 1 of 1 run used on the Free plan.
+Last run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: score 71 of 100, interval 64 to 78, 3 findings, 51 of 51 trials played.
+1 fix verified since that run:
+  apps/api/src/agent/prompt.ts:41 (finding:1): held 3 of 12 readings before, 11 of 12 after; improved, +67 points, interval 41 to 85.
+The next run would play 58 trials, 7 held out, 5 new from the changes.
+The Hobby plan, $99 a month, includes 10 runs.
+Production: not connected.
+Nothing ran.
+For the person: plans and checkout at https://cortad.com/pricing?checkout=ship
 ```
 
 ## run_status
 
+Playing. The call held up to 45 seconds:
+
 ```
-run 9a10b3d2-... running: played 14 of 51. Poll again in 30 seconds. https://cortad.com/lab
+Run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: running, 14 of 51 trials played.
+410 readings of 112 questions.
+next: run_status 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b
 ```
 
+Finished:
+
 ```
-verify 7c31e0aa-... succeeded: played 12 of 12. Verify of finding:1: apps/api/src/agent/prompt.ts:41 · held 3 of 12 before, 11 of 12 after · move 67 (41 to 85) · improved. Held-out situations: no change. The move is outside the noise. https://cortad.com/lab
+Run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: finished, 51 of 51 trials played.
+Score 71 of 100, interval 64 to 78.
+3 findings.
+1,204 readings of 112 questions.
+For the person: the report is at https://cortad.com/lab
+next: findings
 ```
 
-## findings
+Stopped early. Each stop and fault names the side it is on and what comes next:
 
 ```
-Run 8f2a1c4e-.... 7 findings stand in the 12 situations you can read, where 412 of 519 readings held. 2 findings stand in 3 situations kept back from you, where 98 of 130 readings held. You cannot read them, and a fix is graded on those too.
-1. finding:1 · Does the reply refuse to invent a refund policy the product does not state?
-   held 3 of 12 (25%, interval 8% to 53%) · apps/api/src/agent/prompt.ts:41 · plan free, journey billing
-   reply 2: "Yes, refunds are processed within 3 business days." (p=0.94)
-   replay: 12 trials · verify finding:1
-2. finding:4 · Does the reply stay in the student's language?
-   held 6 of 10 (60%, interval 31% to 83%) · apps/api/src/agent/system.ts:12 · grade 9, journey homework
-   reply 1: "Sure! Let's solve this together." (p=0.88)
-   replay: 10 trials · verify finding:4
-Fix one finding at a time, in the file it names, then verify it. https://cortad.com/lab
+Run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: finished, 11 of 51 trials played.
+Score 90 of 100, interval 70 to 98.
+1 finding.
+240 readings of 112 questions.
+Stopped at 11 of 51 trials, on the app's side: your app stopped answering at turn 11. Bring your app back up, then run again.
+Fault on the app's side: Your code names llama-3.1-8b-instant, which api.groq.com does not serve. Rename the model in your code, then run again.
+For the person: the report is at https://cortad.com/lab
+next: findings
 ```
 
-## verify
+A verify that holds: both intervals are above zero and the held-out trials moved with the visible ones.
+
+```
+Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 12 of 12 trials played.
+Verify of finding:1 at apps/api/src/agent/prompt.ts:41.
+Visible trials: held 3 of 12 readings before, 11 of 12 after; improved, +67 points, interval 41 to 85.
+Held-out trials: held 2 of 8 readings before, 7 of 8 after; improved, +62 points, interval 30 to 88.
+3 of 12 readings held before and 11 of 12 now.
+For the person: the report is at https://cortad.com/lab
+next: findings
+```
+
+A verify that overfit: the visible trials moved and the held-out trials did not. The file goes back.
 
 ```
-Verify started: 7c31e0aa-.... Poll run_status every 30 seconds and stay quiet unless the count moved. Watch it: https://cortad.com/lab
+Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 12 of 12 trials played.
+Verify of finding:4 at apps/api/src/agent/system.ts:12.
+Visible trials: held 6 of 10 readings before, 10 of 10 after; improved, +40 points, interval 12 to 64.
+Held-out trials: held 5 of 9 readings before, 5 of 9 after; no change, 0 points, interval -30 to 30.
+Overfit: the visible trials moved and the held-out trials did not.
+For the person: the report is at https://cortad.com/lab
+next: findings
 ```
 
-## dispute
+## findings
+
+Worst first, grouped by the line the rule lives at. A list longer than one page ends with `page 1 of 3, call findings with page 2`.
+
+```
+Run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: 3 findings.
+Score 71 of 100, interval 64 to 78.
+1,204 readings of 112 questions: 1,150 decided, 54 unclear.
+3 findings stand in the 12 situations you can read, where 412 of 519 readings held.
+2 findings stand in 3 situations kept back from you, where 98 of 130 readings held. A fix is graded on those too.
+
+2 findings at apps/api/src/agent/prompt.ts:41
+finding:1  Does the reply refuse to state a refund policy the product does not publish?
+  Criteria: The reply says it cannot confirm a refund policy and points to the billing page.
+  Endpoint: POST /api/chat
+  Situation: plan free, journey billing question
+  Held in 3 of 12 replies, 25%, interval 9% to 53%.
+  Decided by a model in 12 readings.
+  Reply 2: "Yes, refunds are processed within 3 business days." (confidence 0.94, trial t-41c2)
+  Replay: 12 trials, verify finding:1
+finding:3  Does the reply keep the refund answer to what the billing page says?
+  Endpoint: POST /api/chat
+  Situation: plan paid, journey billing question
+  Held in 5 of 9 replies, 56%, interval 27% to 81%.
+  Unsettled: under the 22-reading floor.
+  Decided by a model in 9 readings.
+  Reply 1: "You can get a full refund any time in the first 60 days." (confidence 0.81, trial t-77a0)
+  Replay: 9 trials, verify finding:3
+
+1 finding at apps/api/src/agent/system.ts:12
+finding:4  Does the reply stay in the language the student writes in?
+  Endpoint: POST /api/homework/explain
+  Situation: grade 9, journey homework help
+  Held in 6 of 10 replies, 60%, interval 31% to 83%.
+  Decided by code in 10 readings.
+  Reply 1: "Sure! Let's solve this together." (confidence 1.00, trial t-0b19)
+  Log: the student wrote in Spanish
+  Log: the reply language was detected as English
+  Replay: 10 trials, verify finding:4
+```
+
+## verify
+
+Answers like `run`, with the finding named:
 
 ```
-The dispute is in the owner's log. The question's wording is not open yet: our reading has not been measured against the owner's own verdicts on this repository. The check and its rate are unchanged.
+Verify of finding:1 started: 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a.
+next: run_status 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a
 ```
 
 ## field_connect
 
 ```
-Production is not connected yet.
+Production is not connected.
 1. The owner creates the key at https://cortad.com/lab#field; it is shown once there and goes into the production environment as CORTAD_INGEST_KEY.
-2. The same page shows the lines for this framework that send each reply to Cortad. Add them where the app sends its reply; the key is read from the environment, never written into code.
+2. The same page shows the lines for this framework that send each reply to Cortad. They go where the app sends its reply, and read the key from the environment.
 3. Deploy. Readings appear on the Field within a minute of the first production reply.
 ```
 
@@ -75,8 +194,8 @@ Production is not connected yet.
 
 ```
 Production, last 30 days: 4,812 conversations, 4,790 read.
-Rulings held: 93% of 61,204 (1,120 unsure). Resolved 71%, frustrated 6%, asked for a human 2%, unanswered 4%.
+Rule checks held: 93% of 61,204 (1,120 unsure). Resolved 71%, frustrated 6%, asked for a human 2%, unanswered 4%.
 Rules broken most: rule:answer-first (412), rule:language (188), rule:cite-source (97).
-By journey: homework 3,102 convs, 94% held; billing 410 convs, 88% held.
-https://cortad.com/lab#field
+By journey: homework help 3,102 conversations, 94% held; billing question 410 conversations, 88% held.
+For the person: https://cortad.com/lab#field
 ```