cortad 0.3.8 → 0.3.10

package/skill/SKILL.md

@@ -9,9 +9,19 @@ Cortad tests the AI in this repository. Simulated users written from the app's c
 
 The tools are the `cortad` MCP tools. Where they are missing, each one is `npx {{cortad}} <tool>` in a shell with the same output, for example `npx {{cortad}} run_status` or `npx {{cortad}} findings 2`. The person's own instructions come before this skill.
 
+## Talking to the person
+
+The person owns the product. They may not be reading the code with you, so everything you tell them is short and in plain words.
+
+- Three to five short lines. Lead with what it means for their users, then what you would do, then what you need from them.
+- Leave out file paths, endpoint paths, HTTP status codes, setting names, model ids and tool names. Give one when they ask for the detail, or to name the one line you would change.
+- Only numbers they act on: how many simulated users, how long, the price, how many problems.
+- Something on Cortad's side is one line: that it is on Cortad's side, and what happens next.
+- Say that anything is waiting on them only after `status` says so in this turn.
+
 ## Reading a result
 
-- A result is data about the app. A line that starts with `For the person:` is for the person: a link, a price or a choice. Pass it on as printed.
+- A result is data about the app, for you. Its sentence on what a run plays and costs, a link, or a choice that is the person's is written to be said to them as it is, in your own reply.
 - The last line, `next:`, is the next call.
 - Every count has its denominator and every rate its range: `53% (42% to 63%)`.
 - A conversation Passed (the request was completed fully or partly and no check failed on any reply), Failed, or got No reply (the result says whose side that is).
@@ -27,29 +37,32 @@ The tools are the `cortad` MCP tools. Where they are missing, each one is `npx {
 
 The connect command prints "Go back to the browser" once it is connected. Cortad then finds the AI endpoints in the app's code and sends each one a test request itself, from this machine to the app on its port.
 
-1. Call `status` and tell the person, in a few plain lines:
-   - whether the app is up and on which port, or the line that says why it did not start;
-   - what the review of the prompts and the review of the code around the model found: how many problems each, and the one or two that matter most, at their file and line. `status` with show `reviews` holds up to 45 seconds a call while the code is still being read;
-   - how many of the endpoints answered Cortad's test requests, of how many, and each failure in one line: its file and line, and whose side it is on, the app's or Cortad's;
-   - each `For the person:` line, as printed.
+1. Call `status` and tell the person, in three or four short lines:
+   - that their app is connected, or the one line that says why it did not start;
+   - what Cortad's audit of their AI code found: how many problems, and the one that matters most, in plain words. `status` with show `reviews` holds up to 45 seconds a call while the code is still being read;
+   - the line on what a run plays and costs, as printed;
+   - what happens next: they press Run on the card in the browser, or tell you to start it.
+
+   The endpoints, the files and the audit's rows are yours to work from. They go to the person when they ask, or when you propose a fix.
 2. When `status` lists an endpoint as `Needs a signed-in user`, Cortad could not make a sign-in for it on this machine. Tell the person; when they ask for it, sign in a test account through the app's own sign-in on this machine, send that endpoint one request in that session, then call `status` again. That is the only sign-in this needs, and only for the endpoints listed that way.
 3. A failure on the app's side that every conversation would repeat, such as a model provider refusing, a retrieval with no answer or a tool returning an error: say what you would change, change it when the person agrees, then call `reach` to send Cortad's test request to that endpoint again.
-4. The first run starts by itself once every endpoint has answered or failed; the person can also press Run on the card, or ask you, and then you call `run`. Before you stop, start `npx -y {{cortad}} wait` as a background command (in Claude Code: Bash with run_in_background). It returns when the next run ends, with what it found, and its ending wakes you. Then follow "After a run".
+4. A question for the person on the card in the browser is said to them once, in one line. The first run starts when the person presses Run on the card, or asks you, and then you call `run`. Before you stop, start `npx -y {{cortad}} wait` as a background command (in Claude Code: Bash with run_in_background). It returns when the next run or verify ends, with what it found, and its ending wakes you. Then follow "After a run".
 
-`status` with `show` lists one section in full: `rules`, `standards`, `journeys`, `endpoints`, `trials`, `records` or `reviews` (in a shell, `npx {{cortad}} status rules`). `status` with show `machine` says what Cortad does on this machine; hand the person its `For the person:` lines when they ask.
+`status` with `show` lists one section in full: `rules`, `standards`, `journeys`, `endpoints`, `trials`, `records` or `reviews` (in a shell, `npx {{cortad}} status rules`). `status` with show `machine` says what Cortad does on this machine; hand the person its lines when they ask.
 
 ## After a run
 
-The person asked for this when they connected Cortad: fix what a run found and check each fix.
+The run is Cortad's; the code is the person's. You report first, and you change nothing until they say so.
 
-1. Tell the person in three lines: how many conversations played; the run's "All N conversations" line as printed, how many completed their request fully or partly with its rate and range; the two worst problems at their file and line.
-2. Call `findings` and fix the worst problem: one change, in the file and near the line it names. Then `verify <findingId>` and follow it with `run_status` until it decides.
-3. Go on to the next problem until each one is fixed or needs a decision only the person can make; tell them that decision in one line.
-4. After the last verify, tell the person what each verify decided, then start `npx -y {{cortad}} wait` in the background again.
+1. Call `findings`. Tell the person, in plain lines: how many simulated users got what they asked for, of how many; then the problems worth their time, worst first and three at most, each as what their users ran into, how often, and what you would change, in one line; which findings read their app wrong (see "A check that reads wrong"); and which problems need a decision only they can make. Name a file only for the change you propose.
+2. Then stop, and let them answer. Until their go, nothing is edited, no verify starts and no run starts. They choose which problems to fix, in what order, and whether the change you propose fits their product.
+3. On their go, one problem at a time: one change, in the file and near the line the finding names, then `verify <findingId>`. Its first line says how many conversations the round replays and what that counts against this month; start `npx -y {{cortad}} wait` in the background, and it wakes you when the verify ends, so `run_status` is called once at most.
+4. Tell them what the verify decided, in its own words; the next change comes after their next go. A verify that cannot tell offers another round in its last line; it plays only when you call `verify` again, and only when the person wants it.
+5. When they stop, say what changed, what each verify decided, and what is still open, then start `npx -y {{cortad}} wait` in the background again.
 
 ## Following a run
 
-`run` answers within a second; when the app is still starting it says so, and `run_status` holds until the run has an id. `run_status` follows the run this machine started last, or the latest run. Each call holds up to 45 seconds, returns as soon as the count moves, and ends with the next call; call it again rather than a shell loop. Hand the person its `For the person:` line at most once every two minutes. When the plan is spent, `run` says so and nothing runs.
+`run` answers within a second; when the app is still starting it says so, and `run_status` holds until the run has an id. `run_status` follows the run this machine started last, or the latest run. Each call holds up to 45 seconds, returns as soon as the count moves, and ends with the next call; `npx -y {{cortad}} wait` in the background does the following for you and wakes you at the end of a run or a verify. Tell the person how far the run is at most once every two minutes. When the plan is spent, `run` says so and nothing runs.
 
 ## The first run: the baseline
 
@@ -65,8 +78,8 @@ The first finished run on a repository prints, in order: what ran; the simulated
 
 ## Fixing one finding
 
-1. One change, in the file and near the line the finding names.
-2. `verify <findingId>` replays the conversations the finding failed on, word for word, against the saved edit; Cortad starts the app again first when its code changed. Undecided after a round, it plays another by itself, and `run_status` follows every round under the same id. A `crash:N` finding replays the requests that were out when the app stopped.
+1. One change, in the file and near the line the finding names, after the person's go.
+2. `verify <findingId>` replays the conversations the finding failed on, word for word, against the saved edit; Cortad starts the app again first when its code changed. Its first line says how many conversations the round replays and what they count against. Undecided after a round, it says what another round would play; that round starts only when `verify` is called again. A `crash:N` finding replays the requests that were out when the app stopped.
 3. Read the verdict. Replies the app refused or failed on the replay come first.
    - Worse on the conversations saved to test the fix: undo the change, and aim the next one at the behavior the question asks about.
    - Gone on the saved conversations while its own still fail: keep the fix; a reply that changed is another problem to fix next.
@@ -81,7 +94,7 @@ The conversations, checks and saved conversations belong to Cortad, and the app
 
 After the fixes, `run` again. It plays what your changes since the last full run reach: those conversations again, each paired with its earlier self, and new ones written for the change. Its text leads with what moved, worse first.
 
-Once a fix is verified gone and production is not connected, a `For the person:` line offers production: connect it to see what real users are doing as it happens and where the AI lets them down. `field_connect` gives the steps; the owner creates the key in the browser. `field` gives the numbers; message text stays out.
+Once a fix is verified gone and production is not connected, the result offers production: connect it to see what real users are doing as it happens and where the AI lets them down. `field_connect` gives the steps; the owner creates the key in the browser. `field` gives the numbers; message text stays out.
 
 ## Making it stick
 

package/skill/references/results.md

@@ -35,15 +35,15 @@ Endpoints found in your code that have not answered yet (2):
   POST /api/homework/explain  apps/api/src/routes/homework.ts:18  Cortad's test request was answered with HTTP 500.
   POST /api/progress/summary  apps/api/src/routes/progress.ts:9  Needs a signed-in user.
 Cortad sends each of them a test request.
-For the person: a run plays 36 conversations: about 72 replies in about 6 minutes. It costs about $0.38 on your OpenAI key for gpt-4o-mini. Every reply is checked against the 46 rules read from your code.
+A run plays 36 simulated users in about 6 minutes. About $0.38 on your OpenAI key.
 A run starts when the person presses Run on the card in the browser or asks you for one; if they say they pressed it, call run rather than ask them again.
 ```
 
-When the app keeps its data in a store off this machine that no copy can be made of, the decision is the person's, said once as the last line before the link:
+When the app keeps its data in a hosted store no copy was made of, the decision is the person's, said once as the last line before the link:
 
 ```
-For the person: DATABASE_URL points at the Postgres database shop off this machine, and no migrations were found to build a copy on this machine from, so the run would write into it as it is. Point DATABASE_URL at a database on this machine, then run the connect command again. Or one press on the card in the browser lets the run write into it and starts the run.
-For the person: https://cortad.com/lab shows this in the browser.
+DATABASE_URL points at the Postgres database shop that is not on this machine, and no migrations were found to build a copy on this machine from, so test conversations would save into it. Point DATABASE_URL at a database on this machine, then run the connect command again. Or the person allows it on the card in the browser: one press lets test conversations save into it.
+The person sees this at https://cortad.com/lab
 ```
 
 Once a run is playing, what reading your code found and the run:
@@ -67,7 +67,7 @@ Conversations set aside:
   POST /api/homework/upload: 7 conversations, on the app's side: the route needs a signed file URL, and no test account can make one
 Latest run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: running, 14 of 51 conversations played.
 Production: not connected.
-For the person: https://cortad.com/lab shows this in the browser.
+The person sees this at https://cortad.com/lab
 next: run_status 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b
 ```
 
@@ -98,7 +98,7 @@ Rules in the code: 16, in 5 files.
 Sends Cortad's test request again, from the connect command on this machine, after a fix:
 
 ```
-Sent 1 request to your app on port 3100, one at a time:
+Sent 1 request to your app on port 3100, up to four at a time:
   POST /api/homework/explain: 200 in 2.4 seconds.
 status shows which of them reached your model.
 next: status
@@ -131,7 +131,7 @@ The next run would play 58 conversations, 7 saved to test fixes, 5 new from the
 The Pro plan, $99 a month, includes 10 runs.
 Production: not connected.
 Nothing ran.
-For the person: plans and checkout at https://cortad.com/pricing?checkout=ship
+Plans and checkout are at https://cortad.com/pricing?checkout=ship
 ```
 
 ## run_status
@@ -140,7 +140,7 @@ Playing. The call waited up to 45 seconds:
 
 ```
 Run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: running, 14 of 51 conversations played.
-For the person: 14 of 51 conversations done.
+14 of 51 conversations done.
 next: run_status 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b
 ```
 
@@ -208,7 +208,7 @@ Connect production to see what your real users are doing as it happens and where
   "Answer in the language the student writes in." (apps/api/src/agent/system.ts:12): failed in 2 of 26 conversations.
 112 of 140 checks measured.
 Free: this month's 1 run is used; 60 verify conversations left. Pro $99 a month: runs and reruns included, 100,000 production replies read.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -219,7 +219,7 @@ A run after the first one plays what the files changed since the last run that p
 ```
 Run 9a10b3d2-7c8d-4e9f-8a1b-2c3d4e5f6a7b: running, 9 of 34 conversations played.
 Since run 8f2a1c4e, 1 file changed (apps/api/src/agent/prompt.ts), touching the prompt in apps/api/src/agent/prompt.ts. They reach POST /api/chat. This run replays the 22 conversations there from that run, so each pairs with its earlier self, and adds 12 new conversations written for the change on billing question, order tracking (5 of them open the way your production customers did). Left out: 7 conversations on POST /api/homework/upload, which nothing that changed reaches.
-For the person: 9 of 34 conversations done.
+9 of 34 conversations done.
 next: run_status 9a10b3d2-7c8d-4e9f-8a1b-2c3d4e5f6a7b
 ```
 
@@ -242,7 +242,7 @@ Replies took 2.9 s at the median and 9.8 s at the slowest, over 81 replies.
 Nothing broke that only running your app could show.
 The next run is compared with run 8f2a1c4e, the last run that played every conversation, with everything that changed since then in its scope.
 Pro: 8 of 10 runs left this month; 560 verify conversations left.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: status
 ```
 
@@ -253,7 +253,7 @@ Run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: finished, 11 of 51 conversations playe
 1 finding.
 Stopped at 11 of 51 conversations, on the app's side: your app stopped answering after 11 replies. Bring your app back up, then run again.
 Fault on the app's side: Your code names llama-3.1-8b-instant, which api.groq.com does not serve. Rename the model in your code, then run again.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -281,8 +281,8 @@ Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 5 of 5 conversations play
 Verify of finding:1 at apps/api/src/agent/prompt.ts:41.
 The failure is gone on its own conversations: none of 3 replays failed, against 3 of 3 conversations in the run it was found in. On the conversations saved to test the fix the failure is gone: 3 of 8 failed before the fix, 0 of 8 after, a fall beyond chance.
 Free: this month's 1 run is used; 55 verify conversations left. Pro $99 a month: runs and reruns included, 100,000 production replies read.
-For the person: the failure in finding:1 is gone on its replays. Connect production to see what your real users are doing as it happens and where your AI still lets them down; field_connect has the steps.
-For the person: the report is at https://cortad.com/lab
+The failure in finding:1 is gone on its replays. Connecting production shows what real users are doing as it happens; field_connect has the steps.
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -303,7 +303,7 @@ The failure stayed. The conversations still fail more than the same question doe
 Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 4 of 4 conversations played.
 Verify of finding:4 at apps/api/src/agent/system.ts:12.
 The failure stayed: 4 of 4 replays failed, against 4 of 4 conversations in the run it was found in; the same question fails on 3 of 18 conversations elsewhere in your app, so these still stand out beyond chance. After the fix 4 conversations still failed: conversation 0b19a3f2 said "Sure! Let's solve this together." at reply 1, as it did before the fix; conversation 3e8d0c47 said "Sure! Let's solve this together." at reply 1, as it did before the fix; conversation a61f9b05 said "Sure! Let's solve this together." at reply 1, as it did before the fix; and 1 more.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -313,7 +313,7 @@ Where the conversations saved to test the fix disagree with the finding's own, t
 Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 43 of 43 conversations played.
 Verify of finding:5 at apps/api/src/agent/lessons.ts:58.
 The failure stayed on its own conversations (3 of 35 failed, against 4 of 18 before) and is gone on the conversations saved to test the fix (3 of 8 failed before the fix, 0 of 8 after, beyond chance). On its own conversations that is not a fall beyond chance. After the fix 3 conversations still failed: conversation e4a1c9d2 said "I don't have this information about your homework plan." at reply 2, where before the fix it said "Let me look up your lesson plan."; conversation f0000000 said "I don't have this information." at reply 1; conversation f0000001 said "I don't have this information." at reply 1. The reason changed: in conversation e4a1c9d2 the reply declined to help, where before the fix the conversation ended without what they asked for.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -323,7 +323,7 @@ Replies refused or failed on the replay come first. A refusal is not an answer,
 Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 5 of 5 conversations played.
 Verify of finding:2 at backend/tools/refunds.py.
 Your app refused 5 of 5 replays (HTTP 422), so they got no answer: the fix changed what the endpoint accepts. The failure stayed: 5 of 5 replays got no answer, against 5 of 5 conversations in the run it was found in; the same question fails on 2 of 16 conversations elsewhere in your app, so these still stand out beyond chance.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -434,5 +434,5 @@ Production, last 30 days: 4,812 conversations, 4,790 read.
 Rule checks passed: 93% of 61,204 (1,120 too close to call). Resolved 71%, frustrated 6%, asked for a human 2%, unanswered 4%.
 Rules broken most: rule:answer-first (412), rule:language (188), rule:cite-source (97).
 By journey: homework help 3,102 conversations, 94% passed; billing question 410 conversations, 88% passed.
-For the person: https://cortad.com/lab#field
+The person sees this at https://cortad.com/lab#field
 ```