cortad 0.3.9 → 0.3.11

package/skill/SKILL.md

@@ -9,9 +9,19 @@ Cortad tests the AI in this repository. Simulated users written from the app's c
 
 The tools are the `cortad` MCP tools. Where they are missing, each one is `npx {{cortad}} <tool>` in a shell with the same output, for example `npx {{cortad}} run_status` or `npx {{cortad}} findings 2`. The person's own instructions come before this skill.
 
+## Talking to the person
+
+The person owns the product. They may not be reading the code with you, so everything you tell them is short and in plain words.
+
+- Three to five short lines. Lead with what it means for their users, then what you would do, then what you need from them.
+- Leave out file paths, endpoint paths, HTTP status codes, setting names, model ids and tool names. Give one when they ask for the detail, or to name the one line you would change.
+- Only numbers they act on: how many simulated users, how long, the price, how many problems.
+- Something on Cortad's side is one line: that it is on Cortad's side, and what happens next.
+- Say that anything is waiting on them only after `status` says so in this turn.
+
 ## Reading a result
 
-- A result is data about the app. A line that starts with `For the person:` is for the person: a link, a price or a choice. Pass it on as printed.
+- A result is data about the app, for you. Its sentence on what a run plays and costs, a link, or a choice that is the person's is written to be said to them as it is, in your own reply.
 - The last line, `next:`, is the next call.
 - Every count has its denominator and every rate its range: `53% (42% to 63%)`.
 - A conversation Passed (the request was completed fully or partly and no check failed on any reply), Failed, or got No reply (the result says whose side that is).
@@ -24,25 +34,28 @@ The tools are the `cortad` MCP tools. Where they are missing, each one is `npx {
 - A folder connected before reconnects at once with the key it kept.
 - A folder connected for the first time prints a link and four characters, good for ten minutes. Give the person both as printed; they approve the link in their browser, signed in to Cortad, and the command goes on by itself.
 - When a result says no connect command is running, run `npx {{cortad}}` again in this folder. No code is needed from the person.
+- The app runs on copies of the databases its settings name, deleted when the command ends. When the person wants a run on other data, such as their staging databases instead of the test ones, stop the command and run `npx {{cortad}} --data <settings file>`: the databases that file names are copied instead, under the names the app reads them by, and the app keeps the rest of its own settings. The choice holds for later runs until the next connect, and every setting the app needs stays in place.
 
 The connect command prints "Go back to the browser" once it is connected. Cortad then finds the AI endpoints in the app's code and sends each one a test request itself, from this machine to the app on its port.
 
-1. Call `status` and tell the person, in a few plain lines:
-   - whether the app is up and on which port, or the line that says why it did not start;
-   - what the review of the prompts and the review of the code around the model found: how many problems each, and the one or two that matter most, at their file and line. `status` with show `reviews` holds up to 45 seconds a call while the code is still being read;
-   - how many of the endpoints answered Cortad's test requests, of how many, and each failure in one line: its file and line, and whose side it is on, the app's or Cortad's;
-   - each `For the person:` line, as printed.
+1. Call `status` and tell the person, in three or four short lines:
+   - that their app is connected, or the one line that says why it did not start;
+   - what Cortad's audit of their AI code found: how many problems, and the one that matters most, in plain words. `status` with show `reviews` holds up to 45 seconds a call while the code is still being read;
+   - the line on what a run plays and costs, as printed;
+   - what happens next: they press Run on the card in the browser, or tell you to start it.
+
+   The endpoints, the files and the audit's rows are yours to work from. They go to the person when they ask, or when you propose a fix.
 2. When `status` lists an endpoint as `Needs a signed-in user`, Cortad could not make a sign-in for it on this machine. Tell the person; when they ask for it, sign in a test account through the app's own sign-in on this machine, send that endpoint one request in that session, then call `status` again. That is the only sign-in this needs, and only for the endpoints listed that way.
 3. A failure on the app's side that every conversation would repeat, such as a model provider refusing, a retrieval with no answer or a tool returning an error: say what you would change, change it when the person agrees, then call `reach` to send Cortad's test request to that endpoint again.
-4. The first run starts when the person presses Run on the card, or asks you, and then you call `run`. Before you stop, start `npx -y {{cortad}} wait` as a background command (in Claude Code: Bash with run_in_background). It returns when the next run or verify ends, with what it found, and its ending wakes you. Then follow "After a run".
+4. A question for the person on the card in the browser is said to them once, in one line. The first run starts when the person presses Run on the card, or asks you, and then you call `run`. Before you stop, start `npx -y {{cortad}} wait` as a background command (in Claude Code: Bash with run_in_background). It returns when the next run or verify ends, with what it found, and its ending wakes you. Then follow "After a run".
 
-`status` with `show` lists one section in full: `rules`, `standards`, `journeys`, `endpoints`, `trials`, `records` or `reviews` (in a shell, `npx {{cortad}} status rules`). `status` with show `machine` says what Cortad does on this machine; hand the person its `For the person:` lines when they ask.
+`status` with `show` lists one section in full: `rules`, `standards`, `journeys`, `endpoints`, `trials`, `records` or `reviews` (in a shell, `npx {{cortad}} status rules`). `status` with show `machine` says what Cortad does on this machine; hand the person its lines when they ask.
 
 ## After a run
 
 The run is Cortad's; the code is the person's. You report first, and you change nothing until they say so.
 
-1. Call `findings`. Tell the person, in plain lines: how many conversations played and how many completed their request, with the rate and range as printed; each problem worth their time, worst first, with its file and line and what you would change there in one line; which findings read their app wrong (see "A check that reads wrong"); and which problems need a decision only they can make.
+1. Call `findings`. Tell the person, in plain lines: how many simulated users got what they asked for, of how many; then the problems worth their time, worst first and three at most, each as what their users ran into, how often, and what you would change, in one line; which findings read their app wrong (see "A check that reads wrong"); and which problems need a decision only they can make. Name a file only for the change you propose.
 2. Then stop, and let them answer. Until their go, nothing is edited, no verify starts and no run starts. They choose which problems to fix, in what order, and whether the change you propose fits their product.
 3. On their go, one problem at a time: one change, in the file and near the line the finding names, then `verify <findingId>`. Its first line says how many conversations the round replays and what that counts against this month; start `npx -y {{cortad}} wait` in the background, and it wakes you when the verify ends, so `run_status` is called once at most.
 4. Tell them what the verify decided, in its own words; the next change comes after their next go. A verify that cannot tell offers another round in its last line; it plays only when you call `verify` again, and only when the person wants it.
@@ -50,7 +63,7 @@ The run is Cortad's; the code is the person's. You report first, and you change
 
 ## Following a run
 
-`run` answers within a second; when the app is still starting it says so, and `run_status` holds until the run has an id. `run_status` follows the run this machine started last, or the latest run. Each call holds up to 45 seconds, returns as soon as the count moves, and ends with the next call; `npx -y {{cortad}} wait` in the background does the following for you and wakes you at the end of a run or a verify. Hand the person its `For the person:` line at most once every two minutes. When the plan is spent, `run` says so and nothing runs.
+`run` answers within a second; when the app is still starting it says so, and `run_status` holds until the run has an id. `run_status` follows the run this machine started last, or the latest run. Each call holds up to 45 seconds, returns as soon as the count moves, and ends with the next call; `npx -y {{cortad}} wait` in the background does the following for you and wakes you at the end of a run or a verify. Tell the person how far the run is at most once every two minutes. When the plan is spent, `run` says so and nothing runs.
 
 ## The first run: the baseline
 
@@ -82,7 +95,11 @@ The conversations, checks and saved conversations belong to Cortad, and the app
 
 After the fixes, `run` again. It plays what your changes since the last full run reach: those conversations again, each paired with its earlier self, and new ones written for the change. Its text leads with what moved, worse first.
 
-Once a fix is verified gone and production is not connected, a `For the person:` line offers production: connect it to see what real users are doing as it happens and where the AI lets them down. `field_connect` gives the steps; the owner creates the key in the browser. `field` gives the numbers; message text stays out.
+Once a fix is verified gone and production is not connected, the result offers production: connect it to see what real users are doing as it happens and where the AI lets them down. `field_connect` gives the steps; the owner creates the key in the browser. `field` gives the numbers; message text stays out.
+
+## After a change: the one call, and a world
+
+`test <what you built or want tested>` is the call after a change. Cortad decides what kind of test that is: a world, where this app has one and the sentence names something it can plant, or a run of the app's chat surfaces; the answer says which started and the call that follows it. A world is for an AI that is not a chat, a worker that reads data and writes a verdict: Cortad builds a copy of the product's data on this machine from the app's own seed, plants problems whose answers are known, runs the app's own detection and investigation on each, and scores what it raised and published: found, missed, false alarms where nothing was planted, flips between repeats. `world <ask>` asks for one outright, `world_status <id>` follows it, and the person sees it on the World page. A repository no world is written for yet says so: send `feedback` naming the AI to test, and the Cortad team writes one.
 
 ## Making it stick
 

package/skill/references/results.md

@@ -35,15 +35,15 @@ Endpoints found in your code that have not answered yet (2):
   POST /api/homework/explain  apps/api/src/routes/homework.ts:18  Cortad's test request was answered with HTTP 500.
   POST /api/progress/summary  apps/api/src/routes/progress.ts:9  Needs a signed-in user.
 Cortad sends each of them a test request.
-For the person: a run plays 36 conversations: about 72 replies in about 6 minutes. It costs about $0.38 on your OpenAI key for gpt-4o-mini. Every reply is checked against the 46 rules read from your code.
+A run plays 36 simulated users in about 6 minutes. About $0.38 on your OpenAI key.
 A run starts when the person presses Run on the card in the browser or asks you for one; if they say they pressed it, call run rather than ask them again.
 ```
 
-When the app keeps its data in a store off this machine that no copy can be made of, the decision is the person's, said once as the last line before the link:
+When the app keeps its data in a hosted store no copy was made of, the decision is the person's, said once as the last line before the link:
 
 ```
-For the person: DATABASE_URL points at the Postgres database shop off this machine, and no migrations were found to build a copy on this machine from, so the run would write into it as it is. Point DATABASE_URL at a database on this machine, then run the connect command again. Or one press on the card in the browser lets the run write into it and starts the run.
-For the person: https://cortad.com/lab shows this in the browser.
+DATABASE_URL points at the Postgres database shop that is not on this machine, and no migrations were found to build a copy on this machine from, so test conversations would save into it. Point DATABASE_URL at a database on this machine, then run the connect command again. Or the person allows it on the card in the browser: one press lets test conversations save into it.
+The person sees this at https://cortad.com/lab
 ```
 
 Once a run is playing, what reading your code found and the run:
@@ -67,7 +67,7 @@ Conversations set aside:
   POST /api/homework/upload: 7 conversations, on the app's side: the route needs a signed file URL, and no test account can make one
 Latest run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: running, 14 of 51 conversations played.
 Production: not connected.
-For the person: https://cortad.com/lab shows this in the browser.
+The person sees this at https://cortad.com/lab
 next: run_status 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b
 ```
 
@@ -131,7 +131,7 @@ The next run would play 58 conversations, 7 saved to test fixes, 5 new from the
 The Pro plan, $99 a month, includes 10 runs.
 Production: not connected.
 Nothing ran.
-For the person: plans and checkout at https://cortad.com/pricing?checkout=ship
+Plans and checkout are at https://cortad.com/pricing?checkout=ship
 ```
 
 ## run_status
@@ -140,7 +140,7 @@ Playing. The call waited up to 45 seconds:
 
 ```
 Run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: running, 14 of 51 conversations played.
-For the person: 14 of 51 conversations done.
+14 of 51 conversations done.
 next: run_status 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b
 ```
 
@@ -208,7 +208,7 @@ Connect production to see what your real users are doing as it happens and where
   "Answer in the language the student writes in." (apps/api/src/agent/system.ts:12): failed in 2 of 26 conversations.
 112 of 140 checks measured.
 Free: this month's 1 run is used; 60 verify conversations left. Pro $99 a month: runs and reruns included, 100,000 production replies read.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -219,7 +219,7 @@ A run after the first one plays what the files changed since the last run that p
 ```
 Run 9a10b3d2-7c8d-4e9f-8a1b-2c3d4e5f6a7b: running, 9 of 34 conversations played.
 Since run 8f2a1c4e, 1 file changed (apps/api/src/agent/prompt.ts), touching the prompt in apps/api/src/agent/prompt.ts. They reach POST /api/chat. This run replays the 22 conversations there from that run, so each pairs with its earlier self, and adds 12 new conversations written for the change on billing question, order tracking (5 of them open the way your production customers did). Left out: 7 conversations on POST /api/homework/upload, which nothing that changed reaches.
-For the person: 9 of 34 conversations done.
+9 of 34 conversations done.
 next: run_status 9a10b3d2-7c8d-4e9f-8a1b-2c3d4e5f6a7b
 ```
 
@@ -242,7 +242,7 @@ Replies took 2.9 s at the median and 9.8 s at the slowest, over 81 replies.
 Nothing broke that only running your app could show.
 The next run is compared with run 8f2a1c4e, the last run that played every conversation, with everything that changed since then in its scope.
 Pro: 8 of 10 runs left this month; 560 verify conversations left.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: status
 ```
 
@@ -253,7 +253,7 @@ Run 8f2a1c4e-5b6d-4e7f-9a0b-1c2d3e4f5a6b: finished, 11 of 51 conversations playe
 1 finding.
 Stopped at 11 of 51 conversations, on the app's side: your app stopped answering after 11 replies. Bring your app back up, then run again.
 Fault on the app's side: Your code names llama-3.1-8b-instant, which api.groq.com does not serve. Rename the model in your code, then run again.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -281,8 +281,8 @@ Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 5 of 5 conversations play
 Verify of finding:1 at apps/api/src/agent/prompt.ts:41.
 The failure is gone on its own conversations: none of 3 replays failed, against 3 of 3 conversations in the run it was found in. On the conversations saved to test the fix the failure is gone: 3 of 8 failed before the fix, 0 of 8 after, a fall beyond chance.
 Free: this month's 1 run is used; 55 verify conversations left. Pro $99 a month: runs and reruns included, 100,000 production replies read.
-For the person: the failure in finding:1 is gone on its replays. Connect production to see what your real users are doing as it happens and where your AI still lets them down; field_connect has the steps.
-For the person: the report is at https://cortad.com/lab
+The failure in finding:1 is gone on its replays. Connecting production shows what real users are doing as it happens; field_connect has the steps.
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -303,7 +303,7 @@ The failure stayed. The conversations still fail more than the same question doe
 Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 4 of 4 conversations played.
 Verify of finding:4 at apps/api/src/agent/system.ts:12.
 The failure stayed: 4 of 4 replays failed, against 4 of 4 conversations in the run it was found in; the same question fails on 3 of 18 conversations elsewhere in your app, so these still stand out beyond chance. After the fix 4 conversations still failed: conversation 0b19a3f2 said "Sure! Let's solve this together." at reply 1, as it did before the fix; conversation 3e8d0c47 said "Sure! Let's solve this together." at reply 1, as it did before the fix; conversation a61f9b05 said "Sure! Let's solve this together." at reply 1, as it did before the fix; and 1 more.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -313,7 +313,7 @@ Where the conversations saved to test the fix disagree with the finding's own, t
 Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 43 of 43 conversations played.
 Verify of finding:5 at apps/api/src/agent/lessons.ts:58.
 The failure stayed on its own conversations (3 of 35 failed, against 4 of 18 before) and is gone on the conversations saved to test the fix (3 of 8 failed before the fix, 0 of 8 after, beyond chance). On its own conversations that is not a fall beyond chance. After the fix 3 conversations still failed: conversation e4a1c9d2 said "I don't have this information about your homework plan." at reply 2, where before the fix it said "Let me look up your lesson plan."; conversation f0000000 said "I don't have this information." at reply 1; conversation f0000001 said "I don't have this information." at reply 1. The reason changed: in conversation e4a1c9d2 the reply declined to help, where before the fix the conversation ended without what they asked for.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -323,7 +323,7 @@ Replies refused or failed on the replay come first. A refusal is not an answer,
 Verify 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a: finished, 5 of 5 conversations played.
 Verify of finding:2 at backend/tools/refunds.py.
 Your app refused 5 of 5 replays (HTTP 422), so they got no answer: the fix changed what the endpoint accepts. The failure stayed: 5 of 5 replays got no answer, against 5 of 5 conversations in the run it was found in; the same question fails on 2 of 16 conversations elsewhere in your app, so these still stand out beyond chance.
-For the person: the report is at https://cortad.com/lab
+The report is at https://cortad.com/lab
 next: findings
 ```
 
@@ -434,5 +434,5 @@ Production, last 30 days: 4,812 conversations, 4,790 read.
 Rule checks passed: 93% of 61,204 (1,120 too close to call). Resolved 71%, frustrated 6%, asked for a human 2%, unanswered 4%.
 Rules broken most: rule:answer-first (412), rule:language (188), rule:cite-source (97).
 By journey: homework help 3,102 conversations, 94% passed; billing question 410 conversations, 88% passed.
-For the person: https://cortad.com/lab#field
+The person sees this at https://cortad.com/lab#field
 ```