cortad 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -33
- package/lib/identities.mjs +1 -1
- package/lib/mint.mjs +1 -1
- package/lib/replay.mjs +1 -1
- package/lib/sample.mjs +2 -2
- package/lib/service.mjs +2 -2
- package/lib/trace.cjs +2 -2
- package/local.mjs +3 -3
- package/package.json +1 -1
- package/skill/references/results.md +1 -1
package/README.md
CHANGED
|
@@ -1,24 +1,16 @@
|
|
|
1
1
|
# cortad
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Runs Cortad's test conversations against the AI app on your machine, and gives your coding agent the tools to run them.
|
|
4
4
|
|
|
5
5
|
```
|
|
6
6
|
npx cortad <code>
|
|
7
7
|
```
|
|
8
8
|
|
|
9
|
-
Run it in your app's folder with the code from cortad.com.
|
|
9
|
+
Run it in your app's folder with the code from cortad.com. It starts your app, connects it to Cortad, and adds Cortad to Claude Code, Codex and Cursor on this machine. The first run starts on its own. Ctrl-C disconnects.
|
|
10
10
|
|
|
11
|
-
##
|
|
11
|
+
## Your coding agent
|
|
12
12
|
|
|
13
|
-
|
|
14
|
-
- Sends Cortad's test conversations to your app on localhost and returns the replies.
|
|
15
|
-
- Signs in as a test account when your app needs one: an account made for the session, never an existing user and never an admin.
|
|
16
|
-
- Makes Cortad known to the coding agents on this machine (Claude Code, Codex, Cursor): an MCP entry and a skill in each one's own home folder, so `npx cortad mcp` answers them from then on. Nothing is written into your repository.
|
|
17
|
-
- Never writes your files. The agent that edits your code is your own.
|
|
18
|
-
|
|
19
|
-
## For your coding agent
|
|
20
|
-
|
|
21
|
-
After the first connect, your agent has eight tools: `status`, `run`, `run_status`, `findings`, `verify`, `dispute`, `field_connect`, `field`. The same eight work as shell commands:
|
|
13
|
+
After that, your agent has eight tools: `status`, `run`, `run_status`, `findings`, `verify`, `dispute`, `field_connect`, `field`. The same eight work as commands:
|
|
22
14
|
|
|
23
15
|
```
|
|
24
16
|
npx cortad status
|
|
@@ -27,26 +19,7 @@ npx cortad findings
|
|
|
27
19
|
npx cortad verify <findingId>
|
|
28
20
|
```
|
|
29
21
|
|
|
30
|
-
A run needs your app up.
|
|
31
|
-
|
|
32
|
-
## What Cortad receives
|
|
33
|
-
|
|
34
|
-
- The files git would commit, once. Nothing git ignores, and no env, key or data files.
|
|
35
|
-
- Your app's replies to the test conversations, with values from your env files hidden and cookies removed.
|
|
36
|
-
|
|
37
|
-
## What stays on your machine
|
|
38
|
-
|
|
39
|
-
- Your env files and their values.
|
|
40
|
-
- Tokens and cookies. A signed-in test request gets its token added here.
|
|
41
|
-
- The key in `~/.cortad/<project>/token` (readable by you only). It starts runs and reads findings for this one repository and nothing else; revoke it from your account page.
|
|
42
|
-
|
|
43
|
-
## The test shell
|
|
44
|
-
|
|
45
|
-
Commands a run needs on this machine (your own test suite, a WebSocket door) are confined by the operating system (Seatbelt on macOS, bubblewrap on Linux). They can read your project and your toolchains, write only to temp and build folders, and reach only localhost.
|
|
46
|
-
|
|
47
|
-
## Loaded into your app
|
|
48
|
-
|
|
49
|
-
When cortad starts your app it preloads one short file, `lib/trace.cjs` for Node or `lib/pyhook/sitecustomize.py` for Python. It notes which request called a model, so Cortad finds your chat route. It writes only to cortad's temp folder.
|
|
22
|
+
A run needs your app up. If nothing on this machine is holding it, the command starts it with the key in `~/.cortad` and stops it ten minutes after the last run.
|
|
50
23
|
|
|
51
24
|
## Flags
|
|
52
25
|
|
|
@@ -63,4 +36,4 @@ When cortad starts your app it preloads one short file, `lib/trace.cjs` for Node
|
|
|
63
36
|
- `~/.cortad/identity.key` the seed for the session's test accounts
|
|
64
37
|
- `$TMPDIR/cortad-<pid>/` removed on exit
|
|
65
38
|
|
|
66
|
-
macOS and Linux, Node 20
|
|
39
|
+
macOS and Linux, Node 20 or later. No dependencies. What is sent and what stays on your machine: `npx cortad --explain`, and cortad.com/privacy.
|
package/lib/identities.mjs
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// The same file lives on the Cortad side. Keep the two in step: the recipes the
|
|
2
2
|
// cloud sends are gated here again, on the machine that holds the secrets.
|
|
3
3
|
// Signing in as the customer's own callers, from inside the box, through their own doors.
|
|
4
4
|
//
|
package/lib/mint.mjs
CHANGED
|
@@ -137,7 +137,7 @@ export function makeIdentities({ root, work, envFiles, sourceFiles = () => [], s
|
|
|
137
137
|
// nothing and says so; port the program to this file if that turns out to be common.
|
|
138
138
|
// A Firebase caller with no emulator on this machine is signed in at their real project, below.
|
|
139
139
|
// Handed to the program it waited its whole deadline on an emulator that was never there, and
|
|
140
|
-
//
|
|
140
|
+
// one app's sign-in arrived after the world had given up waiting for it.
|
|
141
141
|
const merged = Object.assign({}, ...[...envs().all].reverse());
|
|
142
142
|
const emulated = Boolean(merged.FIREBASE_AUTH_EMULATOR_HOST);
|
|
143
143
|
const refused = new Map();
|
package/lib/replay.mjs
CHANGED
|
@@ -19,7 +19,7 @@ export function makeCapture({ work, keepSecret, onDoor }) {
|
|
|
19
19
|
let held = null;
|
|
20
20
|
let alive = false;
|
|
21
21
|
// The ports a hooked process listens on. Loading is not listening: turbo's own Node launcher loads
|
|
22
|
-
// the hook, and
|
|
22
|
+
// the hook, and one app was told we would see its messages while its API on Bun carried none.
|
|
23
23
|
const ports = new Set();
|
|
24
24
|
// Bun reads BUN_OPTIONS and splits it on spaces, quotes included, and only the `--preload=` form
|
|
25
25
|
// leaves `bun run <script>` working. A hook path with a space in it is copied to one without.
|
package/lib/sample.mjs
CHANGED
|
@@ -20,7 +20,7 @@ const PER_STORE = 6;
|
|
|
20
20
|
const PER_STORE_MAX = 60;
|
|
21
21
|
// A few rows from MANY collections, never many rows from few: a store whose collection names are
|
|
22
22
|
// the product's own grid (jordan_grade-10_sem-2_math) is only read when every cell is read. Forty
|
|
23
|
-
// eight of them, twenty two rows apiece, filled the whole answer with seven of
|
|
23
|
+
// eight of them, twenty two rows apiece, filled the whole answer with seven of one app's cells and
|
|
24
24
|
// left grade 10 semester 2 maths unread, so the depth follows the breadth: how many rows a
|
|
25
25
|
// collection gets is the answer's room divided by how many collections there are.
|
|
26
26
|
const COLLECTIONS = 200;
|
|
@@ -122,7 +122,7 @@ async function qdrant(plan, names, env, per, until) {
|
|
|
122
122
|
const each = Math.max(PER_MIN, Math.min(per, Math.floor(BUDGET / ROW_BYTES / Math.max(1, asking.length))));
|
|
123
123
|
const rows = [];
|
|
124
124
|
let read = 0;
|
|
125
|
-
// Where each collection that had rows left off. On
|
|
125
|
+
// Where each collection that had rows left off. On one app a third of the 177 collections are
|
|
126
126
|
// empty, so the even split left half the answer unspent at three pages a cell.
|
|
127
127
|
const more = new Map();
|
|
128
128
|
const scroll = async (name, limit, offset) => {
|
package/lib/service.mjs
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// Some apps keep their sign-in on a different service than the one that serves the AI:
|
|
1
|
+
// Some apps keep their sign-in on a different service than the one that serves the AI: one app
|
|
2
2
|
// serves chat from apps/api and mounts better-auth in apps/dashboard, and the command starts only
|
|
3
3
|
// apps/api. A recipe that names such a service (src/setup/road-identities.ts) is minted against
|
|
4
4
|
// that service instead, so nobody is asked to send a message by hand. Cookies set on localhost
|
|
@@ -62,7 +62,7 @@ export async function waitForPort(port, deadlineMs = 90_000) {
|
|
|
62
62
|
}
|
|
63
63
|
|
|
64
64
|
// The page a sign-in believes it is being asked from. Nearly every auth library refuses a request
|
|
65
|
-
// whose Origin it does not trust: better-auth answered
|
|
65
|
+
// whose Origin it does not trust: better-auth answered one app's own sign-up with 403
|
|
66
66
|
// INVALID_ORIGIN because the ask went to 127.0.0.1 with no page behind it. Their own settings name
|
|
67
67
|
// the page; where they do not, it is that port on localhost, which is what their own browser sends.
|
|
68
68
|
export function originFor(port, origins = []) {
|
package/lib/trace.cjs
CHANGED
|
@@ -76,7 +76,7 @@ if (FILE) {
|
|
|
76
76
|
return als.run(ctx, () => emit.call(this, type, req, ...rest));
|
|
77
77
|
};
|
|
78
78
|
|
|
79
|
-
// Bun serves through Bun.serve, never node:http, so the patch above saw no request at all:
|
|
79
|
+
// Bun serves through Bun.serve, never node:http, so the patch above saw no request at all: an
|
|
80
80
|
// Elysia API on Bun took the customer's messages and nothing was written. Bun.serve is wrapped
|
|
81
81
|
// instead, both the fetch handler and the per-route handlers Bun can dispatch to directly. The body
|
|
82
82
|
// is read from a clone, so the app's own reader is untouched.
|
|
@@ -176,7 +176,7 @@ if (FILE) {
|
|
|
176
176
|
|
|
177
177
|
// Outbound to a service their own settings name (a vector store, a database API, a search
|
|
178
178
|
// host): which setting, which host and whether it answered. No body, no headers, no query. A
|
|
179
|
-
// run that tested
|
|
179
|
+
// run that tested a tutoring app while every lookup failed inside Node never knew; this is
|
|
180
180
|
// the witness that lets the run say retrieval was not tested instead of grading without it.
|
|
181
181
|
let depHosts = null, depSeen = -1;
|
|
182
182
|
const depOf = (input) => {
|
package/local.mjs
CHANGED
|
@@ -440,7 +440,7 @@ const onPath = (bin) => (process.env.PATH ?? "").split(":").some((dir) => dir &&
|
|
|
440
440
|
let appDir = root;
|
|
441
441
|
|
|
442
442
|
// ---- a sign-in mounted in another workspace
|
|
443
|
-
//
|
|
443
|
+
// One app served its AI from apps/api and mounted sign-in in apps/dashboard: a test account can
|
|
444
444
|
// only be made where the sign-in is, and posting a sign-up at the AI's own port is a 404.
|
|
445
445
|
const PORT_IN_SCRIPT = /(?:^|\s)(?:PORT=|-p[ =]|--port[ =])(\d{2,5})\b/;
|
|
446
446
|
// The port a framework serves on when nobody names one. sveltekit and astro before vite: both bring
|
|
@@ -684,7 +684,7 @@ async function start(waitMs) {
|
|
|
684
684
|
const tail = () => seen.replace(/\x1b\[[0-9;]*m/g, "").split("\n").filter(Boolean).slice(-25).join("\n");
|
|
685
685
|
if (exited !== null) return { port: null, exited, tail: tail() };
|
|
686
686
|
// nodemon and its kind outlive the app they watch: the app is gone, the process is not, and
|
|
687
|
-
// the wait ran its whole three minutes on
|
|
687
|
+
// the wait ran its whole three minutes on one app with the reason sitting in the output.
|
|
688
688
|
// Also: the port is taken, under a watcher that does not exit when its app cannot listen. The app
|
|
689
689
|
// that holds the port is theirs and already running, which startApp turns into attaching to it.
|
|
690
690
|
if (/app crashed - waiting for file changes|waiting for (?:file )?changes before restart|Failed running|EADDRINUSE|address already in use/i.test(seen)) {
|
|
@@ -737,7 +737,7 @@ async function listening(pid) {
|
|
|
737
737
|
} catch { return []; }
|
|
738
738
|
}
|
|
739
739
|
// Your app and everything it started, stopped for certain. A process group is not enough: nodemon
|
|
740
|
-
// gives the real server a group of its own, and
|
|
740
|
+
// gives the real server a group of its own, and one app's server outlived this command as an
|
|
741
741
|
// orphan still holding its port and its production connections. Asked first, then made to.
|
|
742
742
|
async function stopApp(pid) {
|
|
743
743
|
const family = await familyOf(pid).catch(() => [pid]);
|
package/package.json
CHANGED
|
@@ -5,7 +5,7 @@ One result per verb, as the tool returns it. The numbers are from a walk of a tu
|
|
|
5
5
|
## status
|
|
6
6
|
|
|
7
7
|
```
|
|
8
|
-
Cortad ·
|
|
8
|
+
Cortad · your-app · Free: 0 of 1 run left this month, 60 of 60 verify trials.
|
|
9
9
|
App: Your app answered during startup on port 3100. This is the last recorded state, not a new health check.
|
|
10
10
|
Conversations written: 51. A run can start.
|
|
11
11
|
Latest run 8f2a1c4e-... succeeded: played 51 of 51. Score 71 of 100. 7 findings; call findings. https://cortad.com/lab
|