@viraatdas/rudder 2.10.21 → 2.10.23
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +42 -0
- package/dist/improve/advisor.d.ts +10 -0
- package/dist/improve/advisor.js +115 -0
- package/dist/improve/advisor.js.map +1 -0
- package/dist/improve/collect.d.ts +10 -0
- package/dist/improve/collect.js +202 -0
- package/dist/improve/collect.js.map +1 -0
- package/dist/improve/gate.d.ts +17 -0
- package/dist/improve/gate.js +60 -0
- package/dist/improve/gate.js.map +1 -0
- package/dist/improve/index.d.ts +9 -0
- package/dist/improve/index.js +336 -0
- package/dist/improve/index.js.map +1 -0
- package/dist/improve/judge.d.ts +22 -0
- package/dist/improve/judge.js +101 -0
- package/dist/improve/judge.js.map +1 -0
- package/dist/improve/mine.d.ts +25 -0
- package/dist/improve/mine.js +173 -0
- package/dist/improve/mine.js.map +1 -0
- package/dist/improve/propose.d.ts +51 -0
- package/dist/improve/propose.js +150 -0
- package/dist/improve/propose.js.map +1 -0
- package/dist/improve/rank.d.ts +16 -0
- package/dist/improve/rank.js +49 -0
- package/dist/improve/rank.js.map +1 -0
- package/dist/improve/schedule.d.ts +1 -0
- package/dist/improve/schedule.js +93 -0
- package/dist/improve/schedule.js.map +1 -0
- package/dist/improve/ship.d.ts +25 -0
- package/dist/improve/ship.js +82 -0
- package/dist/improve/ship.js.map +1 -0
- package/dist/improve/state.d.ts +149 -0
- package/dist/improve/state.js +261 -0
- package/dist/improve/state.js.map +1 -0
- package/dist/main.js +11 -0
- package/dist/main.js.map +1 -1
- package/dist/native/darwin-arm64/rudder-native +0 -0
- package/dist/native/darwin-x64/rudder-native +0 -0
- package/dist/native/linux-x64/rudder-native +0 -0
- package/dist/state.js +2 -0
- package/dist/state.js.map +1 -1
- package/dist/task-summary.d.ts +6 -0
- package/dist/task-summary.js +6 -1
- package/dist/task-summary.js.map +1 -1
- package/dist/types.d.ts +16 -0
- package/dist/types.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -16,6 +16,11 @@ isolated, reviewable, and easy to merge. It opens a native three-pane dashboard,
|
|
|
16
16
|
gives every task its own git worktree, and runs real Claude Code or Codex
|
|
17
17
|
processes in the worker pane.
|
|
18
18
|
|
|
19
|
+
> **AI agents and contributors:** this README is for using Rudder. If you are
|
|
20
|
+
> an AI agent (or a human) working on the Rudder codebase itself, read
|
|
21
|
+
> [`AGENTS.md`](./AGENTS.md) first; it is the engineering reference and the
|
|
22
|
+
> source of truth for architecture, conventions, and release rules.
|
|
23
|
+
|
|
19
24
|
## Install
|
|
20
25
|
|
|
21
26
|
```bash
|
|
@@ -181,6 +186,7 @@ running, Rudder also exposes matching project skills; the orchestrator can write
|
|
|
181
186
|
| `/review-all` | Combine completed worktrees and start a Codex review-all agent |
|
|
182
187
|
| `/merge-all` | Merge all completed worktrees |
|
|
183
188
|
| `/automerge` | Toggle automatic clean merges (persists across sessions; shown in the agents pane header) |
|
|
189
|
+
| `/color terminal\|paper` | Choose the native dashboard color mode; `terminal` uses your terminal foreground/background, `paper` restores the white canvas |
|
|
184
190
|
| `/login` | Browser login for Rudder Cloud |
|
|
185
191
|
| `/cloud` | Onload the current workspace or start a fresh cloud worker |
|
|
186
192
|
| `/cloud list` | List cloud workers |
|
|
@@ -350,6 +356,42 @@ rudder cleanup
|
|
|
350
356
|
`rudder mouse-test parsed`. Set `RUDDER_WHEEL_SCROLL_ROWS=<n>` to change the
|
|
351
357
|
scroll step, or `RUDDER_MOUSE_DEBUG=1` to inspect routing.
|
|
352
358
|
|
|
359
|
+
## Continual improvement loop
|
|
360
|
+
|
|
361
|
+
Rudder improves itself from its own telemetry. Every run already leaves a
|
|
362
|
+
record on disk (`.rudder/runs/`, event logs, verifier results, steer history,
|
|
363
|
+
token usage). The improvement loop, `rudder improve`, is a scheduled local
|
|
364
|
+
batch job (launchd on macOS, not a resident daemon) that:
|
|
365
|
+
|
|
366
|
+
1. **Collects** recent session telemetry across your registered projects,
|
|
367
|
+
redacting secrets at the source.
|
|
368
|
+
2. **Mines** it into ranked friction findings (failed runs, user redirects,
|
|
369
|
+
merge conflicts, verifier misses) using the advisor pattern: a Sonnet
|
|
370
|
+
executor consults a Fable 5 advisor mid-generation, so most tokens bill at
|
|
371
|
+
the executor rate.
|
|
372
|
+
3. **Proposes** fixes with headless agents in isolated worktrees of the
|
|
373
|
+
rudder repo, each briefed with a rich context pack (finding, evidence,
|
|
374
|
+
surface map, prior failed attempts, repo conventions).
|
|
375
|
+
4. **Judges** each candidate with the repo's full test gates plus a
|
|
376
|
+
three-lens adversarial LLM panel that fails closed.
|
|
377
|
+
5. **Ships** survivors automatically: rebase onto `origin/main`,
|
|
378
|
+
`npm version patch`, push main + tag, and the normal tag-driven CI
|
|
379
|
+
publishes the new version to npm. Later cycles verify the targeted metric
|
|
380
|
+
actually improved and flag regressions for revert.
|
|
381
|
+
|
|
382
|
+
```bash
|
|
383
|
+
rudder improve run --dry-run # see what it would do, propose nothing
|
|
384
|
+
rudder improve schedule install # nightly cycle at 03:30 via launchd
|
|
385
|
+
rudder improve status # shipped versions, metrics trend, ledger
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
Everything stays on your machine, spend is hard-capped per cycle
|
|
389
|
+
(`improve.budgetUsd`, default $5), autonomy is configurable
|
|
390
|
+
(`improve.autonomy`: `observe` | `propose` | `ship`), and `RUDDER_IMPROVE=0`
|
|
391
|
+
disables it. The full harness spec is in
|
|
392
|
+
[`docs/continual-improvement.md`](./docs/continual-improvement.md);
|
|
393
|
+
implementation details live in [`AGENTS.md`](./AGENTS.md) section 15.
|
|
394
|
+
|
|
353
395
|
## Building from source
|
|
354
396
|
|
|
355
397
|
```bash
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
import type { SpendMeter } from "./state.js";
|
|
2
|
+
export declare function callAdvisedTextModel(params: {
|
|
3
|
+
executorModel: string;
|
|
4
|
+
advisorModel: string;
|
|
5
|
+
system: string;
|
|
6
|
+
user: string;
|
|
7
|
+
maxTokens?: number;
|
|
8
|
+
timeoutMs?: number;
|
|
9
|
+
meter?: SpendMeter;
|
|
10
|
+
}): Promise<string>;
|
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
import { callTextModel, resolveAnthropicApiKey } from "../task-summary.js";
|
|
2
|
+
/**
|
|
3
|
+
* Advisor-pattern text call (Anthropic advisor tool, beta
|
|
4
|
+
* advisor-tool-2026-03-01): a cheaper executor model generates the output and
|
|
5
|
+
* consults a higher-intelligence advisor mid-generation for strategic
|
|
6
|
+
* guidance. Most tokens bill at the executor rate; the loop gets near
|
|
7
|
+
* advisor-grade judgment on its mining and judging calls at a fraction of the
|
|
8
|
+
* cost of running the big model end to end.
|
|
9
|
+
*
|
|
10
|
+
* Degradation path: the deterministic test hook and the no-API-key case (CLI
|
|
11
|
+
* auth) fall back to callTextModel, and any advisor-request failure falls
|
|
12
|
+
* back to a plain executor-only call, so the loop never hard-depends on the
|
|
13
|
+
* beta.
|
|
14
|
+
*/
|
|
15
|
+
const ADVISOR_BETA = "advisor-tool-2026-03-01";
|
|
16
|
+
const ADVISOR_MAX_TOKENS = 2048;
|
|
17
|
+
const MAX_PAUSE_RESUMES = 3;
|
|
18
|
+
const ADVISOR_SYSTEM_NUDGE = `
|
|
19
|
+
|
|
20
|
+
You have access to an \`advisor\` tool backed by a stronger reviewer model. It takes NO parameters; your full context is forwarded automatically. Consult it once before committing to your answer, weigh its guidance seriously, then produce the final output yourself in the requested format.`;
|
|
21
|
+
const ADVISOR_USER_SUFFIX = "\n\n(Advisor: please keep your guidance under 200 words; a focused verdict on the hard part beats a comprehensive plan.)";
|
|
22
|
+
export async function callAdvisedTextModel(params) {
|
|
23
|
+
const plainFallback = () => callTextModel({
|
|
24
|
+
model: params.executorModel,
|
|
25
|
+
system: params.system,
|
|
26
|
+
user: params.user,
|
|
27
|
+
maxTokens: params.maxTokens,
|
|
28
|
+
timeoutMs: params.timeoutMs,
|
|
29
|
+
});
|
|
30
|
+
// Deterministic test hook and disabled-advisor path go straight through.
|
|
31
|
+
if (process.env.RUDDER_FAKE_MODEL_OUTPUT || !params.advisorModel) {
|
|
32
|
+
return await plainFallback();
|
|
33
|
+
}
|
|
34
|
+
const apiKey = await resolveAnthropicApiKey();
|
|
35
|
+
if (!apiKey) {
|
|
36
|
+
// CLI-auth users have no raw key to hit the API with; use the CLI path.
|
|
37
|
+
return await plainFallback();
|
|
38
|
+
}
|
|
39
|
+
const tools = [
|
|
40
|
+
{
|
|
41
|
+
type: "advisor_20260301",
|
|
42
|
+
name: "advisor",
|
|
43
|
+
model: params.advisorModel,
|
|
44
|
+
max_tokens: ADVISOR_MAX_TOKENS,
|
|
45
|
+
},
|
|
46
|
+
];
|
|
47
|
+
const messages = [
|
|
48
|
+
{ role: "user", content: params.user + ADVISOR_USER_SUFFIX },
|
|
49
|
+
];
|
|
50
|
+
const timeoutMs = params.timeoutMs ?? 240000;
|
|
51
|
+
const collected = [];
|
|
52
|
+
for (let attempt = 0; attempt <= MAX_PAUSE_RESUMES; attempt += 1) {
|
|
53
|
+
let data;
|
|
54
|
+
try {
|
|
55
|
+
const controller = new AbortController();
|
|
56
|
+
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
|
57
|
+
const response = await fetch("https://api.anthropic.com/v1/messages", {
|
|
58
|
+
method: "POST",
|
|
59
|
+
headers: {
|
|
60
|
+
"content-type": "application/json",
|
|
61
|
+
"x-api-key": apiKey,
|
|
62
|
+
"anthropic-version": "2023-06-01",
|
|
63
|
+
"anthropic-beta": ADVISOR_BETA,
|
|
64
|
+
},
|
|
65
|
+
body: JSON.stringify({
|
|
66
|
+
model: params.executorModel,
|
|
67
|
+
max_tokens: params.maxTokens ?? 2048,
|
|
68
|
+
system: params.system + ADVISOR_SYSTEM_NUDGE,
|
|
69
|
+
tools,
|
|
70
|
+
messages,
|
|
71
|
+
}),
|
|
72
|
+
signal: controller.signal,
|
|
73
|
+
});
|
|
74
|
+
clearTimeout(timer);
|
|
75
|
+
if (!response.ok) {
|
|
76
|
+
// Invalid pair / beta unavailable / overloaded: degrade to plain.
|
|
77
|
+
return await plainFallback();
|
|
78
|
+
}
|
|
79
|
+
data = (await response.json());
|
|
80
|
+
}
|
|
81
|
+
catch {
|
|
82
|
+
return await plainFallback();
|
|
83
|
+
}
|
|
84
|
+
meterIterations(params, data);
|
|
85
|
+
for (const block of data.content ?? []) {
|
|
86
|
+
if (block.type === "text" && typeof block.text === "string") {
|
|
87
|
+
collected.push(block.text);
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
if (data.stop_reason === "pause_turn" && attempt < MAX_PAUSE_RESUMES) {
|
|
91
|
+
// Resume the paused turn: round-trip the assistant content verbatim.
|
|
92
|
+
messages.push({ role: "assistant", content: data.content ?? [] });
|
|
93
|
+
continue;
|
|
94
|
+
}
|
|
95
|
+
break;
|
|
96
|
+
}
|
|
97
|
+
const text = collected.join("").trim();
|
|
98
|
+
return text || (await plainFallback());
|
|
99
|
+
}
|
|
100
|
+
function meterIterations(params, data) {
|
|
101
|
+
if (!params.meter)
|
|
102
|
+
return;
|
|
103
|
+
const iterations = data.usage?.iterations;
|
|
104
|
+
if (iterations && iterations.length > 0) {
|
|
105
|
+
for (const iteration of iterations) {
|
|
106
|
+
const model = iteration.type === "advisor_message"
|
|
107
|
+
? (iteration.model ?? params.advisorModel)
|
|
108
|
+
: params.executorModel;
|
|
109
|
+
params.meter.addModelTokens(model, iteration.input_tokens ?? 0, iteration.output_tokens ?? 0);
|
|
110
|
+
}
|
|
111
|
+
return;
|
|
112
|
+
}
|
|
113
|
+
params.meter.addModelTokens(params.executorModel, data.usage?.input_tokens ?? 0, data.usage?.output_tokens ?? 0);
|
|
114
|
+
}
|
|
115
|
+
//# sourceMappingURL=advisor.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"advisor.js","sourceRoot":"","sources":["../../src/improve/advisor.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,sBAAsB,EAAE,MAAM,oBAAoB,CAAC;AAG3E;;;;;;;;;;;;GAYG;AAEH,MAAM,YAAY,GAAG,yBAAyB,CAAC;AAC/C,MAAM,kBAAkB,GAAG,IAAI,CAAC;AAChC,MAAM,iBAAiB,GAAG,CAAC,CAAC;AAE5B,MAAM,oBAAoB,GAAG;;kSAEqQ,CAAC;AAEnS,MAAM,mBAAmB,GACvB,0HAA0H,CAAC;AAmB7H,MAAM,CAAC,KAAK,UAAU,oBAAoB,CAAC,MAQ1C;IACC,MAAM,aAAa,GAAG,GAAG,EAAE,CACzB,aAAa,CAAC;QACZ,KAAK,EAAE,MAAM,CAAC,aAAa;QAC3B,MAAM,EAAE,MAAM,CAAC,MAAM;QACrB,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,SAAS,EAAE,MAAM,CAAC,SAAS;QAC3B,SAAS,EAAE,MAAM,CAAC,SAAS;KAC5B,CAAC,CAAC;IAEL,yEAAyE;IACzE,IAAI,OAAO,CAAC,GAAG,CAAC,wBAAwB,IAAI,CAAC,MAAM,CAAC,YAAY,EAAE,CAAC;QACjE,OAAO,MAAM,aAAa,EAAE,CAAC;IAC/B,CAAC;IACD,MAAM,MAAM,GAAG,MAAM,sBAAsB,EAAE,CAAC;IAC9C,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,wEAAwE;QACxE,OAAO,MAAM,aAAa,EAAE,CAAC;IAC/B,CAAC;IAED,MAAM,KAAK,GAAG;QACZ;YACE,IAAI,EAAE,kBAAkB;YACxB,IAAI,EAAE,SAAS;YACf,KAAK,EAAE,MAAM,CAAC,YAAY;YAC1B,UAAU,EAAE,kBAAkB;SAC/B;KACF,CAAC;IACF,MAAM,QAAQ,GAA8C;QAC1D,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;KAC7D,CAAC;IACF,MAAM,SAAS,GAAG,MAAM,CAAC,SAAS,IAAI,MAAM,CAAC;IAC7C,MAAM,SAAS,GAAa,EAAE,CAAC;IAE/B,KAAK,IAAI,OAAO,GAAG,CAAC,EAAE,OAAO,IAAI,iBAAiB,EAAE,OAAO,IAAI,CAAC,EAAE,CAAC;QACjE,IAAI,IAAsB,CAAC;QAC3B,IAAI,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,eAAe,EAAE,CAAC;YACzC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,UAAU,CAAC,KAAK,EAAE,EAAE,SAAS,CAAC,CAAC;YAC9D,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,uCAAuC,EAAE;gBACpE,MAAM,EAAE,MAAM;gBACd,OAAO,EAAE;oBACP,cAAc,EAAE,kBAAkB;oBAClC,WAAW,EAAE,MAAM;oBACnB,mBAAmB,EAAE,YAAY;oBACjC,gBAAgB,EAAE,YAAY;iBAC/B;gBACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;oBACnB,KAAK,EAAE,MAAM,CAAC,aAAa;oBAC3B,UAAU,EAAE,MAAM,CAAC,SAAS,IAAI,IAAI;oBACpC,MAAM,EAAE,MAAM,CAAC,MAAM,GAAG,oBAAoB;oBAC5C,KAAK;oBACL,QAAQ;iBACT,CAAC;gBACF,MAAM,EAAE,UAAU,CAAC,MAAM;aAC1B,CAAC,CAAC;YACH,YAAY,CAAC,KAAK,CAAC,CAAC;YACpB,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,kEAAkE;gBAClE,OAAO,MAAM,aAAa,EAAE,CAAC;YAC/B,CAAC;YACD,IAAI,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAqB,CAAC;QACrD,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,MAAM,aAAa,EAAE,CAAC;QAC/B,CAAC;QAED,eAAe,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;QAC9B,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,OAAO,IAAI,EAAE,EAAE,CAAC;YACvC,IAAI,KAAK,CAAC,IAAI,KAAK,MAAM,IAAI,OAAO,KAAK,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAC5D,SAAS,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;YAC7B,CAAC;QACH,CAAC;QACD,IAAI,IAAI,CAAC,WAAW,KAAK,YAAY,IAAI,OAAO,GAAG,iBAAiB,EAAE,CAAC;YACrE,qEAAqE;YACrE,QAAQ,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,WAAW,EAAE,OAAO,EAAE,IAAI,CAAC,OAAO,IAAI,EAAE,EAAE,CAAC,CAAC;YAClE,SAAS;QACX,CAAC;QACD,MAAM;IACR,CAAC;IAED,MAAM,IAAI,GAAG,SAAS,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;IACvC,OAAO,IAAI,IAAI,CAAC,MAAM,aAAa,EAAE,CAAC,CAAC;AACzC,CAAC;AAED,SAAS,eAAe,CACtB,MAA2E,EAC3E,IAAsB;IAEtB,IAAI,CAAC,MAAM,CAAC,KAAK;QAAE,OAAO;IAC1B,MAAM,UAAU,GAAG,IAAI,CAAC,KAAK,EAAE,UAAU,CAAC;IAC1C,IAAI,UAAU,IAAI,UAAU,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACxC,KAAK,MAAM,SAAS,IAAI,UAAU,EAAE,CAAC;YACnC,MAAM,KAAK,GACT,SAAS,CAAC,IAAI,KAAK,iBAAiB;gBAClC,CAAC,CAAC,CAAC,SAAS,CAAC,KAAK,IAAI,MAAM,CAAC,YAAY,CAAC;gBAC1C,CAAC,CAAC,MAAM,CAAC,aAAa,CAAC;YAC3B,MAAM,CAAC,KAAK,CAAC,cAAc,CAAC,KAAK,EAAE,SAAS,CAAC,YAAY,IAAI,CAAC,EAAE,SAAS,CAAC,aAAa,IAAI,CAAC,CAAC,CAAC;QAChG,CAAC;QACD,OAAO;IACT,CAAC;IACD,MAAM,CAAC,KAAK,CAAC,cAAc,CACzB,MAAM,CAAC,aAAa,EACpB,IAAI,CAAC,KAAK,EAAE,YAAY,IAAI,CAAC,EAC7B,IAAI,CAAC,KAAK,EAAE,aAAa,IAAI,CAAC,CAC/B,CAAC;AACJ,CAAC"}
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
import type { ImproveSettings, MetricsSnapshot, SessionRecord, Watermark } from "./state.js";
|
|
2
|
+
/**
|
|
3
|
+
* Walk registered projects and build redacted session records for every run
|
|
4
|
+
* newer than the watermark. Pure telemetry read; never writes into a project.
|
|
5
|
+
*/
|
|
6
|
+
export declare function collectSessions(settings: ImproveSettings, watermark: Watermark): Promise<{
|
|
7
|
+
sessions: SessionRecord[];
|
|
8
|
+
nextWatermark: Watermark;
|
|
9
|
+
}>;
|
|
10
|
+
export declare function computeMetrics(cycleId: string, sessions: SessionRecord[]): MetricsSnapshot;
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
import path from "node:path";
|
|
2
|
+
import { promises as fs } from "node:fs";
|
|
3
|
+
import { loadProjects, runRecordPath, runsDir, verifierPath } from "../state.js";
|
|
4
|
+
import { readJson } from "../util.js";
|
|
5
|
+
import { redactTaskSummarySecrets } from "../task-summary.js";
|
|
6
|
+
const MAX_SESSIONS_PER_PROJECT = 200;
|
|
7
|
+
const MAX_ERROR_EXCERPTS = 3;
|
|
8
|
+
const ERROR_EXCERPT_CHARS = 300;
|
|
9
|
+
const EVENTS_TAIL_BYTES = 64 * 1024;
|
|
10
|
+
/**
|
|
11
|
+
* Walk registered projects and build redacted session records for every run
|
|
12
|
+
* newer than the watermark. Pure telemetry read; never writes into a project.
|
|
13
|
+
*/
|
|
14
|
+
export async function collectSessions(settings, watermark) {
|
|
15
|
+
const projects = await loadProjects();
|
|
16
|
+
const sessions = [];
|
|
17
|
+
const nextWatermark = { version: 1, projects: { ...watermark.projects } };
|
|
18
|
+
for (const project of projects) {
|
|
19
|
+
if (!project.repoRoot)
|
|
20
|
+
continue;
|
|
21
|
+
if (settings.excludeProjects.some((pattern) => matchesGlobish(project.repoRoot, pattern))) {
|
|
22
|
+
continue;
|
|
23
|
+
}
|
|
24
|
+
let exists = false;
|
|
25
|
+
try {
|
|
26
|
+
exists = (await fs.stat(project.repoRoot)).isDirectory();
|
|
27
|
+
}
|
|
28
|
+
catch {
|
|
29
|
+
exists = false;
|
|
30
|
+
}
|
|
31
|
+
if (!exists)
|
|
32
|
+
continue;
|
|
33
|
+
const since = watermark.projects[project.repoRoot] ?? "";
|
|
34
|
+
const runs = await readRunsRaw(project.repoRoot);
|
|
35
|
+
const fresh = runs
|
|
36
|
+
.filter((run) => (run.updatedAt ?? "") > since)
|
|
37
|
+
// Skip runs still in flight; their story is not over yet.
|
|
38
|
+
.filter((run) => !["created", "running", "steering", "verifying"].includes(run.status))
|
|
39
|
+
.sort((a, b) => (a.updatedAt < b.updatedAt ? -1 : 1))
|
|
40
|
+
.slice(-MAX_SESSIONS_PER_PROJECT);
|
|
41
|
+
for (const run of fresh) {
|
|
42
|
+
sessions.push(await sessionFromRun(project.name ?? project.slug ?? project.repoRoot, run));
|
|
43
|
+
}
|
|
44
|
+
const newest = fresh.at(-1)?.updatedAt;
|
|
45
|
+
if (newest && newest > (nextWatermark.projects[project.repoRoot] ?? "")) {
|
|
46
|
+
nextWatermark.projects[project.repoRoot] = newest;
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
return { sessions, nextWatermark };
|
|
50
|
+
}
|
|
51
|
+
/**
|
|
52
|
+
* Read run records WITHOUT the state.ts load path: loadRunRecord fires the
|
|
53
|
+
* background LLM task summarizer per record, which both costs model calls and
|
|
54
|
+
* rewrites run.json (bumping updatedAt past the watermark). Telemetry
|
|
55
|
+
* collection must be a pure read.
|
|
56
|
+
*/
|
|
57
|
+
async function readRunsRaw(repoRoot) {
|
|
58
|
+
let entries = [];
|
|
59
|
+
try {
|
|
60
|
+
const dirents = await fs.readdir(runsDir(repoRoot), { withFileTypes: true });
|
|
61
|
+
entries = dirents.filter((entry) => entry.isDirectory()).map((entry) => entry.name);
|
|
62
|
+
}
|
|
63
|
+
catch {
|
|
64
|
+
return [];
|
|
65
|
+
}
|
|
66
|
+
const runs = await Promise.all(entries.map((id) => readJson(runRecordPath(repoRoot, id))));
|
|
67
|
+
return runs.filter((run) => Boolean(run?.id && run?.status && run?.updatedAt));
|
|
68
|
+
}
|
|
69
|
+
async function sessionFromRun(project, run) {
|
|
70
|
+
const steerCount = (run.turns ?? []).filter((turn) => turn.source === "steerer" || turn.source === "regoal").length;
|
|
71
|
+
const startedAt = run.process?.startedAt ?? run.createdAt;
|
|
72
|
+
const endedAt = run.process?.endedAt ?? run.updatedAt;
|
|
73
|
+
let durationMs;
|
|
74
|
+
const startMs = Date.parse(startedAt ?? "");
|
|
75
|
+
const endMs = Date.parse(endedAt ?? "");
|
|
76
|
+
if (Number.isFinite(startMs) && Number.isFinite(endMs) && endMs >= startMs) {
|
|
77
|
+
durationMs = endMs - startMs;
|
|
78
|
+
}
|
|
79
|
+
let verifierSatisfied;
|
|
80
|
+
let verifierMissing;
|
|
81
|
+
const verification = run.verification ??
|
|
82
|
+
(await readJson(verifierPath(run.repoRoot, run.id))) ??
|
|
83
|
+
undefined;
|
|
84
|
+
if (verification && typeof verification.satisfied === "boolean") {
|
|
85
|
+
verifierSatisfied = verification.satisfied;
|
|
86
|
+
verifierMissing = (verification.missing ?? []).slice(0, 5).map(redactAndTrim);
|
|
87
|
+
}
|
|
88
|
+
const interesting = run.status === "failed" ||
|
|
89
|
+
run.status === "cancelled" ||
|
|
90
|
+
run.status === "merge-conflict" ||
|
|
91
|
+
verifierSatisfied === false;
|
|
92
|
+
return {
|
|
93
|
+
project,
|
|
94
|
+
repoRoot: run.repoRoot,
|
|
95
|
+
runId: run.id,
|
|
96
|
+
task: redactAndTrim(run.taskSummary ?? run.task ?? "", 160),
|
|
97
|
+
backend: run.backend,
|
|
98
|
+
model: run.model,
|
|
99
|
+
effort: run.effort,
|
|
100
|
+
status: run.status,
|
|
101
|
+
createdAt: run.createdAt,
|
|
102
|
+
updatedAt: run.updatedAt,
|
|
103
|
+
durationMs,
|
|
104
|
+
steerCount,
|
|
105
|
+
autoSteerCount: run.autoSteer?.count ?? 0,
|
|
106
|
+
tokensIn: run.tokens?.input ?? 0,
|
|
107
|
+
tokensOut: run.tokens?.output ?? 0,
|
|
108
|
+
verifierSatisfied,
|
|
109
|
+
verifierMissing,
|
|
110
|
+
mergeStatus: run.merge?.status,
|
|
111
|
+
errorExcerpts: interesting ? await readErrorExcerpts(run) : [],
|
|
112
|
+
};
|
|
113
|
+
}
|
|
114
|
+
/**
|
|
115
|
+
* Read the tail of a run's events.ndjson and pull out error-shaped events.
|
|
116
|
+
* Redaction happens here, before anything reaches a model.
|
|
117
|
+
*/
|
|
118
|
+
async function readErrorExcerpts(run) {
|
|
119
|
+
const eventsFile = path.join(run.repoRoot, ".rudder", "runs", run.id, "events.ndjson");
|
|
120
|
+
let raw = "";
|
|
121
|
+
try {
|
|
122
|
+
const handle = await fs.open(eventsFile, "r");
|
|
123
|
+
try {
|
|
124
|
+
const stat = await handle.stat();
|
|
125
|
+
const start = Math.max(0, stat.size - EVENTS_TAIL_BYTES);
|
|
126
|
+
const buffer = Buffer.alloc(stat.size - start);
|
|
127
|
+
await handle.read(buffer, 0, buffer.length, start);
|
|
128
|
+
raw = buffer.toString("utf8");
|
|
129
|
+
}
|
|
130
|
+
finally {
|
|
131
|
+
await handle.close();
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
catch {
|
|
135
|
+
return [];
|
|
136
|
+
}
|
|
137
|
+
const excerpts = [];
|
|
138
|
+
for (const line of raw.split("\n").reverse()) {
|
|
139
|
+
if (excerpts.length >= MAX_ERROR_EXCERPTS)
|
|
140
|
+
break;
|
|
141
|
+
const trimmed = line.trim();
|
|
142
|
+
if (!trimmed)
|
|
143
|
+
continue;
|
|
144
|
+
let event = null;
|
|
145
|
+
try {
|
|
146
|
+
event = JSON.parse(trimmed);
|
|
147
|
+
}
|
|
148
|
+
catch {
|
|
149
|
+
continue;
|
|
150
|
+
}
|
|
151
|
+
const type = event?.type ?? "";
|
|
152
|
+
if (!/error|fail/i.test(type))
|
|
153
|
+
continue;
|
|
154
|
+
const text = event?.text ?? event?.message ?? "";
|
|
155
|
+
if (!text.trim())
|
|
156
|
+
continue;
|
|
157
|
+
excerpts.push(`[${type}] ${redactAndTrim(text, ERROR_EXCERPT_CHARS)}`);
|
|
158
|
+
}
|
|
159
|
+
return excerpts;
|
|
160
|
+
}
|
|
161
|
+
export function computeMetrics(cycleId, sessions) {
|
|
162
|
+
const n = sessions.length;
|
|
163
|
+
const rate = (count) => (n === 0 ? 0 : round4(count / n));
|
|
164
|
+
const failed = sessions.filter((s) => s.status === "failed").length;
|
|
165
|
+
const cancelled = sessions.filter((s) => s.status === "cancelled").length;
|
|
166
|
+
const conflicts = sessions.filter((s) => s.status === "merge-conflict" || s.mergeStatus === "conflict").length;
|
|
167
|
+
const verifierMisses = sessions.filter((s) => s.verifierSatisfied === false).length;
|
|
168
|
+
const steered = sessions.filter((s) => s.steerCount > 0).length;
|
|
169
|
+
const durations = sessions
|
|
170
|
+
.map((s) => s.durationMs)
|
|
171
|
+
.filter((d) => typeof d === "number")
|
|
172
|
+
.sort((a, b) => a - b);
|
|
173
|
+
return {
|
|
174
|
+
ts: new Date().toISOString(),
|
|
175
|
+
cycleId,
|
|
176
|
+
runsSeen: n,
|
|
177
|
+
failedRate: rate(failed),
|
|
178
|
+
cancelledRate: rate(cancelled),
|
|
179
|
+
steerRate: rate(steered),
|
|
180
|
+
mergeConflictRate: rate(conflicts),
|
|
181
|
+
verifierMissRate: rate(verifierMisses),
|
|
182
|
+
medianDurationMs: durations.length ? durations[Math.floor(durations.length / 2)] : 0,
|
|
183
|
+
totalTokensIn: sessions.reduce((sum, s) => sum + s.tokensIn, 0),
|
|
184
|
+
totalTokensOut: sessions.reduce((sum, s) => sum + s.tokensOut, 0),
|
|
185
|
+
};
|
|
186
|
+
}
|
|
187
|
+
function redactAndTrim(value, maxChars = 240) {
|
|
188
|
+
const redacted = redactTaskSummarySecrets(value).replace(/\s+/g, " ").trim();
|
|
189
|
+
return redacted.length > maxChars ? `${redacted.slice(0, maxChars - 1)}…` : redacted;
|
|
190
|
+
}
|
|
191
|
+
function round4(value) {
|
|
192
|
+
return Math.round(value * 10000) / 10000;
|
|
193
|
+
}
|
|
194
|
+
/** Minimal glob support: `*` wildcards only, matched against the repo path. */
|
|
195
|
+
function matchesGlobish(value, pattern) {
|
|
196
|
+
const regex = new RegExp(`^${pattern.split("*").map(escapeRegExp).join(".*")}$`);
|
|
197
|
+
return regex.test(value);
|
|
198
|
+
}
|
|
199
|
+
function escapeRegExp(value) {
|
|
200
|
+
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
201
|
+
}
|
|
202
|
+
//# sourceMappingURL=collect.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"collect.js","sourceRoot":"","sources":["../../src/improve/collect.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,QAAQ,IAAI,EAAE,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AACjF,OAAO,EAAE,QAAQ,EAAE,MAAM,YAAY,CAAC;AACtC,OAAO,EAAE,wBAAwB,EAAE,MAAM,oBAAoB,CAAC;AAI9D,MAAM,wBAAwB,GAAG,GAAG,CAAC;AACrC,MAAM,kBAAkB,GAAG,CAAC,CAAC;AAC7B,MAAM,mBAAmB,GAAG,GAAG,CAAC;AAChC,MAAM,iBAAiB,GAAG,EAAE,GAAG,IAAI,CAAC;AAEpC;;;GAGG;AACH,MAAM,CAAC,KAAK,UAAU,eAAe,CACnC,QAAyB,EACzB,SAAoB;IAEpB,MAAM,QAAQ,GAAG,MAAM,YAAY,EAAE,CAAC;IACtC,MAAM,QAAQ,GAAoB,EAAE,CAAC;IACrC,MAAM,aAAa,GAAc,EAAE,OAAO,EAAE,CAAC,EAAE,QAAQ,EAAE,EAAE,GAAG,SAAS,CAAC,QAAQ,EAAE,EAAE,CAAC;IAErF,KAAK,MAAM,OAAO,IAAI,QAAQ,EAAE,CAAC;QAC/B,IAAI,CAAC,OAAO,CAAC,QAAQ;YAAE,SAAS;QAChC,IAAI,QAAQ,CAAC,eAAe,CAAC,IAAI,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,cAAc,CAAC,OAAO,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC,EAAE,CAAC;YAC1F,SAAS;QACX,CAAC;QACD,IAAI,MAAM,GAAG,KAAK,CAAC;QACnB,IAAI,CAAC;YACH,MAAM,GAAG,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC;QAC3D,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,GAAG,KAAK,CAAC;QACjB,CAAC;QACD,IAAI,CAAC,MAAM;YAAE,SAAS;QAEtB,MAAM,KAAK,GAAG,SAAS,CAAC,QAAQ,CAAC,OAAO,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC;QACzD,MAAM,IAAI,GAAG,MAAM,WAAW,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACjD,MAAM,KAAK,GAAG,IAAI;aACf,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,SAAS,IAAI,EAAE,CAAC,GAAG,KAAK,CAAC;YAC/C,0DAA0D;aACzD,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,EAAE,SAAS,EAAE,UAAU,EAAE,WAAW,CAAC,CAAC,QAAQ,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;aACtF,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,GAAG,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;aACpD,KAAK,CAAC,CAAC,wBAAwB,CAAC,CAAC;QAEpC,KAAK,MAAM,GAAG,IAAI,KAAK,EAAE,CAAC;YACxB,QAAQ,CAAC,IAAI,CAAC,MAAM,cAAc,CAAC,OAAO,CAAC,IAAI,IAAI,OAAO,CAAC,IAAI,IAAI,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC,CAAC;QAC7F,CAAC;QACD,MAAM,MAAM,GAAG,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,SAAS,CAAC;QACvC,IAAI,MAAM,IAAI,MAAM,GAAG,CAAC,aAAa,CAAC,QAAQ,CAAC,OAAO,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC;YACxE,aAAa,CAAC,QAAQ,CAAC,OAAO,CAAC,QAAQ,CAAC,GAAG,MAAM,CAAC;QACpD,CAAC;IACH,CAAC;IACD,OAAO,EAAE,QAAQ,EAAE,aAAa,EAAE,CAAC;AACrC,CAAC;AAED;;;;;GAKG;AACH,KAAK,UAAU,WAAW,CAAC,QAAgB;IACzC,IAAI,OAAO,GAAa,EAAE,CAAC;IAC3B,IAAI,CAAC;QACH,MAAM,OAAO,GAAG,MAAM,EAAE,CAAC,OAAO,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAC;QAC7E,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,WAAW,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IACtF,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,EAAE,CAAC;IACZ,CAAC;IACD,MAAM,IAAI,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,QAAQ,CAAY,aAAa,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;IACtG,OAAO,IAAI,CAAC,MAAM,CAAC,CAAC,GAAG,EAAoB,EAAE,CAAC,OAAO,CAAC,GAAG,EAAE,EAAE,IAAI,GAAG,EAAE,MAAM,IAAI,GAAG,EAAE,SAAS,CAAC,CAAC,CAAC;AACnG,CAAC;AAED,KAAK,UAAU,cAAc,CAAC,OAAe,EAAE,GAAc;IAC3D,MAAM,UAAU,GAAG,CAAC,GAAG,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,MAAM,CACzC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,KAAK,SAAS,IAAI,IAAI,CAAC,MAAM,KAAK,QAAQ,CAChE,CAAC,MAAM,CAAC;IACT,MAAM,SAAS,GAAG,GAAG,CAAC,OAAO,EAAE,SAAS,IAAI,GAAG,CAAC,SAAS,CAAC;IAC1D,MAAM,OAAO,GAAG,GAAG,CAAC,OAAO,EAAE,OAAO,IAAI,GAAG,CAAC,SAAS,CAAC;IACtD,IAAI,UAA8B,CAAC;IACnC,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,SAAS,IAAI,EAAE,CAAC,CAAC;IAC5C,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC;IACxC,IAAI,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,OAAO,EAAE,CAAC;QAC3E,UAAU,GAAG,KAAK,GAAG,OAAO,CAAC;IAC/B,CAAC;IAED,IAAI,iBAAsC,CAAC;IAC3C,IAAI,eAAqC,CAAC;IAC1C,MAAM,YAAY,GAChB,GAAG,CAAC,YAAY;QAChB,CAAC,MAAM,QAAQ,CAAqB,YAAY,CAAC,GAAG,CAAC,QAAQ,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC;QACxE,SAAS,CAAC;IACZ,IAAI,YAAY,IAAI,OAAO,YAAY,CAAC,SAAS,KAAK,SAAS,EAAE,CAAC;QAChE,iBAAiB,GAAG,YAAY,CAAC,SAAS,CAAC;QAC3C,eAAe,GAAG,CAAC,YAAY,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC;IAChF,CAAC;IAED,MAAM,WAAW,GACf,GAAG,CAAC,MAAM,KAAK,QAAQ;QACvB,GAAG,CAAC,MAAM,KAAK,WAAW;QAC1B,GAAG,CAAC,MAAM,KAAK,gBAAgB;QAC/B,iBAAiB,KAAK,KAAK,CAAC;IAE9B,OAAO;QACL,OAAO;QACP,QAAQ,EAAE,GAAG,CAAC,QAAQ;QACtB,KAAK,EAAE,GAAG,CAAC,EAAE;QACb,IAAI,EAAE,aAAa,CAAC,GAAG,CAAC,WAAW,IAAI,GAAG,CAAC,IAAI,IAAI,EAAE,EAAE,GAAG,CAAC;QAC3D,OAAO,EAAE,GAAG,CAAC,OAAO;QACpB,KAAK,EAAE,GAAG,CAAC,KAAK;QAChB,MAAM,EAAE,GAAG,CAAC,MAAM;QAClB,MAAM,EAAE,GAAG,CAAC,MAAM;QAClB,SAAS,EAAE,GAAG,CAAC,SAAS;QACxB,SAAS,EAAE,GAAG,CAAC,SAAS;QACxB,UAAU;QACV,UAAU;QACV,cAAc,EAAE,GAAG,CAAC,SAAS,EAAE,KAAK,IAAI,CAAC;QACzC,QAAQ,EAAE,GAAG,CAAC,MAAM,EAAE,KAAK,IAAI,CAAC;QAChC,SAAS,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,IAAI,CAAC;QAClC,iBAAiB;QACjB,eAAe;QACf,WAAW,EAAE,GAAG,CAAC,KAAK,EAAE,MAAM;QAC9B,aAAa,EAAE,WAAW,CAAC,CAAC,CAAC,MAAM,iBAAiB,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE;KAC/D,CAAC;AACJ,CAAC;AAED;;;GAGG;AACH,KAAK,UAAU,iBAAiB,CAAC,GAAc;IAC7C,MAAM,UAAU,GAAG,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ,EAAE,SAAS,EAAE,MAAM,EAAE,GAAG,CAAC,EAAE,EAAE,eAAe,CAAC,CAAC;IACvF,IAAI,GAAG,GAAG,EAAE,CAAC;IACb,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,MAAM,EAAE,CAAC,IAAI,CAAC,UAAU,EAAE,GAAG,CAAC,CAAC;QAC9C,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;YACjC,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,GAAG,iBAAiB,CAAC,CAAC;YACzD,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,CAAC;YAC/C,MAAM,MAAM,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC;YACnD,GAAG,GAAG,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;QAChC,CAAC;gBAAS,CAAC;YACT,MAAM,MAAM,CAAC,KAAK,EAAE,CAAC;QACvB,CAAC;IACH,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,EAAE,CAAC;IACZ,CAAC;IAED,MAAM,QAAQ,GAAa,EAAE,CAAC;IAC9B,KAAK,MAAM,IAAI,IAAI,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC;QAC7C,IAAI,QAAQ,CAAC,MAAM,IAAI,kBAAkB;YAAE,MAAM;QACjD,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,CAAC;QAC5B,IAAI,CAAC,OAAO;YAAE,SAAS;QACvB,IAAI,KAAK,GAA8D,IAAI,CAAC;QAC5E,IAAI,CAAC;YACH,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAuD,CAAC;QACpF,CAAC;QAAC,MAAM,CAAC;YACP,SAAS;QACX,CAAC;QACD,MAAM,IAAI,GAAG,KAAK,EAAE,IAAI,IAAI,EAAE,CAAC;QAC/B,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC;YAAE,SAAS;QACxC,MAAM,IAAI,GAAG,KAAK,EAAE,IAAI,IAAI,KAAK,EAAE,OAAO,IAAI,EAAE,CAAC;QACjD,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE;YAAE,SAAS;QAC3B,QAAQ,CAAC,IAAI,CAAC,IAAI,IAAI,KAAK,aAAa,CAAC,IAAI,EAAE,mBAAmB,CAAC,EAAE,CAAC,CAAC;IACzE,CAAC;IACD,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED,MAAM,UAAU,cAAc,CAAC,OAAe,EAAE,QAAyB;IACvE,MAAM,CAAC,GAAG,QAAQ,CAAC,MAAM,CAAC;IAC1B,MAAM,IAAI,GAAG,CAAC,KAAa,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,CAAC;IAClE,MAAM,MAAM,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,KAAK,QAAQ,CAAC,CAAC,MAAM,CAAC;IACpE,MAAM,SAAS,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,KAAK,WAAW,CAAC,CAAC,MAAM,CAAC;IAC1E,MAAM,SAAS,GAAG,QAAQ,CAAC,MAAM,CAC/B,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,KAAK,gBAAgB,IAAI,CAAC,CAAC,WAAW,KAAK,UAAU,CACrE,CAAC,MAAM,CAAC;IACT,MAAM,cAAc,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,iBAAiB,KAAK,KAAK,CAAC,CAAC,MAAM,CAAC;IACpF,MAAM,OAAO,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,UAAU,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC;IAChE,MAAM,SAAS,GAAG,QAAQ;SACvB,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,UAAU,CAAC;SACxB,MAAM,CAAC,CAAC,CAAC,EAAe,EAAE,CAAC,OAAO,CAAC,KAAK,QAAQ,CAAC;SACjD,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IACzB,OAAO;QACL,EAAE,EAAE,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE;QAC5B,OAAO;QACP,QAAQ,EAAE,CAAC;QACX,UAAU,EAAE,IAAI,CAAC,MAAM,CAAC;QACxB,aAAa,EAAE,IAAI,CAAC,SAAS,CAAC;QAC9B,SAAS,EAAE,IAAI,CAAC,OAAO,CAAC;QACxB,iBAAiB,EAAE,IAAI,CAAC,SAAS,CAAC;QAClC,gBAAgB,EAAE,IAAI,CAAC,cAAc,CAAC;QACtC,gBAAgB,EAAE,SAAS,CAAC,MAAM,CAAC,CAAC,CAAC,SAAS,CAAC,IAAI,CAAC,KAAK,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QACpF,aAAa,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC,GAAG,GAAG,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC;QAC/D,cAAc,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC,GAAG,GAAG,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC;KAClE,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,KAAa,EAAE,QAAQ,GAAG,GAAG;IAClD,MAAM,QAAQ,GAAG,wBAAwB,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;IAC7E,OAAO,QAAQ,CAAC,MAAM,GAAG,QAAQ,CAAC,CAAC,CAAC,GAAG,QAAQ,CAAC,KAAK,CAAC,CAAC,EAAE,QAAQ,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,QAAQ,CAAC;AACvF,CAAC;AAED,SAAS,MAAM,CAAC,KAAa;IAC3B,OAAO,IAAI,CAAC,KAAK,CAAC,KAAK,GAAG,KAAK,CAAC,GAAG,KAAK,CAAC;AAC3C,CAAC;AAED,+EAA+E;AAC/E,SAAS,cAAc,CAAC,KAAa,EAAE,OAAe;IACpD,MAAM,KAAK,GAAG,IAAI,MAAM,CACtB,IAAI,OAAO,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,YAAY,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CACvD,CAAC;IACF,OAAO,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;AAC3B,CAAC;AAED,SAAS,YAAY,CAAC,KAAa;IACjC,OAAO,KAAK,CAAC,OAAO,CAAC,qBAAqB,EAAE,MAAM,CAAC,CAAC;AACtD,CAAC"}
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
import type { Proposal } from "./propose.js";
|
|
2
|
+
export type GateResult = {
|
|
3
|
+
passed: boolean;
|
|
4
|
+
steps: Array<{
|
|
5
|
+
name: string;
|
|
6
|
+
code: number;
|
|
7
|
+
timedOut: boolean;
|
|
8
|
+
failureTail?: string;
|
|
9
|
+
}>;
|
|
10
|
+
};
|
|
11
|
+
/**
|
|
12
|
+
* The repo's own deterministic gates, run inside the proposal worktree before
|
|
13
|
+
* any judge tokens are spent. Order matches AGENTS.md section 11: typecheck,
|
|
14
|
+
* native tests when native/ changed, then the node test suite (which builds).
|
|
15
|
+
*/
|
|
16
|
+
export declare function runGates(proposal: Proposal): Promise<GateResult>;
|
|
17
|
+
export declare function gateSummary(gate: GateResult): string;
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
import path from "node:path";
|
|
2
|
+
import { execStep, logsDir, tail } from "./state.js";
|
|
3
|
+
const MINUTE = 60 * 1000;
|
|
4
|
+
/**
|
|
5
|
+
* The repo's own deterministic gates, run inside the proposal worktree before
|
|
6
|
+
* any judge tokens are spent. Order matches AGENTS.md section 11: typecheck,
|
|
7
|
+
* native tests when native/ changed, then the node test suite (which builds).
|
|
8
|
+
*/
|
|
9
|
+
export async function runGates(proposal) {
|
|
10
|
+
const logFile = path.join(logsDir(), `${proposal.finding.id}-gates.log`);
|
|
11
|
+
const nativeChanged = proposal.changedFiles.some((file) => file.startsWith("native/"));
|
|
12
|
+
const steps = [
|
|
13
|
+
{
|
|
14
|
+
name: "npm ci",
|
|
15
|
+
command: "npm",
|
|
16
|
+
args: ["ci", "--no-audit", "--no-fund"],
|
|
17
|
+
timeoutMs: 15 * MINUTE,
|
|
18
|
+
},
|
|
19
|
+
{ name: "npm run check", command: "npm", args: ["run", "check"], timeoutMs: 10 * MINUTE },
|
|
20
|
+
...(nativeChanged
|
|
21
|
+
? [
|
|
22
|
+
{
|
|
23
|
+
name: "cargo test",
|
|
24
|
+
command: "cargo",
|
|
25
|
+
args: ["test", "--manifest-path", "native/Cargo.toml"],
|
|
26
|
+
timeoutMs: 40 * MINUTE,
|
|
27
|
+
},
|
|
28
|
+
]
|
|
29
|
+
: []),
|
|
30
|
+
{ name: "npm test", command: "npm", args: ["test"], timeoutMs: 40 * MINUTE },
|
|
31
|
+
];
|
|
32
|
+
const result = { passed: true, steps: [] };
|
|
33
|
+
for (const step of steps) {
|
|
34
|
+
const outcome = await execStep({
|
|
35
|
+
command: step.command,
|
|
36
|
+
args: step.args,
|
|
37
|
+
cwd: proposal.worktree,
|
|
38
|
+
timeoutMs: step.timeoutMs,
|
|
39
|
+
logFile,
|
|
40
|
+
});
|
|
41
|
+
const entry = {
|
|
42
|
+
name: step.name,
|
|
43
|
+
code: outcome.code,
|
|
44
|
+
timedOut: outcome.timedOut,
|
|
45
|
+
...(outcome.code !== 0 ? { failureTail: tail(outcome.output, 4000) } : {}),
|
|
46
|
+
};
|
|
47
|
+
result.steps.push(entry);
|
|
48
|
+
if (outcome.code !== 0) {
|
|
49
|
+
result.passed = false;
|
|
50
|
+
break;
|
|
51
|
+
}
|
|
52
|
+
}
|
|
53
|
+
return result;
|
|
54
|
+
}
|
|
55
|
+
export function gateSummary(gate) {
|
|
56
|
+
return gate.steps
|
|
57
|
+
.map((step) => `${step.name}: ${step.code === 0 ? "ok" : step.timedOut ? "TIMEOUT" : `exit ${step.code}`}`)
|
|
58
|
+
.join(", ");
|
|
59
|
+
}
|
|
60
|
+
//# sourceMappingURL=gate.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"gate.js","sourceRoot":"","sources":["../../src/improve/gate.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,YAAY,CAAC;AAQrD,MAAM,MAAM,GAAG,EAAE,GAAG,IAAI,CAAC;AAEzB;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,QAAkB;IAC/C,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,EAAE,GAAG,QAAQ,CAAC,OAAO,CAAC,EAAE,YAAY,CAAC,CAAC;IACzE,MAAM,aAAa,GAAG,QAAQ,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC,IAAY,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC,CAAC;IAC/F,MAAM,KAAK,GAAgF;QACzF;YACE,IAAI,EAAE,QAAQ;YACd,OAAO,EAAE,KAAK;YACd,IAAI,EAAE,CAAC,IAAI,EAAE,YAAY,EAAE,WAAW,CAAC;YACvC,SAAS,EAAE,EAAE,GAAG,MAAM;SACvB;QACD,EAAE,IAAI,EAAE,eAAe,EAAE,OAAO,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,KAAK,EAAE,OAAO,CAAC,EAAE,SAAS,EAAE,EAAE,GAAG,MAAM,EAAE;QACzF,GAAG,CAAC,aAAa;YACf,CAAC,CAAC;gBACE;oBACE,IAAI,EAAE,YAAY;oBAClB,OAAO,EAAE,OAAO;oBAChB,IAAI,EAAE,CAAC,MAAM,EAAE,iBAAiB,EAAE,mBAAmB,CAAC;oBACtD,SAAS,EAAE,EAAE,GAAG,MAAM;iBACvB;aACF;YACH,CAAC,CAAC,EAAE,CAAC;QACP,EAAE,IAAI,EAAE,UAAU,EAAE,OAAO,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,MAAM,CAAC,EAAE,SAAS,EAAE,EAAE,GAAG,MAAM,EAAE;KAC7E,CAAC;IAEF,MAAM,MAAM,GAAe,EAAE,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC;IACvD,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,MAAM,OAAO,GAAG,MAAM,QAAQ,CAAC;YAC7B,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,IAAI,EAAE,IAAI,CAAC,IAAI;YACf,GAAG,EAAE,QAAQ,CAAC,QAAQ;YACtB,SAAS,EAAE,IAAI,CAAC,SAAS;YACzB,OAAO;SACR,CAAC,CAAC;QACH,MAAM,KAAK,GAAG;YACZ,IAAI,EAAE,IAAI,CAAC,IAAI;YACf,IAAI,EAAE,OAAO,CAAC,IAAI;YAClB,QAAQ,EAAE,OAAO,CAAC,QAAQ;YAC1B,GAAG,CAAC,OAAO,CAAC,IAAI,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SAC3E,CAAC;QACF,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACzB,IAAI,OAAO,CAAC,IAAI,KAAK,CAAC,EAAE,CAAC;YACvB,MAAM,CAAC,MAAM,GAAG,KAAK,CAAC;YACtB,MAAM;QACR,CAAC;IACH,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,MAAM,UAAU,WAAW,CAAC,IAAgB;IAC1C,OAAO,IAAI,CAAC,KAAK;SACd,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,GAAG,IAAI,CAAC,IAAI,KAAK,IAAI,CAAC,IAAI,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,IAAI,CAAC,IAAI,EAAE,EAAE,CAAC;SAC1G,IAAI,CAAC,IAAI,CAAC,CAAC;AAChB,CAAC"}
|