razzou-harness 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -3,6 +3,50 @@
3
3
  The notable changes in each release of razzou-harness. Before 1.0, a new minor version
4
4
  may change how a project is set up, and its notes say how.
5
5
 
6
+ ## 0.4.0 (2026-11-08)
7
+
8
+ **Upgrading a project from 0.3.x:** install the new version, then rebuild the agents'
9
+ image with `harness doctor --build-images`, or run `harness setup`, which does it: the
10
+ image's tag follows the harness's version. The doctor now removes the 0.3.0 image once
11
+ no container uses it. Nothing else changes in a project.
12
+
13
+ - A **Sandboxes** view in the dashboard lists the sandboxes on the OpenShell gateway:
14
+ this project's, with what each is for and whether anything still uses it, then the
15
+ others'. A left-over sandbox of this project, which a crash or a failed delete left,
16
+ can be deleted there; one in use, or another project's, can't. Selecting a sandbox
17
+ shows the network requests its policy refused, from its log: the program, the host,
18
+ why, and how often. The daemon labels the sandboxes it makes with the project's name
19
+ (`harness.project`) to tell them apart; a left-over sandbox from 0.3.0 or earlier has
20
+ no label, so it shows among the others.
21
+ - The view shows what each sandbox uses of the machine now, its CPUs and memory against
22
+ its caps, and the total for every OpenShell sandbox, read from `docker stats` every
23
+ five seconds while the view is open.
24
+ - **Caps on each sandbox**, set in the Sandboxes view: how many CPUs and how much memory
25
+ each new sandbox may use, so that Developers, check runs and reviewers in parallel
26
+ leave the machine usable. The daemon records them, and creates every sandbox with
27
+ them from then on; those already there keep theirs. There is no cap until you set one.
28
+ - An attempt keeps the network requests its sandbox's policy refused. OpenShell keeps
29
+ only a sandbox's last 2,000 log lines, so the daemon reads the log every minute while
30
+ the Developer works, and once more before the sandbox goes. The attempt's card shows
31
+ them, and the Manager gets them with a failed attempt. A refused host is the
32
+ Manager's to resolve: it guides the next Developer around the host, or asks the owner
33
+ to allow it, since only the owner widens the network policy.
34
+
35
+ Disk space:
36
+ - The git bundles the daemon hands to sandboxes, each a copy of the plan branch's whole
37
+ history, are deleted once uploaded, and the ones it takes back once fetched. One was
38
+ kept for every attempt and every review, and never deleted, so `.harness/` grew with
39
+ each. The daemon deletes the ones earlier versions left in `.harness/modules/` when it
40
+ starts.
41
+ - The daemon deletes this project's left-over sandboxes when it starts and every hour:
42
+ those the Sandboxes view offers to delete, so a delete that failed is tried again. It
43
+ spares an Architect's, which holds the owner's conversation with it.
44
+ - `harness doctor --build-images` removes the agents' images the new one replaces, about
45
+ 2 GB each, once no container uses them: the other versions' tags, and the untagged
46
+ images that building a tag again leaves. It also shows the size of Docker's build
47
+ cache, with `docker builder prune` to free it; that cache is shared with every other
48
+ build on the machine, so the doctor leaves it alone.
49
+
6
50
  ## 0.3.0 (2026-11-07)
7
51
 
8
52
  **Upgrading a project from 0.2.x:** install the new version, then run `harness setup`
package/README.md CHANGED
@@ -261,7 +261,18 @@ behind a token. Its views:
261
261
  - **Project:** modules, proposals, plans and features, with attempts, findings, CI,
262
262
  deploys and `progress.txt`.
263
263
  - **Agents:** the Architect's terminal, the Manager's and the Developers' live
264
- transcripts, a message box for the Manager, and a shell in any agent's sandbox.
264
+ transcripts, a message box for the Manager, and a shell in any agent's sandbox. An
265
+ attempt's card keeps the network requests its sandbox's policy refused.
266
+ - **Sandboxes:** what runs on the OpenShell gateway. This project's sandboxes, with what
267
+ each is for, whether anything still uses it, and the CPUs and memory it uses now,
268
+ read from `docker stats` every five seconds. A left-over one, which a crash or a
269
+ failed delete left, has a button to delete it; the daemon deletes them itself when it
270
+ starts and every hour, except an Architect's. Select one to see the network requests
271
+ its policy refused, from its log: the program, the host, why, and how often. Then the
272
+ other projects' sandboxes, which the dashboard only lists, and the gateway's status.
273
+ Above them, the caps on each new sandbox: how many CPUs and how much memory it may
274
+ use, so that agents and checks in parallel leave the machine usable. There is no cap
275
+ until you set one.
265
276
  - **Events:** the event log, live, with a filter.
266
277
 
267
278
  The header pauses and resumes the pipeline: a pause lets what runs finish, and starts
@@ -421,7 +432,7 @@ through an `Arch:` pull request, or a commit of your own on `main`.
421
432
 
422
433
  ```json
423
434
  {
424
- "harnessVersion": "0.3.0",
435
+ "harnessVersion": "0.4.0",
425
436
  "project": { "name": "shop", "repo": "<org>/shop", "bot": "<org>-harness[bot]" },
426
437
  "roles": {
427
438
  "architect": { "model": "claude-opus-5-5", "effort": "max" },
@@ -56,7 +56,7 @@ export class Architects {
56
56
  throw new Error(`could not take the design: ${(result.stderr || result.stdout).trim()}`);
57
57
  const local = join(this.localDir(module.module), 'design.bundle');
58
58
  await o.driver.download(module.architect.sandbox, `${p.out}/design.bundle`, local);
59
- return { head: await o.repo.fetchBundle(local, `refs/architect/${module.module}`) };
59
+ return { head: await o.fetchBundle(local, `refs/architect/${module.module}`) };
60
60
  }
61
61
  async syncModule(module) {
62
62
  const o = this.o;
@@ -93,7 +93,7 @@ export class Architects {
93
93
  const branch = `arch/${module.module}`;
94
94
  const reviewing = module.status === 'in_review' && (await o.repo.revParse(`refs/heads/${branch}`)) !== undefined;
95
95
  await o.repo.bundle(reviewing ? ['refs/heads/main', `refs/heads/${branch}`] : 'refs/heads/main', join(dir, 'main.bundle'));
96
- await o.driver.upload(sandbox, join(dir, 'main.bundle'), p.harness);
96
+ await o.uploadBundle(sandbox, join(dir, 'main.bundle'), p.harness);
97
97
  const clone = await o.exec(sandbox, [
98
98
  `git clone --quiet --branch main ${sh(`${p.harness}/main.bundle`)} ${sh(p.work)}`,
99
99
  `cd ${sh(p.work)}`,
@@ -199,7 +199,7 @@ export class Architects {
199
199
  const p = o.paths(module.architect.sandbox);
200
200
  const local = join(this.localDir(module.module), 'arch.bundle');
201
201
  await o.repo.bundle(`refs/heads/arch/${module.module}`, local);
202
- await o.driver.upload(module.architect.sandbox, local, p.harness);
202
+ await o.uploadBundle(module.architect.sandbox, local, p.harness);
203
203
  const result = await o.exec(module.architect.sandbox, [
204
204
  `cd ${sh(p.work)}`,
205
205
  `git fetch --quiet --force ${sh(`${p.harness}/arch.bundle`)} ${sh(`refs/heads/arch/${module.module}:refs/remotes/harness/arch`)}`,
@@ -252,7 +252,7 @@ export class Architects {
252
252
  await o.forge.fetchMain(o.repo);
253
253
  const local = join(this.localDir(module.module), 'main.bundle');
254
254
  await o.repo.bundle('refs/heads/main', local);
255
- await o.driver.upload(module.architect.sandbox, local, p.harness);
255
+ await o.uploadBundle(module.architect.sandbox, local, p.harness);
256
256
  await o.exec(module.architect.sandbox, [
257
257
  `cd ${sh(p.work)}`,
258
258
  `git fetch --quiet --force ${sh(`${p.harness}/main.bundle`)} refs/heads/main:refs/heads/main`,
@@ -116,6 +116,7 @@ function reviewItem(o, feature, attempt) {
116
116
  `The Developer reports done: ${attempt.summary ?? '(no summary)'}`,
117
117
  ...(attempt.notes ? ['', `Its notes: ${attempt.notes}`] : []),
118
118
  ...questionsAsked(attempt),
119
+ ...refusedLines(attempt, false),
119
120
  '',
120
121
  'Every check passed in a fresh sandbox:',
121
122
  ...checkLines(attempt.checks ?? []),
@@ -155,6 +156,7 @@ function failureItem(o, feature, attempt) {
155
156
  ? ['', `Its finding (${finding.category}): ${finding.summary}`, ...(finding.tried?.length ? [`Tried: ${finding.tried.join('; ')}`] : []), ...(finding.needs ? [`Needs: ${finding.needs}`] : [])]
156
157
  : []),
157
158
  ...questionsAsked(attempt),
159
+ ...refusedLines(attempt, true),
158
160
  ...(failing.length ? ['', 'Failing checks:', ...checkLines(failing, true)] : []),
159
161
  '',
160
162
  head
@@ -245,6 +247,26 @@ function questionsAsked(attempt) {
245
247
  return [];
246
248
  return ['', 'Its questions:', ...asked.map((q) => `- ${clip(q.text, 500)} → ${q.answer === undefined ? '(unanswered)' : clip(q.answer, 500)}`)];
247
249
  }
250
+ /**
251
+ * The requests the attempt's sandbox policy refused. A host the feature needs and the
252
+ * policy doesn't allow explains many failures; the Manager resolves it, around the host
253
+ * or with the owner, who alone widens the policy.
254
+ */
255
+ function refusedLines(attempt, failed) {
256
+ const refused = attempt.refused ?? [];
257
+ if (refused.length === 0)
258
+ return [];
259
+ const shown = refused.slice(0, 12);
260
+ return [
261
+ '',
262
+ "Its sandbox's network policy refused:",
263
+ ...shown.map((d) => `- ${d.target} ${d.program ? `from ${d.program}` : '(a DNS query)'}, ${d.count} time${d.count === 1 ? '' : 's'}: ${d.reason.replace(/_/g, ' ')}`),
264
+ ...(refused.length > shown.length ? [`- and ${refused.length - shown.length} more`] : []),
265
+ ...(failed
266
+ ? ["A sandbox reaches only the hosts of network.bundles and network.extraHosts in harness/config.json, and the next attempt meets the same policy. If the feature can do without a host, give the way around it in your guidance. If it can't, ask_owner to allow the host, with blocking: true when no attempt can succeed without it: only the owner widens the policy."]
267
+ : []),
268
+ ];
269
+ }
248
270
  function checkLines(results, withOutput = false) {
249
271
  return results.flatMap((r) => {
250
272
  const verdict = r.exit === 0 ? 'passed' : r.timedOut ? 'timed out' : `exited ${r.exit}`;
@@ -132,7 +132,8 @@ export class Managers {
132
132
  await o.repo.bundle(`refs/heads/${branch}`, join(dir, 'clone.bundle'));
133
133
  copyFileSync(join(o.options.packageDir, 'roles', 'manager.md'), join(dir, 'manager.md'));
134
134
  writeFileSync(join(dir, 'settings.json'), managerSettings());
135
- for (const file of ['clone.bundle', 'manager.md', 'settings.json'])
135
+ await o.uploadBundle(manager.sandbox, join(dir, 'clone.bundle'), p.harness);
136
+ for (const file of ['manager.md', 'settings.json'])
136
137
  await o.driver.upload(manager.sandbox, join(dir, file), p.harness);
137
138
  await o.uploadAgentFiles(manager.sandbox, 'manager', dir);
138
139
  const clone = await o.exec(manager.sandbox, `git clone --quiet --branch ${sh(branch)} ${sh(`${p.harness}/clone.bundle`)} ${sh(p.work)}`);
@@ -254,7 +255,7 @@ export class Managers {
254
255
  const local = join(this.localDir(manager), 'review.bundle');
255
256
  await o.repo.bundle([`refs/heads/${branch}`, ...(head && attemptRef ? [attemptRef] : [])], local);
256
257
  const p = this.paths(manager);
257
- await o.driver.upload(manager.sandbox, local, p.harness);
258
+ await o.uploadBundle(manager.sandbox, local, p.harness);
258
259
  const result = await o.exec(manager.sandbox, [
259
260
  `cd ${sh(p.work)}`,
260
261
  `git fetch --quiet --force ${sh(`${p.harness}/review.bundle`)} '+refs/heads/*:refs/remotes/harness/*' '+refs/attempts/*:refs/remotes/harness/attempts/*'`,
@@ -1,6 +1,6 @@
1
1
  import { randomUUID } from 'node:crypto';
2
2
  import { EventEmitter } from 'node:events';
3
- import { copyFileSync, existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
3
+ import { copyFileSync, existsSync, mkdirSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs';
4
4
  import { join } from 'node:path';
5
5
  import { claudeCode, mcpConfig, readResult, reportJsonSchema } from '../agents/claude-code.js';
6
6
  import { loadClaudeToken, readUsage } from '../anthropic/usage.js';
@@ -13,23 +13,28 @@ import { GitHubClient, loadAppCredentials } from '../github/app.js';
13
13
  import { GitHubForge, LocalForge } from '../github/forge.js';
14
14
  import { rulesProblems } from '../github/setup.js';
15
15
  import { toolsManifest } from '../roles.js';
16
+ import { PROJECT_LABEL } from '../sandbox/driver.js';
16
17
  import { BUNDLES } from '../sandbox/allowlist.js';
17
18
  import { buildPolicy } from '../sandbox/policy.js';
19
+ import { Denials, LOG_LINES } from '../sandbox/refusals.js';
18
20
  import { EventStore } from '../state/events.js';
19
21
  import { activeAttempts, apply, countedAttempts, featureKey, interruptions, isActive, managersOf, planKey, planMerged, project, writeSnapshot, } from '../state/state.js';
20
- import { clip, sh, sleep } from '../util.js';
22
+ import { clip, machineSize, sh, sleep } from '../util.js';
21
23
  import { Architects } from './architects.js';
22
24
  import { AgentChannel } from './channel.js';
23
25
  import { architectRef } from './design.js';
24
26
  import { parseCostEstimate } from '../review/cost-estimate.js';
25
27
  import { activeReview, reviewStep, stepReview } from './review.js';
26
28
  import { Managers } from './manager.js';
29
+ import { sweepLeftOvers } from './sandboxes.js';
27
30
  import { attemptRefName, preMerge, probeScript, refOf, sandboxName, waitForFile } from './refs.js';
28
31
  import { handleRequest } from './requests.js';
29
32
  import { buildTask } from './task.js';
30
33
  import { webRead } from './web-read.js';
31
34
  /** After this many errors in a row on the same attempt or plan (infrastructure, not the agent), the attempt is interrupted, or the pipeline pauses. */
32
35
  const STEP_ERROR_LIMIT = 5;
36
+ /** How often the daemon deletes this project's left-over sandboxes, besides when it starts. */
37
+ const SWEEP_EVERY_MS = 3_600_000;
33
38
  /**
34
39
  * Runs the loop in code (proposal v2 §7). Each pass looks at the state and takes the
35
40
  * next step for each attempt under way, or starts the next piece of work, then brings
@@ -48,6 +53,8 @@ export class Orchestrator {
48
53
  repo;
49
54
  options;
50
55
  driver;
56
+ /** The check sandboxes in use, with the ref each checks: the state doesn't name them. */
57
+ checking = new Map();
51
58
  agent;
52
59
  managers;
53
60
  architects;
@@ -66,6 +73,8 @@ export class Orchestrator {
66
73
  again = false;
67
74
  stopping = false;
68
75
  stepErrors = { key: '', count: 0 };
76
+ /** When the left-over sandboxes are next deleted: at the first pass, then every hour. */
77
+ nextSweep = 0;
69
78
  /** Errors in a row of each plan's step, and when it is tried again: one plan that keeps failing never stops the others. */
70
79
  planErrors = new Map();
71
80
  /** The heads of Arch and Fix PRs already given the reviewers' statuses as "not reviewed". */
@@ -74,6 +83,8 @@ export class Orchestrator {
74
83
  developers = new Map();
75
84
  /** The reviewers under way, by sandbox: plans in parallel can be reviewed at once. */
76
85
  reviewers = new Map();
86
+ /** What the policy refused the attempts under way, by sandbox, added up from their logs every minute. */
87
+ refusals = new Map();
77
88
  constructor(options) {
78
89
  this.options = options;
79
90
  this.driver = options.driver;
@@ -101,9 +112,33 @@ export class Orchestrator {
101
112
  throw new Error(`harness/config.json says the bot is ${config.project.bot}, but the App here is ${this.forge.bot}`);
102
113
  }
103
114
  await this.refreshDesign();
115
+ this.deleteOldBundles();
104
116
  this.record({ type: 'DAEMON_STARTED', payload: { version: this.options.version, pid: process.pid } });
105
117
  this.takePendingSetup();
106
118
  }
119
+ /**
120
+ * The git bundles earlier versions kept in `.harness/modules/`: a copy of the plan
121
+ * branch's history for each attempt and each review, though each was read once, by the
122
+ * sandbox. Nothing uses one before the first pass, so all of them go.
123
+ */
124
+ deleteOldBundles() {
125
+ const modules = join(this.options.harnessDir, 'modules');
126
+ if (!existsSync(modules))
127
+ return;
128
+ let count = 0;
129
+ let bytes = 0;
130
+ for (const file of readdirSync(modules, { recursive: true, encoding: 'utf8' }).filter((f) => f.endsWith('.bundle'))) {
131
+ const path = join(modules, file);
132
+ const stat = statSync(path, { throwIfNoEntry: false });
133
+ if (!stat?.isFile())
134
+ continue;
135
+ rmSync(path, { force: true });
136
+ bytes += stat.size;
137
+ count += 1;
138
+ }
139
+ if (count > 0)
140
+ this.log(`deleted ${count} git bundle${count === 1 ? '' : 's'} left in .harness/modules, ${(bytes / 1024 ** 2).toFixed(1)} MiB`);
141
+ }
107
142
  /**
108
143
  * What `harness setup`, `harness budget` or `harness mode` left for the daemon while it
109
144
  * was down (v4 §5), taken once: the owner's hours, which setup only offers, so they don't
@@ -204,6 +239,10 @@ export class Orchestrator {
204
239
  await this.architects.sync();
205
240
  if (!this.stopping)
206
241
  await this.verifyActions().catch((error) => this.log(`owner action: ${error.message}`));
242
+ if (!this.stopping && this.now() >= this.nextSweep) {
243
+ this.nextSweep = this.now() + SWEEP_EVERY_MS;
244
+ await sweepLeftOvers(this).catch((error) => this.log(`left-over sandboxes: ${error.message}`));
245
+ }
207
246
  } while (this.again && !this.stopping);
208
247
  }
209
248
  // ── Owner actions ────────────────────────────────────────────────────────────
@@ -261,6 +300,22 @@ export class Orchestrator {
261
300
  }
262
301
  return changed;
263
302
  }
303
+ /**
304
+ * The owner caps what each new sandbox may use of the machine, CPUs and memory; null
305
+ * lifts a cap. The sandboxes already there keep theirs. Returns why not, if not.
306
+ */
307
+ setSandboxLimits(limits) {
308
+ const { cpu, memoryGiB } = limits;
309
+ const machine = machineSize();
310
+ if (cpu !== null && !(Number.isFinite(cpu) && cpu >= 0.5 && cpu <= machine.cpus))
311
+ return `CPUs: from 0.5 to ${machine.cpus}, this machine's`;
312
+ if (memoryGiB !== null && !(Number.isFinite(memoryGiB) && memoryGiB >= 1 && memoryGiB <= machine.memoryGiB))
313
+ return `memory: from 1 to ${Math.floor(machine.memoryGiB)} GiB, this machine's`;
314
+ const current = this.state.sandboxLimits;
315
+ if ((current?.cpu ?? null) !== cpu || (current?.memoryGiB ?? null) !== memoryGiB)
316
+ this.record({ type: 'SANDBOX_LIMITS_SET', payload: { cpu, memoryGiB } });
317
+ return undefined;
318
+ }
264
319
  /** The owner grants one more attempt to an escalated feature (`module/plan/feature`). */
265
320
  retry(key, note) {
266
321
  const feature = this.state.features[key];
@@ -754,7 +809,8 @@ export class Orchestrator {
754
809
  }));
755
810
  copyFileSync(join(this.options.packageDir, 'roles', 'developer.md'), join(dir, 'developer.md'));
756
811
  writeFileSync(join(dir, 'report.schema.json'), reportJsonSchema());
757
- for (const file of ['plan.bundle', 'task.md', 'developer.md', 'report.schema.json']) {
812
+ await this.uploadBundle(sandbox, join(dir, 'plan.bundle'), p.harness);
813
+ for (const file of ['task.md', 'developer.md', 'report.schema.json']) {
758
814
  await this.driver.upload(sandbox, join(dir, file), p.harness);
759
815
  }
760
816
  await this.uploadAgentFiles(sandbox, 'developer', dir);
@@ -805,8 +861,10 @@ export class Orchestrator {
805
861
  this.record({ type: 'ATTEMPT_ENDED', ref, payload: { outcome: 'aborted', summary: attempt.abort.reason } });
806
862
  return 'progress';
807
863
  }
808
- if (this.now() < Date.parse(attempt.deadline))
864
+ if (this.now() < Date.parse(attempt.deadline)) {
865
+ await this.readRefusals(attempt.sandbox);
809
866
  return 'wait';
867
+ }
810
868
  await this.killAgent(attempt.sandbox);
811
869
  this.record({ type: 'ATTEMPT_ENDED', ref, payload: { outcome: 'timed_out' } });
812
870
  return 'progress';
@@ -836,6 +894,7 @@ export class Orchestrator {
836
894
  /** Step 7: take the Developer's commits out as a bundle, whatever the outcome, and refuse changes to protected files. */
837
895
  async collect(state, attempt) {
838
896
  this.stopFollowing(attempt.sandbox);
897
+ await this.keepRefusals(state, attempt);
839
898
  const ref = refOf(state, attempt);
840
899
  if (attempt.outcome === 'usage_limit') {
841
900
  // Not the Developer's doing, but the sandbox writes the stream it is read from: like any
@@ -1330,31 +1389,38 @@ export class Orchestrator {
1330
1389
  /** Runs checks on a commit of the daemon's repository, in a fresh sandbox with no agent and no credentials. */
1331
1390
  async runChecksIn(name, gitRef, dir, checks) {
1332
1391
  const p = this.paths(name);
1333
- await this.driver.remove(name);
1334
- await this.driver.create({
1335
- name,
1336
- image: this.design.config.sandbox.image,
1337
- ...this.access({ claude: false, aws: checks.some((c) => c.needs.length > 0) }),
1338
- command: ['sleep', 'infinity'],
1339
- });
1392
+ // In use from before its creation to after its removal, so the dashboard never offers to delete it.
1393
+ this.checking.set(name, gitRef.replace(/^refs\/(heads\/)?/, ''));
1340
1394
  try {
1341
- await this.repo.bundle(gitRef, join(dir, 'candidate.bundle'));
1342
- await this.driver.upload(name, join(dir, 'candidate.bundle'), p.harness);
1343
- const fetch = await this.exec(name, [
1344
- `git init --quiet ${sh(p.work)}`,
1345
- `cd ${sh(p.work)}`,
1346
- `git fetch --quiet ${sh(`${p.harness}/candidate.bundle`)} ${sh(`${gitRef}:refs/heads/candidate`)}`,
1347
- 'git checkout --quiet candidate',
1348
- ].join(' && '));
1349
- if (fetch.code !== 0)
1350
- throw new Error(`could not check out the candidate: ${fetch.stderr.trim()}`);
1351
- const init = await initEnvironment(this.driver, name, p.work);
1352
- if (init && init.exit !== 0)
1353
- return [init];
1354
- return await runChecks(this.driver, name, p.work, checks, this.checkOptions(name));
1395
+ await this.driver.remove(name);
1396
+ await this.driver.create({
1397
+ name,
1398
+ image: this.design.config.sandbox.image,
1399
+ ...this.access({ claude: false, aws: checks.some((c) => c.needs.length > 0) }),
1400
+ command: ['sleep', 'infinity'],
1401
+ });
1402
+ try {
1403
+ await this.repo.bundle(gitRef, join(dir, 'candidate.bundle'));
1404
+ await this.uploadBundle(name, join(dir, 'candidate.bundle'), p.harness);
1405
+ const fetch = await this.exec(name, [
1406
+ `git init --quiet ${sh(p.work)}`,
1407
+ `cd ${sh(p.work)}`,
1408
+ `git fetch --quiet ${sh(`${p.harness}/candidate.bundle`)} ${sh(`${gitRef}:refs/heads/candidate`)}`,
1409
+ 'git checkout --quiet candidate',
1410
+ ].join(' && '));
1411
+ if (fetch.code !== 0)
1412
+ throw new Error(`could not check out the candidate: ${fetch.stderr.trim()}`);
1413
+ const init = await initEnvironment(this.driver, name, p.work);
1414
+ if (init && init.exit !== 0)
1415
+ return [init];
1416
+ return await runChecks(this.driver, name, p.work, checks, this.checkOptions(name));
1417
+ }
1418
+ finally {
1419
+ await this.driver.remove(name);
1420
+ }
1355
1421
  }
1356
1422
  finally {
1357
- await this.driver.remove(name);
1423
+ this.checking.delete(name);
1358
1424
  }
1359
1425
  }
1360
1426
  // ── Outcomes and brakes (§7.4) ───────────────────────────────────────────────
@@ -1370,6 +1436,7 @@ export class Orchestrator {
1370
1436
  }
1371
1437
  /** Not the Developer's fault: doesn't count, except from the third interruption of the same feature. */
1372
1438
  async interrupt(state, attempt, reason) {
1439
+ await this.keepRefusals(state, attempt);
1373
1440
  const counts = interruptions(state, attempt.definitionHash) >= 2;
1374
1441
  this.record({ type: 'ATTEMPT_INTERRUPTED', ref: refOf(state, attempt), payload: { reason, counts } });
1375
1442
  // Followers start again when their stream ends: one left on a removed sandbox would go on trying.
@@ -1379,6 +1446,7 @@ export class Orchestrator {
1379
1446
  }
1380
1447
  /** Claude Code exited before writing anything: a flag, the token or the image is wrong, and retrying can't help. */
1381
1448
  async agentDidNotStart(state, attempt, stderr) {
1449
+ await this.keepRefusals(state, attempt);
1382
1450
  const reason = `Claude Code did not start: ${clip(stderr.trim().replace(/\s+/g, ' ') || 'it wrote nothing', 400)}`;
1383
1451
  this.record({ type: 'ATTEMPT_INTERRUPTED', ref: refOf(state, attempt), payload: { reason, counts: false } });
1384
1452
  this.pausePipeline(`environment: ${reason}`);
@@ -1387,6 +1455,8 @@ export class Orchestrator {
1387
1455
  return 'progress';
1388
1456
  }
1389
1457
  async baselineFailed(state, attempt, reason, results) {
1458
+ // init.sh runs in the attempt's sandbox: a dependency it couldn't fetch shows here.
1459
+ await this.keepRefusals(state, attempt);
1390
1460
  const ref = refOf(state, attempt);
1391
1461
  this.record({ type: 'BASELINE_FAILED', ref, payload: { reason, results } });
1392
1462
  this.record({ type: 'ATTEMPT_INTERRUPTED', ref, payload: { reason, counts: false } });
@@ -1395,6 +1465,38 @@ export class Orchestrator {
1395
1465
  await this.driver.remove(attempt.sandbox);
1396
1466
  return 'progress';
1397
1467
  }
1468
+ /**
1469
+ * Reads the sandbox's log for the requests its policy refused, at most once a minute
1470
+ * unless `now`: OpenShell keeps only a sandbox's last lines, which a long attempt
1471
+ * outgrows. Never throws; what was read before stays.
1472
+ */
1473
+ async readRefusals(sandbox, now = false) {
1474
+ const known = this.refusals.get(sandbox) ?? { denials: new Denials(), readAt: 0 };
1475
+ this.refusals.set(sandbox, known);
1476
+ if (!now && this.now() - known.readAt < 60_000)
1477
+ return known.denials;
1478
+ known.readAt = this.now();
1479
+ try {
1480
+ known.denials.add(await this.driver.logs(sandbox, LOG_LINES));
1481
+ }
1482
+ catch (error) {
1483
+ this.log(`the log of ${sandbox} could not be read: ${error.message}`);
1484
+ }
1485
+ return known.denials;
1486
+ }
1487
+ /**
1488
+ * Before the attempt's sandbox goes, what its policy refused is recorded with the
1489
+ * attempt: the Manager sees it with a failure, the owner on the attempt's card. Only
1490
+ * while the sandbox is there, before its checks start; once per attempt.
1491
+ */
1492
+ async keepRefusals(state, attempt) {
1493
+ if (attempt.refused || !['starting', 'running', 'collecting'].includes(attempt.status))
1494
+ return;
1495
+ const requests = (await this.readRefusals(attempt.sandbox, true)).list();
1496
+ this.refusals.delete(attempt.sandbox);
1497
+ if (requests.length > 0)
1498
+ this.record({ type: 'NETWORK_REFUSED', ref: refOf(state, attempt), payload: { requests } });
1499
+ }
1398
1500
  /** An error outside any attempt: logged, and the loop goes on. */
1399
1501
  async onStepError(error) {
1400
1502
  this.stepErrors = { key: 'idle', count: this.stepErrors.key === 'idle' ? this.stepErrors.count + 1 : 1 };
@@ -1547,6 +1649,28 @@ export class Orchestrator {
1547
1649
  mcpConfig: `${harness}/mcp.json`,
1548
1650
  };
1549
1651
  }
1652
+ /**
1653
+ * Uploads a git bundle the daemon wrote for a sandbox, then deletes it here: the
1654
+ * sandbox clones or fetches from its own copy, and the daemon's repository has the
1655
+ * history. Kept, they would grow without end, a copy per attempt and per review.
1656
+ */
1657
+ async uploadBundle(sandbox, file, remoteDir) {
1658
+ try {
1659
+ await this.driver.upload(sandbox, file, remoteDir);
1660
+ }
1661
+ finally {
1662
+ rmSync(file, { force: true });
1663
+ }
1664
+ }
1665
+ /** Fetches a bundle a sandbox made into `ref`, then deletes it: the ref holds its commits. */
1666
+ async fetchBundle(file, ref) {
1667
+ try {
1668
+ return await this.repo.fetchBundle(file, ref);
1669
+ }
1670
+ finally {
1671
+ rmSync(file, { force: true });
1672
+ }
1673
+ }
1550
1674
  /** The shim and the role's tools, so the agent can reach the daemon. */
1551
1675
  async uploadAgentFiles(sandbox, role, localDir) {
1552
1676
  const p = this.paths(sandbox);
@@ -1603,10 +1727,16 @@ export class Orchestrator {
1603
1727
  */
1604
1728
  access(options) {
1605
1729
  const aws = options.aws ? this.design.config.aws : undefined;
1730
+ const limits = this.state.sandboxLimits;
1606
1731
  return {
1607
1732
  providers: [...(options.claude ? [this.design.config.sandbox.claudeProvider] : []), ...(aws ? [aws.provider] : [])],
1608
1733
  policyFile: this.policyFile({ allBundles: options.allBundles ?? false, aws: aws !== undefined }),
1609
1734
  env: this.sandboxEnv(),
1735
+ // Every sandbox the daemon makes says whose it is: the dashboard tells them from other projects'.
1736
+ labels: { [PROJECT_LABEL]: this.design.config.project.name },
1737
+ // The owner's caps, so that agents and checks in parallel leave the machine usable.
1738
+ ...(limits?.cpu !== undefined ? { cpu: `${Math.round(limits.cpu * 1000)}m` } : {}),
1739
+ ...(limits?.memoryGiB !== undefined ? { memory: `${Math.round(limits.memoryGiB * 1024)}Mi` } : {}),
1610
1740
  };
1611
1741
  }
1612
1742
  /** What checks need to plan with Terraform in a sandbox. */
@@ -1660,7 +1790,7 @@ export class Orchestrator {
1660
1790
  throw new Error(`could not bundle the attempt's commits: ${(bundle.stderr || bundle.stdout).trim()}`);
1661
1791
  const local = join(this.attemptDir(ref), 'result.bundle');
1662
1792
  await this.driver.download(attempt.sandbox, `${p.out}/result.bundle`, local);
1663
- return this.repo.fetchBundle(local, attemptRefName(ref));
1793
+ return this.fetchBundle(local, attemptRefName(ref));
1664
1794
  }
1665
1795
  async killAgent(sandbox) {
1666
1796
  const p = this.paths(sandbox);
@@ -87,7 +87,7 @@ async function prepare(o, plan, review) {
87
87
  command: ['sleep', 'infinity'],
88
88
  });
89
89
  await o.repo.bundle(`refs/heads/${plan.branch}`, join(dir, 'plan.bundle'));
90
- await o.driver.upload(sandbox, join(dir, 'plan.bundle'), p.harness);
90
+ await o.uploadBundle(sandbox, join(dir, 'plan.bundle'), p.harness);
91
91
  const clone = await o.exec(sandbox, [
92
92
  `git clone --quiet --branch ${sh(plan.branch)} ${sh(`${p.harness}/plan.bundle`)} ${sh(p.work)}`,
93
93
  `cd ${sh(p.work)}`,
@@ -0,0 +1,115 @@
1
+ import { PROJECT_LABEL } from '../sandbox/driver.js';
2
+ import { LOG_LINES, parseDenials } from '../sandbox/refusals.js';
3
+ import { machineSize } from '../util.js';
4
+ const LIVE_ATTEMPT = ['starting', 'running', 'collecting', 'checking', 'reviewing'];
5
+ const ROLES = { arc: 'Architect', mgr: 'Manager', dev: 'Developer', rev: 'Reviewer', chk: 'Checks' };
6
+ /** Every sandbox the state names, and the check runs under way, with what each is for and whether it is still meant to be there. */
7
+ export function knownSandboxes(state, checking) {
8
+ const known = new Map();
9
+ for (const module of Object.values(state.modules)) {
10
+ // An Architect stays for as long as its module: the owner designs with it, and it answers the Managers.
11
+ known.set(module.architect.sandbox, { role: 'Architect', title: module.module, agent: `architect:${module.module}`, live: true });
12
+ }
13
+ for (const manager of Object.values(state.managers)) {
14
+ const key = manager.plan ? `${manager.module}/${manager.plan}` : manager.module;
15
+ known.set(manager.sandbox, { role: 'Manager', title: key, agent: `manager:${key}`, live: manager.status !== 'stopped' });
16
+ }
17
+ for (const feature of Object.values(state.features)) {
18
+ for (const attempt of feature.attempts) {
19
+ const key = `${feature.module}/${feature.plan}/${feature.feature}`;
20
+ known.set(attempt.sandbox, { role: 'Developer', title: `${key} #${attempt.n}`, agent: `developer:${key}/a${attempt.n}`, live: LIVE_ATTEMPT.includes(attempt.status) });
21
+ }
22
+ }
23
+ for (const plan of Object.values(state.plans)) {
24
+ for (const review of plan.reviews ?? []) {
25
+ const head = review.head.slice(0, 8);
26
+ known.set(review.sandbox, {
27
+ role: 'Reviewer', title: `${plan.module}/${plan.plan} · ${review.role} · ${head}`, agent: `${review.role}:${head}`, live: review.status === 'starting' || review.status === 'running',
28
+ });
29
+ }
30
+ }
31
+ for (const [name, ref] of checking)
32
+ known.set(name, { role: 'Checks', title: `checks on ${ref}`, live: true });
33
+ return known;
34
+ }
35
+ /** The gateway's sandboxes, each told apart as this project's in use, left over, or another's. Throws when OpenShell can't list them. */
36
+ export async function sandboxRows(o) {
37
+ const listed = await o.driver.list();
38
+ const known = knownSandboxes(o.state, o.checking);
39
+ const project = o.design.config.project.name;
40
+ const order = { 'in use': 0, 'left over': 1, other: 2 };
41
+ return listed
42
+ .map((sandbox) => row(sandbox, known.get(sandbox.name), project))
43
+ .sort((a, b) => order[a.use] - order[b.use] || (a.createdAt ?? '').localeCompare(b.createdAt ?? '', 'en') || a.name.localeCompare(b.name, 'en'));
44
+ }
45
+ export async function sandboxOverview(o) {
46
+ const gateway = await o.driver.gateway();
47
+ const settings = { limits: o.state.sandboxLimits ?? {}, machine: machineSize() };
48
+ try {
49
+ return { gateway, sandboxes: await sandboxRows(o), ...settings };
50
+ }
51
+ catch (error) {
52
+ return { gateway, sandboxes: [], problem: error.message, ...settings };
53
+ }
54
+ }
55
+ /** The requests a sandbox of this project was refused, from the last lines OpenShell keeps of its log; undefined for any other sandbox. */
56
+ export async function sandboxDenials(o, name) {
57
+ const found = (await sandboxRows(o)).find((s) => s.name === name && s.use !== 'other');
58
+ return found ? parseDenials(await o.driver.logs(name, LOG_LINES)) : undefined;
59
+ }
60
+ /**
61
+ * One reading of what the sandboxes use at a time, shared for a few seconds: each open
62
+ * dashboard asks every few seconds, and `docker stats` takes two. Never rejects.
63
+ */
64
+ export function sharedReading(read, ttlMs = 3000, now = Date.now) {
65
+ let last;
66
+ return () => {
67
+ if (last && now() - last.at < ttlMs)
68
+ return last.reading;
69
+ const reading = read().catch((error) => ({ at: new Date(now()).toISOString(), sandboxes: {}, total: { cpu: 0, memoryBytes: 0 }, problem: error.message }));
70
+ last = { at: now(), reading };
71
+ return reading;
72
+ };
73
+ }
74
+ /**
75
+ * Deletes this project's left-over sandboxes, those the Sandboxes view offers to delete:
76
+ * a crash or a failed delete leaves them, each holding its container's disk. The daemon
77
+ * runs it when it starts and every hour, so a delete that failed is tried again. It
78
+ * spares an Architect's, which holds the owner's conversation with it: only a lost
79
+ * `.harness/` leaves one, and the owner deletes it from the view. Every sandbox is in
80
+ * the state, or among the check runs, before it is created, so none being made is
81
+ * taken for a left-over one. Returns the names deleted.
82
+ */
83
+ export async function sweepLeftOvers(o) {
84
+ const leftOver = (await sandboxRows(o)).filter((s) => s.use === 'left over' && s.role !== 'Architect');
85
+ for (const sandbox of leftOver) {
86
+ await o.driver.remove(sandbox.name);
87
+ o.log(`deleting the left-over sandbox ${sandbox.name}${sandbox.role ? `, ${sandbox.role}${sandbox.title ? ` · ${sandbox.title}` : ''}` : ''}`);
88
+ }
89
+ return leftOver.map((s) => s.name);
90
+ }
91
+ /** Deletes a left-over sandbox of this project, judged afresh: never one in use, nor another project's. */
92
+ export async function removeLeftOver(o, name) {
93
+ const found = (await sandboxRows(o)).find((s) => s.name === name);
94
+ if (!found)
95
+ return { done: false, error: 'the gateway has no such sandbox' };
96
+ if (found.use === 'in use')
97
+ return { done: false, error: 'it is in use' };
98
+ if (found.use === 'other')
99
+ return { done: false, error: "it isn't this project's" };
100
+ await o.driver.remove(name);
101
+ o.log(`the owner deleted the left-over sandbox ${name}`);
102
+ return { done: true };
103
+ }
104
+ function row(sandbox, known, project) {
105
+ const base = { name: sandbox.name, phase: sandbox.phase, ...(sandbox.createdAt ? { createdAt: sandbox.createdAt } : {}) };
106
+ if (known)
107
+ return { ...base, role: known.role, title: known.title, ...(known.agent ? { agent: known.agent } : {}), use: known.live ? 'in use' : 'left over' };
108
+ const label = sandbox.labels[PROJECT_LABEL];
109
+ // This project's, but the state no longer names it: what a crash or a failed delete leaves.
110
+ if (label === project) {
111
+ const role = ROLES[sandbox.name.split('-')[0] ?? ''];
112
+ return { ...base, ...(role ? { role } : {}), use: 'left over' };
113
+ }
114
+ return { ...base, use: 'other', ...(label ? { project: label } : {}) };
115
+ }