@erclx/aitk 3.53.0 → 3.55.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -75,3 +75,13 @@ Coverage is what it buys over the poll. A worker that finishes goes idle and a w
|
|
|
75
75
|
It classifies nothing and routes nothing. A line it prints says a pull request opened or a worker moved, and the routing block above is still what decides whether a review follows, so the two compose rather than replace each other.
|
|
76
76
|
|
|
77
77
|
Every session in the repository holding a branch other than the base one counts as a worker, whoever launched it. The prototype matched the `orchestrator-` prefix instead, which reads a dispatched worker and misses every hand-launched one. A failed read of either source reports itself on a `watch:` line and leaves the baseline untouched, since reading an empty result as current state would report every worker gone on the pass after.
|
|
78
|
+
|
|
79
|
+
## The stall alarm
|
|
80
|
+
|
|
81
|
+
`watch.sh` prints `WORKER-STOPPED <name> <branch> <dwell>s` once a `waiting` row crosses `STALL_THRESHOLD_S`, and `WORKER-UNMEASURABLE <name> <branch>` for one whose record carries neither timestamp the dwell falls back to. Both are prints, not alerts, so the operator learns of one only when a session already reading the loop's output relays it further. The toolkit ships no notification verb, since the surface a stall reaches the operator through is a session tool rather than a command a shell loop can call.
|
|
82
|
+
|
|
83
|
+
The two lines carry different confidence and the push has to say so rather than treat them as one signal. `WORKER-STOPPED` fires only once the dwell has already crossed the threshold, so it reports a wait already confirmed long. `WORKER-UNMEASURABLE` has no dwell to threshold on, so it fires on the first pass that meets a `waiting` row carrying neither stamp, whether that row has sat five seconds or fifty minutes.
|
|
84
|
+
|
|
85
|
+
On meeting either line, push a notification to the operator through whatever notification surface the client offers, `PushNotification` in this repository's client and one example among the surfaces a different client exposes. Name the worker, the branch, which of the two lines fired, and the dwell where `WORKER-STOPPED` carries one, and send it once per stall the same way `watch.sh` prints it once, rather than repeating it on every interval the row stays stopped.
|
|
86
|
+
|
|
87
|
+
The bound stays open. The alarm reaches the operator only through the controller's own read, so a controller mid-turn does not see the line for as long as the turn runs, and a controller that is itself stopped never does. Neither case closes here, since the watch loop and the notification surface both live inside the same session that has to be free to act on either.
|
package/docs/agents/sandbox.md
CHANGED
|
@@ -25,16 +25,20 @@ Scenario categories: `infra:*` (domain flows), `git:*`, `scaffold:*`. `create` s
|
|
|
25
25
|
aitk sandbox check claude:docs drift --json
|
|
26
26
|
```
|
|
27
27
|
|
|
28
|
-
| Flag | Effect
|
|
29
|
-
| ------------------- |
|
|
30
|
-
| `--envelope <file>` | Read `is_error`, `num_turns`, denials, and the reply text
|
|
31
|
-
| `--writes <file>` | Newline-delimited paths the session wrote, for write scope
|
|
32
|
-
| `--
|
|
33
|
-
| `--
|
|
28
|
+
| Flag | Effect |
|
|
29
|
+
| ------------------- | --------------------------------------------------------------------------- |
|
|
30
|
+
| `--envelope <file>` | Read `is_error`, `num_turns`, denials, and the reply text |
|
|
31
|
+
| `--writes <file>` | Newline-delimited paths the session wrote, for write scope |
|
|
32
|
+
| `--escapes <file>` | Newline-delimited paths written to a watched toolkit root, for escape scope |
|
|
33
|
+
| `--escapes-watched` | At least one watched root held a target this run |
|
|
34
|
+
| `--json` | Emit the verdict record on stdout |
|
|
35
|
+
| `--strict` | Exit 1 on `unchecked` instead of 0 |
|
|
34
36
|
|
|
35
37
|
The verdict `state` is `pass`, `fail`, or `unchecked`. An arm with no `expect.toml` is `unchecked` and exits 0, so the harness stays usable while expectations roll out. A declaration that exists but asserts nothing is a failure, since an expectation file that asserts nothing passes every run.
|
|
36
38
|
|
|
37
|
-
Omitting `--writes
|
|
39
|
+
Omitting `--writes`, `--escapes`, or `--envelope` does not silently drop the assertion kinds that need them. Write scope, escape scope, the turn ceiling, and the reply assertion report as unchecked and appear in the count, so the standalone command cannot claim more coverage than it had. Supplying `--escapes` without `--escapes-watched` reports the same way: a zero-escape result with no root confirmed watched is unmeasured rather than a pass. A verdict never reports `pass` with zero assertions.
|
|
40
|
+
|
|
41
|
+
An arm declaring `escape_scope` asserts a bound on `run.sh`'s own escape watch rather than on the sandbox tree. `write_scope` skips when the run wrote nothing, since a required output missing is itself a finding, but `escape_scope` passes on zero escapes outright, since a clean run producing none is the expected outcome for a destination nothing requires a skill to touch, provided `--escapes-watched` confirms a root held something to watch. Declaring `escape_scope = []` asserts that a correct run reaches none of the watched destinations at all. `.claude/context/sandbox/overview.md` names what the watch reaches and what it cannot.
|
|
38
42
|
|
|
39
43
|
An envelope that parses but carries no `result` field skips the reply assertion the same way an absent file does. An envelope carrying an empty `result` fails it, since a run that returned no text is a finding rather than a gap in the input.
|
|
40
44
|
|
package/package.json
CHANGED
package/src/commands/sandbox.ts
CHANGED
|
@@ -76,6 +76,8 @@ const CLEAN_ENVELOPE: RunEnvelope = { isError: false, turns: 0, denials: 0 }
|
|
|
76
76
|
interface CheckOptions {
|
|
77
77
|
readonly envelope?: string
|
|
78
78
|
readonly writes?: string
|
|
79
|
+
readonly escapes?: string
|
|
80
|
+
readonly escapesWatched?: boolean
|
|
79
81
|
readonly json?: boolean
|
|
80
82
|
readonly strict?: boolean
|
|
81
83
|
}
|
|
@@ -152,11 +154,13 @@ function readEnvelope(path: string | undefined): RunEnvelope | undefined {
|
|
|
152
154
|
}
|
|
153
155
|
|
|
154
156
|
/**
|
|
155
|
-
*
|
|
156
|
-
*
|
|
157
|
-
*
|
|
157
|
+
* Shared by `--writes` and `--escapes`, which are both a newline-delimited
|
|
158
|
+
* path list written by `run.sh`. Undefined when no file was given, which is
|
|
159
|
+
* not the same as a run that produced no paths. Write scope and escape scope
|
|
160
|
+
* both need that distinction: an empty list is a finding, an absent list is a
|
|
161
|
+
* gap in what the caller supplied.
|
|
158
162
|
*/
|
|
159
|
-
function
|
|
163
|
+
function readPathList(path: string | undefined): string[] | undefined {
|
|
160
164
|
if (path === undefined) return undefined
|
|
161
165
|
if (!existsSync(path)) return []
|
|
162
166
|
|
|
@@ -360,7 +364,12 @@ function runCheck(
|
|
|
360
364
|
expectFilePath(PROJECT_ROOT, parsed.category, parsed.command, arm ?? ''),
|
|
361
365
|
{
|
|
362
366
|
sandboxDir,
|
|
363
|
-
writes:
|
|
367
|
+
writes: readPathList(options.writes),
|
|
368
|
+
escapes: readPathList(options.escapes),
|
|
369
|
+
escapesWatched:
|
|
370
|
+
options.escapes === undefined
|
|
371
|
+
? undefined
|
|
372
|
+
: options.escapesWatched === true,
|
|
364
373
|
envelope: readEnvelope(options.envelope),
|
|
365
374
|
},
|
|
366
375
|
)
|
|
@@ -404,6 +413,14 @@ export function register(program: Command): void {
|
|
|
404
413
|
.helpOption('-h, --help', 'Show this help message')
|
|
405
414
|
.option('--envelope <file>', 'Run envelope JSON from claude -p')
|
|
406
415
|
.option('--writes <file>', 'Newline-delimited paths the session wrote')
|
|
416
|
+
.option(
|
|
417
|
+
'--escapes <file>',
|
|
418
|
+
'Newline-delimited paths written to a watched toolkit root, for escape scope',
|
|
419
|
+
)
|
|
420
|
+
.option(
|
|
421
|
+
'--escapes-watched',
|
|
422
|
+
'At least one watched root held a target this run, so a zero-escape file is a clean watch rather than one with nothing to watch',
|
|
423
|
+
)
|
|
407
424
|
.option('--json', 'Emit the verdict as JSON on stdout')
|
|
408
425
|
.option('--strict', 'Exit non-zero when the arm declares no expectation')
|
|
409
426
|
.addHelpText(
|
package/src/sandbox/expect.ts
CHANGED
|
@@ -19,6 +19,13 @@ export interface Expectation {
|
|
|
19
19
|
readonly absent: readonly string[]
|
|
20
20
|
readonly content: readonly ContentAssertion[]
|
|
21
21
|
readonly writeScope: readonly string[]
|
|
22
|
+
/**
|
|
23
|
+
* Undefined means the arm makes no claim about escapes, which is every arm
|
|
24
|
+
* today. Present, even as `[]`, means the arm asserts a bound: the empty
|
|
25
|
+
* form declares that a correct run produces none, so `stringArray`'s
|
|
26
|
+
* collapse of "absent" and "empty" into one `[]` would erase that claim.
|
|
27
|
+
*/
|
|
28
|
+
readonly escapeScope?: readonly string[]
|
|
22
29
|
readonly reply: readonly string[]
|
|
23
30
|
readonly manual: readonly string[]
|
|
24
31
|
readonly maxTurns?: number
|
|
@@ -54,15 +61,23 @@ export interface Verdict {
|
|
|
54
61
|
}
|
|
55
62
|
|
|
56
63
|
/**
|
|
57
|
-
* `writes` and `envelope` are absent when the caller supplied no
|
|
58
|
-
* which is not the same as a run that wrote nothing
|
|
59
|
-
* assertion kinds that depend on them report
|
|
60
|
-
* dropping out of the count, so the cheap
|
|
61
|
-
* coverage than it had.
|
|
64
|
+
* `writes`, `escapes`, and `envelope` are absent when the caller supplied no
|
|
65
|
+
* data for them, which is not the same as a run that wrote nothing, escaped
|
|
66
|
+
* nowhere, or reported nothing. The assertion kinds that depend on them report
|
|
67
|
+
* as skipped rather than silently dropping out of the count, so the cheap
|
|
68
|
+
* standalone path cannot claim more coverage than it had.
|
|
62
69
|
*/
|
|
63
70
|
export interface CheckInput {
|
|
64
71
|
readonly sandboxDir: string
|
|
65
72
|
readonly writes?: readonly string[]
|
|
73
|
+
readonly escapes?: readonly string[]
|
|
74
|
+
/**
|
|
75
|
+
* Whether any watched escape root held one of the four directories this run.
|
|
76
|
+
* Undefined when the caller supplied no escapes at all, which already skips.
|
|
77
|
+
* False is what separates a watch that ran and found nothing from one with
|
|
78
|
+
* nothing to watch, both of which produce the same empty `escapes` list.
|
|
79
|
+
*/
|
|
80
|
+
readonly escapesWatched?: boolean
|
|
66
81
|
readonly envelope?: RunEnvelope
|
|
67
82
|
}
|
|
68
83
|
|
|
@@ -187,6 +202,10 @@ export function parseExpectation(source: string): Expectation {
|
|
|
187
202
|
absent: stringArray(parsed.absent),
|
|
188
203
|
content: contentArray(parsed.content),
|
|
189
204
|
writeScope: stringArray(parsed.write_scope),
|
|
205
|
+
escapeScope:
|
|
206
|
+
parsed.escape_scope === undefined
|
|
207
|
+
? undefined
|
|
208
|
+
: stringArray(parsed.escape_scope),
|
|
190
209
|
reply: stringArray(parsed.reply),
|
|
191
210
|
manual: stringArray(parsed.manual),
|
|
192
211
|
maxTurns:
|
|
@@ -205,6 +224,7 @@ export function countMechanicalAssertions(expectation: Expectation): number {
|
|
|
205
224
|
expectation.absent.length +
|
|
206
225
|
expectation.content.length +
|
|
207
226
|
expectation.writeScope.length +
|
|
227
|
+
(expectation.escapeScope === undefined ? 0 : 1) +
|
|
208
228
|
expectation.reply.length
|
|
209
229
|
)
|
|
210
230
|
}
|
|
@@ -326,7 +346,7 @@ function checkWriteScope(
|
|
|
326
346
|
// none, and without this the declaration vanishes from the verdict entirely:
|
|
327
347
|
// no result, no skipped entry, and no contribution to the unchecked count that
|
|
328
348
|
// exists to surface exactly this. The `undefined` branch above cannot stand in,
|
|
329
|
-
// since `run.sh` always passes `--writes` and `
|
|
349
|
+
// since `run.sh` always passes `--writes` and `readPathList` returns `[]` for an
|
|
330
350
|
// empty file. An arm whose output escaped the snapshot reads as a clean run,
|
|
331
351
|
// which is the vacuous pass the harness exists to remove.
|
|
332
352
|
if (writes.length === 0) {
|
|
@@ -348,6 +368,65 @@ function checkWriteScope(
|
|
|
348
368
|
}
|
|
349
369
|
}
|
|
350
370
|
|
|
371
|
+
/**
|
|
372
|
+
* `run.sh` watches two toolkit roots for a write to shared session scratch
|
|
373
|
+
* during a run, and reports every one it finds as an unattributed escape with
|
|
374
|
+
* no arm able to fail on it. An arm whose skill legitimately reaches outside
|
|
375
|
+
* the sandbox tree declares the destinations here, and the harness asserts
|
|
376
|
+
* them instead of trusting the skill to bound itself.
|
|
377
|
+
*
|
|
378
|
+
* Declaring the scope inverts the empty case against `checkWriteScope`. A
|
|
379
|
+
* write-scope declaration exists to bound required output, so a run that wrote
|
|
380
|
+
* nothing skips rather than passing on a fabricated zero. An escape-scope
|
|
381
|
+
* declaration exists to bound a side effect nothing requires, so zero escapes
|
|
382
|
+
* is the outcome a correct run produces and reports as a pass outright,
|
|
383
|
+
* provided a watched root held something to watch. `run.sh`'s `snapshot_root`
|
|
384
|
+
* returns an empty manifest both when a watch ran clean and when none of the
|
|
385
|
+
* four watched directories existed under a root, and the two produce the same
|
|
386
|
+
* empty `escapes` list. `watched` is what tells them apart: a run that had
|
|
387
|
+
* nothing to watch reports unmeasured rather than passing on a diff it never
|
|
388
|
+
* had the target to take.
|
|
389
|
+
*/
|
|
390
|
+
function checkEscapeScope(
|
|
391
|
+
expectation: Expectation,
|
|
392
|
+
escapes: readonly string[] | undefined,
|
|
393
|
+
watched: boolean | undefined,
|
|
394
|
+
): KindOutcome {
|
|
395
|
+
if (expectation.escapeScope === undefined) return { results: [], skipped: [] }
|
|
396
|
+
|
|
397
|
+
if (escapes === undefined) {
|
|
398
|
+
return {
|
|
399
|
+
results: [],
|
|
400
|
+
skipped: ['escape scope: no escape data supplied, pass --escapes'],
|
|
401
|
+
}
|
|
402
|
+
}
|
|
403
|
+
|
|
404
|
+
if (escapes.length === 0) {
|
|
405
|
+
if (watched === false) {
|
|
406
|
+
return {
|
|
407
|
+
results: [],
|
|
408
|
+
skipped: ['escape scope: no watched root held a target, unmeasured'],
|
|
409
|
+
}
|
|
410
|
+
}
|
|
411
|
+
|
|
412
|
+
return {
|
|
413
|
+
results: [{ ok: true, message: 'no escape during this run' }],
|
|
414
|
+
skipped: [],
|
|
415
|
+
}
|
|
416
|
+
}
|
|
417
|
+
|
|
418
|
+
const globs = expectation.escapeScope.map((glob) => new Bun.Glob(glob))
|
|
419
|
+
|
|
420
|
+
return {
|
|
421
|
+
results: escapes.map((path) =>
|
|
422
|
+
globs.some((glob) => glob.match(path))
|
|
423
|
+
? { ok: true, message: `declared escape: ${path}` }
|
|
424
|
+
: { ok: false, message: `unbounded escape: ${path}` },
|
|
425
|
+
),
|
|
426
|
+
skipped: [],
|
|
427
|
+
}
|
|
428
|
+
}
|
|
429
|
+
|
|
351
430
|
/**
|
|
352
431
|
* Plain substrings, matched case-sensitively, against the text the run replied
|
|
353
432
|
* with. A substring rather than a regex because the pattern a reply assertion
|
|
@@ -431,6 +510,11 @@ export function checkExpectation(
|
|
|
431
510
|
input: CheckInput,
|
|
432
511
|
): Verdict {
|
|
433
512
|
const scope = checkWriteScope(expectation, input.writes)
|
|
513
|
+
const escapeScope = checkEscapeScope(
|
|
514
|
+
expectation,
|
|
515
|
+
input.escapes,
|
|
516
|
+
input.escapesWatched,
|
|
517
|
+
)
|
|
434
518
|
const reply = checkReply(expectation, input.envelope)
|
|
435
519
|
const envelope = checkEnvelope(expectation, input.envelope)
|
|
436
520
|
|
|
@@ -440,9 +524,15 @@ export function checkExpectation(
|
|
|
440
524
|
...checkContent(expectation, input.sandboxDir),
|
|
441
525
|
...reply.results,
|
|
442
526
|
...scope.results,
|
|
527
|
+
...escapeScope.results,
|
|
443
528
|
...envelope.results,
|
|
444
529
|
]
|
|
445
|
-
const skipped = [
|
|
530
|
+
const skipped = [
|
|
531
|
+
...scope.skipped,
|
|
532
|
+
...escapeScope.skipped,
|
|
533
|
+
...reply.skipped,
|
|
534
|
+
...envelope.skipped,
|
|
535
|
+
]
|
|
446
536
|
|
|
447
537
|
const failed = results.filter((result) => !result.ok).length
|
|
448
538
|
|