humanish 0.132.0 → 0.133.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -0
- package/dist/actors/codex/restricted-participant.d.ts +2 -0
- package/dist/actors/codex/restricted-participant.js +6 -2
- package/dist/actors/computer-use/loop/actions.js +1 -0
- package/dist/actors/computer-use/loop/types.d.ts +6 -6
- package/dist/actors/computer-use/loop/watch.js +17 -15
- package/dist/actors/computer-use/openai-context.js +26 -14
- package/dist/actors/computer-use/openai-conversation.d.ts +17 -2
- package/dist/actors/computer-use/openai-conversation.js +47 -7
- package/dist/actors/computer-use/openai-wire.d.ts +21 -0
- package/dist/actors/computer-use/openai-wire.js +82 -9
- package/dist/actors/computer-use/wait.d.ts +8 -6
- package/dist/actors/computer-use/wait.js +10 -8
- package/dist/actors/contract.d.ts +4 -0
- package/dist/actors/local-agent/cli.js +5 -5
- package/dist/evidence/json-split.d.ts +21 -0
- package/dist/evidence/json-split.js +57 -0
- package/dist/evidence/redaction.d.ts +2 -0
- package/dist/evidence/redaction.js +8 -0
- package/dist/evidence/sensitive-json.d.ts +9 -0
- package/dist/evidence/sensitive-json.js +73 -0
- package/dist/routes/computer-use/local-vm.js +9 -3
- package/dist/routes/computer-use/participant-model.js +3 -0
- package/dist/run/sandbox-ids.d.ts +6 -1
- package/dist/run/sandbox-ids.js +65 -5
- package/dist/study/bundle-estimate.d.ts +2 -0
- package/dist/study/bundle-estimate.js +15 -1
- package/dist/study/types.d.ts +5 -4
- package/dist/verify/verify.js +4 -2
- package/docs/architecture/actor-contract.md +1 -0
- package/docs/architecture/browser-control.md +2 -2
- package/docs/contracts/schemas.md +22 -10
- package/package.json +1 -1
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
// longest (actor.maxWaitMs) and how long a wait that names no duration lasts while the participant
|
|
3
3
|
// only waits (actor.idleWaitMs). The loop asks planWait for each wait. A turn with another action
|
|
4
4
|
// sends its waits to the desktop in steps one desktop call carries; a turn that only waits watches
|
|
5
|
-
// the screen instead (loop/watch.ts), for its waits' lengths
|
|
6
|
-
// watchMaxMs, or longer as the watches of idle turns in a row grow.
|
|
5
|
+
// the screen instead (loop/watch.ts), for its waits' lengths and at least idleWaitMs, the named
|
|
6
|
+
// ones together up to watchMaxMs, or longer as the watches of idle turns in a row grow.
|
|
7
7
|
export const CUA_WAIT_LIMITS = Object.freeze({
|
|
8
8
|
/**
|
|
9
9
|
* The longest wait one desktop call is given. A browser-control request must be answered within
|
|
@@ -26,16 +26,18 @@ export const CUA_WAIT_LIMITS = Object.freeze({
|
|
|
26
26
|
defaultWatchMaxMs: 600_000,
|
|
27
27
|
/**
|
|
28
28
|
* How long a wait that names no duration lasts in a turn that only waits or takes screenshots,
|
|
29
|
-
* when a study does not set actor.idleWaitMs. OpenAI's computer
|
|
30
|
-
*
|
|
31
|
-
*
|
|
29
|
+
* when a study does not set actor.idleWaitMs. OpenAI's computer tool `wait` names no duration;
|
|
30
|
+
* its wait_minutes function names one in minutes. A participant told to wait about a minute
|
|
31
|
+
* sent ten computer tool waits in one turn, which lasts 50 s at this length; one wait a turn
|
|
32
|
+
* makes a one-minute wait about six turns. It is also the shortest a watch lasts and how often
|
|
33
|
+
* it looks.
|
|
32
34
|
*/
|
|
33
35
|
defaultIdleMs: 5_000,
|
|
34
36
|
/**
|
|
35
37
|
* How long a wait that names no duration lasts after the participant acts in the same turn, so
|
|
36
|
-
* the screen can settle before the next screenshot. Nearly every OpenAI computer
|
|
37
|
-
* a click or keypress this way, so a longer one would slow every run. A watch looks again
|
|
38
|
-
* soon after a large part of the screen moved.
|
|
38
|
+
* the screen can settle before the next screenshot. Nearly every OpenAI computer tool wait
|
|
39
|
+
* follows a click or keypress this way, so a longer one would slow every run. A watch looks again
|
|
40
|
+
* this soon after a large part of the screen moved.
|
|
39
41
|
*/
|
|
40
42
|
settleMs: 500,
|
|
41
43
|
/**
|
|
@@ -91,6 +91,10 @@ export interface ActorTraceItem {
|
|
|
91
91
|
* ran in place of a turn that only waited. Additive: absent on plain waits, on other items and
|
|
92
92
|
* producers, and in older bundles. */
|
|
93
93
|
watch?: ActorWatch;
|
|
94
|
+
/** On a computer-use wait's `ui_action`: the length in milliseconds the participant named for
|
|
95
|
+
* it, before any cap. Absent on a wait that named no length, such as the OpenAI computer
|
|
96
|
+
* tool's `wait`, on other items and producers, and in older bundles. */
|
|
97
|
+
namedMs?: number;
|
|
94
98
|
}
|
|
95
99
|
/**
|
|
96
100
|
* A watch over the screen while a computer-use participant only waited: the loop looked every
|
|
@@ -190,11 +190,11 @@ export function promptFor(request, screenshotPath) {
|
|
|
190
190
|
"giving up, and put your closing words in message. When done=true, set outcome: reached if " +
|
|
191
191
|
"the task is finished, blocked if something in the app stopped you, not_reached if you are " +
|
|
192
192
|
"stopping for another reason. Otherwise outcome is null.",
|
|
193
|
-
"A turn whose actions are only waits
|
|
194
|
-
"
|
|
195
|
-
"
|
|
196
|
-
"
|
|
197
|
-
"
|
|
193
|
+
"A turn whose actions are only waits and that asks for some wait time asks to wait its waits' " +
|
|
194
|
+
"ms together; an ask shorter than a wait whose ms is null counts as one, and named ms count " +
|
|
195
|
+
"up to the study's longest wait. The note from the harness says how long you waited and why " +
|
|
196
|
+
"it ended. A wait among other actions lasts its ms, and the note says when the study's limit " +
|
|
197
|
+
"for one wait shortened it.",
|
|
198
198
|
`Reply with ONLY a JSON object of this shape: ${shape}`,
|
|
199
199
|
hint,
|
|
200
200
|
].join("\n");
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
/** The JSON text of a value cut around the elements of one of its arrays. */
|
|
2
|
+
export interface ArraySplit {
|
|
3
|
+
/** The array's elements, read as JSON.stringify reads them. */
|
|
4
|
+
readonly elements: readonly Record<string, unknown>[];
|
|
5
|
+
/** The value's JSON text up to and including the array's `[`. */
|
|
6
|
+
before(): string;
|
|
7
|
+
/** The value's JSON text from the array's `]` on. */
|
|
8
|
+
after(): string;
|
|
9
|
+
}
|
|
10
|
+
/**
|
|
11
|
+
* The cut of JSON.stringify(value) around the elements of its array `value[key]`, when that text
|
|
12
|
+
* holds each element's text as JSON.stringify(element) writes it: `value` is a plain object that
|
|
13
|
+
* has `key` among its own enumerable keys, `value[key]` is a plain array with no holes, and each
|
|
14
|
+
* element is a plain object. A plain object has Object.prototype or null as its prototype, a plain
|
|
15
|
+
* array Array.prototype, and neither has a toJSON. Otherwise undefined: the caller needs the whole
|
|
16
|
+
* text. Getters, proxies and a toJSON that changes `value` while it is serialized are not detected:
|
|
17
|
+
* the elements are read before anything is serialized. before() and after() serialize the members
|
|
18
|
+
* on their side of the array when called, so a caller that calls before(), serializes each element
|
|
19
|
+
* and then calls after() runs every toJSON in the order JSON.stringify(value) runs them.
|
|
20
|
+
*/
|
|
21
|
+
export declare function splitAtArray(value: unknown, key: string): ArraySplit | undefined;
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
// When the JSON text of a value can be worked on one element of an array at a time. A run bundle
|
|
2
|
+
// holds every participant's trace in its `streams` array, and the bundle's whole text can pass
|
|
3
|
+
// V8's longest string, 536,870,888 characters on Node 24.
|
|
4
|
+
//
|
|
5
|
+
// JSON.stringify calls a toJSON with the value's key in its parent, and an element serialized
|
|
6
|
+
// alone gets "" in place of its index. It writes a boxed primitive as the primitive. A plain object
|
|
7
|
+
// or array with no toJSON is written as its own keys or items wherever it sits, and every value
|
|
8
|
+
// inside an element gets the same key alone as inside the whole value.
|
|
9
|
+
import { isRecord } from "../run/type-guards.js";
|
|
10
|
+
/**
|
|
11
|
+
* The cut of JSON.stringify(value) around the elements of its array `value[key]`, when that text
|
|
12
|
+
* holds each element's text as JSON.stringify(element) writes it: `value` is a plain object that
|
|
13
|
+
* has `key` among its own enumerable keys, `value[key]` is a plain array with no holes, and each
|
|
14
|
+
* element is a plain object. A plain object has Object.prototype or null as its prototype, a plain
|
|
15
|
+
* array Array.prototype, and neither has a toJSON. Otherwise undefined: the caller needs the whole
|
|
16
|
+
* text. Getters, proxies and a toJSON that changes `value` while it is serialized are not detected:
|
|
17
|
+
* the elements are read before anything is serialized. before() and after() serialize the members
|
|
18
|
+
* on their side of the array when called, so a caller that calls before(), serializes each element
|
|
19
|
+
* and then calls after() runs every toJSON in the order JSON.stringify(value) runs them.
|
|
20
|
+
*/
|
|
21
|
+
export function splitAtArray(value, key) {
|
|
22
|
+
// JSON.stringify writes only own enumerable keys.
|
|
23
|
+
if (!isPlainRecord(value))
|
|
24
|
+
return undefined;
|
|
25
|
+
const keys = Object.keys(value);
|
|
26
|
+
const at = keys.indexOf(key);
|
|
27
|
+
if (at === -1)
|
|
28
|
+
return undefined;
|
|
29
|
+
const array = value[key];
|
|
30
|
+
if (!isPlainArray(array))
|
|
31
|
+
return undefined;
|
|
32
|
+
// JSON.stringify reads the length once and then each index, so a hole reads as undefined and an
|
|
33
|
+
// own iterator is never called. Array.from would call that iterator, and every skips holes.
|
|
34
|
+
const elements = [];
|
|
35
|
+
for (let index = 0, length = array.length; index < length; index += 1)
|
|
36
|
+
elements.push(array[index]);
|
|
37
|
+
if (!elements.every(isPlainRecord))
|
|
38
|
+
return undefined;
|
|
39
|
+
// fromEntries keeps a key such as __proto__ an own property.
|
|
40
|
+
const members = (from, to) => Object.fromEntries(keys.slice(from, to).map((name) => [name, value[name]]));
|
|
41
|
+
return {
|
|
42
|
+
elements,
|
|
43
|
+
before: () => JSON.stringify({ ...members(0, at), [key]: [] }).slice(0, -"]}".length),
|
|
44
|
+
after: () => {
|
|
45
|
+
const text = JSON.stringify(members(at + 1));
|
|
46
|
+
return text === "{}" ? "]}" : `],${text.slice(1)}`;
|
|
47
|
+
},
|
|
48
|
+
};
|
|
49
|
+
}
|
|
50
|
+
const hasToJson = (value) => typeof value.toJSON === "function";
|
|
51
|
+
function isPlainRecord(value) {
|
|
52
|
+
if (!isRecord(value) || hasToJson(value))
|
|
53
|
+
return false;
|
|
54
|
+
const prototype = Object.getPrototypeOf(value);
|
|
55
|
+
return prototype === Object.prototype || prototype === null;
|
|
56
|
+
}
|
|
57
|
+
const isPlainArray = (value) => Array.isArray(value) && Object.getPrototypeOf(value) === Array.prototype && !hasToJson(value);
|
|
@@ -14,6 +14,8 @@ export declare const REDACTION_MARKERS: {
|
|
|
14
14
|
export declare function sensitivePatterns(): readonly RegExp[];
|
|
15
15
|
/** True if the text contains a credential. Local paths and a URL on an E2B host alone do not count. */
|
|
16
16
|
export declare function containsCredential(text: string): boolean;
|
|
17
|
+
/** True if the text contains a URL on an E2B host, as containsSensitive reads one. */
|
|
18
|
+
export declare function containsSandboxUrl(text: string): boolean;
|
|
17
19
|
/** True if the text contains any secret-shaped token or known local path. */
|
|
18
20
|
export declare function containsSensitive(text: string): boolean;
|
|
19
21
|
/**
|
|
@@ -76,6 +76,10 @@ const E2B_PUBLIC_PAGE = [
|
|
|
76
76
|
// A stream URL's auth key rides its `password` parameter, 16 characters from the SDK, which the
|
|
77
77
|
// credential parameter pattern matches as well.
|
|
78
78
|
const SANDBOX_URL = new RegExp(String.raw `(?!https:\/\/(?:${E2B_PUBLIC_PAGE})[.,;:!?'")]*(?:\s|$))https?:\\?\/\\?\/(?:[^/\s@]*@)*[^/\s\\"'<>@]*e2b[^)\s]+`, "gi");
|
|
79
|
+
// src/evidence/sensitive-json.ts scans a JSON array one element at a time and relies on which of
|
|
80
|
+
// these patterns can match across the `},{` between two elements. A new pattern that can cross
|
|
81
|
+
// those characters needs a case in tests/evidence/sensitive-json.test.ts.
|
|
82
|
+
//
|
|
79
83
|
// A pattern that matches a credential by its context puts the context in a `keep` group, which
|
|
80
84
|
// redaction leaves in place: `?token=[REDACTED_SECRET]`. Context goes in a group and not in a
|
|
81
85
|
// lookbehind, because a pattern that starts with a lookbehind is tried at every position of the
|
|
@@ -174,6 +178,10 @@ const CREDENTIAL_PATTERNS = SECRET_PATTERNS.filter((pattern) => pattern !== SAND
|
|
|
174
178
|
export function containsCredential(text) {
|
|
175
179
|
return CREDENTIAL_PATTERNS.some((pattern) => matchesPattern(pattern, text));
|
|
176
180
|
}
|
|
181
|
+
/** True if the text contains a URL on an E2B host, as containsSensitive reads one. */
|
|
182
|
+
export function containsSandboxUrl(text) {
|
|
183
|
+
return matchesPattern(SANDBOX_URL, text);
|
|
184
|
+
}
|
|
177
185
|
/** True if the text contains any secret-shaped token or known local path. */
|
|
178
186
|
export function containsSensitive(text) {
|
|
179
187
|
return (containsSecret(text) || LOCAL_PATH_PATTERNS.some(([pattern]) => matchesPattern(pattern, text)));
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* containsSensitive(JSON.stringify(value)), with each element of the array `value[key]` serialized
|
|
3
|
+
* and scanned on its own, so no one string holds two elements. Only an array of plain objects is
|
|
4
|
+
* scanned that way. Any other array, and a value, array or element with a toJSON or a prototype of
|
|
5
|
+
* its own, is scanned as the whole text, which can be longer than one string holds. Getters and
|
|
6
|
+
* proxies are not detected, so the answer is the whole text's for values as JSON.parse returns
|
|
7
|
+
* them.
|
|
8
|
+
*/
|
|
9
|
+
export declare function containsSensitiveJson(value: unknown, key: string): boolean;
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
// containsSensitive over a JSON document that may not fit in one string. V8's longest string is
|
|
2
|
+
// 536,870,888 characters on Node 24, and a run with every participant's trace loaded can pass it.
|
|
3
|
+
//
|
|
4
|
+
// The document is scanned as pieces: the text before the array and its first element, each
|
|
5
|
+
// element after that, and the last element with the text after the array. Every cut then falls on
|
|
6
|
+
// the `,` between two objects, `},{`. Most patterns stop at a quote, so a match stays inside one
|
|
7
|
+
// JSON string. Those that read a name and its value cross only the quotes around its `:`. A
|
|
8
|
+
// credential variable's unquoted value runs over brackets and braces up to a comma, so at a cut it
|
|
9
|
+
// stops, but after the last element it can run over the array's `]`. That is why the text after
|
|
10
|
+
// the array joins the last element.
|
|
11
|
+
//
|
|
12
|
+
// The sandbox URL can cross a cut. Its user info runs over quotes, braces and commas up to an `@`:
|
|
13
|
+
// `https://host"},{"contact":"ops@e2b-host`. From the end of its scheme to one character past
|
|
14
|
+
// `e2b`, that match holds no slash and no whitespace. So a match across a cut starts at a scheme
|
|
15
|
+
// with no slash or whitespace after it up to the cut, and reaches the next piece only up to its
|
|
16
|
+
// first slash or whitespace. Between that scheme and the cut, the match can only be in its user
|
|
17
|
+
// info. The scheme and every `@` there are inside JSON strings, so a host that starts after one
|
|
18
|
+
// stops at its string's closing quote. Had that host reached `e2b` first, the scan of its piece or
|
|
19
|
+
// an earlier junction would have matched. So the scan keeps only the scheme between pieces. Each
|
|
20
|
+
// cut's junction scans the scheme, `"}` in place of the text after it, the cut's `,` and the next
|
|
21
|
+
// piece up to its first slash or whitespace. A public E2B page after the scheme would need a space
|
|
22
|
+
// or the text's end before the cut, so neither text exempts the URL. The scan holds one piece at a
|
|
23
|
+
// time.
|
|
24
|
+
import { splitAtArray } from "./json-split.js";
|
|
25
|
+
import { containsSandboxUrl, containsSensitive } from "./redaction.js";
|
|
26
|
+
// The sandbox URL's scheme, `https://` or `https:\/\/` as JSON may escape it, at a text's end.
|
|
27
|
+
const SCHEME_AT_END = /https?:\\?\/\\?\/$/i;
|
|
28
|
+
const SLASH_OR_SPACE = /[/\s]/;
|
|
29
|
+
/**
|
|
30
|
+
* containsSensitive(JSON.stringify(value)), with each element of the array `value[key]` serialized
|
|
31
|
+
* and scanned on its own, so no one string holds two elements. Only an array of plain objects is
|
|
32
|
+
* scanned that way. Any other array, and a value, array or element with a toJSON or a prototype of
|
|
33
|
+
* its own, is scanned as the whole text, which can be longer than one string holds. Getters and
|
|
34
|
+
* proxies are not detected, so the answer is the whole text's for values as JSON.parse returns
|
|
35
|
+
* them.
|
|
36
|
+
*/
|
|
37
|
+
export function containsSensitiveJson(value, key) {
|
|
38
|
+
const split = splitAtArray(value, key);
|
|
39
|
+
if (split === undefined)
|
|
40
|
+
return wholeTextSensitive(value);
|
|
41
|
+
const { elements } = split;
|
|
42
|
+
const [prefix, suffix] = [split.before(), split.after()];
|
|
43
|
+
if (elements.length === 0)
|
|
44
|
+
return containsSensitive(prefix + suffix);
|
|
45
|
+
// The scheme of a URL no slash or whitespace has followed, from an earlier piece.
|
|
46
|
+
let scheme;
|
|
47
|
+
for (const [index, element] of elements.entries()) {
|
|
48
|
+
const own = JSON.stringify(element);
|
|
49
|
+
const piece = (index === 0 ? prefix : "") + own + (index === elements.length - 1 ? suffix : "");
|
|
50
|
+
if (containsSensitive(piece))
|
|
51
|
+
return true;
|
|
52
|
+
if (scheme !== undefined && containsSandboxUrl(`${scheme}"},${head(piece)}`))
|
|
53
|
+
return true;
|
|
54
|
+
scheme = schemeAfter(scheme, piece);
|
|
55
|
+
}
|
|
56
|
+
return false;
|
|
57
|
+
}
|
|
58
|
+
const wholeTextSensitive = (value) => containsSensitive(JSON.stringify(value) ?? "");
|
|
59
|
+
/** `piece` up to and including its first slash or whitespace, or all of it when it has none. */
|
|
60
|
+
function head(piece) {
|
|
61
|
+
const at = piece.search(SLASH_OR_SPACE);
|
|
62
|
+
return at === -1 ? piece : piece.slice(0, at + 1);
|
|
63
|
+
}
|
|
64
|
+
/** The scheme a sandbox URL could continue from after `piece`, given the one before it. */
|
|
65
|
+
function schemeAfter(before, piece) {
|
|
66
|
+
let last = piece.length - 1;
|
|
67
|
+
while (last >= 0 && !SLASH_OR_SPACE.test(piece[last]))
|
|
68
|
+
last -= 1;
|
|
69
|
+
if (last === -1)
|
|
70
|
+
return before;
|
|
71
|
+
// `https:\/\/` is the longest scheme: ten characters ending at the slash.
|
|
72
|
+
return SCHEME_AT_END.exec(piece.slice(Math.max(0, last - 9), last + 1))?.[0];
|
|
73
|
+
}
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { collectDesktopRecording } from "../../evidence/desktop-recording-artifact.js";
|
|
2
2
|
import path from "node:path";
|
|
3
|
+
import { participantList } from "../../study/study-fields.js";
|
|
3
4
|
import { inboxRecipientFor, } from "./participant-desktop.js";
|
|
4
5
|
import { createLocalFirecrackerDesktop, } from "../../substrates/local/firecracker-desktop.js";
|
|
5
6
|
import { localBrowserDefaults, localBrowserUnsupportedReason, } from "../../substrates/local/runtime-config.js";
|
|
@@ -139,11 +140,16 @@ function createLocalParticipantDesktop(context, run, warnings, artifactRoot) {
|
|
|
139
140
|
};
|
|
140
141
|
}
|
|
141
142
|
/** The Codex account participant, whose unconfirmed cleanup blocks the study's analysis. */
|
|
142
|
-
function accountProvider(state,
|
|
143
|
-
return async ({ executor }) => {
|
|
143
|
+
function accountProvider(state, config) {
|
|
144
|
+
return async ({ executor, participant: { index } }) => {
|
|
145
|
+
// The wait settings the participant's loop runs with (plan-participants.ts), so its tool
|
|
146
|
+
// description states them.
|
|
147
|
+
const maxWaitMs = config.actor?.maxWaitMs;
|
|
148
|
+
const idleWaitMs = participantList(config)?.[index]?.idleWaitMs ?? config.actor?.idleWaitMs;
|
|
144
149
|
const participant = createRestrictedCodexParticipant({
|
|
145
150
|
speechEnabled: executor.speechEnabled === true,
|
|
146
151
|
...(maxWaitMs === undefined ? {} : { maxWaitMs }),
|
|
152
|
+
...(idleWaitMs === undefined ? {} : { idleWaitMs }),
|
|
147
153
|
});
|
|
148
154
|
state.participants.push(participant);
|
|
149
155
|
const provider = Object.assign(participant.provider, {
|
|
@@ -185,7 +191,7 @@ export function prepareLocalVmRun(options) {
|
|
|
185
191
|
const { config: _config, assets: _assets, signal: _signal, ...runOptions } = options;
|
|
186
192
|
return {
|
|
187
193
|
options: account
|
|
188
|
-
? { ...runOptions, createProvider: accountProvider(state, config
|
|
194
|
+
? { ...runOptions, createProvider: accountProvider(state, config) }
|
|
189
195
|
: runOptions,
|
|
190
196
|
localVm: {
|
|
191
197
|
desktop: (run, warnings, artifactRoot) => createLocalParticipantDesktop(context, run, warnings, artifactRoot),
|
|
@@ -47,6 +47,9 @@ export async function startParticipantModel(spec, deps, executor) {
|
|
|
47
47
|
...(spec.planned.limits.maxWaitMs === undefined
|
|
48
48
|
? {}
|
|
49
49
|
: { maxWaitMs: spec.planned.limits.maxWaitMs }),
|
|
50
|
+
...(spec.planned.limits.idleWaitMs === undefined
|
|
51
|
+
? {}
|
|
52
|
+
: { idleWaitMs: spec.planned.limits.idleWaitMs }),
|
|
50
53
|
session: { env },
|
|
51
54
|
});
|
|
52
55
|
return { provider: codexParticipant.provider, codexParticipant };
|
|
@@ -4,7 +4,12 @@ import { type PreparedRunArtifactPaths } from "./paths.js";
|
|
|
4
4
|
export declare function scrubSandboxIds(text: string, ids: Iterable<string>, label?: (id: string) => string): string;
|
|
5
5
|
/**
|
|
6
6
|
* `value` as it may be written or returned: ids at the sandbox-id keys become the marker and a
|
|
7
|
-
* digest, and any raw id left in free text or a URL becomes its label.
|
|
7
|
+
* digest, and any raw id left in free text, a key or a URL becomes its label. When no label goes
|
|
8
|
+
* in, the result is the key-redacted value. Otherwise it is what JSON.parse returns for the
|
|
9
|
+
* scrubbed JSON text of that value. A run bundle is scrubbed one stream at a time, so no string
|
|
10
|
+
* holds two streams unless a member outside `streams` holds them too. An id that holds one of
|
|
11
|
+
* {}[],:"\ makes the scrub work on the whole text, which throws RangeError for a bundle past V8's
|
|
12
|
+
* longest string. No E2B id holds one.
|
|
8
13
|
*/
|
|
9
14
|
export declare function publicSandboxView<T>(value: T, ids: Iterable<string>): T;
|
|
10
15
|
/**
|
package/dist/run/sandbox-ids.js
CHANGED
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
// so output an agent pastes into an issue never carries an id.
|
|
5
5
|
import { opendir, realpath } from "node:fs/promises";
|
|
6
6
|
import path from "node:path";
|
|
7
|
+
import { splitAtArray } from "../evidence/json-split.js";
|
|
7
8
|
import { readPlainText } from "../evidence/plain-text.js";
|
|
8
9
|
import { collectSandboxIds, redactSandboxIds, sandboxIdDigest, REDACTED_SANDBOX_ID, } from "../evidence/redaction.js";
|
|
9
10
|
import { ContainedReadRefusedError, readContainedRegularFile, refusalText, RUN_ARTIFACT_MAX_BYTES, writeContainedOutputFile, } from "./contained-output.js";
|
|
@@ -26,20 +27,79 @@ export function scrubSandboxIds(text, ids, label = sandboxIdLabel) {
|
|
|
26
27
|
unique.sort((left, right) => right.length - left.length);
|
|
27
28
|
return unique.reduce((current, id) => current.split(id).join(label(id)), text);
|
|
28
29
|
}
|
|
30
|
+
// A run bundle holds every finished participant's trace, and its whole JSON text can pass V8's
|
|
31
|
+
// longest string, 536,870,888 characters on Node 24. publicSandboxView scrubs such a bundle in
|
|
32
|
+
// pieces: the shell, which is the bundle's JSON text with `streams` written as `[]`, and the JSON
|
|
33
|
+
// text of each stream on its own. The pieces give what the whole text gives:
|
|
34
|
+
// - An id that holds none of {}[],:"\ lies inside one token of the whole text. Each cut between
|
|
35
|
+
// pieces falls between two of the characters {}[], and the placeholder adds only [ and ], so no
|
|
36
|
+
// occurrence crosses a cut. A replacement leaves those characters in place, so the next id's
|
|
37
|
+
// occurrences cannot cross one either.
|
|
38
|
+
// - A label holds no quote or backslash, so it never moves the end of a string. A label inside an
|
|
39
|
+
// escape, a number, true, false or null makes its piece fail to parse, and the whole text fails
|
|
40
|
+
// the same way.
|
|
41
|
+
// - JSON.stringify writes a stream's members alone as it writes them inside the bundle, since each
|
|
42
|
+
// gets the same key for its toJSON. Only the stream itself would get "" in place of its index,
|
|
43
|
+
// and splitAtArray takes no stream with a toJSON. The members before `streams`, each stream and
|
|
44
|
+
// the members after are serialized in that order, which is the order the whole text runs each
|
|
45
|
+
// toJSON in.
|
|
46
|
+
// - The parsed shell keeps its keys in the order JSON.parse gives for the whole text, and the
|
|
47
|
+
// streams go in at the place of `streams`. A key a label went into holds the label's `[`, so no
|
|
48
|
+
// other key becomes `streams`. An id inside the key `streams` itself takes the whole text.
|
|
49
|
+
// - Once a label goes in, the view is parsed from the scrubbed pieces, as it was from the whole
|
|
50
|
+
// text. A piece kept as given could run a toJSON or getter again when the view is written.
|
|
51
|
+
// - All of this holds for a bundle that serializing leaves unchanged, as every bundle JSON.parse
|
|
52
|
+
// returns is. splitAtArray reads the streams before any toJSON runs.
|
|
53
|
+
/** The characters that let an id reach past one token of a JSON text or move where a string ends. */
|
|
54
|
+
const CROSSES_TOKENS = /[{}[\],:"\\]/;
|
|
55
|
+
/** Whether every occurrence of `id` in a bundle's JSON text stays inside one piece. */
|
|
56
|
+
const staysInOnePiece = (id) => id.length === 0 || (!CROSSES_TOKENS.test(id) && !"streams".includes(id));
|
|
29
57
|
/**
|
|
30
58
|
* `value` as it may be written or returned: ids at the sandbox-id keys become the marker and a
|
|
31
|
-
* digest, and any raw id left in free text or a URL becomes its label.
|
|
59
|
+
* digest, and any raw id left in free text, a key or a URL becomes its label. When no label goes
|
|
60
|
+
* in, the result is the key-redacted value. Otherwise it is what JSON.parse returns for the
|
|
61
|
+
* scrubbed JSON text of that value. A run bundle is scrubbed one stream at a time, so no string
|
|
62
|
+
* holds two streams unless a member outside `streams` holds them too. An id that holds one of
|
|
63
|
+
* {}[],:"\ makes the scrub work on the whole text, which throws RangeError for a bundle past V8's
|
|
64
|
+
* longest string. No E2B id holds one.
|
|
32
65
|
*/
|
|
33
66
|
export function publicSandboxView(value, ids) {
|
|
34
67
|
const redacted = redactSandboxIds(value);
|
|
35
68
|
const known = [...ids];
|
|
36
69
|
if (known.length === 0)
|
|
37
70
|
return redacted;
|
|
38
|
-
const
|
|
71
|
+
const split = known.every(staysInOnePiece) ? splitAtArray(redacted, "streams") : undefined;
|
|
72
|
+
return (split === undefined ? wholeTextView(redacted, known) : streamwiseView(redacted, split, known));
|
|
73
|
+
}
|
|
74
|
+
/** The scrub of `value`'s whole JSON text, parsed back when a label went in. */
|
|
75
|
+
function wholeTextView(value, ids) {
|
|
76
|
+
const text = JSON.stringify(value);
|
|
39
77
|
if (text === undefined)
|
|
40
|
-
return
|
|
41
|
-
const scrubbed = scrubSandboxIds(text,
|
|
42
|
-
return
|
|
78
|
+
return value;
|
|
79
|
+
const scrubbed = scrubSandboxIds(text, ids);
|
|
80
|
+
return scrubbed === text ? value : JSON.parse(scrubbed);
|
|
81
|
+
}
|
|
82
|
+
/** The bundle's whole-text scrub, built from its shell and each stream scrubbed on its own. */
|
|
83
|
+
function streamwiseView(bundle, split, ids) {
|
|
84
|
+
const before = split.before();
|
|
85
|
+
let changed = false;
|
|
86
|
+
// Each stream is parsed at its turn, even before any label goes in, since a later piece can still
|
|
87
|
+
// take one and by then the earlier streams' text is gone.
|
|
88
|
+
const streams = split.elements.map((stream) => {
|
|
89
|
+
const text = JSON.stringify(stream);
|
|
90
|
+
const scrubbed = scrubSandboxIds(text, ids);
|
|
91
|
+
if (scrubbed !== text)
|
|
92
|
+
changed = true;
|
|
93
|
+
return JSON.parse(scrubbed);
|
|
94
|
+
});
|
|
95
|
+
const shellText = before + split.after();
|
|
96
|
+
const shell = scrubSandboxIds(shellText, ids);
|
|
97
|
+
if (!changed && shell === shellText)
|
|
98
|
+
return bundle;
|
|
99
|
+
// Assigning to `streams` keeps its place among the keys.
|
|
100
|
+
const view = JSON.parse(shell);
|
|
101
|
+
view.streams = streams;
|
|
102
|
+
return view;
|
|
43
103
|
}
|
|
44
104
|
/** The raw sandbox ids a receipts journal names. */
|
|
45
105
|
function receiptSandboxIds(journal) {
|
|
@@ -16,6 +16,8 @@ export interface BundleEstimate {
|
|
|
16
16
|
readonly runFileBytes: number;
|
|
17
17
|
/** The largest participant trace file, which analysis also reads whole. */
|
|
18
18
|
readonly largestTraceBytes: number;
|
|
19
|
+
/** Every participant's trace file together, whose items the Observer page joins in one string. */
|
|
20
|
+
readonly tracesBytes: number;
|
|
19
21
|
/** observer/index.html, which verify reads whole. */
|
|
20
22
|
readonly observerPageBytes: number;
|
|
21
23
|
/** Every file, screenshots included. */
|
|
@@ -16,6 +16,13 @@
|
|
|
16
16
|
// observer-data.json leaves the lists out too and was 0.94 to 1.12 of run.json's size on the live
|
|
17
17
|
// runs measured, so it passes 32 MiB only after run.json passes 16 MiB, and the estimate does not
|
|
18
18
|
// check it.
|
|
19
|
+
// The Observer page and the Observer server join every participant's items in one string before
|
|
20
|
+
// gzip. A bundle export renders the Observer page, so it fails there too once --max-bytes is
|
|
21
|
+
// raised past its 25 MiB default, which refuses such a run first. Writing run.json joins none: the
|
|
22
|
+
// sandbox id scrub (publicSandboxView) works one participant at a time. V8's longest string is
|
|
23
|
+
// 536,870,888 characters on Node 24. A trace file's bytes are at least the characters of its
|
|
24
|
+
// items as compact JSON, so the estimate compares the traces' bytes and may warn a little early.
|
|
25
|
+
import { constants } from "node:buffer";
|
|
19
26
|
import { EVIDENCE_LIMITS } from "../analysis/analysis-limits.js";
|
|
20
27
|
import { RUN_ARTIFACT_MAX_BYTES } from "../run/contained-output.js";
|
|
21
28
|
import { formatDuration } from "../run/projection.js";
|
|
@@ -156,6 +163,7 @@ export function estimateBundle(participants, sessionMs) {
|
|
|
156
163
|
return {
|
|
157
164
|
runFileBytes,
|
|
158
165
|
largestTraceBytes: largestTrace,
|
|
166
|
+
tracesBytes: traces,
|
|
159
167
|
observerPageBytes,
|
|
160
168
|
// observer-data.json is about run.json's size.
|
|
161
169
|
totalBytes: observerPageBytes + screenshots * RATES.screenshotBytes + traces + 2 * runFileBytes,
|
|
@@ -178,6 +186,7 @@ export function estimateBundle(participants, sessionMs) {
|
|
|
178
186
|
export function bundleLimitReason(estimate) {
|
|
179
187
|
const runFile = estimate.runFileBytes > EVIDENCE_LIMITS.sourceBytes;
|
|
180
188
|
const page = estimate.observerPageBytes > RUN_ARTIFACT_MAX_BYTES;
|
|
189
|
+
const joined = estimate.tracesBytes > constants.MAX_STRING_LENGTH;
|
|
181
190
|
const entries = estimate.entries > estimate.entryLimit;
|
|
182
191
|
const passed = [
|
|
183
192
|
...(runFile
|
|
@@ -195,6 +204,11 @@ export function bundleLimitReason(estimate) {
|
|
|
195
204
|
`The Observer page, observer/index.html, may grow to about ${mib(estimate.observerPageBytes)}, past the ${mib(RUN_ARTIFACT_MAX_BYTES)} verify reads in one file.`,
|
|
196
205
|
]
|
|
197
206
|
: []),
|
|
207
|
+
...(joined
|
|
208
|
+
? [
|
|
209
|
+
`All participants' traces together may grow to about ${mib(estimate.tracesBytes)}, past the ${mib(constants.MAX_STRING_LENGTH)} that fits in one string. The Observer page puts all their items in one string, so the page would not build. A bundle export allowed past its 25 MiB default renders that page, so it would fail too.`,
|
|
210
|
+
]
|
|
211
|
+
: []),
|
|
198
212
|
...(entries
|
|
199
213
|
? [
|
|
200
214
|
`The run may hold about ${count(estimate.entries)} files and folders, past the ${count(estimate.entryLimit)} the public-safety scan reads in a run of ${plural(estimate.basis.participants, "participant")}.`,
|
|
@@ -222,7 +236,7 @@ export function bundleLimitReason(estimate) {
|
|
|
222
236
|
...(runFile ? ["shorten the mission (actor.mission)"] : []),
|
|
223
237
|
];
|
|
224
238
|
// A trace is one participant's, so splitting the study helps only the run's own files.
|
|
225
|
-
if (runFile || page || entries || changes.length === 0)
|
|
239
|
+
if (runFile || page || joined || entries || changes.length === 0)
|
|
226
240
|
changes.push("split the participants across studies");
|
|
227
241
|
const advice = changes.length < 2
|
|
228
242
|
? changes.join("")
|
package/dist/study/types.d.ts
CHANGED
|
@@ -414,10 +414,11 @@ export interface StudyActor {
|
|
|
414
414
|
maxWaitMs?: number;
|
|
415
415
|
/**
|
|
416
416
|
* Computer-use and shared-world: how long a `wait` that names no duration lasts when the
|
|
417
|
-
* participant's turn only waits or takes screenshots,
|
|
418
|
-
* while it watches such a turn, in whole milliseconds
|
|
419
|
-
* maxWaitMs. Default 5000. After an action in the same
|
|
420
|
-
* computer
|
|
417
|
+
* participant's turn only waits or takes screenshots, the shortest such a turn is watched for,
|
|
418
|
+
* and how often the loop looks at the screen while it watches such a turn, in whole milliseconds
|
|
419
|
+
* from 1000 to 600000 and no longer than maxWaitMs. Default 5000. After an action in the same
|
|
420
|
+
* turn such a wait lasts 500 ms. OpenAI's computer tool `wait` never names a duration; its
|
|
421
|
+
* wait_minutes function does.
|
|
421
422
|
*/
|
|
422
423
|
idleWaitMs?: number;
|
|
423
424
|
/**
|
package/dist/verify/verify.js
CHANGED
|
@@ -2,7 +2,8 @@ import path from "node:path";
|
|
|
2
2
|
import { listAnalysisExecutions } from "../analysis/store-executions.js";
|
|
3
3
|
import { loadAnalysis } from "../analysis/load.js";
|
|
4
4
|
import { analysisSharingProblems } from "../analysis/sharing.js";
|
|
5
|
-
import {
|
|
5
|
+
import { REDACTED_SANDBOX_ID } from "../evidence/redaction.js";
|
|
6
|
+
import { containsSensitiveJson } from "../evidence/sensitive-json.js";
|
|
6
7
|
import { keyedSandboxIds, readRunSandboxIds } from "../run/sandbox-ids.js";
|
|
7
8
|
import { validatePreparedRunArtifactPaths } from "../run/paths.js";
|
|
8
9
|
import { isReadRunBundleSchema, RUN_BUNDLE_FILE, RUN_BUNDLE_SCHEMA, RUN_BUNDLE_V1_SCHEMA, RUN_BUNDLE_V2_SCHEMA, RUN_BUNDLE_V3_SCHEMA, } from "../run/bundle.js";
|
|
@@ -112,9 +113,10 @@ export async function verifyResolvedRun(cwd, runInput, runPaths) {
|
|
|
112
113
|
addTraceScan(traces, publicSafetyFindings, unscannedArtifacts);
|
|
113
114
|
// JSON escapes can hide a sensitive value from the byte scan while the
|
|
114
115
|
// decoded recording exposes it to Observer, feedback, or analysis input.
|
|
116
|
+
// Each participant is scanned apart: together their traces can pass V8's longest string.
|
|
115
117
|
if (publicSafetyFindings.length < MAX_REPORTED_FINDINGS &&
|
|
116
118
|
recorded !== null &&
|
|
117
|
-
|
|
119
|
+
containsSensitiveJson(recorded, "streams")) {
|
|
118
120
|
publicSafetyFindings.push("sensitive decoded run.json");
|
|
119
121
|
}
|
|
120
122
|
checks.push({
|
|
@@ -176,6 +176,7 @@ export interface ActorTraceItem {
|
|
|
176
176
|
looks: number;
|
|
177
177
|
area?: { x: number; y: number; width: number; height: number }; // pixels, coordinates only
|
|
178
178
|
};
|
|
179
|
+
namedMs?: number; // computer-use wait ui_action: the length the participant named, before any cap
|
|
179
180
|
}
|
|
180
181
|
|
|
181
182
|
export interface ActorCapabilities {
|
|
@@ -110,8 +110,8 @@ with speech, whose heard speech reaches the participant only with a screenshot),
|
|
|
110
110
|
shortened` notice with both durations when it does, and sends the rest as consecutive waits of at
|
|
111
111
|
most 30 seconds. The Codex participant's `humanish_ui` schema leaves a wait's length open for that
|
|
112
112
|
reason; the wire still refuses a single wait over 30 seconds. A wait that names no duration, as
|
|
113
|
-
|
|
114
|
-
in the same turn. A turn that only waits or takes screenshots sends no wait to the desktop: the
|
|
113
|
+
the OpenAI computer tool's `wait` does, reaches the desktop with a length too: 500 ms after an
|
|
114
|
+
action in the same turn. A turn that only waits or takes screenshots sends no wait to the desktop: the
|
|
115
115
|
loop observes the screen every `actor.idleWaitMs` until it changes or the wait's time passes
|
|
116
116
|
(`src/actors/computer-use/loop/watch.ts`), so a waiting participant costs one observe request per
|
|
117
117
|
look.
|
|
@@ -1082,14 +1082,17 @@ Core-owned fields:
|
|
|
1082
1082
|
16 pixels counts once it holds still at a look; one of 24 tiles or fewer once
|
|
1083
1083
|
it has held still for more than 65 s with no tile within 4 tiles of it ticking
|
|
1084
1084
|
or moving with a region of more than 24 tiles
|
|
1085
|
-
(`src/evidence/frame-change.ts`). The first watch
|
|
1086
|
-
length: the named waits' lengths, together up to `actor.maxWaitMs` (else
|
|
1087
|
-
600000), and `idleWaitMs` for each wait that names none
|
|
1088
|
-
|
|
1089
|
-
|
|
1090
|
-
|
|
1091
|
-
|
|
1092
|
-
|
|
1085
|
+
(`src/evidence/frame-change.ts`). The first watch's limit is the turn's
|
|
1086
|
+
planned length: the named waits' lengths, together up to `actor.maxWaitMs` (else
|
|
1087
|
+
600000), and `idleWaitMs` for each wait that names none, and at least
|
|
1088
|
+
`idleWaitMs`. Each further one in a row that ended at its limit, or that a
|
|
1089
|
+
change of 24 tiles or fewer woke, lasts twice as long, up to the larger of its
|
|
1090
|
+
planned length and that cap. An OpenAI participant names a length with the
|
|
1091
|
+
`wait_minutes` function tool; its computer tool's `wait` names none.
|
|
1092
|
+
`limitMs` is how long it could last; `askedMs` is what the turn's
|
|
1093
|
+
waits asked for, the same sum with no cap, so it is above `limitMs` when the
|
|
1094
|
+
named waits passed the cap and below it when the watch grew or the named waits
|
|
1095
|
+
were shorter than `idleWaitMs`; `wake` is `changed`, `speech`,
|
|
1093
1096
|
`condition`, `limit`, `clock` (wind-down or end), `stalled` (a look got no
|
|
1094
1097
|
frame; the next observation is a fresh one), or `stopped` (the study budget or
|
|
1095
1098
|
the session's time ran out); `looks` counts its looks; `area` (`x`, `y`,
|
|
@@ -1100,6 +1103,14 @@ Core-owned fields:
|
|
|
1100
1103
|
waits after the one carrying `watch` are titled `wait 0ms`. The look that
|
|
1101
1104
|
ended a watch is the turn's `turn-NN` screenshot; other looks are not kept.
|
|
1102
1105
|
Additive: absent on plain waits, other producers and older bundles
|
|
1106
|
+
- optional `items[].namedMs`, on the computer-use `ui_action` of each wait that
|
|
1107
|
+
named its length: that length in milliseconds, as the participant named it,
|
|
1108
|
+
before `actor.maxWaitMs` or the watch's cap applied. An OpenAI participant
|
|
1109
|
+
names a length only through `wait_minutes`, so its waits with `namedMs` are
|
|
1110
|
+
that tool's calls. Its waits without it are the computer tool's `wait`, a
|
|
1111
|
+
`wait_minutes` call whose minutes were not a finite number above zero, and
|
|
1112
|
+
a call to a function it does not have. Additive: absent on waits that named
|
|
1113
|
+
none, other items and producers, and older bundles
|
|
1103
1114
|
- optional `items[].page` and `finalPage` (`{ origin, path }`, `TracePage` in
|
|
1104
1115
|
`src/actors/contract.ts`): where a computer-use participant's browser was. The loop puts `page`
|
|
1105
1116
|
on the `screenshot` item of each observation that read a URL: `turn-00-start` is the page before
|
|
@@ -1183,8 +1194,9 @@ Core-owned fields:
|
|
|
1183
1194
|
with the outputs that answered it, and the latest reply. Past an estimated
|
|
1184
1195
|
64,000 input tokens it is cut down to 32,000 in one step: the opening
|
|
1185
1196
|
screenshot is dropped first, then the oldest exchanges become lines of a note,
|
|
1186
|
-
sent as an assistant message, that keeps their reasoning summaries, messages
|
|
1187
|
-
and
|
|
1197
|
+
sent as an assistant message, that keeps their reasoning summaries, messages,
|
|
1198
|
+
actions and `wait_minutes` calls as text, with any text that came back with
|
|
1199
|
+
their screenshots or in a function call's output, such as how a wait ended.
|
|
1188
1200
|
Past 32,000 characters the note keeps its first four lines and its newest
|
|
1189
1201
|
ones, and names the turns between them as not listed.
|
|
1190
1202
|
The two newest exchanges are always carried whole, even when they alone pass
|