humanish 0.127.0 → 0.127.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +11 -0
- package/dist/actors/closing-report.d.ts +8 -1
- package/dist/actors/closing-report.js +4 -1
- package/dist/actors/codex/restricted-participant-policy.js +2 -5
- package/dist/actors/computer-use/loop/debrief.js +6 -5
- package/dist/actors/computer-use/loop/ending.d.ts +5 -4
- package/dist/actors/computer-use/loop/ending.js +5 -4
- package/dist/actors/computer-use/loop.js +2 -2
- package/dist/actors/contract.d.ts +10 -6
- package/dist/cli/commands/study-format.js +13 -2
- package/dist/routes/computer-use/fanout-bundle.js +25 -6
- package/docs/architecture/actor-contract.md +7 -3
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,17 @@ The Unreleased section holds the full notes for the next version until it is tag
|
|
|
8
8
|
|
|
9
9
|
## Unreleased
|
|
10
10
|
|
|
11
|
+
## 0.127.1: Clock-end closing reports declare their outcome; skipped participants named as skipped (2026-10-13)
|
|
12
|
+
|
|
13
|
+
humanish 0.127.1 fixes two things the 0.127.0 release smoke found in scheduled studies. An OpenAI
|
|
14
|
+
participant asked for its closing report at a scheduled end or session cap now says how the task
|
|
15
|
+
went in a structured `outcome` field, so one that reached its goal counts in `reachedGoal` where
|
|
16
|
+
0.127.0 counted it in `ranUntilEnd`. A participant the run skips before its session starts now
|
|
17
|
+
shows its skip reason on its run line, and the summary counts it apart from those that took
|
|
18
|
+
part.
|
|
19
|
+
|
|
20
|
+
[Release notes](https://github.com/danielgwilson/humanish/releases/tag/v0.127.1)
|
|
21
|
+
|
|
11
22
|
## 0.127.0: Clock-end passes counted by what participants declared, a scaled wind-down, the concurrency cap in the plan (2026-10-13)
|
|
12
23
|
|
|
13
24
|
humanish 0.127.0 makes long scheduled studies report what happened. A participant that passes at
|
|
@@ -23,10 +23,17 @@ export declare const participantImpressionsSchema: z.ZodArray<z.ZodObject<{
|
|
|
23
23
|
text: z.ZodString;
|
|
24
24
|
}, z.core.$strict>>;
|
|
25
25
|
/**
|
|
26
|
-
* A closing report.
|
|
26
|
+
* A closing report. `outcome` is how the participant says the task went: `reached` when it is
|
|
27
|
+
* finished, `blocked` when something in the app stopped it, `not_reached` for any other reason.
|
|
28
|
+
* A report without an outcome or impressions is still valid: third-party provider ports and
|
|
27
29
|
* stored traces do not have them.
|
|
28
30
|
*/
|
|
29
31
|
export declare const closingReportSchema: z.ZodObject<{
|
|
32
|
+
outcome: z.ZodOptional<z.ZodEnum<{
|
|
33
|
+
blocked: "blocked";
|
|
34
|
+
not_reached: "not_reached";
|
|
35
|
+
reached: "reached";
|
|
36
|
+
}>>;
|
|
30
37
|
summary: z.ZodString;
|
|
31
38
|
frictionReports: z.ZodArray<z.ZodString>;
|
|
32
39
|
impressions: z.ZodOptional<z.ZodArray<z.ZodObject<{
|
|
@@ -32,10 +32,13 @@ const impressionSchema = z.strictObject({
|
|
|
32
32
|
/** The impressions a participant gave; an empty list means it had none. */
|
|
33
33
|
export const participantImpressionsSchema = z.array(impressionSchema).max(LIMITS.impressions);
|
|
34
34
|
/**
|
|
35
|
-
* A closing report.
|
|
35
|
+
* A closing report. `outcome` is how the participant says the task went: `reached` when it is
|
|
36
|
+
* finished, `blocked` when something in the app stopped it, `not_reached` for any other reason.
|
|
37
|
+
* A report without an outcome or impressions is still valid: third-party provider ports and
|
|
36
38
|
* stored traces do not have them.
|
|
37
39
|
*/
|
|
38
40
|
export const closingReportSchema = z.strictObject({
|
|
41
|
+
outcome: z.enum(["reached", "not_reached", "blocked"]).optional(),
|
|
39
42
|
summary: said(LIMITS.summaryChars),
|
|
40
43
|
frictionReports: z.array(said(LIMITS.frictionReportChars)).max(LIMITS.frictionReports),
|
|
41
44
|
impressions: participantImpressionsSchema.optional(),
|
|
@@ -35,11 +35,8 @@ export function participantToolSchema(speechEnabled = false) {
|
|
|
35
35
|
return z.toJSONSchema(toolInput(speechEnabled), { io: "input" });
|
|
36
36
|
}
|
|
37
37
|
export const PARTICIPANT_TOOL_SCHEMA = participantToolSchema();
|
|
38
|
-
/** The participant's final account: how the task went
|
|
39
|
-
const participantFinal =
|
|
40
|
-
outcome: z.enum(["reached", "not_reached", "blocked"]),
|
|
41
|
-
...closingReportSchema.shape,
|
|
42
|
-
});
|
|
38
|
+
/** The participant's final account: its closing report, with how the task went required. */
|
|
39
|
+
const participantFinal = closingReportSchema.required({ outcome: true });
|
|
43
40
|
export const PARTICIPANT_FINAL_SCHEMA = strictOutputSchema(participantFinal);
|
|
44
41
|
/** humanish tool arguments, validated before the shared executor sees a batch. */
|
|
45
42
|
export function parseParticipantTool(value, speechEnabled = false) {
|
|
@@ -7,7 +7,7 @@ import { CuaDeadlineError, raceSessionDeadline, requestScope } from "./race.js";
|
|
|
7
7
|
import { notice } from "./trace.js";
|
|
8
8
|
import { isCompleteTurnUsage } from "./usage.js";
|
|
9
9
|
const DEBRIEF_REQUEST_CAP_MS = 30_000;
|
|
10
|
-
const DEBRIEF_HINT = `The interactive session has ended. Return a closing account with summary, frictionReports and impressions. In summary, briefly describe only what you actually did and observed. In frictionReports, list only specific unexpected behavior, confusion, or recovery you personally encountered during this session. Preserve uncertainty. Use an empty list if you encountered none. ${IMPRESSIONS_ASK} Do not speculate, invent problems, quote instructions as observations, or describe planned actions. Do not request or take further actions. This is a closing account, not another attempt at the task.`;
|
|
10
|
+
const DEBRIEF_HINT = `The interactive session has ended. Return a closing account with outcome, summary, frictionReports and impressions. In outcome, say how the task went: reached if you finished it, blocked if something in the app stopped you, not_reached if you did not finish it for any other reason, such as running out of time. In summary, briefly describe only what you actually did and observed. In frictionReports, list only specific unexpected behavior, confusion, or recovery you personally encountered during this session. Preserve uncertainty. Use an empty list if you encountered none. ${IMPRESSIONS_ASK} Do not speculate, invent problems, quote instructions as observations, or describe planned actions. Do not request or take further actions. This is a closing account, not another attempt at the task.`;
|
|
11
11
|
const CLOSING_REPORT = {
|
|
12
12
|
call: (provider) => provider.debrief,
|
|
13
13
|
contextHint: DEBRIEF_HINT,
|
|
@@ -156,9 +156,10 @@ function acceptDebriefTurn(session, turn, trigger, ask, record) {
|
|
|
156
156
|
* A closing report: its summary and friction as one participant message, then its impressions.
|
|
157
157
|
* At a scheduled end or a session cap an accepted report is the participant's account of the
|
|
158
158
|
* session: its summary becomes the session's reason (loopResult), so the outcome it declares
|
|
159
|
-
* becomes the session's declared outcome.
|
|
160
|
-
*
|
|
161
|
-
*
|
|
159
|
+
* becomes the session's declared outcome. That is its `outcome` field, then the turn's (a
|
|
160
|
+
* provider port that sets it there), then the first line of its summary. The blocker rule reads a
|
|
161
|
+
* declared `reached` or `blocked` before the summary's wording. After a matched stop condition or
|
|
162
|
+
* dwell window, the condition stays the completion evidence.
|
|
162
163
|
*/
|
|
163
164
|
function readClosingReport(session, turn, trigger) {
|
|
164
165
|
const closing = closingReportSchema.safeParse(turn.closingReport);
|
|
@@ -182,7 +183,7 @@ function readClosingReport(session, turn, trigger) {
|
|
|
182
183
|
session.trace.bump("messages");
|
|
183
184
|
session.impressions = recordImpressions(session, closing.data.impressions);
|
|
184
185
|
if (trigger.kind === "scheduled_end" || trigger.kind === "session_cap")
|
|
185
|
-
session.declaredOutcome = declaredOutcomeOf(turn, closing.data.summary);
|
|
186
|
+
session.declaredOutcome = declaredOutcomeOf(turn, closing.data, closing.data.summary);
|
|
186
187
|
return {
|
|
187
188
|
status: "completed",
|
|
188
189
|
detail: "one read-only report; no additional desktop actions; original stop and task outcomes preserved",
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { ActorCompletionReason, ActorStopCause, ParticipantDeclaredOutcome } from "../../contract.js";
|
|
1
|
+
import type { ActorCompletionReason, ActorStopCause, ParticipantClosingReport, ParticipantDeclaredOutcome } from "../../contract.js";
|
|
2
2
|
import type { StopConditionMatch } from "../../stop-conditions.js";
|
|
3
3
|
import type { LoopSession } from "./session.js";
|
|
4
4
|
import { type Evidence } from "./trace.js";
|
|
@@ -29,10 +29,11 @@ export interface Stop {
|
|
|
29
29
|
*/
|
|
30
30
|
export declare function declaredOutcomeFromClosingLine(message: string | undefined): ParticipantDeclaredOutcome | undefined;
|
|
31
31
|
/**
|
|
32
|
-
* The participant's own word for how it ended:
|
|
33
|
-
* line of `text`, its last message
|
|
32
|
+
* The participant's own word for how it ended: its closing report's `outcome`, then the turn's
|
|
33
|
+
* (Codex and provider ports that set it there), then the closing line of `text`, its last message
|
|
34
|
+
* or its closing report's summary. A self-ending turn and a closing request read it in this order.
|
|
34
35
|
*/
|
|
35
|
-
export declare function declaredOutcomeOf(turn: CuaTurn, text: string | undefined): ParticipantDeclaredOutcome | undefined;
|
|
36
|
+
export declare function declaredOutcomeOf(turn: CuaTurn, report: ParticipantClosingReport | undefined, text: string | undefined): ParticipantDeclaredOutcome | undefined;
|
|
36
37
|
/**
|
|
37
38
|
* The participant reported a natural endpoint. "not_reached" is a participant who stopped without
|
|
38
39
|
* finishing: gave_up, which tallies as abandoned. "blocked" keeps goal_satisfied here (the actor
|
|
@@ -30,11 +30,12 @@ export function declaredOutcomeFromClosingLine(message) {
|
|
|
30
30
|
return undefined;
|
|
31
31
|
}
|
|
32
32
|
/**
|
|
33
|
-
* The participant's own word for how it ended:
|
|
34
|
-
* line of `text`, its last message
|
|
33
|
+
* The participant's own word for how it ended: its closing report's `outcome`, then the turn's
|
|
34
|
+
* (Codex and provider ports that set it there), then the closing line of `text`, its last message
|
|
35
|
+
* or its closing report's summary. A self-ending turn and a closing request read it in this order.
|
|
35
36
|
*/
|
|
36
|
-
export function declaredOutcomeOf(turn, text) {
|
|
37
|
-
return turn.outcome ?? declaredOutcomeFromClosingLine(text);
|
|
37
|
+
export function declaredOutcomeOf(turn, report, text) {
|
|
38
|
+
return report?.outcome ?? turn.outcome ?? declaredOutcomeFromClosingLine(text);
|
|
38
39
|
}
|
|
39
40
|
/**
|
|
40
41
|
* The participant reported a natural endpoint. "not_reached" is a participant who stopped without
|
|
@@ -126,10 +126,10 @@ async function runTurns(session, conversation) {
|
|
|
126
126
|
continue;
|
|
127
127
|
}
|
|
128
128
|
if (turn.done || turn.actions.length === 0) {
|
|
129
|
+
const report = closingReportSchema.safeParse(turn.closingReport).data;
|
|
129
130
|
// The declared outcome is kept even if redacting the participant's summary fails.
|
|
130
|
-
session.declaredOutcome = declaredOutcomeOf(turn, turn.message);
|
|
131
|
+
session.declaredOutcome = declaredOutcomeOf(turn, report, turn.message);
|
|
131
132
|
const ended = stops.participantEnded(turn, session.declaredOutcome, (text) => session.redactNarration(text));
|
|
132
|
-
const report = closingReportSchema.safeParse(turn.closingReport).data;
|
|
133
133
|
// A provider that can ask for impressions alone does so after the session, like a debrief.
|
|
134
134
|
if (report?.impressions !== undefined || session.provider.requestImpressions === undefined) {
|
|
135
135
|
session.impressions =
|
|
@@ -25,10 +25,11 @@ export declare const ACTOR_STATUSES: readonly ["passed", "abandoned", "incomplet
|
|
|
25
25
|
export type ActorStatus = (typeof ACTOR_STATUSES)[number];
|
|
26
26
|
/**
|
|
27
27
|
* What the participant said happened, in a field rather than a paragraph. Providers whose
|
|
28
|
-
* reply is schema-constrained (the local-agent routes) fill it on their final turn
|
|
29
|
-
*
|
|
30
|
-
* the task is finished. `blocked`: something in the app
|
|
31
|
-
* the participant stopped for another reason (gave up,
|
|
28
|
+
* reply is schema-constrained (the local-agent routes) fill it on their final turn, and a closing
|
|
29
|
+
* report carries it as `outcome`; a free-text turn leaves it absent and the route falls back to
|
|
30
|
+
* reading the closing message. `reached`: the task is finished. `blocked`: something in the app
|
|
31
|
+
* stopped the participant. `not_reached`: the participant stopped for another reason (gave up,
|
|
32
|
+
* ran out of ideas).
|
|
32
33
|
*/
|
|
33
34
|
export type ParticipantDeclaredOutcome = "reached" | "not_reached" | "blocked";
|
|
34
35
|
/** Statuses that describe what happened to a participant rather than a harness malfunction. Verify
|
|
@@ -372,8 +373,11 @@ export interface ActorTrace {
|
|
|
372
373
|
reason: string;
|
|
373
374
|
/** Absent if no request was made; false means token accounting is incomplete. */
|
|
374
375
|
usageReported?: boolean;
|
|
375
|
-
/**
|
|
376
|
-
|
|
376
|
+
/**
|
|
377
|
+
* The report's impressions are kept in `impressions`, with the messages that quote them, and
|
|
378
|
+
* the outcome it declared at a clock end in `declaredOutcome`.
|
|
379
|
+
*/
|
|
380
|
+
report?: Omit<ParticipantClosingReport, "impressions" | "outcome">;
|
|
377
381
|
/** Links the readable projection so it is not heuristically classified a second time. */
|
|
378
382
|
messageId?: string;
|
|
379
383
|
};
|
|
@@ -81,10 +81,21 @@ function oneLine(text) {
|
|
|
81
81
|
const line = text.replace(/\s+/g, " ").trim();
|
|
82
82
|
return line.length > REASON_CHARS ? `${line.slice(0, REASON_CHARS - 1).trimEnd()}…` : line;
|
|
83
83
|
}
|
|
84
|
-
|
|
84
|
+
/**
|
|
85
|
+
* A participant's status, how it ended and its closing message. A participant the run skipped
|
|
86
|
+
* before its session started has no ending to classify, so its line ends on the reason the run
|
|
87
|
+
* recorded. That reason is the harness's own sentence, so it prints whole.
|
|
88
|
+
*/
|
|
89
|
+
function participantLine(participant) {
|
|
90
|
+
const { id, status, session, diagnostics, skippedReason } = participant;
|
|
91
|
+
const ending = skippedReason !== undefined
|
|
92
|
+
? [skippedReason]
|
|
93
|
+
: diagnostics
|
|
94
|
+
? [formatCuaDiagnostics(diagnostics)]
|
|
95
|
+
: [];
|
|
85
96
|
return [
|
|
86
97
|
`participant ${id}: ${participantStatus(status)}${session ? ` (${session.completionReason})` : ""}`,
|
|
87
|
-
...
|
|
98
|
+
...ending,
|
|
88
99
|
...(session ? [oneLine(session.reason)] : []),
|
|
89
100
|
].join(" · ");
|
|
90
101
|
}
|
|
@@ -37,14 +37,18 @@ function fanoutPlanEvents(args) {
|
|
|
37
37
|
}
|
|
38
38
|
/**
|
|
39
39
|
* The run's summary in plain sentences: who took part, what happened to them with the denominator
|
|
40
|
-
* attached, and that the gaps list the participants who did not pass.
|
|
40
|
+
* attached, and that the gaps list the participants who did not pass. A participant the run
|
|
41
|
+
* skipped before its session started never used the app, so it is counted in a sentence of its
|
|
42
|
+
* own.
|
|
41
43
|
*/
|
|
42
44
|
function fanoutSummary(args, facts) {
|
|
43
45
|
const count = args.specs.length;
|
|
44
46
|
const people = plural(count, "participant");
|
|
45
47
|
const rerun = args.rerun === undefined ? "" : `Rerun from ${args.rerun.sourceRunId}: `;
|
|
48
|
+
// The live summary is written once, before any participant starts, so it names the plan: by
|
|
49
|
+
// the time it is read a participant may still be waiting, or skipped.
|
|
46
50
|
if (args.inProgress === true) {
|
|
47
|
-
return `${people} ${count === 1 ? "is" : "are
|
|
51
|
+
return `${people} ${count === 1 ? "is planned, in its own copy" : "are planned, each in their own copy"} of the app. The run is still going, so nothing here is final.`;
|
|
48
52
|
}
|
|
49
53
|
if (args.dryRun) {
|
|
50
54
|
return `${rerun}Dry run: ${people} would each use their own copy of ${subjectName(publicSafeAppUrlLabel(args.appUrl), args.aggregateSubject)}. No desktops were launched and $0 was spent.`;
|
|
@@ -52,7 +56,20 @@ function fanoutSummary(args, facts) {
|
|
|
52
56
|
const tasks = facts.tasks === undefined ? "" : ` Tasks: ${facts.tasks}.`;
|
|
53
57
|
// A host suspension comes first: it may explain everything after it.
|
|
54
58
|
const host = args.hostSuspension === undefined ? "" : `${args.hostSuspension.summary} `;
|
|
55
|
-
|
|
59
|
+
const tookPart = count - facts.skipped;
|
|
60
|
+
const outcomes = facts.outcomes ?? `0/${tookPart} finished a session`;
|
|
61
|
+
const gaps = gapsListClause(tookPart - facts.passed, tookPart);
|
|
62
|
+
const sentences = [];
|
|
63
|
+
// A run where every participant was skipped has no one to describe in this sentence.
|
|
64
|
+
if (tookPart > 0)
|
|
65
|
+
sentences.push(`${plural(tookPart, "participant")} took part, ${tookPart === 1 ? "in its own copy" : "each in their own copy"} of the app: ${outcomes}${gaps}.`);
|
|
66
|
+
// The gaps are pointed at once: here only when the sentence before did not.
|
|
67
|
+
const why = gaps === "" ? "; the gaps say why" : "";
|
|
68
|
+
if (facts.skipped === 1)
|
|
69
|
+
sentences.push(`1 participant was skipped before its session started${why}.`);
|
|
70
|
+
if (facts.skipped > 1)
|
|
71
|
+
sentences.push(`${facts.skipped} participants were skipped before their sessions started${why}.`);
|
|
72
|
+
return `${host}${rerun}${sentences.join(" ")}${tasks}`;
|
|
56
73
|
}
|
|
57
74
|
/**
|
|
58
75
|
* The review's gaps: one line for the participants a host suspension likely failed, then a line
|
|
@@ -75,6 +92,7 @@ function fanoutReview(args, streams, judgment) {
|
|
|
75
92
|
// The judge's verdict: live fan-out must prove every participant (judgeParticipants).
|
|
76
93
|
const verdict = args.verdict;
|
|
77
94
|
const passedParticipants = judgment.participants.filter((participant) => participant.judgedStatus === "passed").length;
|
|
95
|
+
const skippedParticipants = (outcomes ?? []).filter((outcome) => outcome?.skippedReason !== undefined).length;
|
|
78
96
|
// What happened to the participants, with the denominator attached. The verdict above has to
|
|
79
97
|
// collapse the run to one word; this does not (docs/principles/three-roles.md).
|
|
80
98
|
const terminalOutcomes = (outcomes ?? []).filter((outcome) => outcome?.session?.status !== undefined);
|
|
@@ -103,9 +121,10 @@ function fanoutReview(args, streams, judgment) {
|
|
|
103
121
|
// sandbox, so the summary carries its digest, which verify accepts in a shared run.
|
|
104
122
|
summary: fanoutSummary(args, {
|
|
105
123
|
passed: passedParticipants,
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
124
|
+
skipped: skippedParticipants,
|
|
125
|
+
...(participants === undefined
|
|
126
|
+
? {}
|
|
127
|
+
: { outcomes: formatParticipantOutcomes(participants, participantEndings) }),
|
|
109
128
|
...(runTasks === undefined ? {} : { tasks: formatRunTaskFunnel(runTasks) }),
|
|
110
129
|
}),
|
|
111
130
|
gaps: args.inProgress === true
|
|
@@ -352,14 +352,18 @@ this paragraph is the record of that decision.
|
|
|
352
352
|
|
|
353
353
|
`ComputerUseProvider.debrief` is an optional read-only request after a structured
|
|
354
354
|
`stopWhen` or dwell stop. It returns a `ComputerUseTurn` with `closingReport` containing
|
|
355
|
-
`summary`, `frictionReports` and `impressions`; an empty friction or impressions list
|
|
356
|
-
and a report without `impressions` (an older provider) is still accepted.
|
|
355
|
+
`outcome`, `summary`, `frictionReports` and `impressions`; an empty friction or impressions list
|
|
356
|
+
is valid, and a report without `outcome` or `impressions` (an older provider) is still accepted.
|
|
357
|
+
After a scheduled end or session cap, `outcome` (`reached`, `not_reached` or `blocked`) is the
|
|
358
|
+
trace's `declaredOutcome`, read before the first line of `summary`. The loop rejects
|
|
357
359
|
actions, pending safety checks, and invalid report shapes. It redacts accepted
|
|
358
360
|
reports, records them in `ActorTrace.debrief`, and projects one readable message
|
|
359
361
|
without invoking action or communication callbacks. Each impression becomes its own message
|
|
360
362
|
and goes to `ActorTrace.impressions`; a session without them records why in the same field.
|
|
361
363
|
A turn that ends the session itself may carry `closingReport` too, and the loop keeps its
|
|
362
|
-
impressions the same way without a second request.
|
|
364
|
+
impressions the same way without a second request. Its `declaredOutcome` is the report's
|
|
365
|
+
`outcome`, then the turn's `outcome`, then the first line of its message, the order a closing
|
|
366
|
+
report at a scheduled end is read in.
|
|
363
367
|
|
|
364
368
|
`ComputerUseProvider.requestImpressions` is the optional request after the participant ended
|
|
365
369
|
the session itself without impressions. It asks for impressions only, returns them as
|