ai-sdk-harness-sessions 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README.md +29 -2
- package/dist/definitions/manager.d.ts +21 -1
- package/dist/definitions/session.d.ts +11 -0
- package/dist/manager/live-session.d.ts +8 -1
- package/dist/manager/session-manager.js +134 -25
- package/dist/turns/approval-answers.d.ts +10 -0
- package/dist/turns/approval-answers.js +22 -0
- package/dist/turns/pending-input.js +24 -0
- package/dist/turns/turn-stream.d.ts +19 -5
- package/dist/turns/turn-stream.js +123 -31
- package/package.json +4 -4
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# ai-sdk-harness-sessions
|
|
2
2
|
|
|
3
|
+
## 0.2.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 05ff36c: The `approve` option answers the approvals a turn asks for before anyone sees them — an approval policy, the grants of the session: a verdict is sent at once and the turn goes on in the same stream, the client seeing the answer; `undefined` leaves the approval to a person. When it answers part of a pause, `pendingInput.approvals` carries its answers in `decision`, and `continue()` sends them with the person's. An approval asked again right after it was answered, with the same input, is left to a person. `update(id, { metadata })` changes the metadata of a session, whatever its status. A session resumed without the turn it waited in tells the agent the answers to its questions in words.
|
|
8
|
+
|
|
3
9
|
## 0.1.0
|
|
4
10
|
|
|
5
11
|
### Minor Changes
|
package/README.md
CHANGED
|
@@ -41,7 +41,7 @@ return turn.toUIMessageStreamResponse();
|
|
|
41
41
|
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
|
|
42
42
|
| A first message | Opens a sandbox, or a view of a shared one, and a harness session, in the background: a message sent meanwhile waits |
|
|
43
43
|
| A message while a turn runs | Refuses it: one turn at a time per session (`SessionConflictError`, status `busy`) |
|
|
44
|
-
| The turn asks approval for a tool call |
|
|
44
|
+
| The turn asks approval for a tool call | Asks your `approve` option first, then keeps the session `awaiting-input` until `continue()` brings the answer |
|
|
45
45
|
| Nobody writes for `idleTimeoutMs` | Suspends the session: its harness session stops with its state, its sandbox is freed. The next message resumes it |
|
|
46
46
|
| Every sandbox slot is taken | Suspends the session idle the longest to make room |
|
|
47
47
|
| Your server stops (`shutdown()`) | Cuts the turns under way short, and suspends every session: the next start resumes them on their next message |
|
|
@@ -226,6 +226,31 @@ const turn =
|
|
|
226
226
|
|
|
227
227
|
A session suspended while it waited keeps what it waits for: `continue()` resumes the session, without its paused turn, whose harness bridge the suspension stopped, and tells the agent the answers in a message. With Claude Code, see [Limitations](#limitations).
|
|
228
228
|
|
|
229
|
+
The agent's questions wait the same way: Claude Code's `AskUserQuestion` reaches you as a call of `askUserQuestions` in `pendingInput.toolCalls`, and its answers are the tool's output, `addToolOutput()` in `useChat`, `toolResultContinuations` otherwise. A session resumed without its turn tells the agent the answers in words, the labels of the options picked.
|
|
230
|
+
|
|
231
|
+
### Answered by a policy
|
|
232
|
+
|
|
233
|
+
The `approve` option answers the approvals before anyone sees them. A verdict is sent at once and the turn goes on, in the same stream: the client sees the request, then its answer, then what the agent did. `undefined` leaves the approval to a person. [`ai-sdk-harness-approval`](https://ai-sdk-harness.pages.dev/docs/packages/harness-approval) gives one from a policy, with the "always allow" of each session kept in its metadata:
|
|
234
|
+
|
|
235
|
+
```ts
|
|
236
|
+
import { type ApprovalGrant, defineApprovalPolicy } from 'ai-sdk-harness-approval';
|
|
237
|
+
|
|
238
|
+
const policy = defineApprovalPolicy({
|
|
239
|
+
commands: { allow: ['ls', 'git status', 'pnpm test'], deny: ['git push', 'rm -rf'] },
|
|
240
|
+
});
|
|
241
|
+
|
|
242
|
+
const sessions = createSessionManager<{ grants?: ApprovalGrant[] }>({
|
|
243
|
+
agent: () => new HarnessAgent({ harness, ...policy.agentSettings(harness) }),
|
|
244
|
+
sandboxes,
|
|
245
|
+
approve: policy.approver({ grants: ({ session }) => session.metadata.grants }),
|
|
246
|
+
});
|
|
247
|
+
|
|
248
|
+
// "Always allow": kept with the session, read by the next approvals.
|
|
249
|
+
await sessions.update(id, { metadata: (metadata) => ({ ...metadata, grants }) });
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
When a pause holds several approvals and `approve` answers some, the session waits for the others: `pendingInput.approvals` carries the answers given in `decision`, and `continue()` sends them with the person's, which cannot overturn them. An approval asked again right after `approve` answered it, with the same input, is left to a person: a harness that does not take the answer would otherwise ask forever.
|
|
253
|
+
|
|
229
254
|
## Sandboxes
|
|
230
255
|
|
|
231
256
|
Where each session runs is up to `sandboxes`:
|
|
@@ -313,6 +338,7 @@ Keep one manager per store: at its start, a manager takes the sessions the store
|
|
|
313
338
|
| `closeSuspendedAfterMs` | never | Close a session suspended for this long |
|
|
314
339
|
| `turnTimeoutMs` | never | Cut a turn short after this long |
|
|
315
340
|
| `suspendIdleWhenFull` | `true` | When every sandbox slot is taken, suspend the session idle the longest rather than refuse a new one |
|
|
341
|
+
| `approve` | none | Answers an approval before anyone sees it: `{ approved, reason }`, or `undefined` to ask a person |
|
|
316
342
|
| `generateId` | `crypto.randomUUID` | The ids of new sessions and of the messages of their turns |
|
|
317
343
|
| `errorMessage` | `getHarnessErrorMessage` | What a client is told of an error in a turn; the session's `error` keeps the full message |
|
|
318
344
|
| `onError` | `console.error` | Called with what fails in the background: a start, a suspension, a write to the store |
|
|
@@ -322,6 +348,7 @@ Keep one manager per store: at its start, a manager takes the sessions the store
|
|
|
322
348
|
| `create({ id, metadata })` | Opens a session, and returns it `preparing` |
|
|
323
349
|
| `send(id, options)` | Starts a turn in reply to a message (`message`, `prompt`, `abortSignal`): a `SessionTurn` |
|
|
324
350
|
| `continue(id, options)` | Resumes a paused turn with its answers (`message`, `toolApprovalContinuations`, `toolResultContinuations`) |
|
|
351
|
+
| `update(id, { metadata })` | Changes the metadata of a session, whatever its status: a value, or a function of the current one |
|
|
325
352
|
| `interrupt(id)` | Cuts the turn under way short |
|
|
326
353
|
| `suspend(id)` | Suspends an idle session now |
|
|
327
354
|
| `close(id)` | Ends a session for good |
|
|
@@ -350,7 +377,7 @@ Each has an `isInstance()` that recognizes it whichever copy of the package thre
|
|
|
350
377
|
|
|
351
378
|
## Development
|
|
352
379
|
|
|
353
|
-
`pnpm --filter ai-sdk-harness-sessions test:e2e` runs real Claude Code sessions in a Docker Sandbox: two side by side, suspended and resumed, across a restart, and waiting for an approval; it needs `sbx` signed in and a Claude credential. See [CONTRIBUTING.md](https://github.com/fpasquet/ai-sdk-harness/blob/main/CONTRIBUTING.md).
|
|
380
|
+
`pnpm --filter ai-sdk-harness-sessions test:e2e` runs real Claude Code sessions in a Docker Sandbox: two side by side, suspended and resumed, across a restart, and waiting for an approval; it needs `sbx` signed in and a Claude credential. `ai-sdk-harness-approval`'s end-to-end tests cover `approve`. See [CONTRIBUTING.md](https://github.com/fpasquet/ai-sdk-harness/blob/main/CONTRIBUTING.md).
|
|
354
381
|
|
|
355
382
|
## License
|
|
356
383
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import type { ToolApprovalResponse, ToolResultPart, UIMessage, UIMessageChunk } from 'ai';
|
|
2
2
|
import type { SessionAgent } from './agent.js';
|
|
3
3
|
import type { SessionSandboxes } from './sandboxes.js';
|
|
4
|
-
import type { SessionRecord, SessionSummary } from './session.js';
|
|
4
|
+
import type { ApprovalVerdict, PendingApproval, SessionRecord, SessionSummary } from './session.js';
|
|
5
5
|
import type { SessionStore } from './store.js';
|
|
6
6
|
export interface SessionManagerOptions<METADATA> {
|
|
7
7
|
/**
|
|
@@ -32,6 +32,18 @@ export interface SessionManagerOptions<METADATA> {
|
|
|
32
32
|
* one, or a resume, rather than refusing it with `SessionCapacityError`. Default: `true`.
|
|
33
33
|
*/
|
|
34
34
|
suspendIdleWhenFull?: boolean;
|
|
35
|
+
/**
|
|
36
|
+
* Answers the approvals a turn asks for before anyone sees them: an approval policy, the grants
|
|
37
|
+
* of the session. A verdict is sent at once and the turn goes on, in the same stream, the client
|
|
38
|
+
* seeing the answer; `undefined` leaves the approval to a person, the session `awaiting-input`.
|
|
39
|
+
* Default: every approval waits for a person.
|
|
40
|
+
*
|
|
41
|
+
* An approval asked again right after it was answered, with the same input, is left to a
|
|
42
|
+
* person: a harness that does not take the answer (see the README) would ask forever.
|
|
43
|
+
*/
|
|
44
|
+
approve?: (approval: PendingApproval, context: {
|
|
45
|
+
session: SessionSummary<METADATA>;
|
|
46
|
+
}) => ApprovalVerdict | PromiseLike<ApprovalVerdict | undefined> | undefined;
|
|
35
47
|
/** The ids of new sessions and of the messages of their turns. Default: `crypto.randomUUID`. */
|
|
36
48
|
generateId?: () => string;
|
|
37
49
|
/**
|
|
@@ -118,6 +130,14 @@ export interface SessionManager<METADATA = unknown> {
|
|
|
118
130
|
* resumed first, its turn with it.
|
|
119
131
|
*/
|
|
120
132
|
continue(id: string, options: ContinueOptions): Promise<SessionTurn<METADATA>>;
|
|
133
|
+
/**
|
|
134
|
+
* Changes what your application keeps with a session, its metadata, whatever its status: a
|
|
135
|
+
* title, the grants of an approval policy… Given a function, it is called with the metadata as
|
|
136
|
+
* they stand.
|
|
137
|
+
*/
|
|
138
|
+
update(id: string, changes: {
|
|
139
|
+
metadata: ((metadata: METADATA) => METADATA) | METADATA;
|
|
140
|
+
}): Promise<SessionSummary<METADATA>>;
|
|
121
141
|
/** Cuts the turn under way short, and resolves once the session is written. */
|
|
122
142
|
interrupt(id: string): Promise<void>;
|
|
123
143
|
/** Suspends a session idle or awaiting input, as idleness would. */
|
|
@@ -29,6 +29,12 @@ export interface SessionUsage {
|
|
|
29
29
|
outputTokens: number;
|
|
30
30
|
totalTokens: number;
|
|
31
31
|
}
|
|
32
|
+
/** An answer to an approval request, as `ToolApprovalResponse` carries it. */
|
|
33
|
+
export interface ApprovalVerdict {
|
|
34
|
+
approved: boolean;
|
|
35
|
+
/** Why: what the agent is told of a denial, what a person reads of an automatic approval. */
|
|
36
|
+
reason?: string;
|
|
37
|
+
}
|
|
32
38
|
/** A tool call the agent asked approval for, waiting for your answer. */
|
|
33
39
|
export interface PendingApproval {
|
|
34
40
|
/** What the answer refers to: `ToolApprovalResponse.approvalId`. */
|
|
@@ -36,6 +42,11 @@ export interface PendingApproval {
|
|
|
36
42
|
toolCallId: string;
|
|
37
43
|
toolName: string;
|
|
38
44
|
input: unknown;
|
|
45
|
+
/**
|
|
46
|
+
* The answer the manager's `approve` option gave already, when it gave one: the turn waits for
|
|
47
|
+
* the answers to the other approvals, and `continue()` sends this one with them.
|
|
48
|
+
*/
|
|
49
|
+
decision?: ApprovalVerdict;
|
|
39
50
|
}
|
|
40
51
|
/** A tool call whose result your application gives, waiting for it. */
|
|
41
52
|
export interface PendingToolCall {
|
|
@@ -2,7 +2,7 @@ import type { HarnessAgentSession } from '@ai-sdk/harness/agent';
|
|
|
2
2
|
import type { UIMessage } from 'ai';
|
|
3
3
|
import type { SessionAgent } from '../definitions/agent.js';
|
|
4
4
|
import type { SessionSandboxLease } from '../definitions/sandboxes.js';
|
|
5
|
-
import type { SessionRecord, SessionStatus, SessionSummary } from '../definitions/session.js';
|
|
5
|
+
import type { PendingInput, SessionRecord, SessionStatus, SessionSummary } from '../definitions/session.js';
|
|
6
6
|
/** A session held in memory, with its harness session and its sandbox once it started. */
|
|
7
7
|
export interface LiveSession<METADATA> {
|
|
8
8
|
record: SessionRecord<METADATA>;
|
|
@@ -18,10 +18,17 @@ export interface LiveSession<METADATA> {
|
|
|
18
18
|
* agent the answers in a message.
|
|
19
19
|
*/
|
|
20
20
|
lostTurn?: boolean;
|
|
21
|
+
/**
|
|
22
|
+
* What the turn paused on, when the manager's `approve` answered part of it: the approvals with
|
|
23
|
+
* the verdicts it gave, for the record.
|
|
24
|
+
*/
|
|
25
|
+
paused?: PendingInput;
|
|
21
26
|
turn?: {
|
|
22
27
|
abort: AbortController;
|
|
23
28
|
done: Promise<void>;
|
|
24
29
|
};
|
|
30
|
+
/** Settles once a close, or a shutdown, that took the session over has written it. */
|
|
31
|
+
ending?: Promise<void>;
|
|
25
32
|
idleTimer?: NodeJS.Timeout;
|
|
26
33
|
}
|
|
27
34
|
/** A live session that started: its agent and harness session are there. */
|
|
@@ -5,6 +5,7 @@ import { SessionCapacityError } from '../errors/session-capacity-error.js';
|
|
|
5
5
|
import { SessionConflictError } from '../errors/session-conflict-error.js';
|
|
6
6
|
import { SessionNotFoundError } from '../errors/session-not-found-error.js';
|
|
7
7
|
import { createMemorySessionStore } from '../store/memory-session-store.js';
|
|
8
|
+
import { decidedChunks, decidedResponses, signatureOf, verdictOf, } from '../turns/approval-answers.js';
|
|
8
9
|
import { describeResponses, pendingInputOf, responsesFrom, withResponses, } from '../turns/pending-input.js';
|
|
9
10
|
import { streamTurn } from '../turns/turn-stream.js';
|
|
10
11
|
import { isLive, isReady, liveSession, messageOf, newRecord, summaryOf, textOf, } from './live-session.js';
|
|
@@ -16,18 +17,33 @@ const NOT_STOPPED = 'The harness session could not be stopped cleanly.';
|
|
|
16
17
|
const NOTHING_TO_RESUME = 'Nothing was kept to resume the session from.';
|
|
17
18
|
/** How long a shutdown waits for a turn it cut short to wind down. */
|
|
18
19
|
const WIND_DOWN_MS = 10_000;
|
|
19
|
-
/**
|
|
20
|
+
/** How many approvals `approve` answers in one turn, at most, before leaving them to a person. */
|
|
21
|
+
const MAX_ANSWERED_PER_TURN = 200;
|
|
22
|
+
/**
|
|
23
|
+
* The answers `continue()` was given, read from `useChat`'s message and given as they are, and
|
|
24
|
+
* the answers `approve` gave already: they stand, whatever the message says.
|
|
25
|
+
*/
|
|
20
26
|
function responsesOf(pending, { message, ...given }) {
|
|
21
|
-
const
|
|
27
|
+
const decided = decidedResponses(pending.approvals);
|
|
28
|
+
const open = {
|
|
29
|
+
...pending,
|
|
30
|
+
approvals: pending.approvals.filter(({ decision }) => decision === undefined),
|
|
31
|
+
};
|
|
32
|
+
const read = message === undefined ? undefined : responsesFrom(message, open);
|
|
33
|
+
const approvals = [
|
|
34
|
+
...(read?.toolApprovalContinuations ?? []),
|
|
35
|
+
...(given.toolApprovalContinuations ?? []),
|
|
36
|
+
].filter(({ approvalId }) => !decided.some((response) => response.approvalId === approvalId));
|
|
37
|
+
const results = [
|
|
38
|
+
...(read?.toolResultContinuations ?? []),
|
|
39
|
+
...(given.toolResultContinuations ?? []),
|
|
40
|
+
];
|
|
22
41
|
return {
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
...(read?.toolResultContinuations ?? []),
|
|
29
|
-
...(given.toolResultContinuations ?? []),
|
|
30
|
-
],
|
|
42
|
+
responses: {
|
|
43
|
+
toolApprovalContinuations: [...decided, ...approvals],
|
|
44
|
+
toolResultContinuations: results,
|
|
45
|
+
},
|
|
46
|
+
answered: approvals.length + results.length,
|
|
31
47
|
};
|
|
32
48
|
}
|
|
33
49
|
/**
|
|
@@ -149,9 +165,8 @@ class HarnessSessionManager {
|
|
|
149
165
|
this.claim(session, 'awaiting-input');
|
|
150
166
|
const { record } = session;
|
|
151
167
|
const pending = record.pendingInput ?? pendingInputOf(record.messages);
|
|
152
|
-
const responses = responsesOf(pending, answers);
|
|
153
|
-
if (
|
|
154
|
-
0) {
|
|
168
|
+
const { responses, answered } = responsesOf(pending, answers);
|
|
169
|
+
if (answered === 0) {
|
|
155
170
|
throw this.conflict(record, 'continue() carries no answer to what the session waits for.');
|
|
156
171
|
}
|
|
157
172
|
record.messages = withResponses(record.messages, responses);
|
|
@@ -167,6 +182,25 @@ class HarnessSessionManager {
|
|
|
167
182
|
})
|
|
168
183
|
: session.agent.stream({ session: session.handle, prompt, abortSignal: signal }), abortSignal);
|
|
169
184
|
};
|
|
185
|
+
update = async (id, { metadata }) => {
|
|
186
|
+
await this.ready();
|
|
187
|
+
const change = (current) => typeof metadata === 'function'
|
|
188
|
+
? metadata(current)
|
|
189
|
+
: metadata;
|
|
190
|
+
const live = this.live.get(id);
|
|
191
|
+
if (live !== undefined)
|
|
192
|
+
return this.changeMetadata(live.record, change(live.record.metadata));
|
|
193
|
+
const kept = await this.store.get(id);
|
|
194
|
+
if (kept === undefined)
|
|
195
|
+
throw new SessionNotFoundError(id);
|
|
196
|
+
const resumeState = kept.status === 'suspended' ? await this.store.getResumeState(id) : undefined;
|
|
197
|
+
const changed = change(kept.metadata);
|
|
198
|
+
// Brought back while the store was read: the live session is the one to change.
|
|
199
|
+
const racing = this.live.get(id);
|
|
200
|
+
if (racing !== undefined)
|
|
201
|
+
return this.changeMetadata(racing.record, changed);
|
|
202
|
+
return this.changeMetadata(kept, changed, resumeState);
|
|
203
|
+
};
|
|
170
204
|
interrupt = async (id) => {
|
|
171
205
|
await this.recovered;
|
|
172
206
|
const session = this.live.get(id);
|
|
@@ -203,12 +237,8 @@ class HarnessSessionManager {
|
|
|
203
237
|
const session = this.live.get(id);
|
|
204
238
|
await session?.ready.catch(() => undefined);
|
|
205
239
|
if (session !== undefined && this.live.get(id) === session) {
|
|
206
|
-
this.
|
|
207
|
-
|
|
208
|
-
await session.turn?.done;
|
|
209
|
-
await this.teardown(session);
|
|
210
|
-
delete session.record.pendingInput;
|
|
211
|
-
await this.transition(session, 'closed');
|
|
240
|
+
session.ending = this.end(session);
|
|
241
|
+
await session.ending;
|
|
212
242
|
return summaryOf(session.record);
|
|
213
243
|
}
|
|
214
244
|
const kept = await this.store.get(id);
|
|
@@ -262,6 +292,13 @@ class HarnessSessionManager {
|
|
|
262
292
|
this.journal.updated(record);
|
|
263
293
|
return persist ? this.journal.write(record, resumeState) : Promise.resolve();
|
|
264
294
|
}
|
|
295
|
+
/** Writes new metadata, without counting it as activity: `updatedAt` stays. */
|
|
296
|
+
async changeMetadata(record, metadata, resumeState) {
|
|
297
|
+
record.metadata = metadata;
|
|
298
|
+
this.journal.updated(record);
|
|
299
|
+
await this.journal.write(record, resumeState);
|
|
300
|
+
return summaryOf(record);
|
|
301
|
+
}
|
|
265
302
|
conflict(record, message) {
|
|
266
303
|
const said = message ?? `The session is ${record.status}${record.error ? `: ${record.error}` : ''}.`;
|
|
267
304
|
return new SessionConflictError(record.id, record.status, said);
|
|
@@ -459,21 +496,24 @@ class HarnessSessionManager {
|
|
|
459
496
|
throw error;
|
|
460
497
|
}
|
|
461
498
|
const { record } = session;
|
|
499
|
+
const answered = { last: new Set(), count: 0, abortSignal: abort.signal };
|
|
462
500
|
const { stream, settled } = streamTurn(result, {
|
|
463
501
|
originalMessages: record.messages,
|
|
464
502
|
generateMessageId: generateId,
|
|
465
503
|
errorMessage,
|
|
504
|
+
next: (messages) => this.answerPause(session, messages, answered),
|
|
466
505
|
onSettled: async ({ messages, failure, usage }) => {
|
|
467
506
|
if (messages !== undefined)
|
|
468
507
|
record.messages = messages;
|
|
469
508
|
record.turns += 1;
|
|
470
|
-
record.usage.inputTokens += usage
|
|
471
|
-
record.usage.outputTokens += usage
|
|
472
|
-
record.usage.totalTokens += usage
|
|
509
|
+
record.usage.inputTokens += usage.inputTokens ?? 0;
|
|
510
|
+
record.usage.outputTokens += usage.outputTokens ?? 0;
|
|
511
|
+
record.usage.totalTokens += usage.totalTokens ?? 0;
|
|
473
512
|
await settle(failure);
|
|
474
513
|
},
|
|
475
514
|
});
|
|
476
|
-
|
|
515
|
+
// A turn cut short by a close or a shutdown is over once they wrote the session.
|
|
516
|
+
const done = settled.then(() => session.ending).then(() => summaryOf(record));
|
|
477
517
|
done.catch(() => undefined);
|
|
478
518
|
return {
|
|
479
519
|
stream,
|
|
@@ -481,6 +521,61 @@ class HarnessSessionManager {
|
|
|
481
521
|
done,
|
|
482
522
|
};
|
|
483
523
|
}
|
|
524
|
+
/**
|
|
525
|
+
* What `approve` makes of the approvals a turn paused on: all answered, the turn goes on; some
|
|
526
|
+
* left to a person, it stops there, `continue()` sending the answers given with theirs.
|
|
527
|
+
*/
|
|
528
|
+
async answerPause(session, messages, answered) {
|
|
529
|
+
session.paused = undefined;
|
|
530
|
+
const pending = pendingInputOf(messages);
|
|
531
|
+
if (this.options.approve === undefined ||
|
|
532
|
+
pending.approvals.length === 0 ||
|
|
533
|
+
!session.handle.hasUnfinishedTurn()) {
|
|
534
|
+
return undefined;
|
|
535
|
+
}
|
|
536
|
+
const approvals = await Promise.all(pending.approvals.map((approval) => this.decide(session, approval, answered)));
|
|
537
|
+
const responses = decidedResponses(approvals);
|
|
538
|
+
if (responses.length === 0)
|
|
539
|
+
return undefined;
|
|
540
|
+
answered.last = new Set(approvals.filter(({ decision }) => decision).map(signatureOf));
|
|
541
|
+
answered.count += responses.length;
|
|
542
|
+
const step = {
|
|
543
|
+
chunks: decidedChunks(approvals),
|
|
544
|
+
messages: withResponses(messages, {
|
|
545
|
+
toolApprovalContinuations: responses,
|
|
546
|
+
toolResultContinuations: [],
|
|
547
|
+
}),
|
|
548
|
+
};
|
|
549
|
+
if (responses.length < approvals.length || pending.toolCalls.length > 0) {
|
|
550
|
+
session.paused = { ...pending, approvals };
|
|
551
|
+
return step;
|
|
552
|
+
}
|
|
553
|
+
const result = await session.agent.continueStream({
|
|
554
|
+
session: session.handle,
|
|
555
|
+
toolApprovalContinuations: responses,
|
|
556
|
+
abortSignal: answered.abortSignal,
|
|
557
|
+
});
|
|
558
|
+
return { ...step, result };
|
|
559
|
+
}
|
|
560
|
+
/** The approval with the verdict of `approve`, unless it is to be left to a person. */
|
|
561
|
+
async decide(session, approval, answered) {
|
|
562
|
+
const { approve } = this.options;
|
|
563
|
+
// Asked again right after it was answered: the harness did not take the answer.
|
|
564
|
+
if (approve === undefined ||
|
|
565
|
+
answered.last.has(signatureOf(approval)) ||
|
|
566
|
+
answered.count >= MAX_ANSWERED_PER_TURN) {
|
|
567
|
+
return approval;
|
|
568
|
+
}
|
|
569
|
+
let decision;
|
|
570
|
+
try {
|
|
571
|
+
decision = await approve(approval, { session: summaryOf(session.record) });
|
|
572
|
+
}
|
|
573
|
+
catch (error) {
|
|
574
|
+
// Left to a person: they decide, as without approve.
|
|
575
|
+
this.report('approve', session.record.id)(error);
|
|
576
|
+
}
|
|
577
|
+
return decision === undefined ? approval : { ...approval, decision: verdictOf(decision) };
|
|
578
|
+
}
|
|
484
579
|
/** After a turn: awaiting input if it paused, idle otherwise. */
|
|
485
580
|
async settleTurn(session, handle, failure) {
|
|
486
581
|
session.turn = undefined;
|
|
@@ -490,9 +585,10 @@ class HarnessSessionManager {
|
|
|
490
585
|
return;
|
|
491
586
|
const unfinished = handle.hasUnfinishedTurn();
|
|
492
587
|
if (unfinished)
|
|
493
|
-
record.pendingInput = pendingInputOf(record.messages);
|
|
588
|
+
record.pendingInput = session.paused ?? pendingInputOf(record.messages);
|
|
494
589
|
else
|
|
495
590
|
delete record.pendingInput;
|
|
591
|
+
session.paused = undefined;
|
|
496
592
|
await this.transition(session, unfinished ? 'awaiting-input' : 'idle', {
|
|
497
593
|
...(failure !== undefined && { error: failure }),
|
|
498
594
|
});
|
|
@@ -520,8 +616,21 @@ class HarnessSessionManager {
|
|
|
520
616
|
await this.closeKept(kept);
|
|
521
617
|
}
|
|
522
618
|
}
|
|
619
|
+
/** Closes a live session, cutting its turn short. */
|
|
620
|
+
async end(session) {
|
|
621
|
+
this.live.delete(session.record.id);
|
|
622
|
+
session.turn?.abort.abort(new Error('The session was closed.'));
|
|
623
|
+
await session.turn?.done;
|
|
624
|
+
await this.teardown(session);
|
|
625
|
+
delete session.record.pendingInput;
|
|
626
|
+
await this.transition(session, 'closed');
|
|
627
|
+
}
|
|
628
|
+
shutdownSession(session) {
|
|
629
|
+
session.ending = this.windDown(session);
|
|
630
|
+
return session.ending;
|
|
631
|
+
}
|
|
523
632
|
/** Suspends a live session for the shutdown, cutting its turn short. */
|
|
524
|
-
async
|
|
633
|
+
async windDown(session) {
|
|
525
634
|
const { record } = session;
|
|
526
635
|
if (record.status === 'preparing') {
|
|
527
636
|
// Its start finds it gone and cleans up after itself; a resume parks it again.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
import type { ToolApprovalResponse, UIMessageChunk } from 'ai';
|
|
2
|
+
import type { ApprovalVerdict, PendingApproval } from '../definitions/session.js';
|
|
3
|
+
/** An approval request as a repeat of another is recognised: the same tool, the same input. */
|
|
4
|
+
export declare const signatureOf: ({ toolName, input }: PendingApproval) => string;
|
|
5
|
+
/** A verdict with only what an approval response carries. */
|
|
6
|
+
export declare const verdictOf: ({ approved, reason }: ApprovalVerdict) => ApprovalVerdict;
|
|
7
|
+
/** The approval responses of the approvals decided already. */
|
|
8
|
+
export declare function decidedResponses(approvals: readonly PendingApproval[]): ToolApprovalResponse[];
|
|
9
|
+
/** The chunks that show a client the approvals decided already, as answered. */
|
|
10
|
+
export declare function decidedChunks(approvals: readonly PendingApproval[]): UIMessageChunk[];
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
/** An approval request as a repeat of another is recognised: the same tool, the same input. */
|
|
2
|
+
export const signatureOf = ({ toolName, input }) => `${toolName}\u0000${JSON.stringify(input)}`;
|
|
3
|
+
/** A verdict with only what an approval response carries. */
|
|
4
|
+
export const verdictOf = ({ approved, reason }) => ({
|
|
5
|
+
approved,
|
|
6
|
+
...(reason !== undefined && { reason }),
|
|
7
|
+
});
|
|
8
|
+
/** The approval responses of the approvals decided already. */
|
|
9
|
+
export function decidedResponses(approvals) {
|
|
10
|
+
return approvals.flatMap(({ approvalId, decision }) => decision === undefined
|
|
11
|
+
? []
|
|
12
|
+
: [{ type: 'tool-approval-response', approvalId, ...verdictOf(decision) }]);
|
|
13
|
+
}
|
|
14
|
+
/** The chunks that show a client the approvals decided already, as answered. */
|
|
15
|
+
export function decidedChunks(approvals) {
|
|
16
|
+
return decidedResponses(approvals).map(({ approvalId, approved, reason }) => ({
|
|
17
|
+
type: 'tool-approval-response',
|
|
18
|
+
approvalId,
|
|
19
|
+
approved,
|
|
20
|
+
...(reason !== undefined && { reason }),
|
|
21
|
+
}));
|
|
22
|
+
}
|
|
@@ -115,6 +115,27 @@ export function withResponses(messages, responses) {
|
|
|
115
115
|
}),
|
|
116
116
|
});
|
|
117
117
|
}
|
|
118
|
+
const isQuestions = (input) => Array.isArray(input?.questions);
|
|
119
|
+
const isAnswers = (output) => typeof output?.action === 'string';
|
|
120
|
+
/**
|
|
121
|
+
* The answers to the agent's questions, `askUserQuestions`, in words: the labels of the options
|
|
122
|
+
* picked rather than their ids, which the agent no longer sees once its turn was lost.
|
|
123
|
+
*/
|
|
124
|
+
function answersOf(input, output) {
|
|
125
|
+
if (!isQuestions(input) || !isAnswers(output))
|
|
126
|
+
return undefined;
|
|
127
|
+
if (output.answers === undefined)
|
|
128
|
+
return `- I ${output.action} your questions.`;
|
|
129
|
+
const { answers } = output;
|
|
130
|
+
return input.questions
|
|
131
|
+
.map(({ id, question, options = [] }) => {
|
|
132
|
+
const answer = answers[id];
|
|
133
|
+
const picked = (answer?.optionIds ?? []).map((optionId) => options.find((option) => option.id === optionId)?.label ?? optionId);
|
|
134
|
+
const said = [...picked, ...(answer?.freeform ? [answer.freeform] : [])].join('; ');
|
|
135
|
+
return `- ${question} — ${said || 'no answer'}`;
|
|
136
|
+
})
|
|
137
|
+
.join('\n');
|
|
138
|
+
}
|
|
118
139
|
/** A short form of a tool call's input, for a message. */
|
|
119
140
|
const inputOf = (input) => {
|
|
120
141
|
const text = typeof input === 'string' ? input : JSON.stringify(input);
|
|
@@ -134,6 +155,9 @@ export function describeResponses(pending, responses) {
|
|
|
134
155
|
const results = responses.toolResultContinuations.map(({ toolCallId, toolName, output }) => {
|
|
135
156
|
const call = pending.toolCalls.find((toolCall) => toolCall.toolCallId === toolCallId);
|
|
136
157
|
const shown = shownOutput(output);
|
|
158
|
+
const answers = shown.error === undefined ? answersOf(call?.input, shown.value) : undefined;
|
|
159
|
+
if (answers !== undefined)
|
|
160
|
+
return answers;
|
|
137
161
|
const value = shown.error === undefined ? inputOf(shown.value) : `error: ${shown.error}`;
|
|
138
162
|
return `- ${call?.toolName ?? toolName}: ${value}`;
|
|
139
163
|
});
|
|
@@ -1,12 +1,23 @@
|
|
|
1
1
|
import type { LanguageModelUsage, UIMessage, UIMessageChunk } from 'ai';
|
|
2
2
|
import type { SessionStreamResult } from '../definitions/agent.js';
|
|
3
|
+
/** The tokens a turn used, added up over its streams. */
|
|
4
|
+
export type TurnUsage = Pick<LanguageModelUsage, 'inputTokens' | 'outputTokens' | 'totalTokens'>;
|
|
3
5
|
/** How a turn ended. */
|
|
4
6
|
export interface TurnOutcome {
|
|
5
7
|
/** The conversation with the turn's answer, when the stream got far enough to give it. */
|
|
6
8
|
messages: UIMessage[] | undefined;
|
|
7
9
|
/** What went wrong, when the turn failed or was cut short. */
|
|
8
10
|
failure: string | undefined;
|
|
9
|
-
usage:
|
|
11
|
+
usage: TurnUsage;
|
|
12
|
+
}
|
|
13
|
+
/**
|
|
14
|
+
* What a turn does once one of its streams ends: answers given in between, sent to the client and
|
|
15
|
+
* written into the conversation, and the stream that continues the turn, if it goes on.
|
|
16
|
+
*/
|
|
17
|
+
export interface TurnStep {
|
|
18
|
+
chunks: UIMessageChunk[];
|
|
19
|
+
messages: UIMessage[];
|
|
20
|
+
result?: SessionStreamResult;
|
|
10
21
|
}
|
|
11
22
|
export interface TurnStreamOptions {
|
|
12
23
|
/** The conversation the turn adds to: its last message is the one the turn answers. */
|
|
@@ -14,14 +25,17 @@ export interface TurnStreamOptions {
|
|
|
14
25
|
generateMessageId: () => string;
|
|
15
26
|
/** What the client is told of an error; the outcome keeps its real message. */
|
|
16
27
|
errorMessage: (error: unknown) => string;
|
|
17
|
-
/**
|
|
28
|
+
/** What follows a stream of the turn that ended with its conversation. Default: the turn ends. */
|
|
29
|
+
next?: (messages: UIMessage[]) => Promise<TurnStep | undefined>;
|
|
30
|
+
/** Runs once, when the turn is over, before the client's stream closes. */
|
|
18
31
|
onSettled: (outcome: TurnOutcome) => Promise<void>;
|
|
19
32
|
}
|
|
20
33
|
/**
|
|
21
|
-
* The turn as a UI message stream
|
|
22
|
-
* client that goes away does not leave the turn hanging, and the outcome is
|
|
34
|
+
* The turn as a UI message stream for the client, read to its end on this side whether the client
|
|
35
|
+
* reads it or not: a client that goes away does not leave the turn hanging, and the outcome is
|
|
36
|
+
* settled either way. A turn may run several streams, one after the other, `next` deciding.
|
|
23
37
|
*/
|
|
24
|
-
export declare function streamTurn(
|
|
38
|
+
export declare function streamTurn(first: SessionStreamResult, options: TurnStreamOptions): {
|
|
25
39
|
stream: ReadableStream<UIMessageChunk>;
|
|
26
40
|
settled: Promise<void>;
|
|
27
41
|
};
|
|
@@ -1,48 +1,140 @@
|
|
|
1
1
|
const messageOf = (error) => error instanceof Error ? error.message : String(error);
|
|
2
|
-
/**
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
let settled;
|
|
10
|
-
const settle = () => {
|
|
11
|
-
settled ??= (async () => {
|
|
12
|
-
const usage = await Promise.resolve(result.totalUsage).catch(() => undefined);
|
|
13
|
-
await options.onSettled({ messages, failure, usage });
|
|
14
|
-
})();
|
|
15
|
-
return settled;
|
|
2
|
+
/** Reads one stream of the turn to its end, sending its chunks on. */
|
|
3
|
+
async function readSegment(result, originalMessages, { first, send, options, }) {
|
|
4
|
+
const outcome = {
|
|
5
|
+
messages: undefined,
|
|
6
|
+
failure: undefined,
|
|
7
|
+
finish: undefined,
|
|
8
|
+
requests: [],
|
|
16
9
|
};
|
|
17
|
-
const
|
|
10
|
+
const holding = options.next !== undefined;
|
|
11
|
+
const reader = result
|
|
18
12
|
.toUIMessageStream({
|
|
19
|
-
originalMessages
|
|
13
|
+
originalMessages,
|
|
20
14
|
// Without it, a turn that continues a message the client cannot tell apart from the others.
|
|
21
15
|
generateMessageId: options.generateMessageId,
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
messages = finished;
|
|
25
|
-
await settle();
|
|
16
|
+
onFinish: ({ messages }) => {
|
|
17
|
+
outcome.messages = messages;
|
|
26
18
|
},
|
|
27
19
|
// Called again with the text it returned, as the error chunk is read: the first is the one.
|
|
28
20
|
onError: (error) => {
|
|
29
|
-
failure ??= messageOf(error);
|
|
21
|
+
outcome.failure ??= messageOf(error);
|
|
30
22
|
return options.errorMessage(error);
|
|
31
23
|
},
|
|
32
24
|
})
|
|
33
|
-
.
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
;
|
|
25
|
+
.getReader();
|
|
26
|
+
try {
|
|
27
|
+
for (let read = await reader.read(); !read.done; read = await reader.read()) {
|
|
28
|
+
const chunk = read.value;
|
|
29
|
+
if (chunk.type === 'finish')
|
|
30
|
+
outcome.finish = chunk;
|
|
31
|
+
else if (holding && chunk.type === 'tool-approval-request')
|
|
32
|
+
outcome.requests.push(chunk);
|
|
33
|
+
// A stream that continues the turn continues the client's message: it is already open.
|
|
34
|
+
else if (first || chunk.type !== 'start')
|
|
35
|
+
send(chunk);
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
catch (error) {
|
|
39
|
+
outcome.failure ??= messageOf(error);
|
|
40
|
+
}
|
|
41
|
+
return outcome;
|
|
42
|
+
}
|
|
43
|
+
/** Adds the tokens a stream of the turn used, once it is over. */
|
|
44
|
+
async function addUsage(usage, result) {
|
|
45
|
+
const spent = await Promise.resolve(result.totalUsage).catch(() => undefined);
|
|
46
|
+
for (const key of ['inputTokens', 'outputTokens', 'totalTokens']) {
|
|
47
|
+
usage[key] = (usage[key] ?? 0) + (spent?.[key] ?? 0);
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
/** Runs a turn's streams one after the other, sending their chunks to the client. */
|
|
51
|
+
class TurnRun {
|
|
52
|
+
options;
|
|
53
|
+
send;
|
|
54
|
+
usage = { inputTokens: 0, outputTokens: 0, totalTokens: 0 };
|
|
55
|
+
messages;
|
|
56
|
+
failure;
|
|
57
|
+
finish;
|
|
58
|
+
constructor(options, send) {
|
|
59
|
+
this.options = options;
|
|
60
|
+
this.send = send;
|
|
61
|
+
}
|
|
62
|
+
async run(first) {
|
|
63
|
+
let result = first;
|
|
64
|
+
let originalMessages = this.options.originalMessages;
|
|
65
|
+
for (let index = 0; result !== undefined; index += 1) {
|
|
66
|
+
const outcome = await readSegment(result, originalMessages, {
|
|
67
|
+
first: index === 0,
|
|
68
|
+
send: this.send,
|
|
69
|
+
options: this.options,
|
|
70
|
+
});
|
|
71
|
+
await addUsage(this.usage, result);
|
|
72
|
+
this.failure ??= outcome.failure;
|
|
73
|
+
this.finish = outcome.finish;
|
|
74
|
+
this.messages = outcome.messages ?? this.messages;
|
|
75
|
+
const step = await this.next(outcome.messages, outcome.requests);
|
|
76
|
+
result = step?.result;
|
|
77
|
+
if (step)
|
|
78
|
+
originalMessages = this.messages = step.messages;
|
|
79
|
+
}
|
|
80
|
+
if (this.finish !== undefined)
|
|
81
|
+
this.send(this.finish);
|
|
82
|
+
return { messages: this.messages, failure: this.failure, usage: this.usage };
|
|
83
|
+
}
|
|
84
|
+
/**
|
|
85
|
+
* What follows a stream that ended well, its approval requests sent first, with the answers
|
|
86
|
+
* `next` gave them; nothing after a failure.
|
|
87
|
+
*/
|
|
88
|
+
async next(messages, requests) {
|
|
89
|
+
let step;
|
|
90
|
+
let failed;
|
|
91
|
+
if (this.failure === undefined && messages !== undefined && this.options.next) {
|
|
92
|
+
try {
|
|
93
|
+
step = await this.options.next(messages);
|
|
94
|
+
}
|
|
95
|
+
catch (error) {
|
|
96
|
+
failed = error;
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
for (const chunk of [...requests, ...(step?.chunks ?? [])])
|
|
100
|
+
this.send(chunk);
|
|
101
|
+
if (failed !== undefined) {
|
|
102
|
+
this.failure = messageOf(failed);
|
|
103
|
+
this.send({ type: 'error', errorText: this.options.errorMessage(failed) });
|
|
39
104
|
}
|
|
40
|
-
|
|
41
|
-
|
|
105
|
+
return step;
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
/**
|
|
109
|
+
* The turn as a UI message stream for the client, read to its end on this side whether the client
|
|
110
|
+
* reads it or not: a client that goes away does not leave the turn hanging, and the outcome is
|
|
111
|
+
* settled either way. A turn may run several streams, one after the other, `next` deciding.
|
|
112
|
+
*/
|
|
113
|
+
export function streamTurn(first, options) {
|
|
114
|
+
let controller;
|
|
115
|
+
let reading = true;
|
|
116
|
+
const stream = new ReadableStream({
|
|
117
|
+
start: (opened) => {
|
|
118
|
+
controller = opened;
|
|
119
|
+
},
|
|
120
|
+
cancel: () => {
|
|
121
|
+
reading = false;
|
|
122
|
+
},
|
|
123
|
+
});
|
|
124
|
+
const send = (chunk) => {
|
|
125
|
+
if (reading)
|
|
126
|
+
controller.enqueue(chunk);
|
|
127
|
+
};
|
|
128
|
+
const settled = (async () => {
|
|
129
|
+
try {
|
|
130
|
+
const outcome = await new TurnRun(options, send).run(first);
|
|
131
|
+
// Settled before the stream closes: the client reads the session once it is written.
|
|
132
|
+
await options.onSettled(outcome);
|
|
42
133
|
}
|
|
43
134
|
finally {
|
|
44
|
-
|
|
135
|
+
if (reading)
|
|
136
|
+
controller.close();
|
|
45
137
|
}
|
|
46
138
|
})();
|
|
47
|
-
return { stream
|
|
139
|
+
return { stream, settled };
|
|
48
140
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ai-sdk-harness-sessions",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.0",
|
|
4
4
|
"description": "Sessions for AI SDK harness agents: the lifecycle of conversations with Claude Code, Codex and friends — one turn at a time, suspended when idle, resumed on the next message and after a restart, several of them sharing one sandbox.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai",
|
|
@@ -55,12 +55,12 @@
|
|
|
55
55
|
"ai": "^7.0.127",
|
|
56
56
|
"typescript": "^6.0.3",
|
|
57
57
|
"vitest": "^5.0.3",
|
|
58
|
-
"@repo/
|
|
58
|
+
"@repo/eslint-config": "0.0.0",
|
|
59
59
|
"@repo/prettier-config": "0.0.0",
|
|
60
60
|
"@repo/typescript-config": "0.0.0",
|
|
61
|
-
"@repo/eslint-config": "0.0.0",
|
|
62
61
|
"@repo/vitest-config": "0.0.0",
|
|
63
|
-
"ai-sdk-sandbox-sbx": "0.1.1"
|
|
62
|
+
"ai-sdk-sandbox-sbx": "0.1.1",
|
|
63
|
+
"@repo/build": "0.0.0"
|
|
64
64
|
},
|
|
65
65
|
"author": {
|
|
66
66
|
"name": "Fabien Pasquet",
|