pi-roundtable 0.9.8 → 0.9.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/docs/plugins.md +42 -4
- package/package.json +1 -1
- package/src/core/attachments/attachment-dir.ts +3 -0
- package/src/core/attachments/attachment-expiry.ts +170 -0
- package/src/core/attachments/attachment-port.ts +1 -0
- package/src/core/attachments/attachment-retention.ts +116 -0
- package/src/core/attachments/attachment-store.ts +32 -35
- package/src/core/attachments/attachment-tally.ts +49 -0
- package/src/core/attachments/tool-attachment.ts +14 -2
- package/src/core/config/attachments.ts +28 -0
- package/src/core/config/config.ts +9 -0
- package/src/core/contract/attachments.ts +33 -0
- package/src/core/define-roundtable.ts +3 -0
- package/src/core/define.ts +8 -0
- package/src/core/domain/ports.ts +7 -0
- package/src/core/host.ts +24 -6
- package/src/core/routing/conversation-turns.ts +7 -0
- package/src/core/runtime/pi-agent-runtime.ts +9 -3
- package/src/core/runtime/session-factory.ts +3 -0
- package/src/core/sessions.ts +6 -0
- package/src/core/shared/attachment-reader.ts +4 -0
- package/src/core/tool-turn.ts +2 -0
- package/src/index.ts +4 -0
- package/src/testing.ts +3 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,24 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [0.9.10] - 2026-10-10
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- `ToolTurn.origin`, `TurnRequest.origin`, `ConversationTurnInput.origin` and `SessionContext.origin()`: where a turn's request came from, as the transport that carried it reported it, such as the browser origin of a web chat connection (`https://chat.example.com`). A tool's `run` and its `hold` read it from the `ToolTurn` they receive, so a tool can label what it creates (an issue's source line) and fail closed when it is undefined. A claim passes it to `context.turns.run({ ..., origin })`; the runtime keeps it with the turn's speaker and clears it when the turn ends. Only the transport sets it, never the message text or the model. A turn whose channel gives none (Discord, a schedule, an agent's turn, the CLI) leaves it undefined, and `testPlugin().runTool` takes `origin` in its options. Additive: nothing changes for a claim or a tool that does not use it, and `SessionContext.origin` is optional on the type so a context built before this still type-checks.
|
|
13
|
+
|
|
14
|
+
## [0.9.9] - 2026-10-10
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
|
|
18
|
+
- `attachments.retention` (`AttachmentsConfig`, `AttachmentRetention`: `{ maxAgeMs, sweepEveryMs? }`) in the configuration, and `RoundtableOptions.attachments` for a host built by hand: the host removes the files turns used once they are `maxAgeMs` old, counted from the turn that took the file, sweeping when it starts and every `sweepEveryMs` (default one hour, at most `maxAgeMs`). It is off by default, where used files live as long as their conversation. It needs `dataDir`; a setting the host cannot keep stops the start and names it. The logs carry counts only (`files`, `bytes`, `unattributedBytes`), never a file name.
|
|
19
|
+
- `AttachmentPort.expireUsed({ olderThanMs })` (`ExpireUsedOptions`, `ExpiredAttachments`), the call the sweep makes: it removes the used files older than the age and takes exactly their bytes off each owner's used-bytes tally for the conversation, under the same per-person serialization as a message taking files, so a sweep never races a turn or a conversation's deletion. The allowance behind `usedBytesLimit` (the web chat's `usedAttachmentBytesPerPrincipal`) no longer stays full for good once a host sweeps its files. The method is optional, so a custom `AttachmentPort` written before it keeps type-checking; a retention setting refuses a port without it.
|
|
20
|
+
|
|
21
|
+
### Changed
|
|
22
|
+
|
|
23
|
+
- A used file's record now carries its owner (a hash, never the principal id), when a turn took it and its size. A file used before this has none: the sweep ages it by its modified time, and gives the bytes back to the one person with a tally in its conversation; with several people it removes the file and reports the bytes as `unattributedBytes` without lowering any tally.
|
|
24
|
+
- `read_attachment` and `ToolTurn.attachment()` answer a file removed after its retention period with a refusal that says so, and not "no attachment named …". The removed file's record keeps only that mark: neither the person's file name nor its type.
|
|
25
|
+
|
|
8
26
|
## [0.9.8] - 2026-10-10
|
|
9
27
|
|
|
10
28
|
### Added
|
package/docs/plugins.md
CHANGED
|
@@ -543,6 +543,7 @@ A plugin whose clients upload a file themselves, over HTTP for instance, has the
|
|
|
543
543
|
| `remove(channel, principalId, file)` | Discards a saved file no turn used; `false` when there is none |
|
|
544
544
|
| `discardPending(olderThan)` | Discards every saved file no turn used that was saved before the date, and returns how many; call it from a service's timer so abandoned uploads do not pile up |
|
|
545
545
|
| `pendingBytes(principalId)` | The bytes the principal has saved that no turn used yet, for a quota of your own |
|
|
546
|
+
| `expireUsed({ olderThanMs })` | Optional on a custom port; the host's own has it. Removes the files turns used more than that long ago and returns `{ files, bytes, unattributedBytes }`; the owners' used-bytes tallies shrink by exactly the bytes removed. See [Retention](#retention-removing-used-files-after-a-period) |
|
|
546
547
|
|
|
547
548
|
A saved file waits for its owner: only the principal that saved it can use it, and only in the conversation it was saved for.
|
|
548
549
|
The conversation's record decides who may save into it: a private conversation refuses every principal but its own with an `AttachmentRefusal` whose `code` is `forbidden`, and a shared one lets each principal save and use their own files.
|
|
@@ -554,14 +555,46 @@ The core limits only the size; the types a plugin accepts, the number of files,
|
|
|
554
555
|
|
|
555
556
|
Once a turn has used a file, the model reads it with `read_attachment`, and a tool reads it with `turn.attachment(file)` (see [`tools`](#tools-what-agents-can-call)).
|
|
556
557
|
|
|
558
|
+
#### Retention: removing used files after a period
|
|
559
|
+
|
|
560
|
+
Left alone, a file a turn used lives as long as its conversation.
|
|
561
|
+
A host that must not keep uploads (for privacy, say) sets a retention period, and the core removes used files that old:
|
|
562
|
+
|
|
563
|
+
```ts
|
|
564
|
+
export default {
|
|
565
|
+
// ...
|
|
566
|
+
attachments: { retention: { maxAgeMs: 7 * 24 * 60 * 60_000 } },
|
|
567
|
+
} satisfies RoundtableConfig;
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
`maxAgeMs` counts from the turn that took the file, not from the upload, so a file that waited a day before a message used it still lives its full period.
|
|
571
|
+
The host sweeps when it starts and then every `sweepEveryMs` (default one hour, never more than `maxAgeMs`); a host built without `defineRoundtable` passes the same object as `RoundtableOptions.attachments`.
|
|
572
|
+
It needs a `dataDir`, and `maxAgeMs` and `sweepEveryMs` must be positive whole numbers of milliseconds: a host that cannot keep the setting stops at start and names it.
|
|
573
|
+
A plugin that wants its own schedule calls `context.attachments.expireUsed({ olderThanMs })` itself.
|
|
574
|
+
|
|
575
|
+
A sweep, for each person who used files:
|
|
576
|
+
|
|
577
|
+
- runs with that person's other attachment calls, one after another, so it never races a message taking files; a conversation deleted meanwhile is skipped, not brought back;
|
|
578
|
+
- removes the file, then replaces its record with a mark that keeps no file name or type, then takes exactly the removed bytes off that person's used-bytes tally for the conversation. A failure part way leaves a tally that is too high, never too low, and the next sweep finishes the job. The `usedBytesLimit` allowance a plugin enforces (the web chat's `usedAttachmentBytesPerPrincipal`) comes back as files expire, instead of filling for good.
|
|
579
|
+
|
|
580
|
+
After that, `read_attachment` and `turn.attachment(file)` answer a clear refusal that the file "was removed after its retention period", with neither a host path nor a bare "not found". The refusal is the only trace; the person's name for the file is gone.
|
|
581
|
+
Files waiting for a turn are not touched: `discardPending` is theirs.
|
|
582
|
+
A file a turn used under an earlier release has no recorded owner; the sweep ages it by the file's modified time and gives its bytes back to the one person with a tally in that conversation. When several people have tallies there, it removes the file but cannot tell whose bytes they were, so no tally shrinks and the bytes are reported as `unattributedBytes`; deleting the conversation clears the tallies.
|
|
583
|
+
Only files taken through `context.attachments` are covered. Files the core downloads from Discord links keep their conversation's lifetime.
|
|
584
|
+
The host logs counts only (`files`, `bytes`, `unattributedBytes`), never a file name, and a failed sweep only its error code.
|
|
585
|
+
|
|
557
586
|
### `tools`: what agents can call
|
|
558
587
|
|
|
559
588
|
`defineTool` takes a name (lowercase words joined by underscores), a description the model reads to decide when to call it, a Typebox parameter schema, the lowest tier that may call it, and the function.
|
|
560
589
|
`run` receives the arguments, already typed, and the turn: the speaker, the channel, the agent, an abort signal, `workspace`, `attachFile`, and `attachment`.
|
|
561
590
|
`turn.workspace` is `{ workspace, scratchDir? }` in a session with a shared workspace (an agent's), the same roots as `SessionContext.workspace`, and undefined elsewhere: a tool that makes a file can save it there, under the scratch dir when there is one, and name its path, so the model sends it on with `attach_file` or a `discord_send_message` file `path` instead of carrying its bytes.
|
|
562
591
|
It returns the text the model reads.
|
|
592
|
+
`turn.origin` is where the turn's request came from, as the transport that carried it reported it, and undefined when the channel gives none: the web chat sets it to the browser origin its connection opened with (`https://chat.example.com`, `chrome-extension://<id>`), a Discord message, a schedule or an agent's turn leave it undefined.
|
|
593
|
+
Only a channel's transport sets it, never the message text or the model, so a tool may write it on a record it creates, such as an issue's source line; a tool that needs it fails closed when it is undefined.
|
|
594
|
+
A claim passes it to `context.turns.run` as `origin`, and the hold of the same call sees it too, since `hold` receives the same `ToolTurn`.
|
|
595
|
+
|
|
563
596
|
`turn.attachment(file)` opens a file someone attached to the conversation, by the name the turn's `## Attachments` block lists, and returns `{ file, name, contentType, size, bytes() }`: a tool passes a user's file on without the model carrying its bytes.
|
|
564
|
-
Only the conversation's own files open; a path, a hidden name, or a missing file throws a `ToolRefusal` the model reads.
|
|
597
|
+
Only the conversation's own files open; a path, a hidden name, or a missing file throws a `ToolRefusal` the model reads, and so does a file the host's [retention period](#retention-removing-used-files-after-a-period) removed, which says so.
|
|
565
598
|
Throw `ToolRefusal` for a call the model should correct; any other error fails the call.
|
|
566
599
|
|
|
567
600
|
<!-- example: examples/tools.ts -->
|
|
@@ -717,7 +750,7 @@ export const cleanup = definePlugin({
|
|
|
717
750
|
#### A hold that looks things up
|
|
718
751
|
|
|
719
752
|
A hold may need something it must fetch as the speaker before it can describe the call, such as the display name of the entity an id stands for, read with the speaker's own credentials.
|
|
720
|
-
A tool's `hold` may then return a promise and takes the turn as a second argument: the `ToolTurn` its `run` receives, with the `speaker`, the `channel` and the `
|
|
753
|
+
A tool's `hold` may then return a promise and takes the turn as a second argument: the `ToolTurn` its `run` receives, with the `speaker`, the `channel`, the `agent` and the `origin` of the turn that makes the call.
|
|
721
754
|
|
|
722
755
|
```ts
|
|
723
756
|
defineTool({
|
|
@@ -1832,6 +1865,7 @@ The factory gets the session's `SessionContext`, fixed when the session is made:
|
|
|
1832
1865
|
- `addressee` is whom tool descriptions should name: the primary owner exactly as configured in their own private conversation, another person by their name and pronouns (their name alone when they gave none), and `THE_SPEAKER` in a shared conversation, or a private one of someone the host knows by no name yet.
|
|
1833
1866
|
- `memory` is `"none"` for a persona that declares it, and `"speaker"` otherwise; a memory tool of your own loads nothing for `"none"`. A task the session runs with `runTask` keeps its conversation's `memory`.
|
|
1834
1867
|
- `speaker()` is the running turn's speaker, undefined between turns; a tool reads it when it runs, never when the session is made, since a session outlives the turn it is built in.
|
|
1868
|
+
- `origin()` is where the running turn's request came from, as `TurnRequest.origin` says, and undefined between turns and for a turn whose transport gives none; read it when the tool runs, like `speaker()`. It is optional on the type so a context built before 0.9.10 still type-checks.
|
|
1835
1869
|
|
|
1836
1870
|
The core's Jev compactor is one such extension:
|
|
1837
1871
|
|
|
@@ -1983,7 +2017,7 @@ Call `context.turns.run(input)` inside your claim's queue task to run a turn in
|
|
|
1983
2017
|
It shows typing and the stop control on the channel's surface, runs the turn, and emits `turnStarted` and `turnEnded` with the turn's `kind`.
|
|
1984
2018
|
It catches runtime errors as failed results and posts the answer, failure notice, or stopped notice through the surface.
|
|
1985
2019
|
Supply `reply(result)` to handle the reply yourself.
|
|
1986
|
-
`input` has `channel`, `kind`, `text`, `speaker`, and optionally `attachments`, `selection` (by default the plugins' `agentSelection`), `steerable`, `interactive`, `confirmed`, `reply`, and `conversation`.
|
|
2020
|
+
`input` has `channel`, `kind`, `text`, `speaker`, and optionally `origin` (where the request came from, as your claim's transport knows it, such as a connection's browser origin; tools read it as `turn.origin`, so never fill it from message text), `attachments`, `selection` (by default the plugins' `agentSelection`), `steerable`, `interactive`, `confirmed`, `reply`, and `conversation`.
|
|
1987
2021
|
It rejects with `NotLinkedError` during `setup`, and with a `PluginError` on a host whose runtime plugin has not provided a runtime.
|
|
1988
2022
|
|
|
1989
2023
|
Before each turn runs, `context.turns.run` records its conversation in the host's registry, `CONVERSATIONS`: at the first turn its key, surface, kind, visibility, owner, and title, and at every later turn only that it was active.
|
|
@@ -2926,7 +2960,7 @@ It returns:
|
|
|
2926
2960
|
| `contribution` | What the plugin added, as the host would collect it: `tools`, `prompt`, `seeds`, `events`, `services`, `http`, and the rest |
|
|
2927
2961
|
| `tools`, `tiers` | The tool names, and the table that says what tier each needs |
|
|
2928
2962
|
| `holds` | The plugin's `holdRules` chained as the host links them (`holdChain`): `holds(tool, input, { workspace?, scratchDir? })` returns the description of a call that must be approved first, or `undefined`; `holds.inTurn(tool, input, context, turn)` asks them as a turn does, and returns a promise when a hold looks things up |
|
|
2929
|
-
| `runTool(name, args, { speaker, channel, workspace }?)` | Runs a tool the way an agent's turn would, in the channel (default `test:1`) for the speaker, and returns the text the model reads; `workspace` (`{ workspace, scratchDir? }`) is the session's `turn.workspace` |
|
|
2963
|
+
| `runTool(name, args, { speaker, origin, channel, workspace }?)` | Runs a tool the way an agent's turn would, in the channel (default `test:1`) for the speaker, and returns the text the model reads; `origin` is the turn's `turn.origin`, and `workspace` (`{ workspace, scratchDir? }`) is the session's `turn.workspace` |
|
|
2930
2964
|
| `files` | Accepted files from `runTool`, recorded as `{ channel, file: ReplyFile }`; inject a surface with `supportsFiles: true` and pass its channel to test attachment tools |
|
|
2931
2965
|
| `events` | The events the plugin itself reported through `context.events`, and those of `context.turns` |
|
|
2932
2966
|
| `conversations`, `turns`, `surfaces` | What the plugin sees as `context.conversations`, `context.turns`, and `context.surfaces`, for a test to drive its claims |
|
|
@@ -3316,7 +3350,9 @@ Import from the entries listed below; source area files are internal.
|
|
|
3316
3350
|
| `AttachmentRef` | `pi-roundtable` | type |
|
|
3317
3351
|
| `AttachmentRefusal` | `pi-roundtable` | value |
|
|
3318
3352
|
| `AttachmentRefusalCode` | `pi-roundtable` | type |
|
|
3353
|
+
| `AttachmentRetention` | `pi-roundtable` | type |
|
|
3319
3354
|
| `AttachmentUpload` | `pi-roundtable` | type |
|
|
3355
|
+
| `AttachmentsConfig` | `pi-roundtable` | type |
|
|
3320
3356
|
| `AvatarMode` | `pi-roundtable` | type |
|
|
3321
3357
|
| `AvatarStudio` | `pi-roundtable` | type |
|
|
3322
3358
|
| `BACKGROUND_TURNS` | `pi-roundtable` | value |
|
|
@@ -3355,6 +3391,8 @@ Import from the entries listed below; source area files are internal.
|
|
|
3355
3391
|
| `DrainOptions` | `pi-roundtable` | type |
|
|
3356
3392
|
| `EventHandlers` | `pi-roundtable` | type |
|
|
3357
3393
|
| `EventSink` | `pi-roundtable` | type |
|
|
3394
|
+
| `ExpireUsedOptions` | `pi-roundtable` | type |
|
|
3395
|
+
| `ExpiredAttachments` | `pi-roundtable` | type |
|
|
3358
3396
|
| `GroupStatus` | `pi-roundtable` | type |
|
|
3359
3397
|
| `HeldActionStore` | `pi-roundtable` | type |
|
|
3360
3398
|
| `HeldCall` | `pi-roundtable` | type |
|
package/package.json
CHANGED
|
@@ -7,6 +7,9 @@ export const STAGING = "attachments-pending";
|
|
|
7
7
|
/** Where each principal's tally of used bytes per conversation is kept. */
|
|
8
8
|
export const USAGE = "attachments-used";
|
|
9
9
|
|
|
10
|
+
/** The record of a saved file, in a directory next to it that no file name can be. */
|
|
11
|
+
export const RECORDS = ".records";
|
|
12
|
+
|
|
10
13
|
/** A channel key as a single path segment. */
|
|
11
14
|
export function channelSegment(channel: ChannelKey): string {
|
|
12
15
|
return channel.replace(/[^A-Za-z0-9_-]/g, "_");
|
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
import { readFile, rename, rm, stat, writeFile } from "node:fs/promises";
|
|
2
|
+
import { join } from "node:path";
|
|
3
|
+
import type { ExpiredAttachments } from "../contract/attachments.ts";
|
|
4
|
+
import { listDir, RECORDS, USAGE } from "./attachment-dir.ts";
|
|
5
|
+
import { isAttachmentName } from "./attachment-name.ts";
|
|
6
|
+
import { shrinkTally } from "./attachment-tally.ts";
|
|
7
|
+
|
|
8
|
+
/** What the core records about a file a turn used; the first two come from its upload. */
|
|
9
|
+
export interface UsedRecord {
|
|
10
|
+
name: string;
|
|
11
|
+
contentType: string;
|
|
12
|
+
/** The owner's directory segment (a hash, never the id), when the core recorded it. */
|
|
13
|
+
principal?: string;
|
|
14
|
+
/** When the turn took the file, in epoch milliseconds. */
|
|
15
|
+
usedAt?: number;
|
|
16
|
+
size?: number;
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
/** What stays of a record once its file is removed: when, and nothing the person named. */
|
|
20
|
+
interface ExpiredMark {
|
|
21
|
+
expiredAt: number;
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
/** The record file of `file` in a conversation's attachment directory. */
|
|
25
|
+
export function recordPath(dir: string, file: string): string {
|
|
26
|
+
return join(dir, RECORDS, `${file}.json`);
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
/** The record as parsed, or undefined when there is none or it is not an object. */
|
|
30
|
+
async function readRecord(path: string): Promise<unknown> {
|
|
31
|
+
try {
|
|
32
|
+
return JSON.parse(await readFile(path, "utf8"));
|
|
33
|
+
} catch (error) {
|
|
34
|
+
if (error instanceof SyntaxError) return undefined;
|
|
35
|
+
if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined;
|
|
36
|
+
throw error;
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
const isExpiredMark = (value: unknown): value is ExpiredMark =>
|
|
41
|
+
typeof value === "object" &&
|
|
42
|
+
value !== null &&
|
|
43
|
+
typeof (value as ExpiredMark).expiredAt === "number";
|
|
44
|
+
|
|
45
|
+
const isUsedRecord = (value: unknown): value is UsedRecord =>
|
|
46
|
+
typeof value === "object" &&
|
|
47
|
+
value !== null &&
|
|
48
|
+
!isExpiredMark(value) &&
|
|
49
|
+
typeof (value as UsedRecord).name === "string";
|
|
50
|
+
|
|
51
|
+
/** Whether the file's retention period ended and the core removed it. */
|
|
52
|
+
export async function wasExpired(dir: string, file: string): Promise<boolean> {
|
|
53
|
+
if (!isAttachmentName(file)) return false;
|
|
54
|
+
return isExpiredMark(await readRecord(recordPath(dir, file)));
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
/** What a tool or the model reads for a file removed after its retention period. */
|
|
58
|
+
export const expiredMessage = (shown: string): string =>
|
|
59
|
+
`"${shown}" was removed after its retention period and cannot be read any more. Ask the person to attach it again.`;
|
|
60
|
+
|
|
61
|
+
export interface ExpiryDeps {
|
|
62
|
+
dataDir: string;
|
|
63
|
+
now(): number;
|
|
64
|
+
/** Runs after the principal's earlier calls, as every call that changes their files does. */
|
|
65
|
+
exclusive<T>(principal: string, run: () => Promise<T>): Promise<T>;
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
/** A used file the sweep found old, with the owner it could tell. */
|
|
69
|
+
interface Candidate {
|
|
70
|
+
channel: string;
|
|
71
|
+
file: string;
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
/** Whoever the owner of an unowned record could only be: a lone tally in the conversation, else none. */
|
|
75
|
+
async function soleHolders(
|
|
76
|
+
dataDir: string,
|
|
77
|
+
): Promise<Map<string, string | null>> {
|
|
78
|
+
const holders = new Map<string, string | null>();
|
|
79
|
+
const root = join(dataDir, USAGE);
|
|
80
|
+
for (const principal of await listDir(root))
|
|
81
|
+
for (const entry of await listDir(join(root, principal))) {
|
|
82
|
+
if (entry.endsWith(".tmp")) continue;
|
|
83
|
+
holders.set(entry, holders.has(entry) ? null : principal);
|
|
84
|
+
}
|
|
85
|
+
return holders;
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/** The used files older than `cutoff`, grouped by the owner whose calls they run with. */
|
|
89
|
+
async function findOld(
|
|
90
|
+
dataDir: string,
|
|
91
|
+
cutoff: number,
|
|
92
|
+
): Promise<Map<string, Candidate[]>> {
|
|
93
|
+
const holders = await soleHolders(dataDir);
|
|
94
|
+
const groups = new Map<string, Candidate[]>();
|
|
95
|
+
const root = join(dataDir, "attachments");
|
|
96
|
+
for (const channel of await listDir(root)) {
|
|
97
|
+
const dir = join(root, channel);
|
|
98
|
+
for (const entry of await listDir(join(dir, RECORDS))) {
|
|
99
|
+
if (!entry.endsWith(".json")) continue;
|
|
100
|
+
const file = entry.slice(0, -".json".length);
|
|
101
|
+
const record = await readRecord(recordPath(dir, file));
|
|
102
|
+
if (!isUsedRecord(record)) continue;
|
|
103
|
+
const usedAt =
|
|
104
|
+
record.usedAt ??
|
|
105
|
+
(await stat(join(dir, file)).catch(() => undefined))?.mtimeMs;
|
|
106
|
+
if (usedAt === undefined || usedAt >= cutoff) continue;
|
|
107
|
+
const owner = record.principal ?? holders.get(channel) ?? "";
|
|
108
|
+
groups.set(owner, [...(groups.get(owner) ?? []), { channel, file }]);
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
return groups;
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/**
|
|
115
|
+
* Removes the files turns used before `now - olderThanMs`. Each owner's files go in one call that
|
|
116
|
+
* runs with their other calls: the file, then its record (which becomes a mark that keeps no
|
|
117
|
+
* name), then the bytes off their tally for the conversation, so a failure part way leaves a
|
|
118
|
+
* tally that is too high, never too low.
|
|
119
|
+
*/
|
|
120
|
+
export async function expireUsedFiles(
|
|
121
|
+
deps: ExpiryDeps,
|
|
122
|
+
olderThanMs: number,
|
|
123
|
+
): Promise<ExpiredAttachments> {
|
|
124
|
+
const { dataDir } = deps;
|
|
125
|
+
const cutoff = deps.now() - olderThanMs;
|
|
126
|
+
const total: ExpiredAttachments = {
|
|
127
|
+
files: 0,
|
|
128
|
+
bytes: 0,
|
|
129
|
+
unattributedBytes: 0,
|
|
130
|
+
};
|
|
131
|
+
for (const [owner, candidates] of await findOld(dataDir, cutoff)) {
|
|
132
|
+
await deps.exclusive(owner || "unattributed", async () => {
|
|
133
|
+
const removed = new Map<string, number>();
|
|
134
|
+
for (const { channel, file } of candidates) {
|
|
135
|
+
const dir = join(dataDir, "attachments", channel);
|
|
136
|
+
const path = recordPath(dir, file);
|
|
137
|
+
const record = await readRecord(path);
|
|
138
|
+
// A turn may have deleted the conversation, or an earlier call this one.
|
|
139
|
+
if (!isUsedRecord(record)) continue;
|
|
140
|
+
const info = await stat(join(dir, file)).catch(() => undefined);
|
|
141
|
+
const size = info?.size ?? record.size ?? 0;
|
|
142
|
+
const usedAt = record.usedAt ?? info?.mtimeMs;
|
|
143
|
+
if (usedAt === undefined || usedAt >= cutoff) continue;
|
|
144
|
+
await rm(join(dir, file), { force: true });
|
|
145
|
+
const mark: ExpiredMark = { expiredAt: deps.now() };
|
|
146
|
+
if (!(await writeMark(path, mark))) continue;
|
|
147
|
+
total.files += 1;
|
|
148
|
+
total.bytes += size;
|
|
149
|
+
if (owner === "") total.unattributedBytes += size;
|
|
150
|
+
else removed.set(channel, (removed.get(channel) ?? 0) + size);
|
|
151
|
+
}
|
|
152
|
+
for (const [channel, bytes] of removed)
|
|
153
|
+
await shrinkTally(join(dataDir, USAGE, owner, channel), bytes);
|
|
154
|
+
});
|
|
155
|
+
}
|
|
156
|
+
return total;
|
|
157
|
+
}
|
|
158
|
+
|
|
159
|
+
/** Replaces a record by its mark; false when the conversation's directory is gone. */
|
|
160
|
+
async function writeMark(path: string, mark: ExpiredMark): Promise<boolean> {
|
|
161
|
+
const temporary = `${path}.tmp`;
|
|
162
|
+
try {
|
|
163
|
+
await writeFile(temporary, JSON.stringify(mark));
|
|
164
|
+
await rename(temporary, path);
|
|
165
|
+
return true;
|
|
166
|
+
} catch (error) {
|
|
167
|
+
if ((error as NodeJS.ErrnoException).code === "ENOENT") return false;
|
|
168
|
+
throw error;
|
|
169
|
+
}
|
|
170
|
+
}
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
import type { AttachmentPort } from "../contract/attachments.ts";
|
|
2
|
+
import { PluginError } from "../errors.ts";
|
|
3
|
+
import type { Logger } from "../log.ts";
|
|
4
|
+
import type { Service } from "../plugin.ts";
|
|
5
|
+
|
|
6
|
+
/** How long used files are kept, and how often the host looks for the ones past it. */
|
|
7
|
+
export interface AttachmentRetention {
|
|
8
|
+
/** Files a turn used more than this many milliseconds ago are removed. */
|
|
9
|
+
maxAgeMs: number;
|
|
10
|
+
/** The wait between two sweeps; default one hour, and never more than `maxAgeMs`. */
|
|
11
|
+
sweepEveryMs?: number;
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
/** The wait between two sweeps when the host sets none. */
|
|
15
|
+
export const DEFAULT_SWEEP_EVERY_MS = 60 * 60_000;
|
|
16
|
+
|
|
17
|
+
/** The port's `expireUsed`; `retentionServices` refuses a port without it before a sweep can run. */
|
|
18
|
+
function expireUsedOf(
|
|
19
|
+
attachments: AttachmentPort,
|
|
20
|
+
): NonNullable<AttachmentPort["expireUsed"]> {
|
|
21
|
+
const expire = attachments.expireUsed;
|
|
22
|
+
if (expire === undefined)
|
|
23
|
+
throw new PluginError("this attachment port cannot expire used files");
|
|
24
|
+
return expire.bind(attachments);
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
/** The wait between two sweeps for `retention`. */
|
|
28
|
+
function sweepPeriod(retention: AttachmentRetention): number {
|
|
29
|
+
return Math.min(
|
|
30
|
+
retention.sweepEveryMs ?? DEFAULT_SWEEP_EVERY_MS,
|
|
31
|
+
retention.maxAgeMs,
|
|
32
|
+
);
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* The service that removes used attachments past their retention period: one sweep when the host
|
|
37
|
+
* starts, then one every period, never two at once. A sweep that fails is logged by its error
|
|
38
|
+
* code alone, since a file system error names the file, and the next sweep tries again. The logs
|
|
39
|
+
* carry counts, never a file's name.
|
|
40
|
+
*/
|
|
41
|
+
export function retentionService(
|
|
42
|
+
attachments: AttachmentPort,
|
|
43
|
+
retention: AttachmentRetention,
|
|
44
|
+
logger: Logger,
|
|
45
|
+
): Service {
|
|
46
|
+
let timer: ReturnType<typeof setInterval> | undefined;
|
|
47
|
+
let sweeping: Promise<void> | undefined;
|
|
48
|
+
const sweep = async (): Promise<void> => {
|
|
49
|
+
try {
|
|
50
|
+
const result = await expireUsedOf(attachments)({
|
|
51
|
+
olderThanMs: retention.maxAgeMs,
|
|
52
|
+
});
|
|
53
|
+
if (result.files > 0)
|
|
54
|
+
logger.info(
|
|
55
|
+
{
|
|
56
|
+
files: result.files,
|
|
57
|
+
bytes: result.bytes,
|
|
58
|
+
unattributedBytes: result.unattributedBytes,
|
|
59
|
+
},
|
|
60
|
+
"used attachments past their retention period removed",
|
|
61
|
+
);
|
|
62
|
+
} catch (error) {
|
|
63
|
+
const code = (error as NodeJS.ErrnoException).code;
|
|
64
|
+
logger.warn(
|
|
65
|
+
code === undefined ? {} : { code },
|
|
66
|
+
"used attachments past their retention period not removed",
|
|
67
|
+
);
|
|
68
|
+
}
|
|
69
|
+
};
|
|
70
|
+
const once = (): Promise<void> => {
|
|
71
|
+
sweeping ??= sweep().finally(() => {
|
|
72
|
+
sweeping = undefined;
|
|
73
|
+
});
|
|
74
|
+
return sweeping;
|
|
75
|
+
};
|
|
76
|
+
return {
|
|
77
|
+
name: "attachment-retention",
|
|
78
|
+
start: async () => {
|
|
79
|
+
await once();
|
|
80
|
+
timer = setInterval(() => void once(), sweepPeriod(retention));
|
|
81
|
+
},
|
|
82
|
+
stop: async () => {
|
|
83
|
+
clearInterval(timer);
|
|
84
|
+
timer = undefined;
|
|
85
|
+
await sweeping;
|
|
86
|
+
},
|
|
87
|
+
};
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
/**
|
|
91
|
+
* The retention service for the host's options, or none when it sets no retention. A setting the
|
|
92
|
+
* host cannot keep stops the boot, naming it: no `dataDir` to keep files in, or an age or period
|
|
93
|
+
* that is not a positive whole number of milliseconds.
|
|
94
|
+
*/
|
|
95
|
+
export function retentionServices(
|
|
96
|
+
retention: AttachmentRetention | undefined,
|
|
97
|
+
dataDir: string | undefined,
|
|
98
|
+
attachments: AttachmentPort,
|
|
99
|
+
logger: Logger,
|
|
100
|
+
): Service[] {
|
|
101
|
+
if (!retention) return [];
|
|
102
|
+
if (dataDir === undefined)
|
|
103
|
+
throw new PluginError(
|
|
104
|
+
"attachments.retention needs a dataDir: set dataDir in the configuration, or the dataDir option of the host.",
|
|
105
|
+
);
|
|
106
|
+
if (attachments.expireUsed === undefined)
|
|
107
|
+
throw new PluginError(
|
|
108
|
+
"attachments.retention needs an attachment port with expireUsed; this one has none.",
|
|
109
|
+
);
|
|
110
|
+
for (const [key, value] of Object.entries(retention))
|
|
111
|
+
if (!Number.isInteger(value) || value < 1)
|
|
112
|
+
throw new PluginError(
|
|
113
|
+
`attachments.retention.${key} must be a positive whole number of milliseconds; got ${String(value)}.`,
|
|
114
|
+
);
|
|
115
|
+
return [retentionService(attachments, retention, logger)];
|
|
116
|
+
}
|
|
@@ -1,11 +1,12 @@
|
|
|
1
|
-
import { createHash } from "node:crypto";
|
|
2
1
|
import { mkdirSync } from "node:fs";
|
|
3
|
-
import {
|
|
4
|
-
import {
|
|
2
|
+
import { rename, rm, stat } from "node:fs/promises";
|
|
3
|
+
import { join } from "node:path";
|
|
5
4
|
import {
|
|
6
5
|
type AttachmentPort,
|
|
7
6
|
AttachmentRefusal,
|
|
8
7
|
type AttachmentUpload,
|
|
8
|
+
type ExpiredAttachments,
|
|
9
|
+
type ExpireUsedOptions,
|
|
9
10
|
type TurnAttachmentOptions,
|
|
10
11
|
} from "../contract/attachments.ts";
|
|
11
12
|
import type { ConversationRegistry } from "../conversations/conversation-registry.ts";
|
|
@@ -19,22 +20,24 @@ import {
|
|
|
19
20
|
channelSegment,
|
|
20
21
|
listDir,
|
|
21
22
|
ownerAttachmentDir,
|
|
23
|
+
RECORDS,
|
|
22
24
|
removeIfEmpty,
|
|
23
25
|
STAGING,
|
|
24
26
|
USAGE,
|
|
25
27
|
} from "./attachment-dir.ts";
|
|
28
|
+
import { expireUsedFiles, type UsedRecord } from "./attachment-expiry.ts";
|
|
26
29
|
import { MAX_ATTACHMENT_BYTES, safeFileName } from "./attachment-fetcher.ts";
|
|
27
30
|
import { isAttachmentName } from "./attachment-name.ts";
|
|
31
|
+
import { principalSegment, readTally, writeTally } from "./attachment-tally.ts";
|
|
28
32
|
import { modelImagesOf } from "./model-images.ts";
|
|
29
33
|
|
|
30
|
-
/** The record of a saved file, in a directory next to it that no file name can be. */
|
|
31
|
-
export const RECORDS = ".records";
|
|
32
|
-
|
|
33
34
|
export interface AttachmentStoreOptions {
|
|
34
35
|
dataDir: string;
|
|
35
36
|
/** Read when used, once the host linked its services; undefined when no registry records conversations. */
|
|
36
37
|
registry(): Pick<ConversationRegistry, "get"> | undefined;
|
|
37
38
|
logger: Logger;
|
|
39
|
+
/** The clock that stamps the turn that takes a file; epoch milliseconds, default `Date.now`. */
|
|
40
|
+
now?: () => number;
|
|
38
41
|
}
|
|
39
42
|
|
|
40
43
|
interface StagedRecord {
|
|
@@ -48,35 +51,6 @@ const isRecord = (value: unknown): value is StagedRecord =>
|
|
|
48
51
|
typeof (value as StagedRecord).name === "string" &&
|
|
49
52
|
typeof (value as StagedRecord).contentType === "string";
|
|
50
53
|
|
|
51
|
-
/** A principal id as one path segment; hashed, so two ids never share a directory. */
|
|
52
|
-
function principalSegment(principalId: string): string {
|
|
53
|
-
return createHash("sha256").update(principalId).digest("hex").slice(0, 32);
|
|
54
|
-
}
|
|
55
|
-
|
|
56
|
-
/** The bytes of a tally file; none when it is missing or unreadable as a number. */
|
|
57
|
-
async function readTally(path: string): Promise<number> {
|
|
58
|
-
try {
|
|
59
|
-
const value = Number(await readFile(path, "utf8"));
|
|
60
|
-
return Number.isFinite(value) && value > 0 ? value : 0;
|
|
61
|
-
} catch (error) {
|
|
62
|
-
if ((error as NodeJS.ErrnoException).code === "ENOENT") return 0;
|
|
63
|
-
throw error;
|
|
64
|
-
}
|
|
65
|
-
}
|
|
66
|
-
|
|
67
|
-
/** Writes a tally whole or not at all, creating its directory if a conversation's deletion removed it meanwhile. */
|
|
68
|
-
async function writeTally(path: string, bytes: number): Promise<void> {
|
|
69
|
-
const temporary = `${path}.tmp`;
|
|
70
|
-
try {
|
|
71
|
-
await writeFile(temporary, String(bytes));
|
|
72
|
-
} catch (error) {
|
|
73
|
-
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
|
|
74
|
-
await mkdir(dirname(path), { recursive: true, mode: 0o700 });
|
|
75
|
-
await writeFile(temporary, String(bytes));
|
|
76
|
-
}
|
|
77
|
-
await rename(temporary, path);
|
|
78
|
-
}
|
|
79
|
-
|
|
80
54
|
/**
|
|
81
55
|
* The files plugins take from people outside a turn. A saved file is staged under its principal and
|
|
82
56
|
* its conversation until a turn uses it, then moved into the conversation's attachment directory,
|
|
@@ -240,6 +214,18 @@ export class AttachmentStore implements AttachmentPort {
|
|
|
240
214
|
join(staging, RECORDS, `${stored.file}.json`),
|
|
241
215
|
join(target, RECORDS, `${stored.file}.json`),
|
|
242
216
|
);
|
|
217
|
+
// Who used the file and when, for the sweep that expires it and gives the tally back.
|
|
218
|
+
const used: UsedRecord = {
|
|
219
|
+
name: stored.name,
|
|
220
|
+
contentType: stored.contentType,
|
|
221
|
+
principal: principalSegment(principalId),
|
|
222
|
+
usedAt: (this.#options.now ?? Date.now)(),
|
|
223
|
+
size: stored.size,
|
|
224
|
+
};
|
|
225
|
+
await Bun.write(
|
|
226
|
+
join(target, RECORDS, `${stored.file}.json`),
|
|
227
|
+
JSON.stringify(used),
|
|
228
|
+
);
|
|
243
229
|
}
|
|
244
230
|
await writeTally(tally, before + added);
|
|
245
231
|
} catch (error) {
|
|
@@ -335,6 +321,17 @@ export class AttachmentStore implements AttachmentPort {
|
|
|
335
321
|
return total;
|
|
336
322
|
}
|
|
337
323
|
|
|
324
|
+
async expireUsed(options: ExpireUsedOptions): Promise<ExpiredAttachments> {
|
|
325
|
+
return expireUsedFiles(
|
|
326
|
+
{
|
|
327
|
+
dataDir: this.#options.dataDir,
|
|
328
|
+
now: this.#options.now ?? Date.now,
|
|
329
|
+
exclusive: (principal, run) => this.#exclusive(principal, run),
|
|
330
|
+
},
|
|
331
|
+
options.olderThanMs,
|
|
332
|
+
);
|
|
333
|
+
}
|
|
334
|
+
|
|
338
335
|
/** The bytes the principal's turns used, across their conversations. */
|
|
339
336
|
async #usedBytes(principalId: string): Promise<number> {
|
|
340
337
|
const dir = this.#tallyDir(principalId);
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
import { createHash } from "node:crypto";
|
|
2
|
+
import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
|
|
3
|
+
import { dirname } from "node:path";
|
|
4
|
+
|
|
5
|
+
/** A principal id as one path segment; hashed, so two ids never share a directory. */
|
|
6
|
+
export function principalSegment(principalId: string): string {
|
|
7
|
+
return createHash("sha256").update(principalId).digest("hex").slice(0, 32);
|
|
8
|
+
}
|
|
9
|
+
|
|
10
|
+
/** The bytes of a tally file; none when it is missing or unreadable as a number. */
|
|
11
|
+
export async function readTally(path: string): Promise<number> {
|
|
12
|
+
try {
|
|
13
|
+
const value = Number(await readFile(path, "utf8"));
|
|
14
|
+
return Number.isFinite(value) && value > 0 ? value : 0;
|
|
15
|
+
} catch (error) {
|
|
16
|
+
if ((error as NodeJS.ErrnoException).code === "ENOENT") return 0;
|
|
17
|
+
throw error;
|
|
18
|
+
}
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
/** Writes a tally whole or not at all, creating its directory if a conversation's deletion removed it meanwhile. */
|
|
22
|
+
export async function writeTally(path: string, bytes: number): Promise<void> {
|
|
23
|
+
const temporary = `${path}.tmp`;
|
|
24
|
+
try {
|
|
25
|
+
await writeFile(temporary, String(bytes));
|
|
26
|
+
} catch (error) {
|
|
27
|
+
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
|
|
28
|
+
await mkdir(dirname(path), { recursive: true, mode: 0o700 });
|
|
29
|
+
await writeFile(temporary, String(bytes));
|
|
30
|
+
}
|
|
31
|
+
await rename(temporary, path);
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/**
|
|
35
|
+
* Takes `bytes` off a tally that is there, never below zero. A tally that is gone, because the
|
|
36
|
+
* conversation was deleted, stays gone: nothing here brings a deleted conversation back.
|
|
37
|
+
*/
|
|
38
|
+
export async function shrinkTally(path: string, bytes: number): Promise<void> {
|
|
39
|
+
const temporary = `${path}.tmp`;
|
|
40
|
+
try {
|
|
41
|
+
const before = await readFile(path, "utf8");
|
|
42
|
+
const value = Number(before);
|
|
43
|
+
const left = Math.max(0, (Number.isFinite(value) ? value : 0) - bytes);
|
|
44
|
+
await writeFile(temporary, String(left));
|
|
45
|
+
await rename(temporary, path);
|
|
46
|
+
} catch (error) {
|
|
47
|
+
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
|
|
48
|
+
}
|
|
49
|
+
}
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { join } from "node:path";
|
|
2
|
+
import { RECORDS } from "./attachment-dir.ts";
|
|
3
|
+
import { expiredMessage, wasExpired } from "./attachment-expiry.ts";
|
|
2
4
|
import { isAttachmentName, plainText } from "./attachment-name.ts";
|
|
3
|
-
import { RECORDS } from "./attachment-store.ts";
|
|
4
5
|
|
|
5
6
|
/** A file someone attached to the conversation, as a tool reads it. */
|
|
6
7
|
export interface ToolAttachment {
|
|
@@ -19,6 +20,15 @@ export class AttachmentLookupError extends Error {
|
|
|
19
20
|
override name = "AttachmentLookupError";
|
|
20
21
|
}
|
|
21
22
|
|
|
23
|
+
/** A file the core removed after its retention period; the message says so, with no path. */
|
|
24
|
+
export class AttachmentExpiredError extends AttachmentLookupError {
|
|
25
|
+
override name = "AttachmentExpiredError";
|
|
26
|
+
|
|
27
|
+
constructor(shown: string) {
|
|
28
|
+
super(expiredMessage(shown));
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
|
|
22
32
|
interface Recorded {
|
|
23
33
|
name?: unknown;
|
|
24
34
|
contentType?: unknown;
|
|
@@ -36,10 +46,12 @@ export async function openAttachment(
|
|
|
36
46
|
if (!isAttachmentName(file))
|
|
37
47
|
throw new AttachmentLookupError(`"${shown}" is not an attachment name`);
|
|
38
48
|
const handle = Bun.file(join(dir, file));
|
|
39
|
-
if (!(await handle.exists()))
|
|
49
|
+
if (!(await handle.exists())) {
|
|
50
|
+
if (await wasExpired(dir, file)) throw new AttachmentExpiredError(shown);
|
|
40
51
|
throw new AttachmentLookupError(
|
|
41
52
|
`no attachment named "${shown}" in this channel`,
|
|
42
53
|
);
|
|
54
|
+
}
|
|
43
55
|
const record = Bun.file(join(dir, RECORDS, `${file}.json`));
|
|
44
56
|
const recorded: Recorded = (await record.exists()) ? await record.json() : {};
|
|
45
57
|
return {
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
import type { AttachmentRetention } from "../attachments/attachment-retention.ts";
|
|
2
|
+
import { integer, optional, shape } from "./schema.ts";
|
|
3
|
+
|
|
4
|
+
/** The configuration's `attachments`: what the host does with the files people send. */
|
|
5
|
+
export interface AttachmentsConfig {
|
|
6
|
+
/**
|
|
7
|
+
* Removes the files turns used once they are `maxAgeMs` old (counted from the turn that took
|
|
8
|
+
* the file), giving the owners' used-bytes allowance back; sweeps when the host starts and every
|
|
9
|
+
* `sweepEveryMs` (default one hour, at most `maxAgeMs`). Left out, used files live as long as
|
|
10
|
+
* their conversation. Files waiting for a turn are not touched; the plugin that took them
|
|
11
|
+
* discards those.
|
|
12
|
+
*/
|
|
13
|
+
retention?: AttachmentRetention;
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
const YEAR_MS = 365 * 24 * 60 * 60_000;
|
|
17
|
+
|
|
18
|
+
/** The `attachments` key's shape. */
|
|
19
|
+
export const attachmentsShape = optional(
|
|
20
|
+
shape({
|
|
21
|
+
retention: optional(
|
|
22
|
+
shape({
|
|
23
|
+
maxAgeMs: integer(1, YEAR_MS),
|
|
24
|
+
sweepEveryMs: optional(integer(1, YEAR_MS)),
|
|
25
|
+
}),
|
|
26
|
+
),
|
|
27
|
+
}),
|
|
28
|
+
);
|
|
@@ -26,6 +26,7 @@ import {
|
|
|
26
26
|
type ResolvedDiscord,
|
|
27
27
|
resolvedDiscord,
|
|
28
28
|
} from "./adapters.ts";
|
|
29
|
+
import { type AttachmentsConfig, attachmentsShape } from "./attachments.ts";
|
|
29
30
|
import {
|
|
30
31
|
bool,
|
|
31
32
|
guarded,
|
|
@@ -98,6 +99,8 @@ export interface RoundtableConfig {
|
|
|
98
99
|
database: { url: string };
|
|
99
100
|
/** Where the process keeps its files, such as pictures, attachments, and skills. */
|
|
100
101
|
dataDir: string;
|
|
102
|
+
/** What the host does with the files people send, such as removing them after a retention period. */
|
|
103
|
+
attachments?: AttachmentsConfig;
|
|
101
104
|
/** Pi's agent directory, holding the model logins; default `<dataDir>/pi`. */
|
|
102
105
|
agentDir?: string;
|
|
103
106
|
/** The model of the agents, written `<provider>/<id>`. */
|
|
@@ -211,6 +214,7 @@ const schema = shape({
|
|
|
211
214
|
discord: optional(discordShape),
|
|
212
215
|
database: shape({ url: text }),
|
|
213
216
|
dataDir: text,
|
|
217
|
+
attachments: attachmentsShape,
|
|
214
218
|
agentDir: optional(text),
|
|
215
219
|
model: text,
|
|
216
220
|
thinking: optional(oneOf(...THINKING_LEVELS)),
|
|
@@ -306,6 +310,8 @@ export interface ResolvedConfig {
|
|
|
306
310
|
discord?: ResolvedDiscord;
|
|
307
311
|
databaseUrl: string;
|
|
308
312
|
dataDir: string;
|
|
313
|
+
/** Absent `retention`: used files live as long as their conversation. */
|
|
314
|
+
attachments: AttachmentsConfig;
|
|
309
315
|
agentDir: string;
|
|
310
316
|
model: ModelRef;
|
|
311
317
|
thinking: ThinkingLevel;
|
|
@@ -421,6 +427,9 @@ export function resolveConfig(input: unknown): ResolvedConfig {
|
|
|
421
427
|
...(discord ? { discord: resolvedDiscord(discord, slug) } : {}),
|
|
422
428
|
databaseUrl: config.database.url,
|
|
423
429
|
dataDir: config.dataDir,
|
|
430
|
+
attachments: config.attachments?.retention
|
|
431
|
+
? { retention: config.attachments.retention }
|
|
432
|
+
: {},
|
|
424
433
|
agentDir: config.agentDir ?? `${config.dataDir}/pi`,
|
|
425
434
|
model,
|
|
426
435
|
thinking,
|
|
@@ -44,6 +44,26 @@ export interface TurnAttachmentOptions {
|
|
|
44
44
|
usedBytesLimit?: number;
|
|
45
45
|
}
|
|
46
46
|
|
|
47
|
+
/** What `expireUsed` asks: how old a used file must be to go. */
|
|
48
|
+
export interface ExpireUsedOptions {
|
|
49
|
+
/** Files a turn used more than this many milliseconds ago are removed. */
|
|
50
|
+
olderThanMs: number;
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
/** What `expireUsed` removed. */
|
|
54
|
+
export interface ExpiredAttachments {
|
|
55
|
+
/** How many used files were removed. */
|
|
56
|
+
files: number;
|
|
57
|
+
/** Their bytes; the owners' used-bytes tallies shrank by as much, less `unattributedBytes`. */
|
|
58
|
+
bytes: number;
|
|
59
|
+
/**
|
|
60
|
+
* Bytes of removed files whose owner the core could not tell, because a turn used them before
|
|
61
|
+
* the core recorded owners and their conversation has more than one person's tally. The files
|
|
62
|
+
* are gone, but no tally shrank for them; deleting the conversation clears the tallies.
|
|
63
|
+
*/
|
|
64
|
+
unattributedBytes: number;
|
|
65
|
+
}
|
|
66
|
+
|
|
47
67
|
/**
|
|
48
68
|
* Files a plugin takes from a person outside a turn, such as an upload over HTTP, kept for the
|
|
49
69
|
* conversation until a turn uses them. A saved file waits for its person: only the principal that
|
|
@@ -89,4 +109,17 @@ export interface AttachmentPort {
|
|
|
89
109
|
discardPending(olderThan: Date): Promise<number>;
|
|
90
110
|
/** The bytes the principal has saved that no turn used yet, across their conversations. */
|
|
91
111
|
pendingBytes(principalId: string): Promise<number>;
|
|
112
|
+
/**
|
|
113
|
+
* Removes the files turns used more than `olderThanMs` ago, and lowers each owner's used-bytes
|
|
114
|
+
* tally for the conversation by exactly the bytes removed, so the allowance `usedBytesLimit`
|
|
115
|
+
* checks comes back. The age counts from the turn that took the file, not from its upload. It
|
|
116
|
+
* runs one after another with that person's other calls, so it never races a message taking
|
|
117
|
+
* files. A removed file leaves a mark without its name: `read_attachment` and
|
|
118
|
+
* `ToolTurn.attachment()` then say the file was removed after its retention period. Files
|
|
119
|
+
* waiting for a turn are not touched; `discardPending` is theirs. A host sets
|
|
120
|
+
* `attachments.retention` to run this on a timer. Optional, so a port a plugin wrote before it
|
|
121
|
+
* existed still type-checks; the host's own port always has it, and a retention setting refuses
|
|
122
|
+
* a port without it.
|
|
123
|
+
*/
|
|
124
|
+
expireUsed?(options: ExpireUsedOptions): Promise<ExpiredAttachments>;
|
|
92
125
|
}
|
|
@@ -279,6 +279,9 @@ export async function defineRoundtable(
|
|
|
279
279
|
},
|
|
280
280
|
database: { url: config.databaseUrl },
|
|
281
281
|
dataDir: config.dataDir,
|
|
282
|
+
...(config.attachments.retention
|
|
283
|
+
? { attachments: { retention: config.attachments.retention } }
|
|
284
|
+
: {}),
|
|
282
285
|
toolTiers: toolTiers(config.toolTiers, logger),
|
|
283
286
|
listeners: [
|
|
284
287
|
...(config.http ? [publicListener(config.http)] : []),
|
package/src/core/define.ts
CHANGED
|
@@ -26,6 +26,14 @@ export interface ToolTurn {
|
|
|
26
26
|
/** The agent whose turn it is. */
|
|
27
27
|
agent: AgentTurnScope | undefined;
|
|
28
28
|
signal: AbortSignal | undefined;
|
|
29
|
+
/**
|
|
30
|
+
* Where the turn's request came from, as its transport reported it, such as the browser origin
|
|
31
|
+
* of a web chat connection (`https://chat.example.com`); undefined for a turn whose channel gives
|
|
32
|
+
* none, such as a Discord message or a schedule. Only the transport sets it, never the message
|
|
33
|
+
* text or the model, so a tool may label a record with it; a tool that needs one fails closed
|
|
34
|
+
* when it is undefined.
|
|
35
|
+
*/
|
|
36
|
+
origin?: string;
|
|
29
37
|
/**
|
|
30
38
|
* The session's shared workspace and scratch dir, where a tool may save a file for the model to
|
|
31
39
|
* pass on by path; undefined for a session without a workspace.
|
package/src/core/domain/ports.ts
CHANGED
|
@@ -30,6 +30,13 @@ export interface TurnRequest {
|
|
|
30
30
|
* starts a turn on someone's behalf gets theirs from `IDENTITY.speakerFor`.
|
|
31
31
|
*/
|
|
32
32
|
speaker: Speaker;
|
|
33
|
+
/**
|
|
34
|
+
* Where the request came from, as the transport that carried it knows: the browser origin of a
|
|
35
|
+
* web chat connection, such as `https://chat.example.com`. A channel with no such thing leaves it
|
|
36
|
+
* out. Only a channel's transport sets it, never the message text or the model; a tool reads it
|
|
37
|
+
* as `ToolTurn.origin`.
|
|
38
|
+
*/
|
|
39
|
+
origin?: string;
|
|
33
40
|
/**
|
|
34
41
|
* Who the conversation belongs to, as the host records it: `private` to one principal, or
|
|
35
42
|
* `shared` by whoever its claim admits, which it is taken to be when absent. A private
|
package/src/core/host.ts
CHANGED
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
import type { SQL } from "bun";
|
|
2
2
|
import { attachmentPort } from "./attachments/attachment-port.ts";
|
|
3
|
+
import {
|
|
4
|
+
type AttachmentRetention,
|
|
5
|
+
retentionServices,
|
|
6
|
+
} from "./attachments/attachment-retention.ts";
|
|
3
7
|
import type { ConversationPort } from "./contract/channels.ts";
|
|
4
8
|
import type { SurfacePort } from "./contract/surface.ts";
|
|
5
9
|
import { openPool, runMigrations } from "./db/migrations.ts";
|
|
@@ -82,6 +86,13 @@ export interface RoundtableOptions {
|
|
|
82
86
|
database?: { url: string };
|
|
83
87
|
/** The directory the host keeps files in; `context.attachments` keeps its files there. Without it that port refuses every call. */
|
|
84
88
|
dataDir?: string;
|
|
89
|
+
/**
|
|
90
|
+
* What the host does with the attachments it keeps. `retention` removes the files turns used once
|
|
91
|
+
* they are `maxAgeMs` old, sweeping at start and every `sweepEveryMs` (default one hour), and gives
|
|
92
|
+
* the owners' used-bytes allowance back; without it used files live as long as their conversation.
|
|
93
|
+
* Needs `dataDir`.
|
|
94
|
+
*/
|
|
95
|
+
attachments?: { retention?: AttachmentRetention };
|
|
85
96
|
/** Receives the work a shutdown drain gave up on, before any service stops. */
|
|
86
97
|
aborted?: (left: string[]) => Promise<void>;
|
|
87
98
|
/**
|
|
@@ -270,6 +281,17 @@ export class Roundtable {
|
|
|
270
281
|
this.#services = new ServiceRegistry(this.#active);
|
|
271
282
|
this.#services.checkRequires();
|
|
272
283
|
const providers = resolveProviders(this.#active, this.#options.judgeModel);
|
|
284
|
+
const attachments = attachmentPort({
|
|
285
|
+
dataDir: this.#options.dataDir,
|
|
286
|
+
registry: () => this.#services?.find(CONVERSATIONS),
|
|
287
|
+
logger,
|
|
288
|
+
});
|
|
289
|
+
const retention = retentionServices(
|
|
290
|
+
this.#options.attachments?.retention,
|
|
291
|
+
this.#options.dataDir,
|
|
292
|
+
attachments,
|
|
293
|
+
logger,
|
|
294
|
+
);
|
|
273
295
|
await this.#migrate();
|
|
274
296
|
this.#registry = await collectContributions(
|
|
275
297
|
this.#active,
|
|
@@ -287,11 +309,7 @@ export class Roundtable {
|
|
|
287
309
|
toolTiers: this.#tiers,
|
|
288
310
|
events: this.#events.sink,
|
|
289
311
|
conversations: this.#conversations(),
|
|
290
|
-
attachments
|
|
291
|
-
dataDir: this.#options.dataDir,
|
|
292
|
-
registry: () => this.#services?.find(CONVERSATIONS),
|
|
293
|
-
logger,
|
|
294
|
-
}),
|
|
312
|
+
attachments,
|
|
295
313
|
surfaces: this.#surfaces(),
|
|
296
314
|
turns: this.#turns(),
|
|
297
315
|
database: () => {
|
|
@@ -331,7 +349,7 @@ export class Roundtable {
|
|
|
331
349
|
});
|
|
332
350
|
const http = new HttpListeners(listeners, routes, logger);
|
|
333
351
|
for (const plugin of this.#active) await plugin.preflight?.();
|
|
334
|
-
for (const service of this.#registry.services) {
|
|
352
|
+
for (const service of [...this.#registry.services, ...retention]) {
|
|
335
353
|
await service.start?.();
|
|
336
354
|
this.#started.push(service);
|
|
337
355
|
}
|
|
@@ -29,6 +29,12 @@ export interface ConversationTurnInput {
|
|
|
29
29
|
text: string;
|
|
30
30
|
/** Who the turn is for; its tier limits the tools. */
|
|
31
31
|
speaker: Speaker;
|
|
32
|
+
/**
|
|
33
|
+
* Where the request came from, as the claim's transport knows it, such as a web chat
|
|
34
|
+
* connection's browser origin; tools read it as `ToolTurn.origin`. Leave it out for a turn the
|
|
35
|
+
* transport gives none, and never fill it from message text.
|
|
36
|
+
*/
|
|
37
|
+
origin?: string;
|
|
32
38
|
/** Files and images the message carries. */
|
|
33
39
|
attachments?: TurnAttachments;
|
|
34
40
|
/** The turn's tools; by default the plugins' `agentSelection`, read at the turn. */
|
|
@@ -184,6 +190,7 @@ export function conversationTurns(
|
|
|
184
190
|
},
|
|
185
191
|
text: input.text,
|
|
186
192
|
speaker,
|
|
193
|
+
...(input.origin === undefined ? {} : { origin: input.origin }),
|
|
187
194
|
...(conversation ? { conversation } : {}),
|
|
188
195
|
...(input.attachments
|
|
189
196
|
? { attachments: input.attachments }
|
|
@@ -71,10 +71,10 @@ export class PiAgentRuntime implements AgentRuntime {
|
|
|
71
71
|
readonly #tiers: ToolTiers;
|
|
72
72
|
readonly #factory: SessionFactory;
|
|
73
73
|
readonly #sessions: ConversationSessions;
|
|
74
|
-
/** The tier and
|
|
74
|
+
/** The tier, speaker and origin of each conversation's running turn. */
|
|
75
75
|
readonly #turns = new Map<
|
|
76
76
|
ChannelKey,
|
|
77
|
-
{ tier: Tier; speaker: Speaker; withheld: boolean }
|
|
77
|
+
{ tier: Tier; speaker: Speaker; origin?: string; withheld: boolean }
|
|
78
78
|
>();
|
|
79
79
|
/** Each conversation's turn in progress, which the owner may steer or stop. */
|
|
80
80
|
readonly #running = new Map<ChannelKey, SteerableRun>();
|
|
@@ -87,6 +87,7 @@ export class PiAgentRuntime implements AgentRuntime {
|
|
|
87
87
|
this.#tiers = options.toolTiers ?? toolTiers();
|
|
88
88
|
this.#factory = new SessionFactory(options, {
|
|
89
89
|
speaker: (channel) => this.#turns.get(channel)?.speaker,
|
|
90
|
+
origin: (channel) => this.#turns.get(channel)?.origin,
|
|
90
91
|
withholdsMemory: (channel) => this.#turns.get(channel)?.withheld === true,
|
|
91
92
|
runTask: (scope, task) => this.#runTask(scope, task),
|
|
92
93
|
});
|
|
@@ -192,7 +193,12 @@ export class PiAgentRuntime implements AgentRuntime {
|
|
|
192
193
|
loadsMemory(registered),
|
|
193
194
|
);
|
|
194
195
|
this.#memoryTurns.set(key, markPrivateMemory);
|
|
195
|
-
this.#turns.set(key, {
|
|
196
|
+
this.#turns.set(key, {
|
|
197
|
+
tier,
|
|
198
|
+
speaker: request.speaker,
|
|
199
|
+
...(request.origin === undefined ? {} : { origin: request.origin }),
|
|
200
|
+
withheld,
|
|
201
|
+
});
|
|
196
202
|
const missing = wanted.filter((name) => !registered.has(name));
|
|
197
203
|
if (missing.length > 0) {
|
|
198
204
|
logger.warn(
|
|
@@ -73,6 +73,8 @@ import {
|
|
|
73
73
|
export interface SessionFactoryDeps {
|
|
74
74
|
/** The person the conversation's running turn is for. */
|
|
75
75
|
speaker(channel: ChannelKey): Speaker | undefined;
|
|
76
|
+
/** Where the conversation's running turn came from, as its transport reported it. */
|
|
77
|
+
origin(channel: ChannelKey): string | undefined;
|
|
76
78
|
/** Whether the conversation's running turn loads no memory, as the agent server's option keeps a speaker out of it. */
|
|
77
79
|
withholdsMemory(channel: ChannelKey): boolean;
|
|
78
80
|
/**
|
|
@@ -355,6 +357,7 @@ export class SessionFactory {
|
|
|
355
357
|
return state.memory;
|
|
356
358
|
},
|
|
357
359
|
speaker: () => this.#deps.speaker(turnKey),
|
|
360
|
+
origin: () => this.#deps.origin(turnKey),
|
|
358
361
|
runTask: (task) =>
|
|
359
362
|
this.#deps.runTask({ turn: turnKey, home: channel }, task),
|
|
360
363
|
};
|
package/src/core/sessions.ts
CHANGED
|
@@ -92,6 +92,12 @@ export interface SessionContext {
|
|
|
92
92
|
attachmentDir?: string;
|
|
93
93
|
/** The person the session's running turn is for; undefined between turns. */
|
|
94
94
|
speaker(): Speaker | undefined;
|
|
95
|
+
/**
|
|
96
|
+
* Where the running turn's request came from, as `TurnRequest.origin` says; undefined between
|
|
97
|
+
* turns and for a turn whose transport gives none. A context built before 0.9.10 has no such
|
|
98
|
+
* method, which reads as undefined.
|
|
99
|
+
*/
|
|
100
|
+
origin?(): string | undefined;
|
|
95
101
|
/** Runs a task beside this session, under its confirmation gate. */
|
|
96
102
|
runTask(task: TransientTask): Promise<string>;
|
|
97
103
|
}
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
import { join } from "node:path";
|
|
2
2
|
import { extractText, getDocumentProxy } from "unpdf";
|
|
3
|
+
import { wasExpired } from "../attachments/attachment-expiry.ts";
|
|
3
4
|
import { isAttachmentName, plainText } from "../attachments/attachment-name.ts";
|
|
5
|
+
import { AttachmentExpiredError } from "../attachments/tool-attachment.ts";
|
|
4
6
|
|
|
5
7
|
/** Characters returned per read; the model can ask for a later offset. */
|
|
6
8
|
export const READ_LIMIT = 60_000;
|
|
@@ -72,6 +74,8 @@ export async function readAttachment(
|
|
|
72
74
|
}
|
|
73
75
|
const handle = Bun.file(join(dir, file));
|
|
74
76
|
if (!(await handle.exists())) {
|
|
77
|
+
if (await wasExpired(dir, file))
|
|
78
|
+
throw new AttachmentExpiredError(plainText(file));
|
|
75
79
|
throw new Error(`no attachment named "${file}" in this channel`);
|
|
76
80
|
}
|
|
77
81
|
let bytes: Uint8Array;
|
package/src/core/tool-turn.ts
CHANGED
|
@@ -15,11 +15,13 @@ export function toolTurn(
|
|
|
15
15
|
context: SessionContext,
|
|
16
16
|
signal: AbortSignal | undefined,
|
|
17
17
|
): ToolTurn {
|
|
18
|
+
const origin = context.origin?.();
|
|
18
19
|
return {
|
|
19
20
|
speaker: context.speaker(),
|
|
20
21
|
channel: context.turnChannel,
|
|
21
22
|
agent: context.agent,
|
|
22
23
|
signal,
|
|
24
|
+
...(origin === undefined ? {} : { origin }),
|
|
23
25
|
...(context.workspace ? { workspace: context.workspace } : {}),
|
|
24
26
|
attachFile: attachReplyFile,
|
|
25
27
|
attachment: async (file) => {
|
package/src/index.ts
CHANGED
|
@@ -16,6 +16,7 @@ export type {
|
|
|
16
16
|
GroupStatus,
|
|
17
17
|
TeamStatus,
|
|
18
18
|
} from "./core/agents/team-status.ts";
|
|
19
|
+
export type { AttachmentRetention } from "./core/attachments/attachment-retention.ts";
|
|
19
20
|
export type { ToolAttachment } from "./core/attachments/tool-attachment.ts";
|
|
20
21
|
export {
|
|
21
22
|
AGENT_SERVER_PLUGIN,
|
|
@@ -26,6 +27,7 @@ export type {
|
|
|
26
27
|
AccessOwnerConfig,
|
|
27
28
|
AccessTierConfig,
|
|
28
29
|
} from "./core/config/access.ts";
|
|
30
|
+
export type { AttachmentsConfig } from "./core/config/attachments.ts";
|
|
29
31
|
export type {
|
|
30
32
|
AdapterConfig,
|
|
31
33
|
BackgroundConfig,
|
|
@@ -38,6 +40,8 @@ export type {
|
|
|
38
40
|
AttachmentPort,
|
|
39
41
|
AttachmentRefusalCode,
|
|
40
42
|
AttachmentUpload,
|
|
43
|
+
ExpiredAttachments,
|
|
44
|
+
ExpireUsedOptions,
|
|
41
45
|
TurnAttachmentOptions,
|
|
42
46
|
} from "./core/contract/attachments.ts";
|
|
43
47
|
export { AttachmentRefusal } from "./core/contract/attachments.ts";
|
package/src/testing.ts
CHANGED
|
@@ -270,6 +270,8 @@ export interface TestPluginResult {
|
|
|
270
270
|
/** `workspace` gives the session a workspace and scratch dir, as an agent's session has. */
|
|
271
271
|
options?: {
|
|
272
272
|
speaker?: Speaker;
|
|
273
|
+
/** Where the turn's request came from, as `ToolTurn.origin` shows it to the tool. */
|
|
274
|
+
origin?: string;
|
|
273
275
|
channel?: ChannelKey;
|
|
274
276
|
workspace?: SessionContext["workspace"];
|
|
275
277
|
},
|
|
@@ -640,6 +642,7 @@ export async function testPlugin(
|
|
|
640
642
|
...(runOptions?.workspace ? { workspace: runOptions.workspace } : {}),
|
|
641
643
|
attachmentDir: ownerAttachmentDir(attachmentDir, channel),
|
|
642
644
|
speaker: () => runOptions?.speaker,
|
|
645
|
+
origin: () => runOptions?.origin,
|
|
643
646
|
runTask: async () => {
|
|
644
647
|
throw new Error("test session cannot run tasks");
|
|
645
648
|
},
|