threadwire 0.1.15 → 0.1.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/README.md +6 -6
- package/docs/delegated-result-protocol.md +20 -0
- package/docs/evidence-artifacts.md +37 -5
- package/package.json +1 -1
- package/scripts/install-local-launcher.js +23 -1
- package/src/cli.js +173 -34
- package/src/delegated-result-admission.js +13 -0
- package/src/evidence-store.js +93 -23
- package/src/provider-capacity-codex.js +2 -1
- package/src/providers/executable.js +17 -4
- package/src/telegram-ingress/core.js +180 -61
- package/src/worker-control.js +11 -0
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.1.17 - 2026-08-06
|
|
6
|
+
|
|
7
|
+
- Persist a normalized `provider_result` evidence artifact before final notification drain, separating it from quota-limited raw prompt/provider-stream evidence and separating `provider_completed`/`provider_failed` from authenticated `delivery_pending`/`delivery_succeeded`/`delivery_failed` state. Only durably finalized raw handles enter normalized results. Exhausted Telegram Retry-After/backoff attempts no longer rewrite provider success or discard its continuation/evidence handles, and ingress evidence-handle notices now use the existing bounded notice queue instead of an unbounded direct send. Isolated completion rejection and deadline cancellation now durably publish `provider_failed` with already observed assistant, continuation, and raw evidence before propagating the execution failure. Delivery transitions recover the exact sealed artifact redaction policy across restart, preserve its durable reservation, and record setup failures as `delivery_failed`. Add the token-free `file:<absolute-path>` target for private, fsynced, byte-identical normalized stdout capture without message delivery; result/transcript and activity paths now reject final symlink and parent-directory aliases before opening either output. Delivery failure records lack of a confirmed response, not exactly-once Telegram semantics after an ambiguous network failure.
|
|
8
|
+
|
|
9
|
+
## 0.1.16 - 2026-08-04
|
|
10
|
+
|
|
11
|
+
- Validate provider executable overrides at resolution time instead of deferring to spawn: `providerExecutable()` checks that a `THREADWIRE_*_BIN` override path exists, is executable (`X_OK`), and is a regular file (not a directory), falling back to the default with a diagnostic when the validation fails. This catches bogus overrides, stale temporary adapter paths, and directory targets before `runWorker` hits EACCES. The full test suite now uses real temporary executables instead of fake paths that relied on the old unchecked pass-through.
|
|
12
|
+
|
|
5
13
|
## 0.1.15 - 2026-08-03
|
|
6
14
|
|
|
7
15
|
- Add machine-readable run health supervision: the existing append-only `--activity-log` now records safe timestamped controller/provider/session/activity/health/terminal facts with an explicit `controller-started` fact (the run controller's own PID, not an injected status-process PID). The `threadwire status --activity-log <absolute-path>` read-only command emits exactly one closed versioned JSON document (`version`, `state`, `provider`, `controllerPid`, `providerPid`, `continuationHandle`, `startedAt`, `lastActivityAt`, `quietForMs`, `health`, `terminal`). PIDs are recorded non-authoritative facts; status never inspects or signals processes. Health events carry a bounded disposition (`retrying`/`blocked`) plus safe category (`authentication`/`permission`/`rate-limit`/`quota`/`billing`/`model`/`network`/`protocol`/`unknown`) and optional `retryAfterMs` in milliseconds, extracted only from an explicit small list of safe provider record containers (top-level record and known nested `error`/`part`/`part.error`/`.data` shapes) — never recursive traversal and never message/text regex. Generic HTTP 429 → retrying/rate-limit; structured `exceeded_current_quota_error`/`insufficient_quota`/`insufficient_balance` → blocked quota/billing; ambiguous short fields such as `retry_after` (seconds) are never interpreted as milliseconds. If a provider protocol suppresses upstream error details (e.g., real Moonshot OpenCode sessions), no health fact is recorded and status stays running/unknown; Hermes must perform an independent bounded provider probe. Activity records are throttled (5 s) with guaranteed first-activity recording. Terminal facts always override earlier retrying/blocked health state and are written exactly once for local and isolated-runtime run paths; signal-derived exit codes 130/143 record as `cancelled`, all others as `completed` (0) or `failed` (nonzero). A later normal activity record clears a prior `retrying` health back to running; `blocked` stays sticky until terminal overrides it. The status command reads the log within a documented 1 MiB byte cap, never opens the file writable, skips at most one incomplete trailing JSONL record, rejects malformed/schema-unsafe records and overlong/malformed continuation handles, validates timestamps as safe non-negative Date-range integers and the injected clock similarly, and never claims stalled/dead from silence. Existing backward-compatible records without timestamps remain readable.
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Threadwire
|
|
2
2
|
|
|
3
|
-
Threadwire is a Node.js CLI that launches a Codex, Claude, Kimi, or OpenCode worker and streams a concise, human-readable activity feed to one explicitly selected Telegram destination
|
|
3
|
+
Threadwire is a Node.js CLI that launches a Codex, Claude, Kimi, or OpenCode worker and either streams a concise, human-readable activity feed to one explicitly selected Telegram destination or writes a normalized result to an explicit local file target. The default Telegram feed includes the assistant's response prose, lifecycle outcomes, and diagnostics. Provider tool-status messages are disabled by default.
|
|
4
4
|
|
|
5
5
|
For the container-native webhook deployment, including its named-volume worktree,
|
|
6
6
|
non-root Codex provider, Docker secrets, loopback TLS proxy, verification, and
|
|
@@ -68,7 +68,7 @@ Use the local launcher. Install or refresh it after updating Threadwire with `np
|
|
|
68
68
|
-- --model example-model
|
|
69
69
|
```
|
|
70
70
|
|
|
71
|
-
The target
|
|
71
|
+
The target has the form `telegram:<chat-id>:<thread-id>` for topics, `telegram:<chat-id>` for direct messages (DMs) and ordinary chats, or `file:<absolute-path>` for local no-message execution. A file target requires a normalized absolute file path, needs no Telegram token, creates the file exclusively with mode `0600`, fsyncs it and its directory, and writes the exact same single normalized `delegated_result` record as stdout. It cannot be combined with `--transcript`. Telegram chat IDs and optional thread IDs keep their existing validation. `--provider` must be exactly one of `codex`, `claude`, `kimi`, or `opencode`. Malformed or missing provider or target arguments are hard failures (exit code 2) reported through Threadwire's normal error path before any provider is launched. `--process-number` accepts an explicit positive safe integer for automation; when omitted, Threadwire uses its launcher PID. Every Telegram message is labeled consistently, such as `[P42] …`, including chunked continuations.
|
|
72
72
|
|
|
73
73
|
`--cwd` is the complete workspace contract for every provider. It may name any caller-accessible directory, including a non-Git directory. Threadwire passes it unchanged to the native provider and performs no profile, repository, source-identity, task-binding, or root-allowlist lookup. `--workspace-profile` has been removed and is rejected; migrate invocations by replacing it with `--cwd <directory>`.
|
|
74
74
|
|
|
@@ -84,7 +84,7 @@ The relay contract: the caller supplies `THREADWIRE_TARGET` (`telegram:<chat-id>
|
|
|
84
84
|
|
|
85
85
|
Pass `--resume-session <exact-provider-session-id>` to continue a stored Codex, Claude, Kimi, or OpenCode session. Threadwire preserves the provider's native resume ordering and streaming options and never uses implicit latest-session behavior.
|
|
86
86
|
|
|
87
|
-
Pass `--transcript <path>` to capture the exact normalized, caller-facing JSONL protocol written to run stdout. The file is created privately (mode `0600`), must not already exist, and contains the same records in the same byte order as stdout.
|
|
87
|
+
Pass `--transcript <path>` with a Telegram target to capture the exact normalized, caller-facing JSONL protocol written to run stdout. The file is created privately (mode `0600`), must not already exist, and contains the same records in the same byte order as stdout. A `file:<absolute-path>` target provides this capture directly and therefore rejects a separate `--transcript`. Neither form contains provider JSONL, reasoning, tool payloads, prompts, stderr, or credentials. The result/transcript path and `--activity-log` path must not canonically alias; final-component symlinks and aliased parent directories are rejected during preflight before either output is opened.
|
|
88
88
|
|
|
89
89
|
`--activity-log <path>` is different: it retains its append contract and writes safe timestamped lifecycle facts—provider start (`provider`, child `pid`), available native session ID, throttled activity markers, structured health events (bounded `retrying`/`blocked` disposition with safe category), and exactly one terminal fact. It is local-only and never enters Telegram or normalized stdout.
|
|
90
90
|
|
|
@@ -94,7 +94,7 @@ Provider stdout is decoded as JSONL with a finite 1 MiB UTF-8 byte limit for eac
|
|
|
94
94
|
|
|
95
95
|
For normal `threadwire run` execution, Threadwire stdout is the parent-model admission plane. It emits exactly one versioned terminal `delegated_result` JSON object and no progress records. The envelope contains a bounded, credential-redacted, control-normalized final conclusion when one was produced, structured terminal state and exit code, and the exact bounded provider session handle when available. Its closed schema can also represent concise blocker/decision requests, opaque artifact handles, commit/URL references, and a concise validation summary; current provider adapters do not infer these optional fields from prose. Unknown, nested, wrongly typed, cardinality-invalid, or oversized admission candidates fail closed. See [Delegated-result protocol](docs/delegated-result-protocol.md) for the complete schema, limits, and caller obligations.
|
|
96
96
|
|
|
97
|
-
Scoped
|
|
97
|
+
Scoped evidence artifacts are separate from both files above. Set the trusted operator setting `THREADWIRE_EVIDENCE_ROOT` for standalone runs; Telegram ingress uses its dedicated evidence volume automatically. CLI and ingress runs reserve separate normalized provider-result and raw prompt/provider-stream artifacts; CLI raw evidence also records final context metrics. A raw handle enters a normalized result only after the raw artifact is durable; if raw metrics append or finalization fails, the error remains visible while the completed provider result retains only its independently durable result handle. The result artifact carries a stable delivery identity and authenticated state, and ingress sends that result handle through the same bounded notice queue. Retrieve later with `threadwire evidence read` or the authenticated `/evidence` Telegram command and an explicit bounded selector. See [Evidence artifacts](docs/evidence-artifacts.md) for permissions, delivery states, quotas, retention, recovery, authorization, redaction, privacy, and cleanup behavior.
|
|
98
98
|
|
|
99
99
|
`threadwire capacity` is a read-only reporting command that probes live account capacity for authenticated Codex (`codex app-server` JSON-RPC `account/rateLimits/read`) and Kimi (`GET https://api.kimi.com/coding/v1/usages` with the installed OAuth credential), normalizes both into one closed short/long-window schema, and prints one deterministic reserve-gated admission selection as a single JSON document. It persists nothing, never prints tokens or account data, takes no session arguments, and does not change `run` provider selection; an external worker ledger consumes the report and owns pinning provider/model/session. See [Capacity admission](docs/capacity-admission.md) for the schema, tie-break, error taxonomy, and security boundary.
|
|
100
100
|
|
|
@@ -104,9 +104,9 @@ The Telegram bot token is used only by Threadwire's notifier and both `TELEGRAM_
|
|
|
104
104
|
|
|
105
105
|
A single trusted renderer is the only code allowed to enqueue Telegram notices. By default, parsed assistant text is forwarded in order with every nonblank rendered line labeled `[P<n>] Assistant: `; no CLI flag is required. This preserves Markdown, code-block indentation, blank lines, and whitespace attached to nonblank content while preventing assistant text from imitating trusted activity on a later line. Streamed fenced JavaScript blocks (` ```js ` / ` ```javascript `) are instead emitted as independently valid, escaped Telegram HTML code messages: the `[P<n>] Assistant:` label stays above the code block, and long blocks are Unicode-safe chunked without exposing raw fences. A short blank-only stream is suppressed at normal close. Because retention is bounded, a blank-only prefix longer than the buffer limit is emitted in oldest-first chunks while at most one limit's worth remains deferred; a later visible assistant character therefore still reconstructs the input exactly, but an indefinitely blank stream cannot be suppressed indefinitely. With `--tool-messages`, tool activity uses a single edit-in-place status line (silent `🛠 …` on start, edited to begin `✅` on finish); without it, both the send and completion edit are suppressed. Worker start/completion/failure and warning/error diagnostics continue to use fixed labeled templates in either mode. Non-assistant provider fields—including summaries, tool names, metadata IDs, inputs, commands, stderr content, raw JSON, reasoning/thinking, tokens, and retry internals—are not forwarded. When tool messages are enabled, the sole provider-data exception is normalized Codex command-completion output: it is appended only to its original tool status edit as an escaped, bounded, expandable Telegram HTML blockquote after shared credential redaction.
|
|
106
106
|
|
|
107
|
-
Assistant deltas from all provider stream IDs share one ordered sentence-boundary buffer, preserving provider-event order. The buffer has a finite 16,384-character default: a continuously active punctuation-free stream emits and clears Unicode-safe fixed-size chunks instead of growing indefinitely. Structured events flush any earlier assistant tail, including retained leading whitespace, before their own fixed notice. Completed boundaries enter delivery without a deliberate batching delay; messages combine only when already naturally queued behind delivery. When Telegram briefly falls behind, Threadwire emits a fixed safe notice that the notifier saturated and then coalesces only eligible assistant progress until headroom returns; lifecycle, diagnostics, and tool activity remain ordered and non-droppable. Notices remain ordered, size-limited, paced, and retried conservatively when Telegram reports throttling or a transient failure.
|
|
107
|
+
Assistant deltas from all provider stream IDs share one ordered sentence-boundary buffer, preserving provider-event order. The buffer has a finite 16,384-character default: a continuously active punctuation-free stream emits and clears Unicode-safe fixed-size chunks instead of growing indefinitely. Structured events flush any earlier assistant tail, including retained leading whitespace, before their own fixed notice. Completed boundaries enter delivery without a deliberate batching delay; messages combine only when already naturally queued behind delivery. When Telegram briefly falls behind, Threadwire emits a fixed safe notice that the notifier saturated and then coalesces only eligible assistant progress until headroom returns; lifecycle, diagnostics, and tool activity remain ordered and non-droppable. Notices remain ordered, size-limited, paced, and retried conservatively when Telegram reports throttling or a transient failure. A provider result is admitted independently: exhausted delivery retries return a delivery failure without rerunning the provider or changing its completed normalized result.
|
|
108
108
|
|
|
109
|
-
The delivery queue has finite defaults of 100 pending notices and 256 KiB of pending UTF-8 text. Message length and capacity settings require positive safe integers; pacing settings require nonnegative safe integers. A notice larger than the byte limit or a limit-plus-one enqueue latches a terminal capacity error, discards queued (not already in-flight) notices, and rejects every later enqueue. Unicode chunks never split a surrogate pair, and every labeled continuation retains its complete process label. If the label cannot fit with content under a configured message length, delivery latches a terminal chunking error before sending. These rules make delivery loss visible and prevent slow or rate-limited transports from consuming unbounded memory.
|
|
109
|
+
The delivery queue has finite defaults of 100 pending notices and 256 KiB of pending UTF-8 text. Message length and capacity settings require positive safe integers; pacing settings require nonnegative safe integers. A notice larger than the byte limit or a limit-plus-one enqueue latches a terminal capacity error, discards queued (not already in-flight) notices, and rejects every later enqueue. Unicode chunks never split a surrogate pair, and every labeled continuation retains its complete process label. If the label cannot fit with content under a configured message length, delivery latches a terminal chunking error before sending. These rules make delivery loss visible and prevent slow or rate-limited transports from consuming unbounded memory. A persisted `delivery_failed` state means Threadwire received no successful response after the bounded attempts. A timeout or dropped response can be ambiguous, so Threadwire guarantees provider non-rerun but does not claim exactly-once Telegram display.
|
|
110
110
|
|
|
111
111
|
## Development
|
|
112
112
|
|
|
@@ -7,6 +7,10 @@ Threadwire has two deliberately separate output planes:
|
|
|
7
7
|
- Standard output for a normal `threadwire run` is the parent-model admission
|
|
8
8
|
plane. It contains exactly one newline-terminated terminal JSON envelope.
|
|
9
9
|
|
|
10
|
+
`--target file:<absolute-path>` disables message delivery and durably writes the
|
|
11
|
+
same admission record to that private, exclusive file before stdout. It requires
|
|
12
|
+
no Telegram token and cannot be combined with `--transcript`.
|
|
13
|
+
|
|
10
14
|
Standard error remains a caller diagnostic channel and is never admission
|
|
11
15
|
content. Provider stdout is private parser input.
|
|
12
16
|
|
|
@@ -108,6 +112,22 @@ references exactly. Streamed conclusion collection is credential-redacted,
|
|
|
108
112
|
control-normalized, and deterministically truncated with a final ellipsis when
|
|
109
113
|
necessary. Code-point accounting never splits a UTF-16 surrogate pair.
|
|
110
114
|
|
|
115
|
+
Provider and delivery outcomes are independent. When evidence is enabled, the
|
|
116
|
+
first artifact handle identifies a normalized `provider_result` containing the
|
|
117
|
+
terminal envelope and final assistant content. That artifact is atomically
|
|
118
|
+
published before final notification drain and records a stable delivery
|
|
119
|
+
identity/state. Any separate raw-evidence handle is admitted only after that
|
|
120
|
+
artifact is durable, so a late raw quota or finalization failure cannot leave a
|
|
121
|
+
stale recovery capability in either result. A provider exit zero remains
|
|
122
|
+
`completed` even when the process returns exit 2 because notification delivery
|
|
123
|
+
exhausted its bounded retries.
|
|
124
|
+
An isolated completion rejection or deadline still publishes `provider_failed`
|
|
125
|
+
with assistant content, continuation, and durable raw evidence observed before
|
|
126
|
+
the failure, then propagates the execution error after bounded cancellation and
|
|
127
|
+
cleanup settlement. Any cleanup or delivery failure remains attached without
|
|
128
|
+
masking that primary execution failure.
|
|
129
|
+
The continuation and evidence handles remain admitted for recovery.
|
|
130
|
+
|
|
111
131
|
A continuation handle consists of one or more colon-separated opaque segments.
|
|
112
132
|
Each segment must start with an ASCII letter or digit and may then contain only
|
|
113
133
|
ASCII letters, digits, `_`, `-`, and `.`. This covers provider UUIDs, token IDs
|
|
@@ -6,11 +6,24 @@ parent admission stream. Telegram uses the dedicated
|
|
|
6
6
|
with the trusted operator setting `THREADWIRE_EVIDENCE_ROOT`; callers cannot
|
|
7
7
|
select a root through CLI arguments.
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
and
|
|
9
|
+
For CLI and Telegram ingress runs, the store publishes a small normalized
|
|
10
|
+
result artifact separately from the prompt and framed provider stdout/stderr
|
|
11
|
+
artifact. The result contains one closed `provider_result` record with `provider_completed` or
|
|
12
|
+
`provider_failed`, the bounded final assistant content, the exact continuation,
|
|
13
|
+
and the terminal `delegated_result`. It is fsynced and atomically published
|
|
14
|
+
before final notification drain. A raw handle is admitted only after its
|
|
15
|
+
artifact is also durable; raw quota or finalization failure is surfaced without
|
|
16
|
+
invalidating the normalized result handle. The raw artifact still contains tool
|
|
17
|
+
calls, commands, raw results, and intermediate narration. CLI raw evidence also
|
|
18
|
+
ends with the final context metrics.
|
|
19
|
+
An isolated completion rejection or deadline first awaits bounded runtime
|
|
20
|
+
cancellation and cleanup settlement, then publishes `provider_failed` with
|
|
21
|
+
assistant content, continuation, and raw evidence observed before the failure
|
|
22
|
+
before propagating the execution error. Cleanup or delivery failures remain
|
|
23
|
+
attached to that primary execution failure.
|
|
24
|
+
Threadwire stdout remains one bounded `delegated_result`; evidence adds only
|
|
25
|
+
unguessable bearer handles. Storage failure fails explicitly and never falls
|
|
26
|
+
back to inlining evidence.
|
|
14
27
|
|
|
15
28
|
## Security and lifecycle
|
|
16
29
|
|
|
@@ -39,6 +52,25 @@ while active. Fsync and atomic rename publish payload before ready metadata.
|
|
|
39
52
|
Startup removes partial/corrupt states and rehydrates only exact-schema,
|
|
40
53
|
unexpired ready artifacts.
|
|
41
54
|
|
|
55
|
+
Result metadata additionally authenticates one stable delivery identity and a
|
|
56
|
+
closed state: `delivery_pending`, `delivery_succeeded`, `delivery_failed`, or
|
|
57
|
+
`delivery_not_requested`. The pending record is part of the initial atomic
|
|
58
|
+
publication. After the existing bounded notice queue settles, Threadwire
|
|
59
|
+
atomically replaces ready metadata and re-seals its redaction-policy AAD with
|
|
60
|
+
the terminal delivery state. The transition decrypts the exact artifact policy
|
|
61
|
+
from ready metadata rather than reconstructing it from runtime configuration or
|
|
62
|
+
an in-memory cache, preserving duplicate and artifact-specific redactions and
|
|
63
|
+
its durable reservation across restart. Retrieval returns this state with
|
|
64
|
+
result-artifact content; legacy artifacts without delivery metadata remain
|
|
65
|
+
readable.
|
|
66
|
+
|
|
67
|
+
`delivery_failed` means no successful transport response was confirmed,
|
|
68
|
+
including when notifier setup fails or the queue exhausts its bounded
|
|
69
|
+
Retry-After/backoff attempts. It does not prove that a
|
|
70
|
+
timed-out or response-lost Telegram request was not displayed. Recovery retries
|
|
71
|
+
must use the persisted provider result and must never rerun the provider;
|
|
72
|
+
Threadwire does not claim exactly-once Telegram send semantics.
|
|
73
|
+
|
|
42
74
|
Defaults are 1 MiB/event; 64 MiB and 100,000 events/artifact; 128 MiB and
|
|
43
75
|
200,000 events/run; 1 GiB, 1,000,000 events, and 10,000 artifacts/store;
|
|
44
76
|
seven-day retention; and a 64 KiB/1,000-line serialized-response ceiling.
|
package/package.json
CHANGED
|
@@ -52,6 +52,28 @@ if [[ "$threadwire_validation_status" -ne 0 ]]; then
|
|
|
52
52
|
exit "$threadwire_validation_status"
|
|
53
53
|
fi
|
|
54
54
|
|
|
55
|
+
threadwire_arguments=("$@")
|
|
56
|
+
threadwire_target=""
|
|
57
|
+
shift
|
|
58
|
+
while [[ "$#" -gt 0 ]]; do
|
|
59
|
+
case "$1" in
|
|
60
|
+
--) break ;;
|
|
61
|
+
--tool-messages|--relay-write) shift ;;
|
|
62
|
+
--target)
|
|
63
|
+
threadwire_target="$2"
|
|
64
|
+
shift 2
|
|
65
|
+
;;
|
|
66
|
+
*)
|
|
67
|
+
if [[ "$#" -lt 2 ]]; then break; fi
|
|
68
|
+
shift 2
|
|
69
|
+
;;
|
|
70
|
+
esac
|
|
71
|
+
done
|
|
72
|
+
|
|
73
|
+
if [[ "$threadwire_target" == file:* ]]; then
|
|
74
|
+
exec node "$THREADWIRE_ENTRYPOINT" "\${threadwire_arguments[@]}"
|
|
75
|
+
fi
|
|
76
|
+
|
|
55
77
|
if [[ ! -r "$THREADWIRE_DOTENV_PATH" ]]; then
|
|
56
78
|
printf 'threadwire: required dotenv is not readable: %s\\n' "$THREADWIRE_DOTENV_PATH" >&2
|
|
57
79
|
exit 2
|
|
@@ -77,7 +99,7 @@ if [[ -z "$telegram_bot_token" ]]; then
|
|
|
77
99
|
fi
|
|
78
100
|
|
|
79
101
|
export THREADWIRE_TELEGRAM_BOT_TOKEN="$telegram_bot_token"
|
|
80
|
-
exec node "$THREADWIRE_ENTRYPOINT" "
|
|
102
|
+
exec node "$THREADWIRE_ENTRYPOINT" "\${threadwire_arguments[@]}"
|
|
81
103
|
`
|
|
82
104
|
}
|
|
83
105
|
|
package/src/cli.js
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
// @ts-check
|
|
2
2
|
|
|
3
|
-
import {readFile} from "node:fs/promises"
|
|
3
|
+
import {lstat, readFile, readlink, realpath} from "node:fs/promises"
|
|
4
4
|
import {randomUUID} from "node:crypto"
|
|
5
|
-
import {isAbsolute, normalize, resolve} from "node:path"
|
|
5
|
+
import {basename, dirname, isAbsolute, join, normalize, resolve} from "node:path"
|
|
6
6
|
import {stdin, stderr, stdout} from "node:process"
|
|
7
7
|
import {createFetchTransport} from "./notifiers/fetch-transport.js"
|
|
8
8
|
import {createTelegramSender, parseTelegramTarget} from "./notifiers/telegram.js"
|
|
@@ -30,7 +30,7 @@ import {
|
|
|
30
30
|
import {probeCodexCapacity} from "./provider-capacity-codex.js"
|
|
31
31
|
import {probeKimiCapacity} from "./provider-capacity-kimi.js"
|
|
32
32
|
|
|
33
|
-
const HELP = `Usage: threadwire run --provider <codex|claude|kimi|opencode> --target telegram:<chat-id> | telegram:<chat-id>:<thread-id>
|
|
33
|
+
const HELP = `Usage: threadwire run --provider <codex|claude|kimi|opencode> --target telegram:<chat-id> | telegram:<chat-id>:<thread-id> | file:<absolute-path>
|
|
34
34
|
[--process-number <positive-integer>] [--cwd <directory>]
|
|
35
35
|
[--relay-write]
|
|
36
36
|
[--tool-messages] [--max-output-length <positive-integer>]
|
|
@@ -305,12 +305,20 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
305
305
|
await store.close()
|
|
306
306
|
}
|
|
307
307
|
}
|
|
308
|
-
|
|
309
|
-
|
|
308
|
+
const destination = parseDeliveryTarget(parsed.target)
|
|
309
|
+
if (destination.type === "file" && parsed.transcript !== undefined) {
|
|
310
|
+
throw new Error("--transcript cannot be combined with a file target")
|
|
310
311
|
}
|
|
312
|
+
const transcriptPath = destination.type === "file" ? destination.path : parsed.transcript
|
|
313
|
+
if (
|
|
314
|
+
transcriptPath !== undefined && parsed.activityLog !== undefined
|
|
315
|
+
&& await canonicalFilesystemPath(transcriptPath) === await canonicalFilesystemPath(parsed.activityLog)
|
|
316
|
+
) {
|
|
317
|
+
throw new Error("Result and activity log paths must resolve to different filesystem paths")
|
|
318
|
+
}
|
|
319
|
+
if (!validateOnly) normalizedOutput = new NormalizedOutput(output, transcriptPath)
|
|
311
320
|
const metrics = new ContextBudgetMetrics()
|
|
312
321
|
runAdmission = normalizedOutput === undefined ? undefined : new DelegatedResultAdmission({output: normalizedOutput, metrics})
|
|
313
|
-
const target = parseTelegramTarget(parsed.target)
|
|
314
322
|
if (parsed.relayWrite) validateRelayWriteProviderArguments(parsed.providerArguments)
|
|
315
323
|
if (validateOnly) return 0
|
|
316
324
|
const usesIsolatedRuntime = parsed.relayWrite
|
|
@@ -335,27 +343,63 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
335
343
|
let activity
|
|
336
344
|
/** @type {Awaited<ReturnType<EvidenceStore["createArtifact"]>> | undefined} */
|
|
337
345
|
let evidence
|
|
346
|
+
/** @type {Awaited<ReturnType<EvidenceStore["createArtifact"]>> | undefined} */
|
|
347
|
+
let resultEvidence
|
|
338
348
|
/** @type {EvidenceStore | undefined} */
|
|
339
349
|
let ownedEvidenceStore
|
|
350
|
+
/** @type {EvidenceStore | undefined} */
|
|
351
|
+
let evidenceStore
|
|
352
|
+
/** @type {{destinationId: string, runId: string} | undefined} */
|
|
353
|
+
let evidenceOwner
|
|
354
|
+
/** @type {WorkerControl | undefined} */
|
|
355
|
+
let control
|
|
356
|
+
/** @type {unknown} */
|
|
357
|
+
let deliveryError
|
|
358
|
+
let resultEvidenceFinalized = false
|
|
340
359
|
let evidencePayloadBytes = 0
|
|
360
|
+
let resultEvidenceBytes = 0
|
|
361
|
+
const deliveryIdentity = randomUUID()
|
|
341
362
|
const launchDeadline = isolatedPreflight?.deadline
|
|
342
363
|
try {
|
|
343
|
-
environment = await boundedLaunch(resolveFileBackedSettings(
|
|
364
|
+
environment = await boundedLaunch(resolveFileBackedSettings(
|
|
365
|
+
sourceEnvironment,
|
|
366
|
+
destination.type === "telegram" ? ["THREADWIRE_TELEGRAM_BOT_TOKEN"] : []
|
|
367
|
+
), launchDeadline)
|
|
368
|
+
if (destination.type === "telegram" && !environment.THREADWIRE_TELEGRAM_BOT_TOKEN) {
|
|
369
|
+
throw new Error("THREADWIRE_TELEGRAM_BOT_TOKEN is required")
|
|
370
|
+
}
|
|
344
371
|
const prompt = await readPrompt(parsed, dependencies.input ?? stdin, launchDeadline?.signal)
|
|
345
372
|
const configuredEvidenceRoot = evidenceRoot(sourceEnvironment.THREADWIRE_EVIDENCE_ROOT)
|
|
346
373
|
if (dependencies.evidenceStore === undefined && configuredEvidenceRoot !== undefined) {
|
|
347
374
|
ownedEvidenceStore = await boundedLaunch(EvidenceStore.open({root: configuredEvidenceRoot}), launchDeadline)
|
|
348
375
|
}
|
|
349
|
-
|
|
376
|
+
evidenceStore = dependencies.evidenceStore ?? ownedEvidenceStore
|
|
350
377
|
if (evidenceStore !== undefined) {
|
|
351
|
-
|
|
352
|
-
destinationId:
|
|
353
|
-
|
|
378
|
+
evidenceOwner = dependencies.evidenceOwnerScope ?? evidenceStore.createOwnerScope({
|
|
379
|
+
destinationId: destination.type === "telegram"
|
|
380
|
+
? `${destination.target.chatId}:${destination.target.threadId ?? "dm"}`
|
|
381
|
+
: "file",
|
|
382
|
+
runId: deliveryIdentity
|
|
354
383
|
})
|
|
384
|
+
const redactions = await boundedLaunch(collectEvidenceRedactions(environment), launchDeadline)
|
|
355
385
|
evidence = await evidenceStore.createArtifact(evidenceOwner, {
|
|
356
386
|
contentType: "text/plain; charset=utf-8",
|
|
357
|
-
redactions
|
|
387
|
+
redactions
|
|
358
388
|
})
|
|
389
|
+
try {
|
|
390
|
+
resultEvidence = await evidenceStore.createArtifact(evidenceOwner, {
|
|
391
|
+
contentType: "application/json",
|
|
392
|
+
redactions,
|
|
393
|
+
delivery: {
|
|
394
|
+
identity: deliveryIdentity,
|
|
395
|
+
state: destination.type === "telegram" ? "delivery_pending" : "delivery_not_requested"
|
|
396
|
+
}
|
|
397
|
+
})
|
|
398
|
+
} catch (error) {
|
|
399
|
+
await evidence.abort()
|
|
400
|
+
evidence = undefined
|
|
401
|
+
throw error
|
|
402
|
+
}
|
|
359
403
|
}
|
|
360
404
|
if (evidence !== undefined) {
|
|
361
405
|
const promptEvidence = `prompt\n${prompt}\nprovider-stream\n`
|
|
@@ -374,17 +418,21 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
374
418
|
providerEnvironment
|
|
375
419
|
)
|
|
376
420
|
if (parsed.resumeSession !== undefined) admission.setContinuationHandle(parsed.resumeSession)
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
421
|
+
if (destination.type === "telegram") {
|
|
422
|
+
const transport = (dependencies.transportFactory ?? createFetchTransport)(
|
|
423
|
+
/** @type {string} */ (environment.THREADWIRE_TELEGRAM_BOT_TOKEN),
|
|
424
|
+
undefined,
|
|
425
|
+
parseTelegramRequestTimeoutMs(environment)
|
|
426
|
+
)
|
|
427
|
+
control = new WorkerControl({
|
|
428
|
+
sender: createTelegramSender(destination.target, transport),
|
|
429
|
+
processNumber: parsed.processNumber ?? process.pid,
|
|
430
|
+
toolMessages: parsed.toolMessages,
|
|
431
|
+
...(dependencies.workerControlOptions ?? {}),
|
|
432
|
+
...(parsed.maxOutputLength === undefined ? {} : {maxOutputLength: parsed.maxOutputLength}),
|
|
433
|
+
metrics
|
|
434
|
+
})
|
|
435
|
+
}
|
|
388
436
|
activity = parsed.activityLog === undefined ? undefined : new ActivityLog(parsed.activityLog)
|
|
389
437
|
activity?.recordController(process.pid)
|
|
390
438
|
/** @type {import("./run-worker.js").RunWorkerOptions} */
|
|
@@ -397,7 +445,7 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
397
445
|
parse: provider.parse,
|
|
398
446
|
...(provider.completion === undefined ? {} : {completion: provider.completion}),
|
|
399
447
|
...(launchDeadline === undefined ? {} : {signal: launchDeadline.signal}),
|
|
400
|
-
onEvent: (event) => {
|
|
448
|
+
onEvent: async (event) => {
|
|
401
449
|
validateNormalizedWorkerEvent(event)
|
|
402
450
|
if (event.type !== "text-delta") {
|
|
403
451
|
metrics.recordRejected(`${event.type}_progress`, Buffer.byteLength(JSON.stringify(event), "utf8"))
|
|
@@ -406,7 +454,13 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
406
454
|
else if (event.type === "tool") activity?.recordActivity(provider.name, "tool")
|
|
407
455
|
else if (event.type === "lifecycle") activity?.recordActivity(provider.name, "lifecycle")
|
|
408
456
|
acceptAdmissionEvent(admission, event)
|
|
409
|
-
|
|
457
|
+
if (control !== undefined) {
|
|
458
|
+
try {
|
|
459
|
+
await control.accept(event)
|
|
460
|
+
} catch (error) {
|
|
461
|
+
deliveryError ??= error
|
|
462
|
+
}
|
|
463
|
+
}
|
|
410
464
|
},
|
|
411
465
|
onSpawn: (pid) => {
|
|
412
466
|
if (activity && pid !== undefined) activity.recordStarted(provider.name, pid)
|
|
@@ -456,27 +510,73 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
456
510
|
} else {
|
|
457
511
|
exitCode = await (dependencies.workerRunner ?? runWorker)(workerOptions)
|
|
458
512
|
}
|
|
459
|
-
await boundedLaunch(control.close(), launchDeadline)
|
|
460
513
|
terminalExitCode = exitCode
|
|
461
514
|
} finally {
|
|
462
|
-
|
|
515
|
+
const terminal = {state: /** @type {"completed" | "failed"} */ (terminalExitCode === 0 ? "completed" : "failed"), exitCode: terminalExitCode}
|
|
516
|
+
if (evidence !== undefined && resultEvidence !== undefined) {
|
|
517
|
+
try {
|
|
518
|
+
admission.addArtifactHandle(resultEvidence.handle)
|
|
519
|
+
const providerResult = admission.createProviderResult(terminal)
|
|
520
|
+
const serializedProviderResult = `${JSON.stringify(providerResult)}\n`
|
|
521
|
+
await resultEvidence.append("provider-result", serializedProviderResult)
|
|
522
|
+
const resultArtifact = await resultEvidence.finalize()
|
|
523
|
+
resultEvidenceBytes = resultArtifact.bytes
|
|
524
|
+
resultEvidenceFinalized = true
|
|
525
|
+
} catch (error) {
|
|
526
|
+
await evidence.abort()
|
|
527
|
+
await resultEvidence.abort()
|
|
528
|
+
admission.removeArtifactHandle(resultEvidence.handle)
|
|
529
|
+
metrics.clearArtifact()
|
|
530
|
+
terminalExitCode = 2
|
|
531
|
+
evidenceError = error
|
|
532
|
+
}
|
|
533
|
+
} else {
|
|
534
|
+
await evidence?.abort()
|
|
535
|
+
await resultEvidence?.abort()
|
|
536
|
+
}
|
|
537
|
+
if (control !== undefined) {
|
|
538
|
+
try {
|
|
539
|
+
await boundedLaunch(control.close(), launchDeadline)
|
|
540
|
+
} catch (error) {
|
|
541
|
+
deliveryError ??= error
|
|
542
|
+
}
|
|
543
|
+
}
|
|
544
|
+
if (evidence !== undefined && resultEvidenceFinalized) {
|
|
545
|
+
admission.addArtifactHandle(evidence.handle)
|
|
463
546
|
try {
|
|
464
|
-
admission.addArtifactHandle(evidence.handle)
|
|
465
|
-
const terminal = {state: /** @type {"completed" | "failed"} */ (terminalExitCode === 0 ? "completed" : "failed"), exitCode: terminalExitCode}
|
|
466
547
|
metrics.recordAdmission(admission.preview(terminal))
|
|
467
|
-
const projectedArtifactBytes = metrics.projectSelfInclusiveArtifact(
|
|
548
|
+
const projectedArtifactBytes = metrics.projectSelfInclusiveArtifact(
|
|
549
|
+
evidencePayloadBytes + resultEvidenceBytes,
|
|
550
|
+
2
|
|
551
|
+
)
|
|
468
552
|
await evidence.append("context-metrics", `${JSON.stringify(metrics.snapshot())}\n`)
|
|
469
553
|
const artifact = await evidence.finalize()
|
|
470
|
-
assertArtifactProjection(artifact.bytes, projectedArtifactBytes)
|
|
554
|
+
assertArtifactProjection(artifact.bytes + resultEvidenceBytes, projectedArtifactBytes)
|
|
471
555
|
} catch (error) {
|
|
472
|
-
await evidence.abort()
|
|
473
556
|
admission.removeArtifactHandle(evidence.handle)
|
|
557
|
+
await evidence.abort()
|
|
474
558
|
metrics.clearArtifact()
|
|
475
|
-
|
|
476
|
-
evidenceError
|
|
559
|
+
metrics.recordArtifact(resultEvidenceBytes, 1)
|
|
560
|
+
evidenceError ??= error
|
|
477
561
|
}
|
|
562
|
+
} else if (resultEvidenceFinalized) {
|
|
563
|
+
metrics.recordArtifact(resultEvidenceBytes, 1)
|
|
478
564
|
}
|
|
479
565
|
admission.complete({state: terminalExitCode === 0 ? "completed" : "failed", exitCode: terminalExitCode})
|
|
566
|
+
if (
|
|
567
|
+
destination.type === "telegram" && resultEvidenceFinalized
|
|
568
|
+
&& evidenceStore !== undefined && evidenceOwner !== undefined && resultEvidence !== undefined
|
|
569
|
+
) {
|
|
570
|
+
try {
|
|
571
|
+
await evidenceStore.recordDelivery(
|
|
572
|
+
evidenceOwner,
|
|
573
|
+
resultEvidence.handle,
|
|
574
|
+
control !== undefined && deliveryError === undefined ? "delivery_succeeded" : "delivery_failed"
|
|
575
|
+
)
|
|
576
|
+
} catch (error) {
|
|
577
|
+
evidenceError ??= error
|
|
578
|
+
}
|
|
579
|
+
}
|
|
480
580
|
errorOutput.write(`threadwire-context-metrics ${JSON.stringify(metrics.conciseDiagnostic())}\n`)
|
|
481
581
|
if (activity !== undefined) {
|
|
482
582
|
activity.recordTerminal(/** @type {"codex" | "claude" | "kimi" | "opencode"} */ (parsed.provider), terminalState(terminalExitCode), terminalExitCode)
|
|
@@ -486,6 +586,7 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
486
586
|
launchDeadline?.close()
|
|
487
587
|
}
|
|
488
588
|
if (evidenceError !== undefined) throw evidenceError
|
|
589
|
+
if (deliveryError !== undefined) throw deliveryError
|
|
489
590
|
return terminalExitCode
|
|
490
591
|
} catch (error) {
|
|
491
592
|
let reportedError = error
|
|
@@ -507,6 +608,44 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
507
608
|
}
|
|
508
609
|
}
|
|
509
610
|
|
|
611
|
+
/** @param {string} value @returns {{type: "telegram", target: import("./types.js").TelegramTarget} | {type: "file", path: string}} */
|
|
612
|
+
function parseDeliveryTarget(value) {
|
|
613
|
+
if (!value.startsWith("file:")) return {type: "telegram", target: parseTelegramTarget(value)}
|
|
614
|
+
const path = value.slice("file:".length)
|
|
615
|
+
if (!isAbsolute(path) || normalize(path) !== path || path.includes("\0") || path === "/") {
|
|
616
|
+
throw new Error("File target must contain a normalized absolute file path")
|
|
617
|
+
}
|
|
618
|
+
return {type: "file", path}
|
|
619
|
+
}
|
|
620
|
+
|
|
621
|
+
/** @param {string} path @param {Set<string>} [seen] @returns {Promise<string>} */
|
|
622
|
+
async function canonicalFilesystemPath(path, seen = new Set()) {
|
|
623
|
+
if (seen.has(path)) throw new Error("Result or activity log path contains a filesystem alias cycle")
|
|
624
|
+
seen.add(path)
|
|
625
|
+
try {
|
|
626
|
+
return await realpath(path)
|
|
627
|
+
} catch (error) {
|
|
628
|
+
if (!missingPath(error)) throw error
|
|
629
|
+
}
|
|
630
|
+
try {
|
|
631
|
+
const metadata = await lstat(path)
|
|
632
|
+
if (metadata.isSymbolicLink()) {
|
|
633
|
+
return canonicalFilesystemPath(resolve(dirname(path), await readlink(path)), seen)
|
|
634
|
+
}
|
|
635
|
+
} catch (error) {
|
|
636
|
+
if (!missingPath(error)) throw error
|
|
637
|
+
}
|
|
638
|
+
const parent = dirname(path)
|
|
639
|
+
if (parent === path) return path
|
|
640
|
+
return join(await canonicalFilesystemPath(parent, seen), basename(path))
|
|
641
|
+
}
|
|
642
|
+
|
|
643
|
+
/** @param {unknown} error */
|
|
644
|
+
function missingPath(error) {
|
|
645
|
+
return typeof error === "object" && error !== null
|
|
646
|
+
&& /** @type {{code?: unknown}} */ (error).code === "ENOENT"
|
|
647
|
+
}
|
|
648
|
+
|
|
510
649
|
/** @param {string | undefined} value */
|
|
511
650
|
function evidenceRoot(value) {
|
|
512
651
|
if (value === undefined) return undefined
|
|
@@ -105,6 +105,19 @@ export class DelegatedResultAdmission {
|
|
|
105
105
|
return this.createEnvelope(terminal)
|
|
106
106
|
}
|
|
107
107
|
|
|
108
|
+
/** @param {{state: TerminalState, exitCode: number}} terminal */
|
|
109
|
+
createProviderResult(terminal) {
|
|
110
|
+
const result = this.createEnvelope(terminal)
|
|
111
|
+
const assistantContent = truncateBoundedString(sanitizeOutput(this.conclusion), CONCLUSION_LIMIT)
|
|
112
|
+
return {
|
|
113
|
+
version: 1,
|
|
114
|
+
type: "provider_result",
|
|
115
|
+
providerState: terminal.state === "completed" ? "provider_completed" : "provider_failed",
|
|
116
|
+
...(assistantContent === undefined ? {} : {assistantContent}),
|
|
117
|
+
result
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
|
|
108
121
|
/** @param {AdmissionReference} reference */
|
|
109
122
|
addReference(reference) {
|
|
110
123
|
this.references = /** @type {AdmissionReference[]} */ (optionalReferences([...this.references, reference]) ?? [])
|
package/src/evidence-store.js
CHANGED
|
@@ -30,6 +30,9 @@ const DEFAULT_LIMITS = Object.freeze({
|
|
|
30
30
|
retentionMs: 7 * 24 * 60 * 60 * 1000
|
|
31
31
|
})
|
|
32
32
|
const TEXT_CONTENT_TYPES = new Set(["application/json", "application/x-ndjson"])
|
|
33
|
+
const DELIVERY_STATES = new Set([
|
|
34
|
+
"delivery_pending", "delivery_succeeded", "delivery_failed", "delivery_not_requested"
|
|
35
|
+
])
|
|
33
36
|
|
|
34
37
|
export class EvidenceAccessError extends Error {
|
|
35
38
|
constructor() {
|
|
@@ -80,8 +83,6 @@ export class EvidenceStore {
|
|
|
80
83
|
this.capabilities = new Map()
|
|
81
84
|
/** @type {WeakSet<object>} */
|
|
82
85
|
this.scopes = new WeakSet()
|
|
83
|
-
/** @type {Map<string, Set<string>>} */
|
|
84
|
-
this.redactions = new Map()
|
|
85
86
|
/** @type {Map<string, number>} */
|
|
86
87
|
this.activeReaders = new Map()
|
|
87
88
|
this.storeBytes = baseBytes
|
|
@@ -177,7 +178,7 @@ export class EvidenceStore {
|
|
|
177
178
|
|
|
178
179
|
/**
|
|
179
180
|
* @param {{destinationId: string, runId: string}} owner
|
|
180
|
-
* @param {{contentType: string, redactions?: string[]}} options
|
|
181
|
+
* @param {{contentType: string, redactions?: string[], delivery?: EvidenceDelivery}} options
|
|
181
182
|
*/
|
|
182
183
|
async createArtifact(owner, options) {
|
|
183
184
|
this.assertScope(owner)
|
|
@@ -186,13 +187,15 @@ export class EvidenceStore {
|
|
|
186
187
|
const id = randomBytes(32).toString("base64url")
|
|
187
188
|
const handle = `evidence_${randomBytes(32).toString("base64url")}`
|
|
188
189
|
const now = this.clock()
|
|
190
|
+
const delivery = options.delivery === undefined ? undefined : validateDelivery(options.delivery)
|
|
189
191
|
const metadata = /** @type {EvidenceMetadata} */ ({
|
|
190
192
|
version: 2, id, state: "pending", owner: {...owner}, contentType,
|
|
191
193
|
capabilityHash: capabilityHash(handle),
|
|
192
194
|
policyVersion: 1, framingVersion: 1,
|
|
193
195
|
redactionPolicy: {algorithm: "aes-256-gcm", nonce: "", ciphertext: "", tag: ""},
|
|
194
196
|
payloadCommitment: "A".repeat(43),
|
|
195
|
-
bytes: 0, storedBytes: 0, lines: 0, events: 0, createdAt: now, expiresAt: now + this.limits.retentionMs
|
|
197
|
+
bytes: 0, storedBytes: 0, lines: 0, events: 0, createdAt: now, expiresAt: now + this.limits.retentionMs,
|
|
198
|
+
...(delivery === undefined ? {} : {delivery})
|
|
196
199
|
})
|
|
197
200
|
metadata.redactionPolicy = encryptRedactions(handle, redactions, metadata)
|
|
198
201
|
const overheadBytes = durableArtifactReservation(metadata)
|
|
@@ -230,11 +233,10 @@ export class EvidenceStore {
|
|
|
230
233
|
if (!file) throw new Error("Evidence pending payload is unavailable")
|
|
231
234
|
this.artifacts.set(id, metadata)
|
|
232
235
|
this.capabilities.set(metadata.capabilityHash, id)
|
|
233
|
-
this.redactions.set(id, new Set(redactions))
|
|
234
236
|
return new EvidenceWriter(this, metadata, file, handle, finishOperation, redactions, owner)
|
|
235
237
|
}
|
|
236
238
|
|
|
237
|
-
/** @param {object} owner @param {unknown} request */
|
|
239
|
+
/** @param {object} owner @param {unknown} request @returns {Promise<EvidenceReadResult>} */
|
|
238
240
|
async read(owner, request) {
|
|
239
241
|
this.assertScope(owner)
|
|
240
242
|
const validated = validateEvidenceReadRequest(request, this.limits)
|
|
@@ -284,14 +286,16 @@ export class EvidenceStore {
|
|
|
284
286
|
data: binary.toString("base64"),
|
|
285
287
|
encoding: "base64",
|
|
286
288
|
redacted,
|
|
287
|
-
truncated: truncated || binary.length < selected.length
|
|
289
|
+
truncated: truncated || binary.length < selected.length,
|
|
290
|
+
...deliveryResult(metadata)
|
|
288
291
|
}, this.limits.maxReadBytes)
|
|
289
292
|
}
|
|
290
293
|
return fitSerializedResult({
|
|
291
294
|
contentType: metadata.contentType,
|
|
292
295
|
data: decodeUtf8(selected),
|
|
293
296
|
redacted,
|
|
294
|
-
truncated
|
|
297
|
+
truncated,
|
|
298
|
+
...deliveryResult(metadata)
|
|
295
299
|
}, this.limits.maxReadBytes)
|
|
296
300
|
} finally {
|
|
297
301
|
const readers = (this.activeReaders.get(id) ?? 1) - 1
|
|
@@ -301,7 +305,7 @@ export class EvidenceStore {
|
|
|
301
305
|
}
|
|
302
306
|
}
|
|
303
307
|
|
|
304
|
-
/** @param {unknown} request */
|
|
308
|
+
/** @param {unknown} request @returns {Promise<EvidenceReadResult>} */
|
|
305
309
|
async readBearer(request) {
|
|
306
310
|
const validated = validateEvidenceReadRequest(request, this.limits)
|
|
307
311
|
const id = this.lookupCapability(validated.handle)
|
|
@@ -317,7 +321,44 @@ export class EvidenceStore {
|
|
|
317
321
|
}
|
|
318
322
|
}
|
|
319
323
|
|
|
320
|
-
/** @param {
|
|
324
|
+
/** @param {object} owner @param {string} handle @param {"delivery_succeeded" | "delivery_failed"} state */
|
|
325
|
+
recordDelivery(owner, handle, state) {
|
|
326
|
+
const finish = this.beginOperation()
|
|
327
|
+
return this.enqueueMaintenance(() => this.recordDeliveryOnce(owner, handle, state)).finally(finish)
|
|
328
|
+
}
|
|
329
|
+
|
|
330
|
+
/** @param {object} owner @param {string} handle @param {"delivery_succeeded" | "delivery_failed"} state */
|
|
331
|
+
async recordDeliveryOnce(owner, handle, state) {
|
|
332
|
+
await this.assertRootIdentity()
|
|
333
|
+
this.assertScope(owner)
|
|
334
|
+
if (state !== "delivery_succeeded" && state !== "delivery_failed") throw new Error("Invalid terminal delivery state")
|
|
335
|
+
const id = this.lookupCapability(handle)
|
|
336
|
+
if (!id) throw new EvidenceAccessError()
|
|
337
|
+
const metadata = this.artifacts.get(id)
|
|
338
|
+
if (!metadata || metadata.state !== "ready" || !sameOwner(metadata.owner, owner) || metadata.delivery === undefined) {
|
|
339
|
+
throw new EvidenceAccessError()
|
|
340
|
+
}
|
|
341
|
+
if (metadata.delivery.state !== "delivery_pending") throw new Error("Evidence delivery state is already terminal")
|
|
342
|
+
const redactions = decryptRedactions(handle, metadata.redactionPolicy, metadata)
|
|
343
|
+
const updated = /** @type {EvidenceMetadata} */ ({
|
|
344
|
+
...metadata,
|
|
345
|
+
delivery: {identity: metadata.delivery.identity, state}
|
|
346
|
+
})
|
|
347
|
+
updated.reservedBytes = durableArtifactReservation(updated)
|
|
348
|
+
updated.redactionPolicy = encryptRedactions(handle, redactions, updated)
|
|
349
|
+
const reservationDelta = updated.reservedBytes - metadata.reservedBytes
|
|
350
|
+
if (reservationDelta > 0) this.reserve(metadata.owner.runId, reservationDelta, 0)
|
|
351
|
+
try {
|
|
352
|
+
await atomicJson(this.root, `${id}${META_SUFFIX}`, updated)
|
|
353
|
+
} catch (error) {
|
|
354
|
+
if (reservationDelta > 0) this.reserve(metadata.owner.runId, -reservationDelta, 0)
|
|
355
|
+
throw error
|
|
356
|
+
}
|
|
357
|
+
if (reservationDelta < 0) this.reserve(metadata.owner.runId, reservationDelta, 0)
|
|
358
|
+
this.artifacts.set(id, updated)
|
|
359
|
+
}
|
|
360
|
+
|
|
361
|
+
/** @param {{chatId: string, threadId: number | null, senderId: string}} requester @param {unknown} request @returns {Promise<EvidenceReadResult>} */
|
|
321
362
|
async readTelegram(requester, request) {
|
|
322
363
|
const validated = validateEvidenceReadRequest(request, this.limits)
|
|
323
364
|
const id = this.lookupCapability(validated.handle)
|
|
@@ -352,7 +393,6 @@ export class EvidenceStore {
|
|
|
352
393
|
this.capabilities.delete(metadata.capabilityHash)
|
|
353
394
|
this.release(metadata)
|
|
354
395
|
this.artifactSlots -= 1
|
|
355
|
-
this.redactions.delete(id)
|
|
356
396
|
}
|
|
357
397
|
}
|
|
358
398
|
|
|
@@ -379,7 +419,6 @@ export class EvidenceStore {
|
|
|
379
419
|
this.capabilities.delete(metadata.capabilityHash)
|
|
380
420
|
this.release(metadata)
|
|
381
421
|
this.artifactSlots -= 1
|
|
382
|
-
this.redactions.delete(id)
|
|
383
422
|
}
|
|
384
423
|
|
|
385
424
|
/** @template T @param {() => Promise<T>} operation @returns {Promise<T>} */
|
|
@@ -435,7 +474,6 @@ export class EvidenceStore {
|
|
|
435
474
|
this.artifactSlots += 1
|
|
436
475
|
this.artifacts.set(id, metadata)
|
|
437
476
|
this.capabilities.set(metadata.capabilityHash, id)
|
|
438
|
-
this.redactions.set(id, new Set(this.runtimeRedactions))
|
|
439
477
|
} catch {
|
|
440
478
|
await Promise.all([...files].map((suffix) => safeUnlink(this.path(id, suffix))))
|
|
441
479
|
}
|
|
@@ -601,7 +639,6 @@ class EvidenceWriter {
|
|
|
601
639
|
this.store.artifactSlots -= 1
|
|
602
640
|
this.store.artifacts.delete(this.metadata.id)
|
|
603
641
|
this.store.capabilities.delete(this.metadata.capabilityHash)
|
|
604
|
-
this.store.redactions.delete(this.metadata.id)
|
|
605
642
|
await safeUnlink(this.store.path(this.metadata.id, PENDING_SUFFIX))
|
|
606
643
|
await safeUnlink(this.store.path(this.metadata.id, PAYLOAD_SUFFIX))
|
|
607
644
|
await safeUnlink(this.store.path(this.metadata.id, META_SUFFIX))
|
|
@@ -632,7 +669,6 @@ class EvidenceWriter {
|
|
|
632
669
|
this.store.artifactSlots -= 1
|
|
633
670
|
this.store.artifacts.delete(this.metadata.id)
|
|
634
671
|
this.store.capabilities.delete(this.metadata.capabilityHash)
|
|
635
|
-
this.store.redactions.delete(this.metadata.id)
|
|
636
672
|
await safeUnlink(this.store.path(this.metadata.id, PENDING_SUFFIX))
|
|
637
673
|
await safeUnlink(this.store.path(this.metadata.id, META_SUFFIX))
|
|
638
674
|
await syncDirectory(this.store.root)
|
|
@@ -875,7 +911,7 @@ function utf8Prefix(buffer, limit) {
|
|
|
875
911
|
return buffer.subarray(0, limit)
|
|
876
912
|
}
|
|
877
913
|
|
|
878
|
-
/** @param {
|
|
914
|
+
/** @param {EvidenceReadResult} result @param {number} limit @returns {EvidenceReadResult} */
|
|
879
915
|
function fitSerializedResult(result, limit) {
|
|
880
916
|
if (Buffer.byteLength(`${JSON.stringify(result)}\n`) <= limit) return result
|
|
881
917
|
let bytes = Buffer.from(result.data, "utf8")
|
|
@@ -1313,14 +1349,16 @@ function decryptRedactions(handle, policy, metadata) {
|
|
|
1313
1349
|
|
|
1314
1350
|
/** @param {EvidenceMetadata} metadata */
|
|
1315
1351
|
function redactionAad(metadata) {
|
|
1316
|
-
|
|
1352
|
+
const values = [
|
|
1317
1353
|
"threadwire-evidence-policy", metadata.policyVersion, metadata.version, metadata.framingVersion, metadata.id,
|
|
1318
1354
|
metadata.state,
|
|
1319
1355
|
metadata.capabilityHash, metadata.owner.destinationId, metadata.owner.runId,
|
|
1320
1356
|
metadata.contentType, metadata.createdAt, metadata.expiresAt,
|
|
1321
1357
|
metadata.bytes, metadata.storedBytes, metadata.reservedBytes,
|
|
1322
1358
|
metadata.lines, metadata.events, metadata.payloadCommitment
|
|
1323
|
-
]
|
|
1359
|
+
]
|
|
1360
|
+
if (metadata.delivery !== undefined) values.push(metadata.delivery.identity, metadata.delivery.state)
|
|
1361
|
+
return Buffer.from(JSON.stringify(values), "utf8")
|
|
1324
1362
|
}
|
|
1325
1363
|
|
|
1326
1364
|
/** @param {EvidenceMetadata} metadata */
|
|
@@ -1374,6 +1412,22 @@ function validateRedactions(values) {
|
|
|
1374
1412
|
return values
|
|
1375
1413
|
}
|
|
1376
1414
|
|
|
1415
|
+
/** @param {unknown} value @returns {EvidenceDelivery} */
|
|
1416
|
+
function validateDelivery(value) {
|
|
1417
|
+
if (
|
|
1418
|
+
!plainObject(value) || !exactDataKeys(value, ["identity", "state"])
|
|
1419
|
+
|| typeof value.identity !== "string"
|
|
1420
|
+
|| !/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/u.test(value.identity)
|
|
1421
|
+
|| typeof value.state !== "string" || !DELIVERY_STATES.has(value.state)
|
|
1422
|
+
) throw new Error("Invalid evidence delivery metadata")
|
|
1423
|
+
return /** @type {EvidenceDelivery} */ ({identity: value.identity, state: value.state})
|
|
1424
|
+
}
|
|
1425
|
+
|
|
1426
|
+
/** @param {EvidenceMetadata} metadata */
|
|
1427
|
+
function deliveryResult(metadata) {
|
|
1428
|
+
return metadata.delivery === undefined ? {} : {delivery: {...metadata.delivery}}
|
|
1429
|
+
}
|
|
1430
|
+
|
|
1377
1431
|
/** @param {Record<string, unknown>} left @param {object} right */
|
|
1378
1432
|
function sameOwner(left, right) {
|
|
1379
1433
|
return left.destinationId === /** @type {{destinationId?: unknown}} */ (right).destinationId
|
|
@@ -1382,11 +1436,13 @@ function sameOwner(left, right) {
|
|
|
1382
1436
|
|
|
1383
1437
|
/** @param {unknown} value @param {string} id @param {EvidenceLimits} limits */
|
|
1384
1438
|
function validMetadata(value, id, limits) {
|
|
1439
|
+
const metadataKeys = [
|
|
1440
|
+
"version", "policyVersion", "framingVersion", "id", "state", "owner", "contentType", "capabilityHash",
|
|
1441
|
+
"redactionPolicy", "payloadCommitment", "bytes", "storedBytes", "reservedBytes", "lines", "events", "createdAt", "expiresAt"
|
|
1442
|
+
]
|
|
1443
|
+
if (plainObject(value) && value.delivery !== undefined) metadataKeys.push("delivery")
|
|
1385
1444
|
return plainObject(value) && value.version === 2 && value.id === id && value.state === "ready"
|
|
1386
|
-
&& exactKeys(value,
|
|
1387
|
-
"version", "policyVersion", "framingVersion", "id", "state", "owner", "contentType", "capabilityHash",
|
|
1388
|
-
"redactionPolicy", "payloadCommitment", "bytes", "storedBytes", "reservedBytes", "lines", "events", "createdAt", "expiresAt"
|
|
1389
|
-
])
|
|
1445
|
+
&& exactKeys(value, metadataKeys)
|
|
1390
1446
|
&& value.policyVersion === 1 && value.framingVersion === 1
|
|
1391
1447
|
&& plainObject(value.owner) && typeof value.owner.destinationId === "string" && validIdentity(value.owner.destinationId)
|
|
1392
1448
|
&& typeof value.owner.runId === "string" && validIdentity(value.owner.runId)
|
|
@@ -1404,6 +1460,17 @@ function validMetadata(value, id, limits) {
|
|
|
1404
1460
|
&& typeof value.events === "number" && nonnegative(value.events) && value.events <= limits.maxArtifactEvents
|
|
1405
1461
|
&& typeof value.createdAt === "number" && typeof value.expiresAt === "number"
|
|
1406
1462
|
&& nonnegative(value.createdAt) && positive(value.expiresAt) && value.createdAt < value.expiresAt
|
|
1463
|
+
&& (value.delivery === undefined || validDelivery(value.delivery))
|
|
1464
|
+
}
|
|
1465
|
+
|
|
1466
|
+
/** @param {unknown} value */
|
|
1467
|
+
function validDelivery(value) {
|
|
1468
|
+
try {
|
|
1469
|
+
validateDelivery(value)
|
|
1470
|
+
return true
|
|
1471
|
+
} catch {
|
|
1472
|
+
return false
|
|
1473
|
+
}
|
|
1407
1474
|
}
|
|
1408
1475
|
|
|
1409
1476
|
/** @param {unknown} value */
|
|
@@ -1455,8 +1522,11 @@ function validateLimits(limits) {
|
|
|
1455
1522
|
* owner: {destinationId: string, runId: string}, contentType: string,
|
|
1456
1523
|
* capabilityHash: string,
|
|
1457
1524
|
* redactionPolicy: {algorithm: "aes-256-gcm", nonce: string, ciphertext: string, tag: string}, payloadCommitment: string,
|
|
1458
|
-
* bytes: number, storedBytes: number, reservedBytes: number, lines: number, events: number, createdAt: number, expiresAt: number
|
|
1525
|
+
* bytes: number, storedBytes: number, reservedBytes: number, lines: number, events: number, createdAt: number, expiresAt: number,
|
|
1526
|
+
* delivery?: EvidenceDelivery
|
|
1459
1527
|
* }} EvidenceMetadata
|
|
1528
|
+
* @typedef {{identity: string, state: "delivery_pending" | "delivery_succeeded" | "delivery_failed" | "delivery_not_requested"}} EvidenceDelivery
|
|
1529
|
+
* @typedef {{contentType: string, data: string, encoding?: string, redacted: boolean, truncated: boolean, delivery?: EvidenceDelivery}} EvidenceReadResult
|
|
1460
1530
|
* @typedef {{
|
|
1461
1531
|
* handle: string,
|
|
1462
1532
|
* bytes?: {offset: number, limit: number},
|
|
@@ -16,6 +16,7 @@ const MAX_LINES = 1_024
|
|
|
16
16
|
* failures classify with fixed, credential-free messages.
|
|
17
17
|
* @param {{
|
|
18
18
|
* env?: NodeJS.ProcessEnv,
|
|
19
|
+
* executable?: string | undefined,
|
|
19
20
|
* spawnImplementation?: typeof spawn | undefined,
|
|
20
21
|
* timeoutMs?: number
|
|
21
22
|
* }} options
|
|
@@ -25,7 +26,7 @@ export async function probeCodexCapacity(options) {
|
|
|
25
26
|
const environment = options.env ?? process.env
|
|
26
27
|
const spawnImplementation = options.spawnImplementation ?? spawn
|
|
27
28
|
const timeoutMs = options.timeoutMs ?? DEFAULT_CAPACITY_TIMEOUT_MS
|
|
28
|
-
const executable = providerExecutable("THREADWIRE_CODEX_BIN", DEFAULT_EXECUTABLE, environment)
|
|
29
|
+
const executable = options.executable ?? providerExecutable("THREADWIRE_CODEX_BIN", DEFAULT_EXECUTABLE, environment)
|
|
29
30
|
const result = await appServerExchange(spawnImplementation, executable, environment, timeoutMs)
|
|
30
31
|
return normalizeCodexRateLimits(result)
|
|
31
32
|
}
|
|
@@ -1,10 +1,11 @@
|
|
|
1
1
|
// @ts-check
|
|
2
2
|
|
|
3
|
+
import {accessSync, constants, statSync} from "node:fs"
|
|
4
|
+
|
|
3
5
|
/**
|
|
4
6
|
* Resolve the terminal provider executable Threadwire spawns. The default is the
|
|
5
|
-
*
|
|
6
|
-
*
|
|
7
|
-
* cutover and rollback (for example, pointing back at the previous shim path).
|
|
7
|
+
* real provider CLI. A non-empty environment override supports staged cutover
|
|
8
|
+
* and rollback, but only when the override path exists and is executable.
|
|
8
9
|
* @param {string} overrideVariable
|
|
9
10
|
* @param {string} defaultPath
|
|
10
11
|
* @param {NodeJS.ProcessEnv} [environment]
|
|
@@ -12,5 +13,17 @@
|
|
|
12
13
|
*/
|
|
13
14
|
export function providerExecutable(overrideVariable, defaultPath, environment = process.env) {
|
|
14
15
|
const override = environment[overrideVariable]
|
|
15
|
-
|
|
16
|
+
if (override !== undefined && override.length > 0) {
|
|
17
|
+
try {
|
|
18
|
+
accessSync(override, constants.X_OK)
|
|
19
|
+
if (!statSync(override).isFile()) {
|
|
20
|
+
console.error(`threadwire: ${overrideVariable}=${override} is a directory, falling back to ${defaultPath}`)
|
|
21
|
+
return defaultPath
|
|
22
|
+
}
|
|
23
|
+
return override
|
|
24
|
+
} catch {
|
|
25
|
+
console.error(`threadwire: ${overrideVariable}=${override} is not executable, falling back to ${defaultPath}`)
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
return defaultPath
|
|
16
29
|
}
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
import {createHash, randomUUID, timingSafeEqual} from "node:crypto"
|
|
4
4
|
import {abortable} from "../absolute-deadline.js"
|
|
5
|
+
import {DelegatedResultAdmission} from "../delegated-result-admission.js"
|
|
5
6
|
import {evidenceTelegramDestination} from "../evidence-store.js"
|
|
6
7
|
import {NoticeQueue} from "../notice-queue.js"
|
|
7
8
|
import {createFetchTransport} from "../notifiers/fetch-transport.js"
|
|
@@ -136,6 +137,17 @@ export async function dispatchWorker(job, config, dependencies = {}) {
|
|
|
136
137
|
let control
|
|
137
138
|
/** @type {Awaited<ReturnType<import("../evidence-store.js").EvidenceStore["createArtifact"]>> | undefined} */
|
|
138
139
|
let evidence
|
|
140
|
+
/** @type {Awaited<ReturnType<import("../evidence-store.js").EvidenceStore["createArtifact"]>> | undefined} */
|
|
141
|
+
let resultEvidence
|
|
142
|
+
/** @type {{destinationId: string, runId: string} | undefined} */
|
|
143
|
+
let evidenceOwner
|
|
144
|
+
let resultEvidenceFinalized = false
|
|
145
|
+
/** @type {unknown} */
|
|
146
|
+
let evidenceError
|
|
147
|
+
/** @type {unknown} */
|
|
148
|
+
let deliveryError
|
|
149
|
+
const deliveryIdentity = randomUUID()
|
|
150
|
+
const admission = new DelegatedResultAdmission({output: {write: () => {}}})
|
|
139
151
|
try {
|
|
140
152
|
provider = createProviderImpl(job.provider, [], job.prompt, undefined, providerEnvironment)
|
|
141
153
|
const transport = createFetchTransportImpl(config.botToken, undefined, config.telegramRequestTimeoutMs)
|
|
@@ -147,18 +159,99 @@ export async function dispatchWorker(job, config, dependencies = {}) {
|
|
|
147
159
|
NoticeQueueClass: NoticeQueueImpl,
|
|
148
160
|
RelayClass: RelayImpl
|
|
149
161
|
})
|
|
150
|
-
|
|
162
|
+
evidenceOwner = dependencies.evidenceStore?.createOwnerScope({
|
|
151
163
|
destinationId: evidenceTelegramDestination(job.target, job.senderId),
|
|
152
|
-
runId:
|
|
153
|
-
})
|
|
154
|
-
evidence = evidenceOwner === undefined ? undefined : await dependencies.evidenceStore?.createArtifact(evidenceOwner, {
|
|
155
|
-
contentType: "text/plain; charset=utf-8",
|
|
156
|
-
redactions: await collectEvidenceRedactions(dependencies.providerEnvironment ?? {})
|
|
164
|
+
runId: deliveryIdentity
|
|
157
165
|
})
|
|
166
|
+
if (evidenceOwner !== undefined) {
|
|
167
|
+
const redactions = await collectEvidenceRedactions(dependencies.providerEnvironment ?? {})
|
|
168
|
+
evidence = await dependencies.evidenceStore?.createArtifact(evidenceOwner, {
|
|
169
|
+
contentType: "text/plain; charset=utf-8",
|
|
170
|
+
redactions
|
|
171
|
+
})
|
|
172
|
+
try {
|
|
173
|
+
resultEvidence = await dependencies.evidenceStore?.createArtifact(evidenceOwner, {
|
|
174
|
+
contentType: "application/json",
|
|
175
|
+
redactions,
|
|
176
|
+
delivery: {identity: deliveryIdentity, state: "delivery_pending"}
|
|
177
|
+
})
|
|
178
|
+
} catch (error) {
|
|
179
|
+
await evidence?.abort()
|
|
180
|
+
evidence = undefined
|
|
181
|
+
throw error
|
|
182
|
+
}
|
|
183
|
+
}
|
|
158
184
|
} catch (error) {
|
|
159
185
|
earlyKimiPreflight?.deadline?.close()
|
|
160
186
|
throw error
|
|
161
187
|
}
|
|
188
|
+
/** @param {import("../types.js").WorkerEvent} event */
|
|
189
|
+
const acceptEvent = async (event) => {
|
|
190
|
+
if (event.type === "text-delta") admission.acceptConclusionEvent(event)
|
|
191
|
+
try {
|
|
192
|
+
await control.accept(event)
|
|
193
|
+
} catch (error) {
|
|
194
|
+
deliveryError ??= error
|
|
195
|
+
}
|
|
196
|
+
}
|
|
197
|
+
/** @param {unknown} record */
|
|
198
|
+
const acceptRecord = (record) => {
|
|
199
|
+
const id = job.provider === "kimi" && isolatedRuntimeClient !== undefined
|
|
200
|
+
? kimiSessionEnvelopeId(record)
|
|
201
|
+
: provider.sessionId(record)
|
|
202
|
+
if (id !== undefined) {
|
|
203
|
+
admission.setContinuationHandle(id)
|
|
204
|
+
dependencies.activity?.recordSession(provider.name, id)
|
|
205
|
+
}
|
|
206
|
+
}
|
|
207
|
+
/** @param {number} exitCode */
|
|
208
|
+
const finalizeEvidence = async (exitCode) => {
|
|
209
|
+
if (resultEvidence === undefined || resultEvidenceFinalized) return
|
|
210
|
+
let rawEvidenceReady = false
|
|
211
|
+
if (evidence !== undefined) {
|
|
212
|
+
try {
|
|
213
|
+
await evidence.finalize()
|
|
214
|
+
rawEvidenceReady = true
|
|
215
|
+
} catch (error) {
|
|
216
|
+
await evidence.abort()
|
|
217
|
+
evidenceError ??= error
|
|
218
|
+
}
|
|
219
|
+
}
|
|
220
|
+
admission.addArtifactHandle(resultEvidence.handle)
|
|
221
|
+
if (rawEvidenceReady) admission.addArtifactHandle(/** @type {NonNullable<typeof evidence>} */ (evidence).handle)
|
|
222
|
+
const terminal = {
|
|
223
|
+
state: /** @type {"completed" | "failed"} */ (exitCode === 0 ? "completed" : "failed"),
|
|
224
|
+
exitCode
|
|
225
|
+
}
|
|
226
|
+
await resultEvidence.append("provider-result", `${JSON.stringify(admission.createProviderResult(terminal))}\n`)
|
|
227
|
+
await resultEvidence.finalize()
|
|
228
|
+
resultEvidenceFinalized = true
|
|
229
|
+
}
|
|
230
|
+
const finishDelivery = async () => {
|
|
231
|
+
if (resultEvidence !== undefined && deliveryError === undefined) {
|
|
232
|
+
try {
|
|
233
|
+
await control.acceptEvidenceHandle(resultEvidence.handle)
|
|
234
|
+
} catch (error) {
|
|
235
|
+
deliveryError ??= error
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
try {
|
|
239
|
+
await control.close()
|
|
240
|
+
} catch (error) {
|
|
241
|
+
deliveryError ??= error
|
|
242
|
+
}
|
|
243
|
+
if (
|
|
244
|
+
resultEvidenceFinalized && resultEvidence !== undefined && evidenceOwner !== undefined
|
|
245
|
+
&& dependencies.evidenceStore !== undefined
|
|
246
|
+
) {
|
|
247
|
+
await dependencies.evidenceStore.recordDelivery(
|
|
248
|
+
evidenceOwner,
|
|
249
|
+
resultEvidence.handle,
|
|
250
|
+
deliveryError === undefined ? "delivery_succeeded" : "delivery_failed"
|
|
251
|
+
)
|
|
252
|
+
}
|
|
253
|
+
if (deliveryError !== undefined) throw deliveryError
|
|
254
|
+
}
|
|
162
255
|
let evidenceTransferred = false
|
|
163
256
|
try {
|
|
164
257
|
await evidence?.append("prompt", `prompt\n${job.prompt}\nprovider-stream\n`)
|
|
@@ -181,33 +274,56 @@ export async function dispatchWorker(job, config, dependencies = {}) {
|
|
|
181
274
|
prompt: job.prompt,
|
|
182
275
|
providerArguments: [],
|
|
183
276
|
...(preflight.deadline === undefined ? {} : {deadline: preflight.deadline}),
|
|
184
|
-
onEvent:
|
|
185
|
-
onRecord: async (record) =>
|
|
186
|
-
const id = job.provider === "kimi" ? kimiSessionEnvelopeId(record) : provider.sessionId(record)
|
|
187
|
-
if (id !== undefined) dependencies.activity?.recordSession(provider.name, id)
|
|
188
|
-
},
|
|
277
|
+
onEvent: acceptEvent,
|
|
278
|
+
onRecord: async (record) => acceptRecord(record),
|
|
189
279
|
...(job.provider === "kimi" ? {} : {
|
|
190
280
|
onStdoutChunk: (chunk) => evidence?.append("provider-stdout", chunk),
|
|
191
281
|
onStderrChunk: (chunk) => evidence?.append("provider-stderr", chunk)
|
|
192
282
|
})
|
|
193
283
|
})
|
|
194
|
-
let exitCode
|
|
284
|
+
let exitCode = 2
|
|
285
|
+
/** @type {Error | undefined} */
|
|
286
|
+
let executionError
|
|
195
287
|
try {
|
|
196
288
|
exitCode = await abortable(isolatedRun.completion, preflight.deadline?.signal)
|
|
197
289
|
} catch (error) {
|
|
198
|
-
|
|
199
|
-
|
|
290
|
+
executionError = error instanceof Error ? error : new Error("Isolated runtime run failed")
|
|
291
|
+
try {
|
|
292
|
+
await isolatedRun.cancel(executionError)
|
|
293
|
+
} catch (cancelError) {
|
|
294
|
+
executionError = new AggregateError(
|
|
295
|
+
[executionError, cancelError],
|
|
296
|
+
"Isolated runtime failed and cleanup could not be confirmed"
|
|
297
|
+
)
|
|
298
|
+
}
|
|
200
299
|
}
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
await
|
|
205
|
-
|
|
300
|
+
await finalizeEvidence(exitCode)
|
|
301
|
+
let propagatedExecutionError = executionError
|
|
302
|
+
try {
|
|
303
|
+
await finishDelivery()
|
|
304
|
+
} catch (error) {
|
|
305
|
+
if (propagatedExecutionError === undefined) throw error
|
|
306
|
+
propagatedExecutionError = new AggregateError(
|
|
307
|
+
[propagatedExecutionError, error],
|
|
308
|
+
propagatedExecutionError.message
|
|
309
|
+
)
|
|
206
310
|
}
|
|
311
|
+
if (evidenceError !== undefined) {
|
|
312
|
+
if (propagatedExecutionError === undefined) throw evidenceError
|
|
313
|
+
propagatedExecutionError = new AggregateError(
|
|
314
|
+
[propagatedExecutionError, evidenceError],
|
|
315
|
+
propagatedExecutionError.message
|
|
316
|
+
)
|
|
317
|
+
}
|
|
318
|
+
if (propagatedExecutionError !== undefined) throw propagatedExecutionError
|
|
319
|
+
if (exitCode !== 0) throw new Error(`Isolated ${job.provider === "kimi" ? "Kimi" : "Codex"} worker exited with status ${exitCode}`)
|
|
207
320
|
return
|
|
208
321
|
} catch (error) {
|
|
209
322
|
await control.close().catch(() => {})
|
|
210
|
-
|
|
323
|
+
if (!resultEvidenceFinalized) {
|
|
324
|
+
await evidence?.abort()
|
|
325
|
+
await resultEvidence?.abort()
|
|
326
|
+
}
|
|
211
327
|
throw error
|
|
212
328
|
} finally {
|
|
213
329
|
isolatedDeadline?.close()
|
|
@@ -260,15 +376,12 @@ export async function dispatchWorker(job, config, dependencies = {}) {
|
|
|
260
376
|
provider: provider.name,
|
|
261
377
|
parse: provider.parse,
|
|
262
378
|
...(provider.completion === undefined ? {} : {completion: provider.completion}),
|
|
263
|
-
onEvent:
|
|
379
|
+
onEvent: acceptEvent,
|
|
264
380
|
onSpawn: (pid) => {
|
|
265
381
|
if (pid !== undefined) dependencies.activity?.recordStarted(provider.name, pid)
|
|
266
382
|
spawnGate.resolve()
|
|
267
383
|
},
|
|
268
|
-
onRecord: async (record) =>
|
|
269
|
-
const id = provider.sessionId(record)
|
|
270
|
-
if (id !== undefined) dependencies.activity?.recordSession(provider.name, id)
|
|
271
|
-
},
|
|
384
|
+
onRecord: async (record) => acceptRecord(record),
|
|
272
385
|
onStdoutChunk: (chunk) => evidence?.append("provider-stdout", chunk),
|
|
273
386
|
onStderrChunk: (chunk) => evidence?.append("provider-stderr", chunk)
|
|
274
387
|
})
|
|
@@ -278,49 +391,61 @@ export async function dispatchWorker(job, config, dependencies = {}) {
|
|
|
278
391
|
// reported. Always notify settlement so concurrency slots are released.
|
|
279
392
|
evidenceTransferred = true
|
|
280
393
|
void (async () => {
|
|
394
|
+
let exitCode = 2
|
|
281
395
|
/** @type {Error | undefined} */
|
|
282
|
-
let
|
|
396
|
+
let failure
|
|
397
|
+
let failureReported = false
|
|
283
398
|
try {
|
|
284
399
|
try {
|
|
285
|
-
|
|
400
|
+
exitCode = await running
|
|
286
401
|
if (job.provider === "kimi" && spawnGate.settled) {
|
|
287
|
-
|
|
288
|
-
await control.accept({
|
|
402
|
+
await acceptEvent({
|
|
289
403
|
type: "lifecycle",
|
|
290
404
|
phase: exitCode === 0 ? "completed" : "failed",
|
|
291
405
|
summary: exitCode === 0 ? "Kimi worker completed" : "Kimi worker failed"
|
|
292
406
|
})
|
|
293
|
-
if (
|
|
407
|
+
if (exitCode !== 0) failure = new Error(`Kimi worker exited with status ${exitCode}`)
|
|
294
408
|
}
|
|
295
409
|
} catch (error) {
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
}
|
|
302
|
-
try {
|
|
303
|
-
await control.close()
|
|
304
|
-
} catch {
|
|
305
|
-
/* Best-effort close after worker failure. */
|
|
306
|
-
}
|
|
307
|
-
return
|
|
410
|
+
failure = error instanceof Error ? error : new Error(safeFailureMessage(error))
|
|
411
|
+
}
|
|
412
|
+
if (!spawnGate.settled && failure === undefined) {
|
|
413
|
+
exitCode = 2
|
|
414
|
+
failure = new Error("Worker exited before spawn was observed")
|
|
308
415
|
}
|
|
309
416
|
|
|
310
417
|
try {
|
|
311
|
-
await
|
|
418
|
+
await finalizeEvidence(exitCode)
|
|
312
419
|
} catch (error) {
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
420
|
+
await evidence?.abort()
|
|
421
|
+
await resultEvidence?.abort()
|
|
422
|
+
failure ??= error instanceof Error ? error : new Error(safeFailureMessage(error))
|
|
423
|
+
}
|
|
424
|
+
if (evidenceError !== undefined) {
|
|
425
|
+
failure ??= evidenceError instanceof Error ? evidenceError : new Error(safeFailureMessage(evidenceError))
|
|
319
426
|
}
|
|
320
427
|
|
|
321
|
-
if (
|
|
322
|
-
|
|
428
|
+
if (spawnGate.settled && failure !== undefined) {
|
|
429
|
+
reportBackgroundFailure(failure)
|
|
430
|
+
failureReported = true
|
|
323
431
|
}
|
|
432
|
+
|
|
433
|
+
if (evidence === undefined || resultEvidenceFinalized) {
|
|
434
|
+
try {
|
|
435
|
+
await finishDelivery()
|
|
436
|
+
} catch (error) {
|
|
437
|
+
failure ??= error instanceof Error ? error : new Error(safeFailureMessage(error))
|
|
438
|
+
}
|
|
439
|
+
} else {
|
|
440
|
+
try {
|
|
441
|
+
await control.close()
|
|
442
|
+
} catch (error) {
|
|
443
|
+
failure ??= error instanceof Error ? error : new Error(safeFailureMessage(error))
|
|
444
|
+
}
|
|
445
|
+
}
|
|
446
|
+
|
|
447
|
+
if (!spawnGate.settled) spawnGate.reject(/** @type {Error} */ (failure))
|
|
448
|
+
else if (failure !== undefined && !failureReported) reportBackgroundFailure(failure)
|
|
324
449
|
} catch (error) {
|
|
325
450
|
if (spawnGate.settled) {
|
|
326
451
|
reportBackgroundFailure(error)
|
|
@@ -328,15 +453,6 @@ export async function dispatchWorker(job, config, dependencies = {}) {
|
|
|
328
453
|
spawnGate.reject(error instanceof Error ? error : new Error(safeFailureMessage(error)))
|
|
329
454
|
}
|
|
330
455
|
} finally {
|
|
331
|
-
if (evidence !== undefined) {
|
|
332
|
-
try {
|
|
333
|
-
await evidence.finalize()
|
|
334
|
-
await sender.send(`Evidence: ${evidence.handle}`)
|
|
335
|
-
} catch (error) {
|
|
336
|
-
await evidence.abort()
|
|
337
|
-
reportBackgroundFailure(error)
|
|
338
|
-
}
|
|
339
|
-
}
|
|
340
456
|
notifySettled()
|
|
341
457
|
}
|
|
342
458
|
})()
|
|
@@ -344,7 +460,10 @@ export async function dispatchWorker(job, config, dependencies = {}) {
|
|
|
344
460
|
await spawned
|
|
345
461
|
} finally {
|
|
346
462
|
try {
|
|
347
|
-
if (!evidenceTransferred)
|
|
463
|
+
if (!evidenceTransferred) {
|
|
464
|
+
await evidence?.abort()
|
|
465
|
+
await resultEvidence?.abort()
|
|
466
|
+
}
|
|
348
467
|
} finally {
|
|
349
468
|
earlyKimiPreflight?.deadline?.close()
|
|
350
469
|
}
|
package/src/worker-control.js
CHANGED
|
@@ -98,6 +98,17 @@ export class WorkerControl {
|
|
|
98
98
|
})
|
|
99
99
|
}
|
|
100
100
|
|
|
101
|
+
/** @param {string} handle */
|
|
102
|
+
acceptEvidenceHandle(handle) {
|
|
103
|
+
if (!/^evidence_[A-Za-z0-9_-]{43}$/u.test(handle)) throw new Error("Invalid evidence handle")
|
|
104
|
+
return this.enqueueOperation(async () => {
|
|
105
|
+
this.ensureHealthy()
|
|
106
|
+
this.renderedNotices.push({text: `Evidence: ${handle}`})
|
|
107
|
+
await this.flushLoop(false)
|
|
108
|
+
this.ensureHealthy()
|
|
109
|
+
})
|
|
110
|
+
}
|
|
111
|
+
|
|
101
112
|
close() {
|
|
102
113
|
if (!this.closePromise) {
|
|
103
114
|
this.closePromise = this.enqueueOperation(async () => {
|