threadwire 0.1.14 → 0.1.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +3 -0
- package/README.md +3 -1
- package/package.json +1 -1
- package/scripts/verify-package.js +1 -0
- package/src/activity-log-status.js +291 -0
- package/src/activity-log.js +91 -3
- package/src/cli.js +25 -3
- package/src/providers/index.js +109 -5
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.1.15 - 2026-08-03
|
|
6
|
+
|
|
7
|
+
- Add machine-readable run health supervision: the existing append-only `--activity-log` now records safe timestamped controller/provider/session/activity/health/terminal facts with an explicit `controller-started` fact (the run controller's own PID, not an injected status-process PID). The `threadwire status --activity-log <absolute-path>` read-only command emits exactly one closed versioned JSON document (`version`, `state`, `provider`, `controllerPid`, `providerPid`, `continuationHandle`, `startedAt`, `lastActivityAt`, `quietForMs`, `health`, `terminal`). PIDs are recorded non-authoritative facts; status never inspects or signals processes. Health events carry a bounded disposition (`retrying`/`blocked`) plus safe category (`authentication`/`permission`/`rate-limit`/`quota`/`billing`/`model`/`network`/`protocol`/`unknown`) and optional `retryAfterMs` in milliseconds, extracted only from an explicit small list of safe provider record containers (top-level record and known nested `error`/`part`/`part.error`/`.data` shapes) — never recursive traversal and never message/text regex. Generic HTTP 429 → retrying/rate-limit; structured `exceeded_current_quota_error`/`insufficient_quota`/`insufficient_balance` → blocked quota/billing; ambiguous short fields such as `retry_after` (seconds) are never interpreted as milliseconds. If a provider protocol suppresses upstream error details (e.g., real Moonshot OpenCode sessions), no health fact is recorded and status stays running/unknown; Hermes must perform an independent bounded provider probe. Activity records are throttled (5 s) with guaranteed first-activity recording. Terminal facts always override earlier retrying/blocked health state and are written exactly once for local and isolated-runtime run paths; signal-derived exit codes 130/143 record as `cancelled`, all others as `completed` (0) or `failed` (nonzero). A later normal activity record clears a prior `retrying` health back to running; `blocked` stays sticky until terminal overrides it. The status command reads the log within a documented 1 MiB byte cap, never opens the file writable, skips at most one incomplete trailing JSONL record, rejects malformed/schema-unsafe records and overlong/malformed continuation handles, validates timestamps as safe non-negative Date-range integers and the injected clock similarly, and never claims stalled/dead from silence. Existing backward-compatible records without timestamps remain readable.
|
|
5
8
|
- Incrementally spool oversized provider JSONL records to private per-record temporary files after the bounded in-memory threshold, preserving byte order, UTF-8 chunk boundaries, malformed/incomplete-record diagnostics, and the exact absolute record cap while cleaning every per-record spill directory on success, failure, and cancellation. Protocol completion stops same-chunk scanning before any trailing record can exceed the cap or emit output. Ordinary valid provider records are no longer constrained to Telegram display sizes. Container-backed cancellation and failure teardown now retain exact Docker ownership through inspect, bounded TERM stop, terminal-state confirmation, and exact-container KILL escalation; CLI and Telegram deadline cancellation close the request side, await bounded terminal cleanup confirmation, and never treat an HTTP abort as cleanup success. Unconfirmed cleanup remains sealed for reconciliation and can never report success. Direct local-process behavior is unchanged.
|
|
6
9
|
- Extend the checked-in development image with build-time `oc-codex-multi-auth`, DeepSeek OpenAI-compatible, and xAI provider runtimes plus a root-owned secret-free OpenCode catalog for GPT-5.6 Sol/Terra/Luna, DeepSeek V4 Pro/Flash, Grok 4.5/Build/Code Fast, and Kimi K3 through OpenCode's built-in `moonshotai` provider. Kimi K3 keeps its 1,048,576-token context, 131,072-token output limit, default `max` reasoning effort, and `low`/`high`/`max` variants. Compose now requires portable source paths for native Codex auth, OpenCode auth, plugin OAuth, and the plugin's global account pool; refreshable files mount narrowly read-write, GitHub/context mounts stay read-only, per-project Codex account copies are disabled, and OpenCode database/session/cache state remains in the dedicated development home. The UID/GID 1000 initializer creates every nested mount target without receiving credentials. Threadwire remains the worker boundary; direct OpenCode is for authentication, discovery, and bounded smoke checks only.
|
|
7
10
|
|
package/README.md
CHANGED
|
@@ -86,7 +86,9 @@ Pass `--resume-session <exact-provider-session-id>` to continue a stored Codex,
|
|
|
86
86
|
|
|
87
87
|
Pass `--transcript <path>` to capture the exact normalized, caller-facing JSONL protocol written to run stdout. The file is created privately (mode `0600`), must not already exist, and contains the same records in the same byte order as stdout. It never contains provider JSONL, reasoning, tool payloads, prompts, stderr, or credentials.
|
|
88
88
|
|
|
89
|
-
`--activity-log <path>` is different: it retains its append contract and writes
|
|
89
|
+
`--activity-log <path>` is different: it retains its append contract and writes safe timestamped lifecycle facts—provider start (`provider`, child `pid`), available native session ID, throttled activity markers, structured health events (bounded `retrying`/`blocked` disposition with safe category), and exactly one terminal fact. It is local-only and never enters Telegram or normalized stdout.
|
|
90
|
+
|
|
91
|
+
Inspect a live or completed activity log with the read-only `threadwire status --activity-log <absolute-path>` command. It emits exactly one closed versioned JSON document with `version`, `state` (`running`/`retrying`/`blocked`/`completed`/`failed`/`cancelled`/`unknown`), `provider`, `controllerPid`, `providerPid`, `continuationHandle`, `startedAt`, `lastActivityAt`, `quietForMs`, `health` (or `null`), and `terminal` (or `null`). The command derives state from the file only, never opens it writable, never inspects or signals processes, and never claims stalled/dead from silence. PIDs are recorded facts with explicit non-authoritative semantics.
|
|
90
92
|
|
|
91
93
|
Provider stdout is decoded as JSONL with a finite 1 MiB UTF-8 byte limit for each pending unterminated record. Exceeding it rejects the run, removes listeners and signal handlers, and terminates the provider without parsing or rendering the oversized content.
|
|
92
94
|
|
package/package.json
CHANGED
|
@@ -0,0 +1,291 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
|
|
3
|
+
import {isAbsolute, resolve} from "node:path"
|
|
4
|
+
import {open} from "node:fs/promises"
|
|
5
|
+
|
|
6
|
+
export const ACTIVITY_LOG_READ_CAPACITY = 1_048_576
|
|
7
|
+
|
|
8
|
+
const KNOWN_TYPES = new Set([
|
|
9
|
+
"provider-started", "controller-started", "session-available", "activity", "health", "terminal"
|
|
10
|
+
])
|
|
11
|
+
|
|
12
|
+
const SESSION_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,511}$/u
|
|
13
|
+
|
|
14
|
+
const VALIDATED_FACT_FIELDS = {
|
|
15
|
+
"provider-started": ["type", "provider", "pid"],
|
|
16
|
+
"controller-started": ["type", "pid"],
|
|
17
|
+
"session-available": ["type", "provider", "sessionId"],
|
|
18
|
+
"activity": ["type", "provider", "kind"],
|
|
19
|
+
"health": ["type", "provider", "disposition", "category"],
|
|
20
|
+
"terminal": ["type", "provider", "state", "exitCode"]
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
const OPTIONAL_FACT_FIELDS = {
|
|
24
|
+
"provider-started": ["at"],
|
|
25
|
+
"controller-started": ["at"],
|
|
26
|
+
"session-available": ["at"],
|
|
27
|
+
"activity": ["at"],
|
|
28
|
+
"health": ["retryAfterMs", "at"],
|
|
29
|
+
"terminal": ["at"]
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
const PROVIDERS = new Set(["codex", "claude", "kimi", "opencode"])
|
|
33
|
+
const ACTIVITY_KINDS = new Set(["delta", "tool", "lifecycle"])
|
|
34
|
+
const HEALTH_DISPOSITIONS = new Set(["retrying", "blocked"])
|
|
35
|
+
const HEALTH_CATEGORIES = new Set([
|
|
36
|
+
"authentication", "permission", "rate-limit", "quota", "billing",
|
|
37
|
+
"model", "network", "protocol", "unknown"
|
|
38
|
+
])
|
|
39
|
+
const TERMINAL_STATES = new Set(["completed", "failed", "cancelled"])
|
|
40
|
+
|
|
41
|
+
const MAX_SAFE_DATE_MS = 8640000000000000 // Date maximum
|
|
42
|
+
|
|
43
|
+
/**
|
|
44
|
+
* @typedef {{version: number, state: "running" | "retrying" | "blocked" | "completed" | "failed" | "cancelled" | "unknown", provider: string | null, controllerPid: number | null, providerPid: number | null, continuationHandle: string | null, startedAt: string | null, lastActivityAt: string | null, quietForMs: number, health: {disposition: "retrying" | "blocked", category: string, retryAfterMs?: number} | null, terminal: {state: "completed" | "failed" | "cancelled", exitCode: number} | null}} StatusDocument
|
|
45
|
+
*/
|
|
46
|
+
|
|
47
|
+
/** @param {string[]} arguments_ */
|
|
48
|
+
export function parseStatusArguments(arguments_) {
|
|
49
|
+
if (arguments_[0] !== "status") throw new Error("Usage: threadwire status --activity-log <absolute-path>")
|
|
50
|
+
const own = arguments_.slice(1)
|
|
51
|
+
if (own.length !== 2 || own[0] !== "--activity-log") throw new Error("Usage: threadwire status --activity-log <absolute-path>")
|
|
52
|
+
const path = own[1]
|
|
53
|
+
if (!path) throw new Error("--activity-log must be an absolute path")
|
|
54
|
+
const resolved = resolve(path)
|
|
55
|
+
if (!isAbsolute(path)) throw new Error("--activity-log must be an absolute path")
|
|
56
|
+
return {activityLog: resolved}
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
/**
|
|
60
|
+
* Read and validate an activity log file, with a bounded read capacity to
|
|
61
|
+
* prevent uncontrolled allocation. Skips at most one incomplete trailing line.
|
|
62
|
+
* Rejects malformed completed records and records with unsafe field names.
|
|
63
|
+
* @param {string} path
|
|
64
|
+
* @returns {Promise<Record<string, unknown>[]>}
|
|
65
|
+
*/
|
|
66
|
+
export async function readActivityLog(path) {
|
|
67
|
+
const fd = await open(path, "r")
|
|
68
|
+
try {
|
|
69
|
+
const stat = await fd.stat()
|
|
70
|
+
if (stat.size > ACTIVITY_LOG_READ_CAPACITY) throw new Error("Activity log exceeds read capacity")
|
|
71
|
+
const buffer = Buffer.alloc(stat.size > 0 ? stat.size : 0)
|
|
72
|
+
let bytesRead = 0
|
|
73
|
+
while (bytesRead < stat.size) {
|
|
74
|
+
const result = await fd.read(buffer, bytesRead, stat.size - bytesRead, bytesRead)
|
|
75
|
+
if (result.bytesRead === 0) break
|
|
76
|
+
bytesRead += result.bytesRead
|
|
77
|
+
}
|
|
78
|
+
return parseActivityLogContent(buffer.toString("utf8", 0, bytesRead))
|
|
79
|
+
} finally {
|
|
80
|
+
await fd.close()
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
/**
|
|
85
|
+
* @param {string} content
|
|
86
|
+
* @returns {Record<string, unknown>[]}
|
|
87
|
+
*/
|
|
88
|
+
function parseActivityLogContent(content) {
|
|
89
|
+
const lines = content.split("\n")
|
|
90
|
+
/** @type {Record<string, unknown>[]} */
|
|
91
|
+
const records = []
|
|
92
|
+
for (let index = 0; index < lines.length; index += 1) {
|
|
93
|
+
const line = /** @type {string} */ (lines[index])
|
|
94
|
+
if (line.trim().length === 0) continue
|
|
95
|
+
let record
|
|
96
|
+
try {
|
|
97
|
+
record = JSON.parse(line)
|
|
98
|
+
} catch {
|
|
99
|
+
if (index === lines.length - 1) break
|
|
100
|
+
throw new Error("Activity log contains an unreadable completed record")
|
|
101
|
+
}
|
|
102
|
+
if (!isRecord(record)) throw new Error("Activity log record is not an object")
|
|
103
|
+
validateRecordSchema(record)
|
|
104
|
+
records.push(record)
|
|
105
|
+
}
|
|
106
|
+
return records
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
/**
|
|
110
|
+
* Pure computation: derives a closed status document from an array of
|
|
111
|
+
* validated activity-log facts.
|
|
112
|
+
* @param {Record<string, unknown>[]} records
|
|
113
|
+
* @param {{now?: () => number}} [options]
|
|
114
|
+
* @returns {StatusDocument}
|
|
115
|
+
*/
|
|
116
|
+
export function computeStatus(records, options = {}) {
|
|
117
|
+
const now = options.now ?? (() => Date.now())
|
|
118
|
+
const clock = now()
|
|
119
|
+
if (!Number.isSafeInteger(clock) || clock < 0 || clock > MAX_SAFE_DATE_MS) throw new Error("Clock must be a safe non-negative Date-range integer")
|
|
120
|
+
|
|
121
|
+
let providerStarted = null
|
|
122
|
+
/** @type {number | null} */
|
|
123
|
+
let controllerPid = null
|
|
124
|
+
/** @type {string | null} */
|
|
125
|
+
let continuationHandle = null
|
|
126
|
+
/** @type {{disposition: "retrying" | "blocked", category: string, retryAfterMs?: number} | null} */
|
|
127
|
+
let health = null
|
|
128
|
+
/** @type {{state: "completed" | "failed" | "cancelled", exitCode: number} | null} */
|
|
129
|
+
let terminal = null
|
|
130
|
+
let sawTerminal = false
|
|
131
|
+
/** @type {number | null} */
|
|
132
|
+
let startedAt = null
|
|
133
|
+
/** @type {number | null} */
|
|
134
|
+
let lastActivityAt = null
|
|
135
|
+
let lastHealthIndex = -1
|
|
136
|
+
let lastActivityIndex = -1
|
|
137
|
+
|
|
138
|
+
for (let index = 0; index < records.length; index += 1) {
|
|
139
|
+
const record = /** @type {Record<string, unknown>} */ (records[index])
|
|
140
|
+
// Validate timestamps on every record, since computeStatus may be
|
|
141
|
+
// called directly (not through readActivityLog).
|
|
142
|
+
const atValue = record.at
|
|
143
|
+
if (atValue !== undefined && (typeof atValue !== "number" || !Number.isSafeInteger(atValue) || atValue < 0 || atValue > MAX_SAFE_DATE_MS)) {
|
|
144
|
+
throw new Error("Invalid timestamp")
|
|
145
|
+
}
|
|
146
|
+
const type = record.type
|
|
147
|
+
if (typeof type !== "string" || !KNOWN_TYPES.has(type)) {
|
|
148
|
+
throw new Error(`Unknown activity log record type: ${String(type)}`)
|
|
149
|
+
}
|
|
150
|
+
if (type === "provider-started") {
|
|
151
|
+
if (providerStarted !== null) throw new Error("Duplicate provider-started record")
|
|
152
|
+
providerStarted = record
|
|
153
|
+
if (typeof atValue === "number" && startedAt === null) startedAt = atValue
|
|
154
|
+
else lastActivityAt = lastActivityAt ?? null
|
|
155
|
+
} else if (type === "controller-started") {
|
|
156
|
+
const pid = record.pid
|
|
157
|
+
if (typeof pid !== "number" || !Number.isSafeInteger(pid) || pid <= 0) {
|
|
158
|
+
throw new Error("Invalid controller PID in activity log")
|
|
159
|
+
}
|
|
160
|
+
controllerPid = pid
|
|
161
|
+
if (typeof atValue === "number" && startedAt === null) startedAt = atValue
|
|
162
|
+
if (typeof atValue === "number") lastActivityAt = atValue
|
|
163
|
+
} else if (type === "session-available") {
|
|
164
|
+
const id = typeof record.sessionId === "string" ? record.sessionId : null
|
|
165
|
+
if (id !== null && !SESSION_ID_PATTERN.test(id)) throw new Error("Invalid continuation handle in activity log")
|
|
166
|
+
continuationHandle = id
|
|
167
|
+
if (typeof atValue === "number") lastActivityAt = atValue
|
|
168
|
+
} else if (type === "activity") {
|
|
169
|
+
lastActivityIndex = index
|
|
170
|
+
if (typeof atValue === "number") lastActivityAt = atValue
|
|
171
|
+
} else if (type === "health") {
|
|
172
|
+
if (sawTerminal) throw new Error("Health record appears after terminal")
|
|
173
|
+
lastHealthIndex = index
|
|
174
|
+
health = {
|
|
175
|
+
disposition: /** @type {"retrying" | "blocked"} */ (record.disposition),
|
|
176
|
+
category: /** @type {string} */ (record.category)
|
|
177
|
+
}
|
|
178
|
+
if (typeof record.retryAfterMs === "number") health.retryAfterMs = record.retryAfterMs
|
|
179
|
+
if (typeof atValue === "number") lastActivityAt = atValue
|
|
180
|
+
// Blocked stays sticky regardless; terminal always overrides later.
|
|
181
|
+
} else if (type === "terminal") {
|
|
182
|
+
if (sawTerminal) throw new Error("Duplicate terminal record")
|
|
183
|
+
sawTerminal = true
|
|
184
|
+
terminal = {
|
|
185
|
+
state: /** @type {"completed" | "failed" | "cancelled"} */ (record.state),
|
|
186
|
+
exitCode: /** @type {number} */ (record.exitCode)
|
|
187
|
+
}
|
|
188
|
+
if (typeof atValue === "number") lastActivityAt = atValue
|
|
189
|
+
}
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
// Activity that appears after a retrying health event clears it back to running.
|
|
193
|
+
// Blocked health stays sticky until terminal overrides it.
|
|
194
|
+
if (health !== null && health.disposition === "retrying" && lastActivityIndex > lastHealthIndex) {
|
|
195
|
+
health = null
|
|
196
|
+
}
|
|
197
|
+
|
|
198
|
+
/** @type {number | null} */
|
|
199
|
+
let effectiveStart = startedAt
|
|
200
|
+
const ps = providerStarted
|
|
201
|
+
if (effectiveStart === null && ps !== null && typeof ps.at === "number") {
|
|
202
|
+
effectiveStart = /** @type {number} */ (ps.at)
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
const state = terminal !== null
|
|
206
|
+
? terminal.state
|
|
207
|
+
: health !== null
|
|
208
|
+
? (health.disposition === "blocked" ? "blocked" : "retrying")
|
|
209
|
+
: ps !== null
|
|
210
|
+
? "running"
|
|
211
|
+
: "unknown"
|
|
212
|
+
|
|
213
|
+
const quietForMs = lastActivityAt === null || lastActivityAt === undefined
|
|
214
|
+
? (effectiveStart === null ? 0 : Math.max(0, clock - effectiveStart))
|
|
215
|
+
: Math.max(0, clock - lastActivityAt)
|
|
216
|
+
|
|
217
|
+
return {
|
|
218
|
+
version: 1,
|
|
219
|
+
state,
|
|
220
|
+
provider: ps !== null && typeof ps.provider === "string"
|
|
221
|
+
? ps.provider
|
|
222
|
+
: null,
|
|
223
|
+
controllerPid,
|
|
224
|
+
providerPid: ps !== null && typeof ps.pid === "number"
|
|
225
|
+
? /** @type {number} */ (ps.pid)
|
|
226
|
+
: null,
|
|
227
|
+
continuationHandle,
|
|
228
|
+
startedAt: effectiveStart === null ? null : new Date(effectiveStart).toISOString(),
|
|
229
|
+
lastActivityAt: lastActivityAt === null ? null : new Date(lastActivityAt).toISOString(),
|
|
230
|
+
quietForMs,
|
|
231
|
+
health,
|
|
232
|
+
terminal
|
|
233
|
+
}
|
|
234
|
+
}
|
|
235
|
+
|
|
236
|
+
/**
|
|
237
|
+
* Validate a single activity-log record against the closed schema.
|
|
238
|
+
* @param {Record<string, unknown>} record
|
|
239
|
+
*/
|
|
240
|
+
function validateRecordSchema(record) {
|
|
241
|
+
const type = record.type
|
|
242
|
+
if (typeof type !== "string") throw new Error("Activity log record missing type")
|
|
243
|
+
const requiredFields = VALIDATED_FACT_FIELDS[/** @type {keyof typeof VALIDATED_FACT_FIELDS} */ (type)]
|
|
244
|
+
if (requiredFields === undefined) throw new Error(`Unknown activity log record type: ${type}`)
|
|
245
|
+
const optionalFields = OPTIONAL_FACT_FIELDS[/** @type {keyof typeof OPTIONAL_FACT_FIELDS} */ (type)] ?? []
|
|
246
|
+
const allowed = new Set([...requiredFields, ...optionalFields])
|
|
247
|
+
|
|
248
|
+
const actual = Object.keys(record)
|
|
249
|
+
for (const field of actual) {
|
|
250
|
+
if (!allowed.has(field)) throw new Error(`Unexpected field in ${type} record: ${field}`)
|
|
251
|
+
}
|
|
252
|
+
for (const field of requiredFields) {
|
|
253
|
+
if (!(field in record)) throw new Error(`Missing required field in ${type} record: ${field}`)
|
|
254
|
+
}
|
|
255
|
+
|
|
256
|
+
if (type === "provider-started") {
|
|
257
|
+
if (typeof record.provider !== "string" || !PROVIDERS.has(record.provider)) throw new Error("Invalid provider")
|
|
258
|
+
if (typeof record.pid !== "number" || !Number.isSafeInteger(record.pid) || record.pid <= 0) throw new Error("Invalid pid")
|
|
259
|
+
} else if (type === "controller-started") {
|
|
260
|
+
if (typeof record.pid !== "number" || !Number.isSafeInteger(record.pid) || record.pid <= 0) throw new Error("Invalid pid")
|
|
261
|
+
} else if (type === "session-available") {
|
|
262
|
+
if (typeof record.provider !== "string" || !PROVIDERS.has(record.provider)) throw new Error("Invalid provider")
|
|
263
|
+
if (typeof record.sessionId !== "string") throw new Error("Invalid sessionId")
|
|
264
|
+
} else if (type === "activity") {
|
|
265
|
+
if (typeof record.provider !== "string" || !PROVIDERS.has(record.provider)) throw new Error("Invalid provider")
|
|
266
|
+
if (typeof record.kind !== "string" || !ACTIVITY_KINDS.has(record.kind)) throw new Error("Invalid activity kind")
|
|
267
|
+
} else if (type === "health") {
|
|
268
|
+
if (typeof record.provider !== "string" || !PROVIDERS.has(record.provider)) throw new Error("Invalid provider")
|
|
269
|
+
if (typeof record.disposition !== "string" || !HEALTH_DISPOSITIONS.has(record.disposition)) throw new Error("Invalid health disposition")
|
|
270
|
+
if (typeof record.category !== "string" || !HEALTH_CATEGORIES.has(record.category)) throw new Error("Invalid health category")
|
|
271
|
+
if (record.retryAfterMs !== undefined) {
|
|
272
|
+
if (typeof record.retryAfterMs !== "number" || !Number.isSafeInteger(record.retryAfterMs) || record.retryAfterMs <= 0) {
|
|
273
|
+
throw new Error("Invalid retryAfterMs")
|
|
274
|
+
}
|
|
275
|
+
}
|
|
276
|
+
} else if (type === "terminal") {
|
|
277
|
+
if (typeof record.provider !== "string" || !PROVIDERS.has(record.provider)) throw new Error("Invalid provider")
|
|
278
|
+
if (typeof record.state !== "string" || !TERMINAL_STATES.has(record.state)) throw new Error("Invalid terminal state")
|
|
279
|
+
if (typeof record.exitCode !== "number" || !Number.isSafeInteger(record.exitCode) || record.exitCode < 0 || record.exitCode > 255) {
|
|
280
|
+
throw new Error("Invalid exitCode")
|
|
281
|
+
}
|
|
282
|
+
}
|
|
283
|
+
if (record.at !== undefined && (typeof record.at !== "number" || !Number.isSafeInteger(record.at) || record.at < 0 || record.at > MAX_SAFE_DATE_MS)) {
|
|
284
|
+
throw new Error("Invalid timestamp")
|
|
285
|
+
}
|
|
286
|
+
}
|
|
287
|
+
|
|
288
|
+
/** @param {unknown} value @returns {value is Record<string, unknown>} */
|
|
289
|
+
function isRecord(value) {
|
|
290
|
+
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
291
|
+
}
|
package/src/activity-log.js
CHANGED
|
@@ -4,11 +4,38 @@ import {closeSync, openSync, writeSync} from "node:fs"
|
|
|
4
4
|
|
|
5
5
|
const SESSION_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,511}$/u
|
|
6
6
|
|
|
7
|
+
const ACTIVITY_KINDS = new Set(["delta", "tool", "lifecycle"])
|
|
8
|
+
const HEALTH_DISPOSITIONS = new Set(["retrying", "blocked"])
|
|
9
|
+
const HEALTH_CATEGORIES = new Set([
|
|
10
|
+
"authentication", "permission", "rate-limit", "quota", "billing",
|
|
11
|
+
"model", "network", "protocol", "unknown"
|
|
12
|
+
])
|
|
13
|
+
const TERMINAL_STATES = new Set(["completed", "failed", "cancelled"])
|
|
14
|
+
|
|
15
|
+
const SIGNAL_CANCELLED_EXIT_CODES = new Set([130, 143])
|
|
16
|
+
|
|
17
|
+
const ACTIVITY_THROTTLE_MS = 5000
|
|
18
|
+
|
|
7
19
|
export class ActivityLog {
|
|
8
|
-
/**
|
|
9
|
-
|
|
20
|
+
/**
|
|
21
|
+
* @param {string} path
|
|
22
|
+
* @param {{now?: () => number}} [options]
|
|
23
|
+
*/
|
|
24
|
+
constructor(path, options = {}) {
|
|
10
25
|
this.fileDescriptor = openSync(path, "a", 0o600)
|
|
11
26
|
this.closed = false
|
|
27
|
+
this.now = options.now ?? (() => Date.now())
|
|
28
|
+
this.terminalWritten = false
|
|
29
|
+
/** @type {string | null} */
|
|
30
|
+
this.lastHealthFingerprint = null
|
|
31
|
+
this.lastActivityAt = -ACTIVITY_THROTTLE_MS
|
|
32
|
+
this.activityCount = 0
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/** @param {number} pid */
|
|
36
|
+
recordController(pid) {
|
|
37
|
+
if (!Number.isSafeInteger(pid) || pid <= 0) throw new Error("Controller PID is unavailable")
|
|
38
|
+
this.write({type: "controller-started", pid})
|
|
12
39
|
}
|
|
13
40
|
|
|
14
41
|
/** @param {"codex" | "claude" | "kimi" | "opencode"} provider @param {number} pid */
|
|
@@ -23,15 +50,76 @@ export class ActivityLog {
|
|
|
23
50
|
this.write({type: "session-available", provider, sessionId})
|
|
24
51
|
}
|
|
25
52
|
|
|
53
|
+
/**
|
|
54
|
+
* @param {"codex" | "claude" | "kimi" | "opencode"} provider
|
|
55
|
+
* @param {"delta" | "tool" | "lifecycle"} kind
|
|
56
|
+
*/
|
|
57
|
+
recordActivity(provider, kind) {
|
|
58
|
+
if (!ACTIVITY_KINDS.has(kind)) throw new Error(`Unknown activity kind: ${kind}`)
|
|
59
|
+
const now = this.now()
|
|
60
|
+
if (now - this.lastActivityAt < ACTIVITY_THROTTLE_MS) return
|
|
61
|
+
this.lastActivityAt = now
|
|
62
|
+
this.activityCount += 1
|
|
63
|
+
this.write({type: "activity", provider, kind})
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
/**
|
|
67
|
+
* @param {"codex" | "claude" | "kimi" | "opencode"} provider
|
|
68
|
+
* @param {{disposition: "retrying" | "blocked", category: "authentication" | "permission" | "rate-limit" | "quota" | "billing" | "model" | "network" | "protocol" | "unknown", retryAfterMs?: number}} health
|
|
69
|
+
*/
|
|
70
|
+
recordHealth(provider, health) {
|
|
71
|
+
if (!HEALTH_DISPOSITIONS.has(health.disposition)) throw new Error("Unknown health disposition")
|
|
72
|
+
if (!HEALTH_CATEGORIES.has(health.category)) throw new Error("Unknown health category")
|
|
73
|
+
if (health.retryAfterMs !== undefined) {
|
|
74
|
+
if (!Number.isSafeInteger(health.retryAfterMs) || health.retryAfterMs <= 0) throw new Error("retryAfterMs must be a positive safe integer")
|
|
75
|
+
if (health.disposition !== "retrying") throw new Error("retryAfterMs is only valid for retrying disposition")
|
|
76
|
+
}
|
|
77
|
+
const fingerprint = `${health.disposition}\0${health.category}\0${health.retryAfterMs ?? ""}`
|
|
78
|
+
if (fingerprint === this.lastHealthFingerprint) return
|
|
79
|
+
this.lastHealthFingerprint = fingerprint
|
|
80
|
+
/** @type {{type: string, provider: string, disposition: string, category: string, retryAfterMs?: number}} */
|
|
81
|
+
const fact = {type: "health", provider, disposition: health.disposition, category: health.category}
|
|
82
|
+
if (health.retryAfterMs !== undefined) fact.retryAfterMs = health.retryAfterMs
|
|
83
|
+
this.write(fact)
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
/**
|
|
87
|
+
* @param {"codex" | "claude" | "kimi" | "opencode"} provider
|
|
88
|
+
* @param {"completed" | "failed" | "cancelled"} state
|
|
89
|
+
* @param {number} exitCode
|
|
90
|
+
*/
|
|
91
|
+
recordTerminal(provider, state, exitCode) {
|
|
92
|
+
if (this.terminalWritten) return
|
|
93
|
+
if (!TERMINAL_STATES.has(state)) throw new Error("Unknown terminal state")
|
|
94
|
+
if (!Number.isSafeInteger(exitCode) || exitCode < 0 || exitCode > 255) throw new Error("exitCode must be an integer between 0 and 255")
|
|
95
|
+
this.terminalWritten = true
|
|
96
|
+
this.write({type: "terminal", provider, state, exitCode})
|
|
97
|
+
}
|
|
98
|
+
|
|
26
99
|
close() {
|
|
27
100
|
if (this.closed) return
|
|
28
101
|
this.closed = true
|
|
29
102
|
closeSync(this.fileDescriptor)
|
|
30
103
|
}
|
|
31
104
|
|
|
32
|
-
/**
|
|
105
|
+
/**
|
|
106
|
+
* @param {Record<string, unknown>} fact
|
|
107
|
+
*/
|
|
33
108
|
write(fact) {
|
|
34
109
|
if (this.closed) throw new Error("Activity log is closed")
|
|
110
|
+
fact.at = this.now()
|
|
35
111
|
writeSync(this.fileDescriptor, `${JSON.stringify(fact)}\n`)
|
|
36
112
|
}
|
|
37
113
|
}
|
|
114
|
+
|
|
115
|
+
/**
|
|
116
|
+
* Classify a terminal exit code. Known signal-derived codes (130 = SIGINT,
|
|
117
|
+
* 143 = SIGTERM) map to cancelled; all others map to completed (0) or failed
|
|
118
|
+
* (nonzero). This is the only place that picks the terminal state constant.
|
|
119
|
+
* @param {number} exitCode
|
|
120
|
+
* @returns {"completed" | "failed" | "cancelled"}
|
|
121
|
+
*/
|
|
122
|
+
export function terminalState(exitCode) {
|
|
123
|
+
if (SIGNAL_CANCELLED_EXIT_CODES.has(exitCode)) return "cancelled"
|
|
124
|
+
return exitCode === 0 ? "completed" : "failed"
|
|
125
|
+
}
|
package/src/cli.js
CHANGED
|
@@ -8,7 +8,8 @@ import {createFetchTransport} from "./notifiers/fetch-transport.js"
|
|
|
8
8
|
import {createTelegramSender, parseTelegramTarget} from "./notifiers/telegram.js"
|
|
9
9
|
import {createProvider, PROVIDERS} from "./providers/index.js"
|
|
10
10
|
import {runWorker} from "./run-worker.js"
|
|
11
|
-
import {ActivityLog} from "./activity-log.js"
|
|
11
|
+
import {ActivityLog, terminalState} from "./activity-log.js"
|
|
12
|
+
import {computeStatus, parseStatusArguments, readActivityLog} from "./activity-log-status.js"
|
|
12
13
|
import {DelegatedResultAdmission, validateContinuationHandle} from "./delegated-result-admission.js"
|
|
13
14
|
import {buildProviderEnvironment, collectEvidenceRedactions, parseTelegramRequestTimeoutMs, resolveFileBackedSettings} from "./telegram-ingress/config.js"
|
|
14
15
|
import {WorkerControl} from "./worker-control.js"
|
|
@@ -41,7 +42,9 @@ const HELP = `Usage: threadwire run --provider <codex|claude|kimi|opencode> --ta
|
|
|
41
42
|
(--bytes <offset>:<limit> | --lines <start>:<limit> | --query <literal> --context-bytes <limit>)
|
|
42
43
|
threadwire capacity [--provider <codex|kimi>]...
|
|
43
44
|
[--short-reserve-percent <0-100>] [--long-reserve-percent <0-100>]
|
|
44
|
-
[--timeout-ms <positive-integer>]
|
|
45
|
+
[--timeout-ms <positive-integer>]
|
|
46
|
+
threadwire status --activity-log <absolute-path>
|
|
47
|
+
(Emits one closed versioned JSON status document from the activity log.)`
|
|
45
48
|
|
|
46
49
|
/** @typedef {{provider: string, target: string, cwd: string, toolMessages: boolean, relayWrite: boolean, prompt?: string, promptFile?: string, processNumber?: number, maxOutputLength?: number, resumeSession?: string, transcript?: string, activityLog?: string, providerArguments: string[]}} ParsedArguments */
|
|
47
50
|
/** @typedef {{evidenceRead: true, request: unknown}} EvidenceParsedArguments */
|
|
@@ -244,6 +247,16 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
244
247
|
/** @type {DelegatedResultAdmission | undefined} */
|
|
245
248
|
let runAdmission
|
|
246
249
|
try {
|
|
250
|
+
if (arguments_[0] === "status") {
|
|
251
|
+
const statusParsed = parseStatusArguments(arguments_)
|
|
252
|
+
if (validateOnly) return 0
|
|
253
|
+
const records = await readActivityLog(statusParsed.activityLog)
|
|
254
|
+
const status = computeStatus(records, {
|
|
255
|
+
...(dependencies.now === undefined ? {} : {now: dependencies.now})
|
|
256
|
+
})
|
|
257
|
+
output.write(`${JSON.stringify(status)}\n`)
|
|
258
|
+
return 0
|
|
259
|
+
}
|
|
247
260
|
const parsed = arguments_[0] === "evidence"
|
|
248
261
|
? parseEvidenceArguments(arguments_)
|
|
249
262
|
: arguments_[0] === "capacity" ? parseCapacityArguments(arguments_) : parseArguments(arguments_)
|
|
@@ -373,6 +386,7 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
373
386
|
metrics
|
|
374
387
|
})
|
|
375
388
|
activity = parsed.activityLog === undefined ? undefined : new ActivityLog(parsed.activityLog)
|
|
389
|
+
activity?.recordController(process.pid)
|
|
376
390
|
/** @type {import("./run-worker.js").RunWorkerOptions} */
|
|
377
391
|
const workerOptions = {
|
|
378
392
|
executable: provider.executable,
|
|
@@ -388,6 +402,9 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
388
402
|
if (event.type !== "text-delta") {
|
|
389
403
|
metrics.recordRejected(`${event.type}_progress`, Buffer.byteLength(JSON.stringify(event), "utf8"))
|
|
390
404
|
}
|
|
405
|
+
if (event.type === "text-delta") activity?.recordActivity(provider.name, "delta")
|
|
406
|
+
else if (event.type === "tool") activity?.recordActivity(provider.name, "tool")
|
|
407
|
+
else if (event.type === "lifecycle") activity?.recordActivity(provider.name, "lifecycle")
|
|
391
408
|
acceptAdmissionEvent(admission, event)
|
|
392
409
|
return control.accept(event)
|
|
393
410
|
},
|
|
@@ -401,6 +418,8 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
401
418
|
admission.setContinuationHandle(id)
|
|
402
419
|
activity?.recordSession(provider.name, id)
|
|
403
420
|
}
|
|
421
|
+
const health = provider.health(record)
|
|
422
|
+
if (health !== undefined) activity?.recordHealth(provider.name, health)
|
|
404
423
|
},
|
|
405
424
|
onStdoutChunk: (chunk) => {
|
|
406
425
|
metrics.recordRawChildChunk("provider_stdout", chunk.length)
|
|
@@ -459,7 +478,10 @@ export async function main(arguments_, dependencies = {}) {
|
|
|
459
478
|
}
|
|
460
479
|
admission.complete({state: terminalExitCode === 0 ? "completed" : "failed", exitCode: terminalExitCode})
|
|
461
480
|
errorOutput.write(`threadwire-context-metrics ${JSON.stringify(metrics.conciseDiagnostic())}\n`)
|
|
462
|
-
activity
|
|
481
|
+
if (activity !== undefined) {
|
|
482
|
+
activity.recordTerminal(/** @type {"codex" | "claude" | "kimi" | "opencode"} */ (parsed.provider), terminalState(terminalExitCode), terminalExitCode)
|
|
483
|
+
activity.close()
|
|
484
|
+
}
|
|
463
485
|
await boundedLaunch(ownedEvidenceStore?.close(), launchDeadline).catch(() => {})
|
|
464
486
|
launchDeadline?.close()
|
|
465
487
|
}
|
package/src/providers/index.js
CHANGED
|
@@ -6,7 +6,8 @@ import {buildKimiCommand, createKimiCompletion, createKimiParser, createKimiSess
|
|
|
6
6
|
import {buildOpenCodeCommand, createOpenCodeParser, createOpenCodeSessionId} from "./opencode.js"
|
|
7
7
|
|
|
8
8
|
/**
|
|
9
|
-
* @typedef {{
|
|
9
|
+
* @typedef {{disposition: "retrying" | "blocked", category: "authentication" | "permission" | "rate-limit" | "quota" | "billing" | "model" | "network" | "protocol" | "unknown", retryAfterMs?: number}} ProviderHealth
|
|
10
|
+
* @typedef {{name: "codex" | "claude" | "kimi" | "opencode", executable: string, arguments: string[], parse: (record: unknown) => import("../types.js").WorkerEvent[], sessionId: (record: unknown) => string | undefined, health: (record: unknown) => ProviderHealth | undefined, completion?: (record: unknown) => boolean}} Provider
|
|
10
11
|
*/
|
|
11
12
|
|
|
12
13
|
/** @type {readonly ["codex", "claude", "kimi", "opencode"]} */
|
|
@@ -16,19 +17,122 @@ export const PROVIDERS = ["codex", "claude", "kimi", "opencode"]
|
|
|
16
17
|
export function createProvider(name, providerArguments, prompt, resumeSession, environment = process.env) {
|
|
17
18
|
if (name === "codex") {
|
|
18
19
|
const command = buildCodexCommand(providerArguments, prompt, resumeSession, environment)
|
|
19
|
-
return {...command, name: "codex", parse: parseCodexEvent, sessionId: codexSessionId}
|
|
20
|
+
return {...command, name: "codex", parse: parseCodexEvent, sessionId: codexSessionId, health: extractProviderHealth}
|
|
20
21
|
}
|
|
21
22
|
if (name === "claude") {
|
|
22
23
|
const command = buildClaudeCommand(providerArguments, prompt, resumeSession, environment)
|
|
23
|
-
return {...command, name: "claude", parse: createClaudeParser(), sessionId: claudeSessionId}
|
|
24
|
+
return {...command, name: "claude", parse: createClaudeParser(), sessionId: claudeSessionId, health: extractProviderHealth}
|
|
24
25
|
}
|
|
25
26
|
if (name === "opencode") {
|
|
26
27
|
const command = buildOpenCodeCommand(providerArguments, prompt, resumeSession, environment)
|
|
27
|
-
return {...command, name: "opencode", parse: createOpenCodeParser(), sessionId: createOpenCodeSessionId()}
|
|
28
|
+
return {...command, name: "opencode", parse: createOpenCodeParser(), sessionId: createOpenCodeSessionId(), health: extractProviderHealth}
|
|
28
29
|
}
|
|
29
30
|
if (name === "kimi") {
|
|
30
31
|
const command = buildKimiCommand(providerArguments, prompt, resumeSession, environment)
|
|
31
|
-
return {...command, name: "kimi", parse: createKimiParser(), sessionId: createKimiSessionId(), completion: createKimiCompletion()}
|
|
32
|
+
return {...command, name: "kimi", parse: createKimiParser(), sessionId: createKimiSessionId(), completion: createKimiCompletion(), health: extractProviderHealth}
|
|
32
33
|
}
|
|
33
34
|
throw new Error(`--provider must be one of: ${PROVIDERS.join(", ")}`)
|
|
34
35
|
}
|
|
36
|
+
|
|
37
|
+
/**
|
|
38
|
+
* Extract safe provider health information from a structured provider record.
|
|
39
|
+
* Inspects only an explicit small list of safe containers — the top-level
|
|
40
|
+
* record and known nested `error`, `part`, `part.error`, or `*.data` objects —
|
|
41
|
+
* never traverses arbitrarily nested objects and never regexes raw message
|
|
42
|
+
* text. Generic HTTP 429 → retrying rate-limit. Structured
|
|
43
|
+
* exceeded_current_quota_error / insufficient_quota / insufficient_balance →
|
|
44
|
+
* blocked quota/billing. If no structured fields are exposed — common when a
|
|
45
|
+
* provider protocol suppresses upstream error details — no health fact is
|
|
46
|
+
* written and the run status remains running/unknown; Hermes must perform an
|
|
47
|
+
* independent bounded provider probe.
|
|
48
|
+
* @param {unknown} record
|
|
49
|
+
* @returns {ProviderHealth | undefined}
|
|
50
|
+
*/
|
|
51
|
+
export function extractProviderHealth(record) {
|
|
52
|
+
if (!isRecord(record)) return undefined
|
|
53
|
+
|
|
54
|
+
// Explicit containers to inspect: top-level record, record.error, record.part,
|
|
55
|
+
// record.part.error. Only these bounded paths; never recursive traversal.
|
|
56
|
+
const containers = [record]
|
|
57
|
+
if (isRecord(record.error)) containers.push(record.error)
|
|
58
|
+
if (isRecord(record.part)) {
|
|
59
|
+
containers.push(record.part)
|
|
60
|
+
if (isRecord(record.part.error)) containers.push(record.part.error)
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
for (const container of containers) {
|
|
64
|
+
const health = extractFromContainer(container)
|
|
65
|
+
if (health !== undefined) return health
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
return undefined
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
/**
|
|
72
|
+
* Inspect one safe container for structured health fields.
|
|
73
|
+
* @param {Record<string, unknown>} container
|
|
74
|
+
* @returns {ProviderHealth | undefined}
|
|
75
|
+
*/
|
|
76
|
+
function extractFromContainer(container) {
|
|
77
|
+
// Inspect structured data sub-object when present.
|
|
78
|
+
if (isRecord(container.data)) return extractFromContainer(container.data)
|
|
79
|
+
|
|
80
|
+
const httpStatus = container.http_status ?? container.httpStatus ?? container.status_code ?? container.statusCode
|
|
81
|
+
if (typeof httpStatus === "number" && Number.isSafeInteger(httpStatus)) {
|
|
82
|
+
if (httpStatus === 429) {
|
|
83
|
+
const retryAfterMs = safeRetryMs(container.retryAfterMs ?? container.retry_after_ms)
|
|
84
|
+
return {disposition: "retrying", category: "rate-limit", ...(retryAfterMs === undefined ? {} : {retryAfterMs})}
|
|
85
|
+
}
|
|
86
|
+
if (httpStatus >= 500 && httpStatus < 600) {
|
|
87
|
+
return {disposition: "retrying", category: "unknown"}
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
const code = container.code ?? container.error_code ?? container.errorCode ?? container.error_type ?? container.errorType
|
|
92
|
+
if (typeof code === "string") {
|
|
93
|
+
if (code === "exceeded_current_quota_error" || code === "insufficient_quota" || code === "quota_exceeded") {
|
|
94
|
+
return {disposition: "blocked", category: "quota"}
|
|
95
|
+
}
|
|
96
|
+
if (code === "insufficient_balance" || code === "billing_error" || code === "payment_required") {
|
|
97
|
+
return {disposition: "blocked", category: "billing"}
|
|
98
|
+
}
|
|
99
|
+
if (code === "invalid_api_key" || code === "unauthorized" || code === "auth_error" || code === "authentication_error") {
|
|
100
|
+
return {disposition: "blocked", category: "authentication"}
|
|
101
|
+
}
|
|
102
|
+
if (code === "forbidden" || code === "permission_denied") {
|
|
103
|
+
return {disposition: "blocked", category: "permission"}
|
|
104
|
+
}
|
|
105
|
+
if (code === "model_not_found" || code === "invalid_model" || code === "model_unavailable") {
|
|
106
|
+
return {disposition: "blocked", category: "model"}
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
if (container.insufficient_quota === true || container.quota_exceeded === true) {
|
|
111
|
+
return {disposition: "blocked", category: "quota"}
|
|
112
|
+
}
|
|
113
|
+
if (container.insufficient_balance === true) {
|
|
114
|
+
return {disposition: "blocked", category: "billing"}
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
return undefined
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
/**
|
|
121
|
+
* Extract a safe retry duration in milliseconds. Only explicit millisecond
|
|
122
|
+
* fields (retryAfterMs, retry_after_ms) are used; ambiguous short fields such
|
|
123
|
+
* as retry_after (seconds) are never interpreted as milliseconds.
|
|
124
|
+
* @param {unknown} value
|
|
125
|
+
*/
|
|
126
|
+
function safeRetryMs(value) {
|
|
127
|
+
if (typeof value === "number" && Number.isSafeInteger(value) && value > 0) return value
|
|
128
|
+
if (typeof value === "string" && /^\d+$/u.test(value)) {
|
|
129
|
+
const parsed = Number(value)
|
|
130
|
+
if (Number.isSafeInteger(parsed) && parsed > 0) return parsed
|
|
131
|
+
}
|
|
132
|
+
return undefined
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
/** @param {unknown} value @returns {value is Record<string, unknown>} */
|
|
136
|
+
function isRecord(value) {
|
|
137
|
+
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
138
|
+
}
|