@sema-agent/server 7.93.0 → 7.93.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/USAGE.md +23 -1
- package/dist/bench/s1/live-deps.js +3 -0
- package/dist/boot/fatal-exit.d.ts +21 -0
- package/dist/boot/fatal-exit.js +11 -0
- package/dist/boot/shutdown.js +24 -8
- package/dist/config-catalog.js +1 -0
- package/dist/config-types.d.ts +7 -1
- package/dist/config.js +2 -1
- package/dist/main.js +2 -4
- package/dist/plugins/host-platform.d.ts +39 -0
- package/dist/plugins/host-platform.js +32 -0
- package/dist/plugins/remote-env-host.js +23 -2
- package/dist/plugins/store-backend.js +3 -2
- package/dist/plugins/store-lock-wait.d.ts +23 -0
- package/dist/plugins/store-lock-wait.js +39 -0
- package/dist/run-local.js +5 -6
- package/package.json +2 -2
package/USAGE.md
CHANGED
|
@@ -376,10 +376,32 @@ MODEL_CASCADE_LADDER=deepseek-flash,deepseek-pro # 目录里的模型名,cheap
|
|
|
376
376
|
三动作:`store.dir_in_use`=另一个活进程持有该目录 → 停掉它(或换数据根/迁 SQL 后端);
|
|
377
377
|
`store.dir_claiming`=另一进程正在接管崩溃者留下的锁 → **瞬态,稍候重试**(接管方要么完成要么
|
|
378
378
|
自己也被收割);`store.lock_unreadable`=锁文件坏形(无 owner 可解析/嵌套接管残留)→ 人工处置,
|
|
379
|
-
message 会点名要删的文件,**不要盲删数据根下其它东西**。server
|
|
379
|
+
message 会点名要删的文件,**不要盲删数据根下其它东西**。server 刻意不把这族错**吞掉**(fail-fast:
|
|
380
380
|
两进程共用 file 数据目录是禁止形,静默容忍比拒启更坏)。
|
|
381
|
+
⚠️ 自 7.93.1 起 `store.dir_in_use` **这一码**多一道有界等待(见下面 `STORE_LOCK_WAIT_MS` 那条):
|
|
382
|
+
拒启形与文案一格不变,只是拒之前会先在窗内反复重取;另两码(`dir_claiming` / `lock_unreadable`)
|
|
383
|
+
与非锁类失败**一秒都不等**,原样抛。
|
|
381
384
|
`DB_BACKEND=memory` 并非全无盘:workflow 相关账本仍挂数据根下——多个 memory 形引擎**不要共用**
|
|
382
385
|
数据根/HOME(1.309 起并发 boot 不再拒启,但账本内容级共享仍不受支持)。
|
|
386
|
+
- **`STORE_LOCK_WAIT_MS`(缺省 `60000`,单位 ms)—— 撞上活持锁人时等一会儿再拒**:
|
|
387
|
+
`DB_BACKEND=local` 的数据根锁被**还活着**的持有者占着时(最常见的成因:壳自己 spawn 引擎,用户关掉壳
|
|
388
|
+
又立刻重开,旧引擎正在 `DRAIN_GRACE_MS` 的优雅排空 + 停机收尾里,锁要到收尾链尾部的 `backend.close()`
|
|
389
|
+
才归还),新实例在这个窗内**反复重取**数据根:拿到就照常起动(启动日志一行
|
|
390
|
+
`store_dir_lock_wait{ownerPid,dir,waitMs}`,**只一行**),到窗仍拿不到才拒启 —— 拒句是原来那一句,
|
|
391
|
+
额外写明 `waited Nms`(运维据此区分「当场拒」与「等满了才拒」)。等待发生在 `listen` **之前**,语义是
|
|
392
|
+
「启动慢了 N 秒」而不是「一边服务一边等」。
|
|
393
|
+
🔴 **方向是多等,不是放行**:窗只决定「拒之前等多久」,任何情形下都不会让第二个写者进同一个数据根。
|
|
394
|
+
代价 = 两个真独立部署误指同一目录时,那句「被 pid N 占着」要等满窗才出现(编排面的 liveness 探针
|
|
395
|
+
initialDelay 请按这个窗放宽)。要立刻响 ⇒ `STORE_LOCK_WAIT_MS=0`(逐字回到 7.93.0 的当场拒)。
|
|
396
|
+
上界不设(愿意等多久是部署自己的事);负值夹到 0 并在启动日志点名。只有 file 后端读它。
|
|
397
|
+
🔴 **两条取数据根的路是同一条腿、同一只旋钮**:HTTP 服务的 local 臂与 `run-local`(一次性 CLI)共用
|
|
398
|
+
同一只等待腿 —— **不为前台形留第二套语义**。前台形要「立刻告诉我谁占着」就设 `STORE_LOCK_WAIT_MS=0`,
|
|
399
|
+
那是文档化的通道,而不是两条路各有一套默认。
|
|
400
|
+
⚠️ **裸 boot(没有显式 `DB_BACKEND`)那一形的时序也跟着变**:它的锁冲突走的不是拒启而是既有的
|
|
401
|
+
「退回内存并 warn」降级臂(`store_db_unreachable_fallback_memory`),所以第二只实例此前**当场**降级、
|
|
402
|
+
现在要**等满窗**才降级 —— 方向仍是好的(窗内拿到就用上真数据根,而不是静默丢掉持久面),但它是一次
|
|
403
|
+
真实的启动时长变化。显式 `DB_BACKEND=local` 不走降级臂(拒启),`0` 关掉两形的等待。
|
|
404
|
+
三问成文见 `docs/DEPLOY-PREREQS.md`。
|
|
383
405
|
- **`RUN_STORE_STRICT_HYDRATE=on|off`(缺省 `off`)—— file 后端 run 账本的 fail-closed 启动门**:
|
|
384
406
|
`DB_BACKEND=local` 的 run 账本在 boot 时把 `<数据根>/runs/` 列一遍重建内存索引。那次 `readdir` 失败
|
|
385
407
|
(权限、挂载丢失、路径被换走)时**缺省行为是照常起动**,只留一行 `file_run_store_hydrate_unreadable`
|
|
@@ -342,6 +342,8 @@ async function runLeaderForTrap(rt, brain, models, roles, pricing, trap, cellId,
|
|
|
342
342
|
"--enable=receive-pack",
|
|
343
343
|
remoteDir,
|
|
344
344
|
], { stdio: "ignore" });
|
|
345
|
+
let daemonSpawnError;
|
|
346
|
+
daemon.on("error", (e) => { daemonSpawnError = e.message; });
|
|
345
347
|
const waitForPort = async () => {
|
|
346
348
|
for (let i = 0; i < 30; i++) {
|
|
347
349
|
const up = await new Promise((resolve) => {
|
|
@@ -387,6 +389,7 @@ async function runLeaderForTrap(rt, brain, models, roles, pricing, trap, cellId,
|
|
|
387
389
|
let res;
|
|
388
390
|
try {
|
|
389
391
|
if (!(await waitForPort())) {
|
|
392
|
+
console.error(`[s1] git daemon did not come up on 127.0.0.1:${port}${daemonSpawnError ? ` — spawn failed: ${daemonSpawnError}` : ""}`);
|
|
390
393
|
return { ok: false, reports: [], infraFailed: true };
|
|
391
394
|
}
|
|
392
395
|
const seedDir = mkdtempSync(join(tmpdir(), "s1-team-seed-"));
|
|
@@ -1,3 +1,24 @@
|
|
|
1
1
|
/** `err` 的 fatal 日志文本:`String(err)` 过形状律一次。纯函数,零副作用。 */
|
|
2
2
|
export declare function buildFatalErrorText(err: unknown): string;
|
|
3
|
+
/**
|
|
4
|
+
* 进程级 fatal 出口的**唯一一只腿**:同一只 {@link buildFatalErrorText} 产物写两个面,然后按崩溃码退。
|
|
5
|
+
*
|
|
6
|
+
* ── 病(S-555,test [ref] P2)────────────────────────────────────────────────────────────────────
|
|
7
|
+
* 出口过去只有那条 JSON 行。而拒启文案是**多行**的 —— S-547 的列集对账会生成
|
|
8
|
+
* `ALTER TABLE <表> ADD COLUMN <列定义>;`,它存在的全部理由就是「运维直接粘进 psql」;塞进 JSON 字符串后
|
|
9
|
+
* 整段变成一串 `\n` 转义,**粘不走**。dist 里也没有 pretty / `isTTY` 分支去把它摊开(有也不该有:出口的
|
|
10
|
+
* 可读性不该取决于终端是谁)。
|
|
11
|
+
*
|
|
12
|
+
* ── 修 ─────────────────────────────────────────────────────────────────────────────────────────
|
|
13
|
+
* **同一只产物**再原样多行写一份到 stderr。刻意不做的三件:不铸第二份文案(两份措辞必然分叉)、不加
|
|
14
|
+
* pretty 旋钮(一个只有出错时才走的出口不该有配置面)、JSON 行**照旧**(机读面零回退,scraper 不动)。
|
|
15
|
+
* 两个面都在 stderr,次序=先机器行后人读段。**这个次序还承重**:`createLogger()` 的构造会装上 stdio 的
|
|
16
|
+
* EPIPE 无害化监听(见 `observability/logger.ts` 那段注 —— 宿主把 stderr 接管道又先退时,下一次写会发一个
|
|
17
|
+
* 异步 `'error'`,无监听即致命),所以那条裸 `process.stderr.write` **必须**排在它之后;同步抛由下面的
|
|
18
|
+
* `try/finally` 兜住。
|
|
19
|
+
*
|
|
20
|
+
* 全仓两个 fatal 出口(`main.ts` 的 `main().catch` 与 `run-local.ts` 的同名尾巴)都经这里 —— 后者此前
|
|
21
|
+
* 连形状律都没过(裸 `String(err)`),两处各写一遍正是「同类不同命」的土壤。
|
|
22
|
+
*/
|
|
23
|
+
export declare function fatalExit(err: unknown): never;
|
|
3
24
|
//# sourceMappingURL=fatal-exit.d.ts.map
|
package/dist/boot/fatal-exit.js
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
1
|
import { redactSecrets } from "../trace/redact.js";
|
|
2
|
+
import { createLogger } from "../observability/logger.js";
|
|
2
3
|
export function buildFatalErrorText(err) {
|
|
3
4
|
return redactSecrets(String(err));
|
|
4
5
|
}
|
|
6
|
+
export function fatalExit(err) {
|
|
7
|
+
const text = buildFatalErrorText(err);
|
|
8
|
+
try {
|
|
9
|
+
createLogger().error("fatal", { err: text });
|
|
10
|
+
process.stderr.write(text + "\n");
|
|
11
|
+
}
|
|
12
|
+
finally {
|
|
13
|
+
process.exit(1);
|
|
14
|
+
}
|
|
15
|
+
}
|
|
5
16
|
//# sourceMappingURL=fatal-exit.js.map
|
package/dist/boot/shutdown.js
CHANGED
|
@@ -2,6 +2,7 @@ import { Runner, defaultTaskRegistry } from "@sema-agent/core";
|
|
|
2
2
|
import { writeCrashLast, clearCrashLast, readCrashLast } from "./crash-last.js";
|
|
3
3
|
import { createSighupIdleHandler } from "../sighup-idle.js";
|
|
4
4
|
import { createParentWatch } from "../parent-watch.js";
|
|
5
|
+
import { reapHostForegroundChildren } from "../plugins/host-platform.js";
|
|
5
6
|
import { createEngineLeaseTracker } from "./engine-lease.js";
|
|
6
7
|
import { writeExitLast, readExitLast, clearExitLast } from "./exit-last.js";
|
|
7
8
|
import { isScriptRealmRejection, describeRejectionReason } from "../orchestration/hardened-vm-runner.js";
|
|
@@ -30,8 +31,29 @@ export function installShutdownHandlers(ctx) {
|
|
|
30
31
|
if (dataRoot)
|
|
31
32
|
writeCrashLast(dataRoot, { version, inflight: drainState.inflight?.(), err });
|
|
32
33
|
};
|
|
34
|
+
const reapOwnChildProcesses = () => {
|
|
35
|
+
try {
|
|
36
|
+
const reaped = defaultTaskRegistry.reapAllSessionBackground();
|
|
37
|
+
if (reaped > 0)
|
|
38
|
+
logger.info("shutdown_background_reaped", { reaped });
|
|
39
|
+
}
|
|
40
|
+
catch (err) {
|
|
41
|
+
logger.warn("shutdown_background_reap_failed", { err: String(err) });
|
|
42
|
+
}
|
|
43
|
+
try {
|
|
44
|
+
const fg = reapHostForegroundChildren();
|
|
45
|
+
if (fg.signalled > 0 || fg.alreadyDead > 0)
|
|
46
|
+
logger.info("shutdown_foreground_reaped", { signalled: fg.signalled, alreadyDead: fg.alreadyDead });
|
|
47
|
+
if (fg.failed.length > 0)
|
|
48
|
+
logger.warn("shutdown_foreground_reap_incomplete", { groups: fg.failed });
|
|
49
|
+
}
|
|
50
|
+
catch (err) {
|
|
51
|
+
logger.warn("shutdown_foreground_reap_failed", { err: String(err) });
|
|
52
|
+
}
|
|
53
|
+
};
|
|
33
54
|
const onUncaughtException = (err) => {
|
|
34
55
|
try {
|
|
56
|
+
reapOwnChildProcesses();
|
|
35
57
|
writeLastWords(err);
|
|
36
58
|
console.error(err);
|
|
37
59
|
}
|
|
@@ -50,6 +72,7 @@ export function installShutdownHandlers(ctx) {
|
|
|
50
72
|
return;
|
|
51
73
|
}
|
|
52
74
|
try {
|
|
75
|
+
reapOwnChildProcesses();
|
|
53
76
|
writeLastWords(err);
|
|
54
77
|
console.error(err);
|
|
55
78
|
}
|
|
@@ -92,14 +115,7 @@ export function installShutdownHandlers(ctx) {
|
|
|
92
115
|
subRunner.sessions.dispose();
|
|
93
116
|
void lspManager?.dispose?.();
|
|
94
117
|
void workflowNotifyJournal?.close?.();
|
|
95
|
-
|
|
96
|
-
const reaped = defaultTaskRegistry.reapAllSessionBackground();
|
|
97
|
-
if (reaped > 0)
|
|
98
|
-
logger.info("shutdown_background_reaped", { reaped });
|
|
99
|
-
}
|
|
100
|
-
catch (err) {
|
|
101
|
-
logger.warn("shutdown_background_reap_failed", { err: String(err) });
|
|
102
|
-
}
|
|
118
|
+
reapOwnChildProcesses();
|
|
103
119
|
void fleetClient?.stop();
|
|
104
120
|
server.close(() => {
|
|
105
121
|
void retentionTeardown.then(() => (backend ? backend.close() : Promise.resolve())).finally(() => process.exit(0));
|
package/dist/config-catalog.js
CHANGED
|
@@ -423,6 +423,7 @@ export const CONFIG_CATALOG = [
|
|
|
423
423
|
r("SSH_PORT", "orchestration", "number", "ssh 端口", { derivedDefaultNote: "缺席 ⇒ 22" }),
|
|
424
424
|
r("SSH_PRIVATE_KEY", "orchestration", "secret", "ssh 私钥(adapter 持有,绝不进 workspace)", { danger: SEC }),
|
|
425
425
|
r("SSH_USER", "orchestration", "string", "ssh 用户(必需件)"),
|
|
426
|
+
r("STORE_LOCK_WAIT_MS", "store", "number", "local 数据根 LOCK 撞上活持锁人时的有界等待窗(ms;0=当场拒;到窗仍拿不到 ⇒ 原码拒启,永不放行第二个写者)", { staticDefault: "60000", danger: DATA, configKey: "storeLockWaitMs" }),
|
|
426
427
|
r("STORE_PROBE_INTERVAL_MS", "store", "number", "/health 活体 store 后台探针间隔(ms;0=显式关;仅 SQL 后端挂环)", { staticDefault: "15000", configKey: "storeProbeIntervalMs" }),
|
|
427
428
|
r("STREAM_APPROVAL_ADHOC_GRACE_MS", "approval", "number", "流内审批 adhoc 腿窗后宽限(ms;须 window+grace < orphanTtl,破坏形拒启)", { staticDefault: "60000" }),
|
|
428
429
|
r("STREAM_APPROVAL_ADMIT_MAX_PER_OWNER", "approval", "number", "流内审批 per-owner 未决帽", { staticDefault: "256" }),
|
package/dist/config-types.d.ts
CHANGED
|
@@ -502,6 +502,12 @@ export interface ServiceConfigFlat {
|
|
|
502
502
|
/** B1 /health 活体 store 探测:后台缓存探针间隔 ms(STORE_PROBE_INTERVAL_MS;0=显式关,负/非数 boot
|
|
503
503
|
* 响亮拒)。只对 SQL 后端接线;探针原语=backend.dbNowMs()(S10 时钟探针的真 DB 往返,零新 SQL 面)。 */
|
|
504
504
|
storeProbeIntervalMs: number;
|
|
505
|
+
/** S-553 —— `DB_BACKEND=local` 数据根 `LOCK` 撞上**活**持锁人时的**有界等待窗** ms
|
|
506
|
+
* (`STORE_LOCK_WAIT_MS`;缺省 60_000,地板 0 = 7.93.0 的当场拒,**不设上界**)。
|
|
507
|
+
* 病 = 壳自 spawn 引擎的单机形:旧引擎排空期间一直抱着锁,新引擎当场 fatal ⇒ 用户「刚关掉就再也起不来」。
|
|
508
|
+
* 失败方向一格不动:到窗仍拿不到 ⇒ 原码 fatal(文案多一句等了多久),**永不**放行第二个写者。
|
|
509
|
+
* 只有 file 后端读它;判据与偏离说明见 `plugins/store-lock-wait.ts` 顶注。 */
|
|
510
|
+
storeLockWaitMs: number;
|
|
505
511
|
/** Long-term memory ([ref] S1, clay 2026-07-08): the injection-first file-based memory ENGINE
|
|
506
512
|
* (core `RunnerDeps.memoryBackend` + `MemoryEngine`) is the ONLY memory plane — the legacy MemoryStore
|
|
507
513
|
* path (MEMORY_BACKEND/EMBEDDING_* SQL/file note stores) was dropped without migration. Default ON;
|
|
@@ -1858,7 +1864,7 @@ export interface ServiceConfigFlat {
|
|
|
1858
1864
|
configLocalDir?: string;
|
|
1859
1865
|
}
|
|
1860
1866
|
/** 组:store(持久化)—— DB 引擎三态、session 后端、SQL coords、快照 BLOB / SendUserFile 对象存储。 */
|
|
1861
|
-
export type ServiceStoreConfig = Pick<ServiceConfigFlat, "sessionBackend" | "sessionCacheTtlSec" | "storeProbeIntervalMs" | "dbBackend" | "dbBackendExplicit" | "localDataRoot" | "runStoreStrictHydrate" | "tidb" | "pg" | "dbQueryTimeoutMs" | "snapshotBlobStore" | "snapshotBlobSqlMaxBytes" | "snapshotBlobAllowSql" | "fileHistory" | "sendUserFile">;
|
|
1867
|
+
export type ServiceStoreConfig = Pick<ServiceConfigFlat, "sessionBackend" | "sessionCacheTtlSec" | "storeProbeIntervalMs" | "storeLockWaitMs" | "dbBackend" | "dbBackendExplicit" | "localDataRoot" | "runStoreStrictHydrate" | "tidb" | "pg" | "dbQueryTimeoutMs" | "snapshotBlobStore" | "snapshotBlobSqlMaxBytes" | "snapshotBlobAllowSql" | "fileHistory" | "sendUserFile">;
|
|
1862
1868
|
/** 组:modelPlane(模型面)—— 网关坐标、Anthropic 路线、韧性旋钮、主/廉价 model entry、role 表、降级梯。 */
|
|
1863
1869
|
export type ServiceModelPlaneConfig = Pick<ServiceConfigFlat, "gatewayBaseUrl" | "gatewayApiKey" | "gatewayFallbackUrls" | "gatewayMaxRetries" | "anthropic" | "resilience" | "model" | "envModel" | "models" | "modelApiKeyEnv" | "modelApiKeys" | "modelQuotaWeights" | "tiers" | "projects" | "roles" | "atModelAllowlist" | "cascadeLadder" | "degrade">;
|
|
1864
1870
|
/** 组:approval(审批 / HITL 门)。`directDoorActive` 无 env 解析腿(装配层三域合取的产物),但语义上
|
package/dist/config.js
CHANGED
|
@@ -725,6 +725,7 @@ function parseStoreDomain(ctx) {
|
|
|
725
725
|
sessionBackend,
|
|
726
726
|
sessionCacheTtlSec: numEnv("SESSION_CACHE_TTL_SEC", "300"),
|
|
727
727
|
storeProbeIntervalMs: numEnvBounded("STORE_PROBE_INTERVAL_MS", "15000", 0, 3_600_000),
|
|
728
|
+
storeLockWaitMs: clampEnvWithWarn("STORE_LOCK_WAIT_MS", "60000", 0),
|
|
728
729
|
dbBackend,
|
|
729
730
|
dbBackendExplicit: dbBackendSet,
|
|
730
731
|
localDataRoot,
|
|
@@ -2030,7 +2031,7 @@ function parseA2aServeSkills(raw) {
|
|
|
2030
2031
|
});
|
|
2031
2032
|
}
|
|
2032
2033
|
const STORE_GROUP_KEYS = [
|
|
2033
|
-
"sessionBackend", "sessionCacheTtlSec", "storeProbeIntervalMs", "dbBackend", "dbBackendExplicit", "localDataRoot",
|
|
2034
|
+
"sessionBackend", "sessionCacheTtlSec", "storeProbeIntervalMs", "storeLockWaitMs", "dbBackend", "dbBackendExplicit", "localDataRoot",
|
|
2034
2035
|
"runStoreStrictHydrate",
|
|
2035
2036
|
"tidb", "pg", "dbQueryTimeoutMs", "snapshotBlobStore", "snapshotBlobSqlMaxBytes", "snapshotBlobAllowSql", "fileHistory", "sendUserFile",
|
|
2036
2037
|
];
|
package/dist/main.js
CHANGED
|
@@ -1,5 +1,4 @@
|
|
|
1
|
-
import {
|
|
2
|
-
import { buildFatalErrorText } from "./boot/fatal-exit.js";
|
|
1
|
+
import { fatalExit } from "./boot/fatal-exit.js";
|
|
3
2
|
import { runStage01 } from "./boot/stage-01-config.js";
|
|
4
3
|
import { runStage02 } from "./boot/stage-02-registry-pull.js";
|
|
5
4
|
import { runStage03 } from "./boot/stage-03-store-backend.js";
|
|
@@ -23,7 +22,6 @@ async function main() {
|
|
|
23
22
|
await runStage10(s09);
|
|
24
23
|
}
|
|
25
24
|
void main().catch((err) => {
|
|
26
|
-
|
|
27
|
-
process.exit(1);
|
|
25
|
+
fatalExit(err);
|
|
28
26
|
});
|
|
29
27
|
//# sourceMappingURL=main.js.map
|
|
@@ -48,6 +48,45 @@ export declare function killTreeSoft(pid: number): void;
|
|
|
48
48
|
/** Graceful tree kill where the CALLER has no escalation timer of its own (currently unused by the host lane —
|
|
49
49
|
* exported for parity with core's surface; the bg driver keeps its own HOST_BG_KILL_GRACE_MS ladder). */
|
|
50
50
|
export { killProcessTree };
|
|
51
|
+
/** 登记回执 = 注销的唯一凭据(纯数据;身份即凭据,见簿注)。 */
|
|
52
|
+
export interface ForegroundChildTicket {
|
|
53
|
+
/** 组领袖 pid = pgid(POSIX `detached:true` 的直接后果)。 */
|
|
54
|
+
readonly pid: number;
|
|
55
|
+
}
|
|
56
|
+
/** 登记一只自起的前台子进程,返回注销用的回执。 */
|
|
57
|
+
export declare function trackHostForegroundChild(pid: number): ForegroundChildTicket;
|
|
58
|
+
/** 注销(`exit` / settle / 领养移交)。按回执身份删 ⇒ 幂等,且**绝不**碰同号的其它登记。 */
|
|
59
|
+
export declare function untrackHostForegroundChild(ticket: ForegroundChildTicket): void;
|
|
60
|
+
/** 一次前台收割的读数。三格分开是**刻意**的:`alreadyDead` 是想要的结果(不是失败),`failed` 是**真的**
|
|
61
|
+
* 没收上来 —— 两者混成一个数就等于把后者静默掉([ref]:这条腿上不许有安静的兜底)。 */
|
|
62
|
+
export interface ForegroundReapReading {
|
|
63
|
+
/** 真发出了杀伤信号的组数。 */
|
|
64
|
+
readonly signalled: number;
|
|
65
|
+
/** `ESRCH` —— 组已经不在了。信号的语义就是「让它死」,它已经死了 ⇒ 记数,不报警。 */
|
|
66
|
+
readonly alreadyDead: number;
|
|
67
|
+
/** 其余拒因,逐条 `pid:code`。本模块没有 logger,所以**交回调用方响亮记下**(它有)。 */
|
|
68
|
+
readonly failed: readonly string[];
|
|
69
|
+
}
|
|
70
|
+
/**
|
|
71
|
+
* 收割簿上全部前台子进程组。返回**发出**信号的读数,不是确认死亡数 —— 确认要等,而本函数的调用点之一
|
|
72
|
+
* 是 uncaught 出口。
|
|
73
|
+
*
|
|
74
|
+
* **同步形、一次到位、不等宽限**:两个调用点(`hardShutdown` / uncaught 双钩)都紧挨着 `process.exit`,
|
|
75
|
+
* `await` 一个宽限梯就是把收割赌在一个永远等不到的续段上。上界 = 簿上条数 × 一次 `kill(-pid)` syscall
|
|
76
|
+
* (簿条数被 host lane 的在飞 turn 数界住),没有定时器、没有 promise。
|
|
77
|
+
*
|
|
78
|
+
* 🔴 **为什么是 `killTreeHard` 而不是 SIGTERM**(codex 对抗复审 [high],亲跑复现后改):软鞍的语义是
|
|
79
|
+
* 「给它一个收尾窗,到点我再升级」,而这两个调用点之后**没有任何人在场**去升级 —— 一个
|
|
80
|
+
* `trap '' TERM` 的组收到 SIGTERM 后 `kill` 不抛(实测 `signalled:1, failed:[]`、随后仍然在跑),于是
|
|
81
|
+
* 「已收割」的读数是**假的**,孤儿照旧永久存活;win32 的软鞍对普通 console tree 更是明确杀不掉任何东西
|
|
82
|
+
* (见 {@link killTreeSoft} 注)。且本仓对**前台 exec** 的既有杀伤姿势(abort / timeout / 流被放弃)
|
|
83
|
+
* 一律就是 `killTreeHard` —— 收割用**同一只**,不新造一种只能兑现一半的姿势(规则集更小,不是更大)。
|
|
84
|
+
* 代价如实写:一条行为良好的命令在这里拿不到收尾窗;它换来的是「引擎死了,它起的命令一定也死」这句话
|
|
85
|
+
* 真的成立。后台腿的宽限梯不受影响(那条腿有自己的属主,且不在本簿里)。
|
|
86
|
+
*/
|
|
87
|
+
export declare function reapHostForegroundChildren(): ForegroundReapReading;
|
|
88
|
+
/** 测试面:簿上现存条数(产品面没有消费者 —— 收割腿只读它自己那只 Set)。 */
|
|
89
|
+
export declare function liveHostForegroundChildCountForTest(): number;
|
|
51
90
|
/**
|
|
52
91
|
* win32 env-key case collapse (design D1). Windows env keys are case-insensitive; Node hands the
|
|
53
92
|
* object to CreateProcess as-is, so a case-sensitive merge that produced BOTH `Path` (inherited) and `PATH`
|
|
@@ -54,6 +54,38 @@ export function killTreeSoft(pid) {
|
|
|
54
54
|
process.kill(-pid, "SIGTERM");
|
|
55
55
|
}
|
|
56
56
|
export { killProcessTree };
|
|
57
|
+
const liveHostForegroundGroups = new Set();
|
|
58
|
+
export function trackHostForegroundChild(pid) {
|
|
59
|
+
const ticket = { pid };
|
|
60
|
+
liveHostForegroundGroups.add(ticket);
|
|
61
|
+
return ticket;
|
|
62
|
+
}
|
|
63
|
+
export function untrackHostForegroundChild(ticket) {
|
|
64
|
+
liveHostForegroundGroups.delete(ticket);
|
|
65
|
+
}
|
|
66
|
+
export function reapHostForegroundChildren() {
|
|
67
|
+
let signalled = 0;
|
|
68
|
+
let alreadyDead = 0;
|
|
69
|
+
const failed = [];
|
|
70
|
+
for (const { pid } of liveHostForegroundGroups) {
|
|
71
|
+
try {
|
|
72
|
+
killTreeHard(pid);
|
|
73
|
+
signalled++;
|
|
74
|
+
}
|
|
75
|
+
catch (err) {
|
|
76
|
+
const code = err?.code;
|
|
77
|
+
if (code === "ESRCH")
|
|
78
|
+
alreadyDead++;
|
|
79
|
+
else
|
|
80
|
+
failed.push(`${pid}:${code ?? String(err)}`);
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
liveHostForegroundGroups.clear();
|
|
84
|
+
return { signalled, alreadyDead, failed };
|
|
85
|
+
}
|
|
86
|
+
export function liveHostForegroundChildCountForTest() {
|
|
87
|
+
return liveHostForegroundGroups.size;
|
|
88
|
+
}
|
|
57
89
|
export function collapseWin32EnvKeys(env) {
|
|
58
90
|
if (!IS_WIN32)
|
|
59
91
|
return env;
|
|
@@ -7,7 +7,7 @@ import { openSync, closeSync, readSync, statSync, truncateSync, unlinkSync, mkdi
|
|
|
7
7
|
import { StringDecoder } from "node:string_decoder";
|
|
8
8
|
import { armPipeDestroyGrace, numEnvOr, SYMLINK_CHAIN_MAX_HOPS } from "./remote-shell.js";
|
|
9
9
|
import { hostBackgroundShellEnabled, hostExecSpoolEnabled } from "../config.js";
|
|
10
|
-
import { resolveHostShell, hostShell, spawnGroupOptions, killTreeHard, killTreeSoft, collapseWin32EnvKeys } from "./host-platform.js";
|
|
10
|
+
import { resolveHostShell, hostShell, spawnGroupOptions, killTreeHard, killTreeSoft, collapseWin32EnvKeys, trackHostForegroundChild, untrackHostForegroundChild } from "./host-platform.js";
|
|
11
11
|
import { BackgroundShellManager, seedMemStream, feedMemStream, drainMemStream } from "./background-shell-support.js";
|
|
12
12
|
import { FileError, ExecutionError, RemoteExecutionError, scrubSecretEnv, RollingTailBuffer, markTruncated, SchedulerError, BackgroundShellError, } from "@sema-agent/core";
|
|
13
13
|
import { recordSecretEnvScrub } from "../observability/secret-env-scrub.js";
|
|
@@ -241,9 +241,9 @@ export class RemoteHostExecutionEnv {
|
|
|
241
241
|
throw new BackgroundShellError("spawn_failed", e instanceof Error ? e.message : String(e), e);
|
|
242
242
|
}
|
|
243
243
|
const pid = child.pid;
|
|
244
|
+
child.on("error", () => ctx.onTerminal(true));
|
|
244
245
|
if (pid != null) {
|
|
245
246
|
child.on("close", (code, signal) => ctx.onTerminal(false, code ?? (signal ? 128 + (signalNumber(signal) ?? 9) : 0)));
|
|
246
|
-
child.on("error", () => ctx.onTerminal(true));
|
|
247
247
|
child.unref();
|
|
248
248
|
}
|
|
249
249
|
await outFh.close().catch(() => { });
|
|
@@ -403,10 +403,15 @@ export class RemoteHostExecutionEnv {
|
|
|
403
403
|
env: this.mergeEnv(options?.env),
|
|
404
404
|
...spawnGroupOptions(),
|
|
405
405
|
});
|
|
406
|
+
const ticket = child.pid != null ? trackHostForegroundChild(child.pid) : undefined;
|
|
407
|
+
if (ticket)
|
|
408
|
+
child.once("exit", () => untrackHostForegroundChild(ticket));
|
|
406
409
|
const finish = (r) => {
|
|
407
410
|
if (settled)
|
|
408
411
|
return;
|
|
409
412
|
settled = true;
|
|
413
|
+
if (ticket)
|
|
414
|
+
untrackHostForegroundChild(ticket);
|
|
410
415
|
clearTimeout(timer);
|
|
411
416
|
options?.abortSignal?.removeEventListener("abort", onAbort);
|
|
412
417
|
removeDetachListener?.();
|
|
@@ -455,6 +460,8 @@ export class RemoteHostExecutionEnv {
|
|
|
455
460
|
if (forceSettleTimer)
|
|
456
461
|
clearTimeout(forceSettleTimer);
|
|
457
462
|
options?.abortSignal?.removeEventListener("abort", onAbort);
|
|
463
|
+
if (ticket)
|
|
464
|
+
untrackHostForegroundChild(ticket);
|
|
458
465
|
settled = true;
|
|
459
466
|
resolve(ok({ stdout: seedOut, stderr: seedErr, exitCode: 0, detached: { shellId } }));
|
|
460
467
|
};
|
|
@@ -521,6 +528,7 @@ export class RemoteHostExecutionEnv {
|
|
|
521
528
|
const errBuf = new RollingTailBuffer();
|
|
522
529
|
let settled = false;
|
|
523
530
|
let handedOver = false;
|
|
531
|
+
let ticket;
|
|
524
532
|
let pump;
|
|
525
533
|
let drainTimer;
|
|
526
534
|
const cursor = { out: 0, err: 0 };
|
|
@@ -613,6 +621,8 @@ export class RemoteHostExecutionEnv {
|
|
|
613
621
|
clearInterval(pump);
|
|
614
622
|
pump = undefined;
|
|
615
623
|
}
|
|
624
|
+
if (ticket)
|
|
625
|
+
untrackHostForegroundChild(ticket);
|
|
616
626
|
if (drainTimer) {
|
|
617
627
|
clearTimeout(drainTimer);
|
|
618
628
|
drainTimer = undefined;
|
|
@@ -665,6 +675,8 @@ export class RemoteHostExecutionEnv {
|
|
|
665
675
|
cleanupSpool();
|
|
666
676
|
throw e;
|
|
667
677
|
}
|
|
678
|
+
if (child.pid != null)
|
|
679
|
+
ticket = trackHostForegroundChild(child.pid);
|
|
668
680
|
closeSpoolFds();
|
|
669
681
|
child.on("error", (e) => {
|
|
670
682
|
finish({ ok: false, error: new ExecutionError("spawn_error", `host spawn failed: ${e.message}`) });
|
|
@@ -753,6 +765,8 @@ export class RemoteHostExecutionEnv {
|
|
|
753
765
|
settleFromTermination(code, signal);
|
|
754
766
|
});
|
|
755
767
|
child.on("exit", (code, signal) => {
|
|
768
|
+
if (ticket)
|
|
769
|
+
untrackHostForegroundChild(ticket);
|
|
756
770
|
exited = true;
|
|
757
771
|
drainTimer = setTimeout(() => settleFromTermination(code, signal), EXIT_SETTLE_DRAIN_MS);
|
|
758
772
|
drainTimer.unref?.();
|
|
@@ -797,6 +811,9 @@ export class RemoteHostExecutionEnv {
|
|
|
797
811
|
const signalReady = () => { const w = wake; wake = undefined; w?.(); };
|
|
798
812
|
const sh = hostShell();
|
|
799
813
|
const child = spawn(sh.shell, [...sh.args, command], { cwd, env: this.mergeEnv(options?.env), ...spawnGroupOptions() });
|
|
814
|
+
const ticket = child.pid != null ? trackHostForegroundChild(child.pid) : undefined;
|
|
815
|
+
if (ticket)
|
|
816
|
+
child.once("exit", () => untrackHostForegroundChild(ticket));
|
|
800
817
|
let pipeDestroyTimer;
|
|
801
818
|
const kill = () => {
|
|
802
819
|
try {
|
|
@@ -831,6 +848,8 @@ export class RemoteHostExecutionEnv {
|
|
|
831
848
|
child.on("close", (code, signal) => {
|
|
832
849
|
if (pipeDestroyTimer)
|
|
833
850
|
clearTimeout(pipeDestroyTimer);
|
|
851
|
+
if (ticket)
|
|
852
|
+
untrackHostForegroundChild(ticket);
|
|
834
853
|
if (finished)
|
|
835
854
|
return;
|
|
836
855
|
exitCode = code ?? (signal ? 128 + (signalNumber(signal) ?? 0) : 1);
|
|
@@ -906,6 +925,8 @@ export class RemoteHostExecutionEnv {
|
|
|
906
925
|
if (wallTimer)
|
|
907
926
|
clearTimeout(wallTimer);
|
|
908
927
|
options?.signal?.removeEventListener("abort", onAbort);
|
|
928
|
+
if (ticket)
|
|
929
|
+
untrackHostForegroundChild(ticket);
|
|
909
930
|
}
|
|
910
931
|
}
|
|
911
932
|
async absolutePath(p) {
|
|
@@ -8,6 +8,7 @@ import { FileOutcomeSink } from "./file-outcome-sink.js";
|
|
|
8
8
|
import { TiDBWorkflowRunStore, TiDBWorkflowCompletionInbox, TiDBWorkflowNotifyJournalStore, TiDBWorkflowAgentSessionIndex, PgWorkflowRunStore, PgWorkflowCompletionInbox, PgWorkflowNotifyJournalStore, PgWorkflowAgentSessionIndex } from "./workflow-run-store-sql.js";
|
|
9
9
|
import { FileRunStore } from "./file-run-store.js";
|
|
10
10
|
import { StoreBootRefusalError } from "./store-contracts.js";
|
|
11
|
+
import { acquireLocalBackendWithWait } from "./store-lock-wait.js";
|
|
11
12
|
import { NO_SQL_MIGRATIONS, runSqlOneTimeMigrations } from "./one-time-migrations.js";
|
|
12
13
|
import { LocalSessionStore } from "./local-session-store.js";
|
|
13
14
|
import { LocalCheckpointStore } from "./local-checkpoint-store.js";
|
|
@@ -215,7 +216,7 @@ class LocalBackend {
|
|
|
215
216
|
this.onWarn = onWarn;
|
|
216
217
|
this.onError = onError;
|
|
217
218
|
try {
|
|
218
|
-
this.fileBackend = new FileStorageBackend({
|
|
219
|
+
this.fileBackend = acquireLocalBackendWithWait(() => new FileStorageBackend({
|
|
219
220
|
root,
|
|
220
221
|
...(config.fileHistory !== undefined ? { fileHistory: config.fileHistory } : {}),
|
|
221
222
|
onCorruptRead: (info) => this.onWarn?.("file_store_corrupt_read", {
|
|
@@ -228,7 +229,7 @@ class LocalBackend {
|
|
|
228
229
|
"layer (deployment policy / hooks / shell gate still applied); session repo = a listing silently omitted " +
|
|
229
230
|
"rows; file-history = a rewind boundary or scope read as missing. Inspect the named path before deciding.",
|
|
230
231
|
}),
|
|
231
|
-
});
|
|
232
|
+
}), config.storeLockWaitMs, (fields) => this.onWarn?.("store_dir_lock_wait", { ...fields }));
|
|
232
233
|
}
|
|
233
234
|
catch (e) {
|
|
234
235
|
if (e instanceof AdoptionError)
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
/** 轮询节拍。250ms:比人感知门槛粗、比最短真等待窗(壳重启窗 1-3s)细一个数量级。 */
|
|
2
|
+
export declare const STORE_LOCK_POLL_MS = 250;
|
|
3
|
+
/** 进入等待腿时的**一次性**披露(不刷屏:结构上只在第一次撞锁时调一次,不是靠一个 flag 挡)。 */
|
|
4
|
+
export type StoreLockWaitSink = (fields: {
|
|
5
|
+
readonly ownerPid: number;
|
|
6
|
+
readonly dir: string;
|
|
7
|
+
readonly waitMs: number;
|
|
8
|
+
}) => void;
|
|
9
|
+
/** 时钟两面。`sleep` 是**同步**的(理由见顶注);两面都是形参,判据不靠真时间。 */
|
|
10
|
+
export interface StoreLockWaitClock {
|
|
11
|
+
now(): number;
|
|
12
|
+
sleep(ms: number): void;
|
|
13
|
+
}
|
|
14
|
+
/**
|
|
15
|
+
* 取数据根(= 跑一次 `construct()`);撞上活持锁人且窗为正 ⇒ 在窗内反复重取,到窗仍取不到 ⇒ 原码 fatal。
|
|
16
|
+
*
|
|
17
|
+
* @param construct 取数据根的那一次构造(本仓唯一调用点:`store-backend.ts` 的 local 臂,它在那里构造 `FileStorageBackend`)。
|
|
18
|
+
* **必须**可重入:core 先取锁后建店,取不到时零副作用(顶注已亲读核实)。
|
|
19
|
+
* @param waitMs `STORE_LOCK_WAIT_MS`。`> 0` 才等;`0` / 非数 ⇒ 7.93.0 的当场拒逐字同形。
|
|
20
|
+
* @param onWait 进入等待腿的一次性披露口。
|
|
21
|
+
*/
|
|
22
|
+
export declare function acquireLocalBackendWithWait<T>(construct: () => T, waitMs: number, onWait: StoreLockWaitSink, clock?: StoreLockWaitClock): T;
|
|
23
|
+
//# sourceMappingURL=store-lock-wait.d.ts.map
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
import { dirname } from "node:path";
|
|
2
|
+
import { FileStoreLockError } from "@sema-agent/core";
|
|
3
|
+
export const STORE_LOCK_POLL_MS = 250;
|
|
4
|
+
const REAL_CLOCK = {
|
|
5
|
+
now: () => performance.now(),
|
|
6
|
+
sleep: (ms) => void Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms),
|
|
7
|
+
};
|
|
8
|
+
function liveOwnerPidOf(err) {
|
|
9
|
+
if (!(err instanceof FileStoreLockError))
|
|
10
|
+
return undefined;
|
|
11
|
+
if (err.code !== "store.dir_in_use")
|
|
12
|
+
return undefined;
|
|
13
|
+
return err.ownerPid;
|
|
14
|
+
}
|
|
15
|
+
export function acquireLocalBackendWithWait(construct, waitMs, onWait, clock = REAL_CLOCK) {
|
|
16
|
+
let startedAt;
|
|
17
|
+
for (;;) {
|
|
18
|
+
try {
|
|
19
|
+
return construct();
|
|
20
|
+
}
|
|
21
|
+
catch (err) {
|
|
22
|
+
const ownerPid = liveOwnerPidOf(err);
|
|
23
|
+
if (ownerPid === undefined || !(waitMs > 0))
|
|
24
|
+
throw err;
|
|
25
|
+
const lockErr = err;
|
|
26
|
+
if (startedAt === undefined) {
|
|
27
|
+
startedAt = clock.now();
|
|
28
|
+
onWait({ ownerPid, dir: dirname(lockErr.lockPath), waitMs });
|
|
29
|
+
}
|
|
30
|
+
const waitedMs = clock.now() - startedAt;
|
|
31
|
+
if (waitedMs >= waitMs) {
|
|
32
|
+
throw new FileStoreLockError(lockErr.code, `${lockErr.message} — waited ${Math.round(waitedMs)}ms for it to release the directory ` +
|
|
33
|
+
`(STORE_LOCK_WAIT_MS=${waitMs}); it is still holding it`, lockErr.lockPath, lockErr.ownerPid);
|
|
34
|
+
}
|
|
35
|
+
clock.sleep(Math.min(STORE_LOCK_POLL_MS, waitMs - waitedMs));
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
//# sourceMappingURL=store-lock-wait.js.map
|
package/dist/run-local.js
CHANGED
|
@@ -17,6 +17,7 @@ import { taskWallClockSec } from "./task-workflow.js";
|
|
|
17
17
|
import { applyCatalogToSource } from "./capabilities/center-prompts.js";
|
|
18
18
|
import { validatePromptsDomain, CORE_ENGINE_VERSION } from "./prompts-domain-validate.js";
|
|
19
19
|
import { buildUnpricedCeilingNotice, loadConfig, logConfigDiagnostics, parseNumOrFailNonNegative } from "./config.js";
|
|
20
|
+
import { acquireLocalBackendWithWait } from "./plugins/store-lock-wait.js";
|
|
20
21
|
import { createConfigProvider } from "./config-provider.js";
|
|
21
22
|
import { applyEffective, applyCenterReadFace, resolveMcpServers, mcpForScenario, resolveA2aPeers, a2aForScenario } from "./config-center/facade.js";
|
|
22
23
|
import { hostExecutionEnvFactory } from "./plugins/remote-env-host.js";
|
|
@@ -41,6 +42,7 @@ import { createOrgMemoryAdmissionWiring } from "./boot/org-memory.js";
|
|
|
41
42
|
import { createPermissionDeniedMeter } from "./observability/tool-trace.js";
|
|
42
43
|
import { createKeyResolver } from "./key-resolver.js";
|
|
43
44
|
import { createLogger } from "./observability/logger.js";
|
|
45
|
+
import { fatalExit } from "./boot/fatal-exit.js";
|
|
44
46
|
import { createMetrics } from "./observability/metrics.js";
|
|
45
47
|
import { installSecretEnvScrubRecorder } from "./observability/secret-env-scrub.js";
|
|
46
48
|
import { buildDisplaySafeUrl } from "./trace/redact.js";
|
|
@@ -253,7 +255,7 @@ export async function runLocal(argv, deps = {}) {
|
|
|
253
255
|
const pricing = buildPricing(config.models);
|
|
254
256
|
let fileBackend;
|
|
255
257
|
try {
|
|
256
|
-
fileBackend = new FileStorageBackend({
|
|
258
|
+
fileBackend = acquireLocalBackendWithWait(() => new FileStorageBackend({
|
|
257
259
|
root,
|
|
258
260
|
...(config.fileHistory !== undefined ? { fileHistory: config.fileHistory } : {}),
|
|
259
261
|
onCorruptRead: (info) => logger.warn("file_store_corrupt_read", {
|
|
@@ -266,7 +268,7 @@ export async function runLocal(argv, deps = {}) {
|
|
|
266
268
|
"layer (deployment policy / hooks / shell gate still applied); session repo = a listing silently omitted " +
|
|
267
269
|
"rows; file-snapshot = a snapshot or scope read as missing. Inspect the named path before deciding.",
|
|
268
270
|
}),
|
|
269
|
-
});
|
|
271
|
+
}), config.storeLockWaitMs, (fields) => logger.warn("store_dir_lock_wait", { ...fields }));
|
|
270
272
|
}
|
|
271
273
|
catch (e) {
|
|
272
274
|
printErr(`cannot open local data dir ${root}: ${e instanceof Error ? e.message : String(e)}`);
|
|
@@ -574,9 +576,6 @@ function isMainModule() {
|
|
|
574
576
|
}
|
|
575
577
|
}
|
|
576
578
|
if (isMainModule()) {
|
|
577
|
-
void main().catch((err) =>
|
|
578
|
-
createLogger().error("fatal", { err: String(err) });
|
|
579
|
-
process.exit(1);
|
|
580
|
-
});
|
|
579
|
+
void main().catch((err) => fatalExit(err));
|
|
581
580
|
}
|
|
582
581
|
//# sourceMappingURL=run-local.js.map
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@sema-agent/server",
|
|
3
|
-
"version": "7.93.
|
|
3
|
+
"version": "7.93.2",
|
|
4
4
|
"description": "Sema Server — the server/API implementation layer for Sema, wiring core, registry, model providers, and cloud agent execution. Built on @sema-agent/core.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "BUSL-1.1",
|
|
@@ -74,7 +74,7 @@
|
|
|
74
74
|
"sharp": "^0.35.3"
|
|
75
75
|
},
|
|
76
76
|
"devDependencies": {
|
|
77
|
-
"@sema-agent/sdk": "11.2.
|
|
77
|
+
"@sema-agent/sdk": "11.2.1",
|
|
78
78
|
"@types/libsodium-wrappers": "^0.7.14",
|
|
79
79
|
"@types/node": "22.10.2",
|
|
80
80
|
"@types/pg": "^8.20.0",
|