@baize-ai/core 0.3.15 → 0.3.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.dockerignore +1 -0
- package/CHANGELOG.md +29 -0
- package/Dockerfile +10 -1
- package/cli/commands/doctor.js +60 -0
- package/cli/commands/init.js +104 -42
- package/cli/commands/runtime.js +9 -1
- package/cli/lib/__tests__/init-base-url.test.js +55 -13
- package/cli/lib/__tests__/runtime-base-url.test.js +3 -1
- package/cli/lib/__tests__/runtime-launch.test.js +4 -2
- package/cli/lib/__tests__/runtime-setup.test.js +53 -6
- package/cli/lib/__tests__/tmux-env.test.js +19 -5
- package/cli/lib/claude-eval.js +2 -1
- package/cli/lib/codex-hooks.js +12 -0
- package/cli/lib/path-bins.js +54 -0
- package/cli/lib/runtime/claude.js +4 -2
- package/cli/lib/runtime/codex.js +19 -0
- package/cli/lib/runtime/tmux-env.js +5 -4
- package/cli/lib/runtime-setup.js +157 -13
- package/docker/entrypoint.sh +4 -2
- package/docs/ops-runbook.md +161 -0
- package/package.json +2 -2
- package/scripts/docker-publish.sh +11 -5
- package/scripts/pack-release.sh +34 -26
- package/skills/activity-monitor/scripts/__tests__/guardian.test.js +51 -0
- package/skills/activity-monitor/scripts/adapters/runtime-components.js +27 -0
- package/skills/activity-monitor/scripts/guardian.js +45 -1
- package/skills/activity-monitor/scripts/monitor-orchestrator.js +8 -1
- package/skills/activity-monitor/scripts/upgrade-check.js +3 -1
- package/skills/web-console/public/app.js +20 -75
- package/skills/web-console/scripts/model-provider.js +97 -59
- package/skills/web-console/scripts/server.js +6 -24
- package/templates/pm2/ecosystem.config.cjs +5 -4
- package/test/model-provider.test.js +150 -40
- package/test/web-console-routes.test.js +11 -13
|
@@ -0,0 +1,161 @@
|
|
|
1
|
+
# Baize Agent 容器运维排查手册
|
|
2
|
+
|
|
3
|
+
> 适用:docker 部署的 baize agent(`baize-core` 镜像)。
|
|
4
|
+
> 原则:先只读诊断定位,再修改;生产环境每步执行前确认。
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## 1. 症状:AI main 会话起不来
|
|
9
|
+
|
|
10
|
+
**表现**:agent 不再响应任务;`web-console` 可能还活着,但调度/消息链路断了。
|
|
11
|
+
|
|
12
|
+
**"main 会话"是什么**:AI 主循环(Claude Code / Codex)跑在持久 **tmux 会话**里
|
|
13
|
+
(`claude-main` / `codex-main`),由 `activity-monitor`(Guardian 看门狗)负责拉起,
|
|
14
|
+
`scheduler` 依赖它收发消息。会话没了 = 看门狗没拉起或拉起失败。
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## 2. 快速排查流程(只读,从外到内)
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
# ① 容器状态(是否在跑 / 重启次数)
|
|
22
|
+
docker ps -a | grep -i baize
|
|
23
|
+
|
|
24
|
+
# ② pm2 服务(web-console / scheduler / c4-dispatcher / activity-monitor)
|
|
25
|
+
docker exec <容器名> pm2 list
|
|
26
|
+
|
|
27
|
+
# ③ tmux 会话(main 会话在不在)
|
|
28
|
+
docker exec <容器名> bash -lc "tmux ls"
|
|
29
|
+
|
|
30
|
+
# ④ 看门狗日志(核心——Guardian 为什么没拉起)
|
|
31
|
+
docker exec <容器名> bash -lc "pm2 logs activity-monitor --lines 30 --nostream"
|
|
32
|
+
|
|
33
|
+
# ⑤ 调度器日志
|
|
34
|
+
docker exec <容器名> bash -lc "pm2 logs scheduler --lines 20 --nostream"
|
|
35
|
+
|
|
36
|
+
# ⑥ 初始化状态(runtime 装没装完)
|
|
37
|
+
docker exec <容器名> bash -lc "cat ~/baize/init-state.json 2>/dev/null"
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
**日志判读**:
|
|
41
|
+
|
|
42
|
+
| 日志 | 含义 | 下一步 |
|
|
43
|
+
|---|---|---|
|
|
44
|
+
| `spawnSync codex ENOENT` / `spawnSync claude ENOENT` | **PATH 问题**(可执行文件不在 PATH)→ 见 §3 | §3 |
|
|
45
|
+
| `tmux ls` 无会话 + pm2 正常 | 看门狗没触发或拉起失败 | ④ 看 Guardian 日志 |
|
|
46
|
+
| `init-state.json` 失败 | 初始化未完成(runtime/模型未配置) | web 控制台模型设置重新安装 |
|
|
47
|
+
| tmux 会话在但卡住 | Claude/Codex 等输入 / 认证过期 / API 错误 | `tmux attach -t claude-main` 看输出 |
|
|
48
|
+
|
|
49
|
+
---
|
|
50
|
+
|
|
51
|
+
## 3. 本次实战:PATH 污染导致 `spawnSync codex ENOENT`
|
|
52
|
+
|
|
53
|
+
### 现象
|
|
54
|
+
|
|
55
|
+
```
|
|
56
|
+
activity-monitor: Guardian: Failed to start Codex: spawnSync codex ENOENT
|
|
57
|
+
scheduler: Waiting for agent runtime (offline or stopped)...
|
|
58
|
+
tmux ls: no server running on /tmp/tmux-1001/default
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### 诊断证据链
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
# ① codex symlink 在不在(npm 全局 bin 通常是 symlink)
|
|
65
|
+
docker exec baize-2 bash -lc "ls -la ~/.npm-global/bin/codex; readlink -f ~/.npm-global/bin/codex"
|
|
66
|
+
# → lrwxrwxrwx ... codex -> ../lib/node_modules/@openai/codex/bin/codex.js (symlink 完好)
|
|
67
|
+
|
|
68
|
+
# ② codex 二进制本身能不能跑(关键——区分"二进制坏了" vs "PATH 找不到")
|
|
69
|
+
docker exec baize-2 bash -lc "~/.npm-global/bin/codex --version; echo EXIT=\$?"
|
|
70
|
+
# → codex-cli 0.153.2 / EXIT=0 (二进制完好!)
|
|
71
|
+
|
|
72
|
+
# ③ pm2 进程实际 PATH(Guardian 用它 spawn codex)
|
|
73
|
+
docker exec baize-2 bash -lc "cat /proc/\$(pgrep -f 'activity-monitor' | head -1)/environ | tr '\0' '\n' | grep '^PATH='"
|
|
74
|
+
# → PATH=...codex-path:/home/baize/baize/bin:...(★ 没有 ~/.npm-global/bin)
|
|
75
|
+
|
|
76
|
+
# ④ 谁污染了 PATH(login shell 的 profile)
|
|
77
|
+
docker exec baize-2 bash -lc "grep -n 'PATH' ~/.bashrc ~/.profile ~/.bash_profile 2>/dev/null"
|
|
78
|
+
# → .bashrc/.profile 里有 export PATH="/home/baize/baize/bin:$PATH"
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
**判定**:`codex` 二进制完好但 **pm2 进程 PATH 缺 `~/.npm-global/bin`**(codex symlink
|
|
82
|
+
所在目录)→ `spawnSync codex` 找不到 → ENOENT。
|
|
83
|
+
|
|
84
|
+
### 根因
|
|
85
|
+
|
|
86
|
+
1. **PATH 被污染**:`~/.bashrc`/`~/.profile` 前置 `baize/bin` + codex npm 包
|
|
87
|
+
postinstall 前插 vendor 路径,把 `~/.npm-global/bin` 挤掉。
|
|
88
|
+
2. **坏环境被固化**:在污染 PATH 的 shell 里执行过 `pm2 start`/`pm2 save`,
|
|
89
|
+
pm2 dump 保存了坏环境,之后 pm2 进程一直继承(`docker restart` 若走 pm2
|
|
90
|
+
resurrect 也不会恢复)。
|
|
91
|
+
3. **谁干的**:使用人在容器内让 AI 执行环境类操作(改 profile / 重装 runtime /
|
|
92
|
+
重跑 pm2)时污染了环境并固化。
|
|
93
|
+
|
|
94
|
+
### 修复(干净 PATH + --update-env 覆盖 pm2 固化环境)
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
# 用干净 PATH 重启看门狗;--update-env 强制用当前 shell 环境覆盖 pm2 dump
|
|
98
|
+
docker exec -e PATH=/home/baize/.npm-global/bin:/home/baize/.local/bin:/usr/local/bin:/usr/bin:/bin \
|
|
99
|
+
<容器名> pm2 restart activity-monitor --update-env
|
|
100
|
+
|
|
101
|
+
# 验证:15s 后会话出现、日志不再 ENOENT
|
|
102
|
+
sleep 15
|
|
103
|
+
docker exec <容器名> bash -lc "tmux ls"
|
|
104
|
+
docker exec <容器名> bash -lc "pm2 logs activity-monitor --lines 6 --nostream"
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
> `docker exec -e PATH=...` 用 `-e` 注入,避免嵌套引号。
|
|
108
|
+
> 其他 pm2 服务(scheduler 等)若同样 PATH 坏,按同法 `--update-env` 重启。
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
## 4. 预防(如何避免再发生)
|
|
113
|
+
|
|
114
|
+
### 操作纪律(人和 AI 都要遵守)
|
|
115
|
+
|
|
116
|
+
1. **容器内操作统一走封装入口**:优先用 web 控制台 / `baize` CLI,避免裸
|
|
117
|
+
`docker exec ... bash -lc`(login shell 会加载 profile,触发 PATH 污染)。
|
|
118
|
+
2. **禁止手工改 `~/.bashrc` / `~/.profile` 的 PATH**:本次污染源头。若必须加
|
|
119
|
+
路径,用 `export PATH="$PATH:<新路径>"` **追加**,且绝不动
|
|
120
|
+
`~/.npm-global/bin` / `~/.local/bin` 两项。
|
|
121
|
+
3. **不在容器内手工 `pm2 save`**:pm2 dump 会固化当时的 PATH 环境;容器重启
|
|
122
|
+
由 entrypoint 统一注入环境(它会重存 `SYSTEM_PATH`)。
|
|
123
|
+
4. **AI 执行环境修改类操作前先备份**:`cp ~/.bashrc ~/.bashrc.bak`,
|
|
124
|
+
记录改动 diff,改完验证 `which codex claude baize node` 全部可解析。
|
|
125
|
+
|
|
126
|
+
### 工程加固(D54 已落地,0.3.17 起)
|
|
127
|
+
|
|
128
|
+
- **PATH 完整性收敛到 init.js**:`baize init` 每次运行修复 shell profile 完整 bin PATH
|
|
129
|
+
(`~/.npm-global/bin` + `.local/bin` + `.claude/bin` + `baize/bin` + `.local/node/bin`),
|
|
130
|
+
幂等 + 存量缺失自动补。裸机/docker/存量升级全部覆盖(docker entrypoint 每次启动跑 init)。
|
|
131
|
+
- **共享 `path-bins.js`**:所有 runtime 二进制 spawn(codex/claude/baize)统一走
|
|
132
|
+
`completeBinPath()`(子进程 env 补完整 PATH)——登录 shell PATH 污染不再导致 ENOENT。
|
|
133
|
+
- **ecosystem `ENHANCED_PATH` 显式含 `~/.npm-global/bin`**:PM2 服务即使继承残缺 PATH 也能
|
|
134
|
+
找到 runtime。
|
|
135
|
+
- **Guardian 告警**:连续 3 次启动失败 → 写 `~/baize/activity-monitor/guardian-alert.json`
|
|
136
|
+
+ 日志;恢复后自动清除。静默宕机不再发生。
|
|
137
|
+
- **doctor 检查**:`baize doctor` 新增 PATH 完整性组(`path_incomplete` issue)。
|
|
138
|
+
- install.sh / Dockerfile **不改**(收敛到 init.js);`--update-env` 从根源安全化。
|
|
139
|
+
|
|
140
|
+
### 巡检(可选定时)
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
# 检查 main 会话与 PATH 健康度
|
|
144
|
+
docker exec <容器名> bash -lc "tmux ls"
|
|
145
|
+
docker exec <容器名> bash -lc "which codex claude baize node"
|
|
146
|
+
# 告警文件存在 = 有未恢复的连续失败(无文件=健康)
|
|
147
|
+
docker exec <容器名> bash -lc "cat ~/baize/activity-monitor/guardian-alert.json 2>/dev/null"
|
|
148
|
+
# doctor PATH 组
|
|
149
|
+
docker exec <容器名> bash -lc "baize doctor --json" | python3 -c "import json,sys;print(json.load(sys.stdin)['groups']['path'])"
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## 5. 其他常见问题速查
|
|
155
|
+
|
|
156
|
+
| 问题 | 快速处理 |
|
|
157
|
+
|---|---|
|
|
158
|
+
| web 控制台打不开 | `pm2 logs web-console`;D51 起检查 `~/.claude/cli/lib` 是否存在(init 部署缺陷已修复,见 0.3.16) |
|
|
159
|
+
| 会话起了但任务不响应 | `tmux attach -t <claude-main\|codex-main>` 看 AI 输出;`pm2 logs scheduler` |
|
|
160
|
+
| 容器一直 restart | `docker logs <容器名>`(entrypoint 崩溃);检查 `init-state.json` |
|
|
161
|
+
| A2A 端口不通 | `docker exec <容器名> pm2 list` 看 baize-a2a;`ss -tlnp | grep 8443` |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@baize-ai/core",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.17",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Baize (白泽) — autonomous AI agent infrastructure",
|
|
6
6
|
"main": "cli/baize.js",
|
|
@@ -46,4 +46,4 @@
|
|
|
46
46
|
"publishConfig": {
|
|
47
47
|
"access": "public"
|
|
48
48
|
}
|
|
49
|
-
}
|
|
49
|
+
}
|
|
@@ -30,15 +30,21 @@ ACR_NAMESPACE="${ACR_NAMESPACE:-baize01}"
|
|
|
30
30
|
|
|
31
31
|
# ── Build ────────────────────────────────────────────────────────────────────
|
|
32
32
|
echo "==> Building baize-core:${VERSION} (from npm @baize-ai/core@${VERSION})"
|
|
33
|
+
# Temp build context: .dockerignore is a whitelist (source stays out of the
|
|
34
|
+
# image), and the optional COPY core.tgz needs a placeholder file present —
|
|
35
|
+
# a temp dir keeps the repo worktree clean.
|
|
36
|
+
BUILD_CTX="$(mktemp -d)"
|
|
37
|
+
trap 'rm -rf "$BUILD_CTX"' EXIT
|
|
38
|
+
cp Dockerfile .dockerignore "$BUILD_CTX"/
|
|
39
|
+
mkdir -p "$BUILD_CTX/templates/pm2" "$BUILD_CTX/docker"
|
|
40
|
+
cp templates/pm2/ecosystem.config.cjs "$BUILD_CTX/templates/pm2/"
|
|
41
|
+
cp docker/entrypoint.sh "$BUILD_CTX/docker/"
|
|
42
|
+
touch "$BUILD_CTX/core.tgz" # placeholder — registry install ignores it (BAZE_CORE_TARBALL empty)
|
|
33
43
|
docker build \
|
|
34
44
|
--build-arg BAZE_CORE_VERSION="${VERSION}" \
|
|
35
45
|
-t "${GHCR_IMAGE}:${VERSION}" \
|
|
36
46
|
-t "${GHCR_IMAGE}:latest" \
|
|
37
|
-
|
|
38
|
-
"${BUILD_ARGS[@]}" \
|
|
39
|
-
-t "${GHCR_IMAGE}:${VERSION}" \
|
|
40
|
-
-t "${GHCR_IMAGE}:latest" \
|
|
41
|
-
.
|
|
47
|
+
"$BUILD_CTX"
|
|
42
48
|
|
|
43
49
|
if [ -n "${ACR_REGISTRY}" ] && [ -n "${ACR_NAMESPACE}" ]; then
|
|
44
50
|
ACR_IMAGE="${ACR_REGISTRY}/${ACR_NAMESPACE}/baize-core"
|
package/scripts/pack-release.sh
CHANGED
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
# 兼容 web 控制台上传与 `baize add --file` 两种安装)
|
|
12
12
|
# baize-admin-workspace-<v>.tgz admin 套件源码:server 源码 + admin-ui 构建产物 +
|
|
13
13
|
# Dockerfile + 部署说明(自建镜像 / 自托管用)
|
|
14
|
-
# baize-admin-workspace-<v>.tar docker save 的 admin
|
|
14
|
+
# baize-admin-workspace-<v>.tar (可选)docker save 的 admin 镜像——仅 BUILD_ADMIN_DOCKER=1 打包时产出
|
|
15
15
|
# INSTALL.md 客户侧安装说明(上传 / --file 安装 + 切集群 + admin 部署)
|
|
16
16
|
#
|
|
17
17
|
# 用法:
|
|
@@ -155,8 +155,14 @@ build_admin_tgz() {
|
|
|
155
155
|
log " → ${out} ($(size "${out}"))"
|
|
156
156
|
}
|
|
157
157
|
|
|
158
|
-
# ── 3. Admin Docker 镜像(docker save
|
|
158
|
+
# ── 3. Admin Docker 镜像(docker save)——可选 ─────────────────────────────
|
|
159
|
+
# 商业版 admin 默认源码交付(tgz 含 server + 已构建 admin-ui/dist)。
|
|
160
|
+
# 仅当 BUILD_ADMIN_DOCKER=1 时才导出 docker save 镜像(需先构建 baize-admin-workspace:test)。
|
|
159
161
|
save_admin_image() {
|
|
162
|
+
if [ "${BUILD_ADMIN_DOCKER:-0}" != "1" ]; then
|
|
163
|
+
step "跳过 admin Docker 镜像导出(源码交付模式;需要时 BUILD_ADMIN_DOCKER=1 重跑)"
|
|
164
|
+
return 0
|
|
165
|
+
fi
|
|
160
166
|
step "导出 admin Docker 镜像"
|
|
161
167
|
local image="baize-admin-workspace:test"
|
|
162
168
|
if ! docker image inspect "${image}" >/dev/null 2>&1; then
|
|
@@ -231,36 +237,35 @@ baize a2a status # 期望:集群·在线
|
|
|
231
237
|
```bash
|
|
232
238
|
baize a2a disable # 停 daemon,保留配置与注册数据,模式回 standalone
|
|
233
239
|
```
|
|
240
|
+
### 方式 A:源码部署(推荐——商业版默认交付形态)
|
|
234
241
|
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
### 方式 A:Docker 镜像(推荐)
|
|
242
|
+
套件已含 server 源码与**构建好的** `admin-ui/dist`,无需前端构建:
|
|
238
243
|
|
|
239
244
|
```bash
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
baize-admin
|
|
245
|
+
tar xzf baize-admin-workspace-__VERSION__.tgz
|
|
246
|
+
cd baize-admin-workspace-__VERSION__
|
|
247
|
+
npm ci --omit=dev # Node ≥ 20.11
|
|
248
|
+
BAIZE_ADMIN_TOKEN='<初始口令>' BAIZE_ADMIN_DIR=/srv/baize-admin \
|
|
249
|
+
pm2 start server/src/index.js --name baize-admin # 或 nohup node server/src/index.js
|
|
245
250
|
```
|
|
246
251
|
|
|
247
|
-
-
|
|
248
|
-
-
|
|
249
|
-
-
|
|
252
|
+
- API 监听 8080;`BAIZE_ADMIN_DIR` 是数据目录(SQLite/CA/签名钥),必须持久化并提前建目录;
|
|
253
|
+
- 前端 `admin-ui/dist` 是静态产物:用 nginx/Caddy 反代到与 API 同源的根路径;
|
|
254
|
+
- 公网部署必须 `TRUST_PROXY=1`(信任反代 X-Forwarded-For,否则限流按代理 IP 计数);
|
|
255
|
+
- 首次登录 `admin` + 初始口令(日志打印),登录后到「设置」改口令。
|
|
250
256
|
|
|
251
|
-
### 方式 B
|
|
257
|
+
### 方式 B:Docker(可选——需自建镜像)
|
|
252
258
|
|
|
253
259
|
```bash
|
|
254
|
-
tar xzf baize-admin-workspace-__VERSION__.tgz
|
|
255
|
-
cd baize-admin-workspace-__VERSION__
|
|
256
260
|
docker build -t baize-admin-workspace . # 多阶段:node:22 构建 admin-ui → node:22-slim 运行
|
|
257
|
-
docker run -d --name baize-admin -p 8080:8080
|
|
261
|
+
docker run -d --name baize-admin -p 8080:8080 \
|
|
262
|
+
-v baize-admin-data:/data -e BAIZE_ADMIN_TOKEN='<初始口令>' baize-admin-workspace
|
|
258
263
|
```
|
|
259
264
|
|
|
260
|
-
|
|
261
|
-
- 前端构建产物已含在 `admin-ui/dist`(容器内 `/app/dist`),可与 API 同源静态提供。
|
|
265
|
+
> 厂商如需分发 Docker 镜像 tar:打包时 `BUILD_ADMIN_DOCKER=1 ./scripts/pack-release.sh`(本默认交付不包含 tar)。
|
|
262
266
|
|
|
263
|
-
|
|
267
|
+
- 前端构建产物已含在 `admin-ui/dist`,与 API 同源静态提供(nginx/Caddy 指到该目录即可)。
|
|
268
|
+
- API 默认 8080(`PORT` 可改);本地直跑:`BAIZE_ADMIN_DIR=/srv/baize-admin npm start`。
|
|
264
269
|
|
|
265
270
|
```bash
|
|
266
271
|
baize a2a disable # 先切回单机 / 停服务
|
|
@@ -332,13 +337,16 @@ verify() {
|
|
|
332
337
|
echo " !! ${admin_tgz} 校验失败(解压或 manifest 不正确)" >&2
|
|
333
338
|
rc=1
|
|
334
339
|
fi
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
340
|
+
# docker save 产物:可选(BUILD_ADMIN_DOCKER=1 打包时才存在)
|
|
341
|
+
if [ -e "${admin_tar}" ] || [ "${BUILD_ADMIN_DOCKER:-0}" = "1" ]; then
|
|
342
|
+
if [ -s "${admin_tar}" ]; then
|
|
343
|
+
log " OK ${admin_tar} ($(size "${admin_tar}")) 存在且非空"
|
|
344
|
+
else
|
|
345
|
+
echo " !! ${admin_tar} 不存在或为空" >&2
|
|
346
|
+
rc=1
|
|
347
|
+
fi
|
|
339
348
|
else
|
|
340
|
-
|
|
341
|
-
rc=1
|
|
349
|
+
log " -- ${admin_tar##*/} 未打包(源码交付模式,BUILD_ADMIN_DOCKER=1 可启用)"
|
|
342
350
|
fi
|
|
343
351
|
|
|
344
352
|
[ -f "${DIST_DIR}/INSTALL.md" ] && log " OK ${DIST_DIR}/INSTALL.md ($(size "${DIST_DIR}/INSTALL.md"))"
|
|
@@ -48,6 +48,7 @@ function createDeps(overrides = {}) {
|
|
|
48
48
|
nowMs: overrides.nowMs ?? (() => 100_000),
|
|
49
49
|
initialRuntimeLaunchAtMs: overrides.initialRuntimeLaunchAtMs ?? 0,
|
|
50
50
|
};
|
|
51
|
+
if (overrides.alert) deps.alert = overrides.alert;
|
|
51
52
|
|
|
52
53
|
return { deps, calls };
|
|
53
54
|
}
|
|
@@ -174,4 +175,54 @@ describe('Guardian', () => {
|
|
|
174
175
|
await Promise.resolve();
|
|
175
176
|
assert.deepEqual(order, ['prepare', 'launch', 'prompt']);
|
|
176
177
|
});
|
|
178
|
+
|
|
179
|
+
it('raises alert after ALERT_AFTER_RESTARTS consecutive ENOENT launch failures (D54 P2)', async () => {
|
|
180
|
+
const alerts = [];
|
|
181
|
+
const adapter = {
|
|
182
|
+
sessionName: 'test-main',
|
|
183
|
+
displayName: 'TestRuntime',
|
|
184
|
+
runtimeId: 'codex',
|
|
185
|
+
isRunning: async () => false,
|
|
186
|
+
launch: async () => { throw Object.assign(new Error('spawnSync codex ENOENT'), { code: 'ENOENT' }); },
|
|
187
|
+
clearStaleState: () => {},
|
|
188
|
+
enqueueStartupPrompt: () => {},
|
|
189
|
+
};
|
|
190
|
+
const { deps } = createDeps({ alert: (info) => alerts.push(info) });
|
|
191
|
+
const guardian = new Guardian(adapter, deps);
|
|
192
|
+
guardian._maybeAlert('spawnSync codex ENOENT'); // below threshold → no alert
|
|
193
|
+
assert.equal(alerts.length, 0);
|
|
194
|
+
|
|
195
|
+
// simulate 3 consecutive failed launches crossing the threshold
|
|
196
|
+
guardian.consecutiveRestarts = 3;
|
|
197
|
+
guardian._maybeAlert('spawnSync codex ENOENT');
|
|
198
|
+
guardian._maybeAlert('spawnSync codex ENOENT'); // no duplicate
|
|
199
|
+
assert.equal(alerts.length, 1, 'alert must fire exactly once per episode');
|
|
200
|
+
assert.equal(alerts[0].runtime, 'codex');
|
|
201
|
+
assert.equal(alerts[0].consecutive_restarts, 3);
|
|
202
|
+
assert.ok(/ENOENT/.test(alerts[0].reason));
|
|
203
|
+
|
|
204
|
+
// recovery resets alerted → next episode may alert again
|
|
205
|
+
guardian.alerted = false;
|
|
206
|
+
guardian._maybeAlert('again');
|
|
207
|
+
assert.equal(alerts.length, 2, 'alert must fire again after recovery');
|
|
208
|
+
});
|
|
209
|
+
|
|
210
|
+
it('logs a repair hint on ENOENT launch failure (D54 P2)', async () => {
|
|
211
|
+
const adapter = {
|
|
212
|
+
sessionName: 'test-main',
|
|
213
|
+
displayName: 'TestRuntime',
|
|
214
|
+
runtimeId: 'codex',
|
|
215
|
+
isRunning: async () => false,
|
|
216
|
+
launch: async () => { throw Object.assign(new Error('spawnSync codex ENOENT'), { code: 'ENOENT' }); },
|
|
217
|
+
clearStaleState: () => {},
|
|
218
|
+
enqueueStartupPrompt: () => {},
|
|
219
|
+
};
|
|
220
|
+
const { deps, calls } = createDeps({ alert: () => {} });
|
|
221
|
+
const guardian = new Guardian(adapter, deps);
|
|
222
|
+
guardian.consecutiveRestarts = 0;
|
|
223
|
+
await guardian.startAgent();
|
|
224
|
+
await Promise.resolve();
|
|
225
|
+
await Promise.resolve();
|
|
226
|
+
assert.ok(calls.log.some(m => m.includes('ENOENT') && m.includes('baize init')), 'must log the repair hint');
|
|
227
|
+
});
|
|
177
228
|
});
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { execFileSync, execSync } from 'child_process';
|
|
2
2
|
import fs from 'fs';
|
|
3
|
+
import path from 'path';
|
|
3
4
|
import { Guardian } from '../guardian.js';
|
|
4
5
|
import { HealthEngine } from '../health-engine.js';
|
|
5
6
|
import { ProcSampler } from '../proc-sampler.js';
|
|
@@ -62,6 +63,7 @@ export function createGuardian(activeAdapter, activeToolPipeline, initialRuntime
|
|
|
62
63
|
hookStateFile,
|
|
63
64
|
log,
|
|
64
65
|
}) {
|
|
66
|
+
const monitorDir = apiActivityFile ? path.dirname(apiActivityFile) : undefined;
|
|
65
67
|
return new Guardian(activeAdapter, {
|
|
66
68
|
log,
|
|
67
69
|
initialRuntimeLaunchAtMs,
|
|
@@ -76,6 +78,31 @@ export function createGuardian(activeAdapter, activeToolPipeline, initialRuntime
|
|
|
76
78
|
}));
|
|
77
79
|
fs.writeFileSync(hookStateFile, JSON.stringify({ active_tools: 0 }));
|
|
78
80
|
},
|
|
81
|
+
// D54 P2: consecutive-failure alert — write a visible file in the monitor
|
|
82
|
+
// dir (alongside agent-status.json) so ops/AI can discover an outage that
|
|
83
|
+
// Guardian keeps retrying. Cleared on recovery via onRecovered.
|
|
84
|
+
alert: (info) => {
|
|
85
|
+
if (!monitorDir) return;
|
|
86
|
+
const alertFile = path.join(monitorDir, 'guardian-alert.json');
|
|
87
|
+
try {
|
|
88
|
+
const tmp = `${alertFile}.tmp.${process.pid}`;
|
|
89
|
+
fs.writeFileSync(tmp, `${JSON.stringify(info, null, 2)}\n`);
|
|
90
|
+
fs.renameSync(tmp, alertFile);
|
|
91
|
+
log(`Guardian: ⚠ ALERT after ${info.consecutive_restarts} consecutive failures — ${info.reason}`);
|
|
92
|
+
} catch (e) {
|
|
93
|
+
log(`Guardian: alert write failed: ${e.message}`);
|
|
94
|
+
}
|
|
95
|
+
},
|
|
96
|
+
onRecovered: () => {
|
|
97
|
+
if (!monitorDir) return;
|
|
98
|
+
const alertFile = path.join(monitorDir, 'guardian-alert.json');
|
|
99
|
+
try {
|
|
100
|
+
if (fs.existsSync(alertFile)) fs.unlinkSync(alertFile);
|
|
101
|
+
log('Guardian: runtime recovered — alert cleared');
|
|
102
|
+
} catch (e) {
|
|
103
|
+
log(`Guardian: alert clear failed: ${e.message}`);
|
|
104
|
+
}
|
|
105
|
+
},
|
|
79
106
|
});
|
|
80
107
|
}
|
|
81
108
|
|
|
@@ -5,6 +5,9 @@ export const MAX_RESTART_DELAY = 60;
|
|
|
5
5
|
export const BACKOFF_RESET_THRESHOLD = 60;
|
|
6
6
|
export const STARTUP_GRACE_TICKS = 30;
|
|
7
7
|
export const MAINTENANCE_WAIT_TIMEOUT = 300;
|
|
8
|
+
// D54 P2: after this many consecutive launch failures, raise an alert —
|
|
9
|
+
// the 09-13 incident stayed silent for 44 attempts / 48 minutes.
|
|
10
|
+
export const ALERT_AFTER_RESTARTS = 3;
|
|
8
11
|
|
|
9
12
|
export function getRunningMaintenance({ execSyncImpl = defaultExecSync } = {}) {
|
|
10
13
|
try {
|
|
@@ -76,6 +79,8 @@ export class Guardian {
|
|
|
76
79
|
execSyncImpl: defaultExecSync,
|
|
77
80
|
nowMs: () => Date.now(),
|
|
78
81
|
initialRuntimeLaunchAtMs: 0,
|
|
82
|
+
alert: () => {}, // D54 P2: consecutive-failure alert sink (default no-op)
|
|
83
|
+
onRecovered: () => {}, // D54 P2: clears the alert once the episode ends
|
|
79
84
|
...deps,
|
|
80
85
|
};
|
|
81
86
|
this.notRunningCount = 0;
|
|
@@ -84,6 +89,7 @@ export class Guardian {
|
|
|
84
89
|
this.startupGrace = 0;
|
|
85
90
|
this.startAgentInProgress = false;
|
|
86
91
|
this.runtimeLaunchAtMs = this.deps.initialRuntimeLaunchAtMs;
|
|
92
|
+
this.alerted = false; // D54 P2: raise alert at most once per outage episode
|
|
87
93
|
}
|
|
88
94
|
|
|
89
95
|
getState() {
|
|
@@ -135,6 +141,8 @@ export class Guardian {
|
|
|
135
141
|
} else if (currentTime - this.stableRunningSince >= BACKOFF_RESET_THRESHOLD) {
|
|
136
142
|
this.consecutiveRestarts = 0;
|
|
137
143
|
this.stableRunningSince = 0;
|
|
144
|
+
this.alerted = false; // D54 P2: outage episode over — next one may alert again
|
|
145
|
+
try { this.deps.onRecovered(); } catch (e) { this.deps.log(`Guardian: recovery callback failed: ${e.message}`); }
|
|
138
146
|
}
|
|
139
147
|
}
|
|
140
148
|
|
|
@@ -148,6 +156,29 @@ export class Guardian {
|
|
|
148
156
|
};
|
|
149
157
|
}
|
|
150
158
|
|
|
159
|
+
/**
|
|
160
|
+
* D54 P2: raise the consecutive-failure alert exactly once per outage
|
|
161
|
+
* episode (threshold ALERT_AFTER_RESTARTS). Sink is injectable — the
|
|
162
|
+
* activity-monitor wires it to a visible status/alert file.
|
|
163
|
+
*/
|
|
164
|
+
_maybeAlert(reason) {
|
|
165
|
+
if (this.alerted) return;
|
|
166
|
+
if (this.consecutiveRestarts < ALERT_AFTER_RESTARTS) return;
|
|
167
|
+
this.alerted = true;
|
|
168
|
+
try {
|
|
169
|
+
this.deps.alert({
|
|
170
|
+
agent: this.adapter.displayName,
|
|
171
|
+
runtime: this.adapter.runtimeId,
|
|
172
|
+
consecutive_restarts: this.consecutiveRestarts,
|
|
173
|
+
reason: String(reason || '').slice(0, 500),
|
|
174
|
+
at: this.deps.nowMs(),
|
|
175
|
+
atIso: new Date(this.deps.nowMs()).toISOString(),
|
|
176
|
+
});
|
|
177
|
+
} catch (e) {
|
|
178
|
+
this.deps.log(`Guardian: alert sink failed: ${e.message}`);
|
|
179
|
+
}
|
|
180
|
+
}
|
|
181
|
+
|
|
151
182
|
_handleNotRunning({ state, message, restartLog }) {
|
|
152
183
|
if (this.startupGrace > 0) {
|
|
153
184
|
this.startupGrace -= 1;
|
|
@@ -204,6 +235,11 @@ export class Guardian {
|
|
|
204
235
|
|
|
205
236
|
this.deps.log(`Guardian: Starting ${this.adapter.displayName}...`);
|
|
206
237
|
|
|
238
|
+
// D54 P2: alert on consecutive startAgent attempts regardless of WHERE
|
|
239
|
+
// the failure lands (launch throw, or silent tmux-session-with-dead-runtime).
|
|
240
|
+
// Raised here (threshold checked inside) so every retry episode counts.
|
|
241
|
+
this._maybeAlert(`startAgent attempt #${this.consecutiveRestarts}`);
|
|
242
|
+
|
|
207
243
|
try {
|
|
208
244
|
this.adapter.clearStaleState?.();
|
|
209
245
|
} catch { }
|
|
@@ -213,7 +249,15 @@ export class Guardian {
|
|
|
213
249
|
} catch { }
|
|
214
250
|
|
|
215
251
|
const reportFailure = (err) => {
|
|
216
|
-
|
|
252
|
+
const msg = err?.message || String(err);
|
|
253
|
+
this.deps.log(`Guardian: Failed to start ${this.adapter.displayName}: ${msg}`);
|
|
254
|
+
// D54 P2: an ENOENT on launch means the runtime binary is missing from
|
|
255
|
+
// PATH (or uninstalled) — surface the repair hint instead of silently
|
|
256
|
+
// retrying; then raise an alert once the failure count crosses the bar.
|
|
257
|
+
if (err?.code === 'ENOENT' || /ENOENT/.test(msg)) {
|
|
258
|
+
this.deps.log(`Guardian: runtime binary not found (ENOENT) — run "baize init" to repair PATH/binaries, or reinstall the ${this.adapter.displayName} runtime`);
|
|
259
|
+
}
|
|
260
|
+
this._maybeAlert(msg);
|
|
217
261
|
};
|
|
218
262
|
const launchPrepared = () => {
|
|
219
263
|
try {
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import fs from 'fs';
|
|
2
|
+
import path from 'path';
|
|
2
3
|
import {
|
|
3
4
|
WATCHDOG_INTERRUPT_AVAILABLE_IN_SEC,
|
|
4
5
|
evaluateToolWatchdogTransition,
|
|
@@ -51,7 +52,13 @@ export class MonitorOrchestrator {
|
|
|
51
52
|
const runtimeLaunchAtMs = Number(initialStatus.runtime_launch_at) || nowMs();
|
|
52
53
|
|
|
53
54
|
const engine = createHealthEngine(adapter, initialStatus);
|
|
54
|
-
|
|
55
|
+
// D54 P2: pass the alert sink wiring (apiActivityFile/hookStateFile/log)
|
|
56
|
+
// so Guardian's consecutive-failure alert writes guardian-alert.json.
|
|
57
|
+
const guardian = createGuardian(adapter, toolPipeline, runtimeLaunchAtMs, {
|
|
58
|
+
apiActivityFile: path.join(monitorDir, 'api-activity.json'),
|
|
59
|
+
hookStateFile: path.join(monitorDir, 'hook-state.json'),
|
|
60
|
+
log,
|
|
61
|
+
});
|
|
55
62
|
|
|
56
63
|
if (initialHealth === 'rate_limited' && initialStatus.cooldown_until) {
|
|
57
64
|
engine.enterRateLimited(initialStatus.cooldown_until, initialStatus.rate_limit_reset || '');
|
|
@@ -10,6 +10,7 @@ import { execFileSync } from 'child_process';
|
|
|
10
10
|
import fs from 'fs';
|
|
11
11
|
import path from 'path';
|
|
12
12
|
import os from 'os';
|
|
13
|
+
import { completeBinPath } from '../../../cli/lib/path-bins.js';
|
|
13
14
|
|
|
14
15
|
const BAIZE_DIR = process.env.BAIZE_DIR || path.join(os.homedir(), 'baize');
|
|
15
16
|
const MONITOR_DIR = path.join(BAIZE_DIR, 'activity-monitor');
|
|
@@ -93,7 +94,8 @@ function main() {
|
|
|
93
94
|
// Check baize-core
|
|
94
95
|
try {
|
|
95
96
|
const coreVersion = execFileSync('baize', ['--version'], {
|
|
96
|
-
encoding: 'utf8', stdio: 'pipe', timeout: 5000
|
|
97
|
+
encoding: 'utf8', stdio: 'pipe', timeout: 5000,
|
|
98
|
+
env: { ...process.env, PATH: completeBinPath() },
|
|
97
99
|
}).trim();
|
|
98
100
|
const result = getLatestTag('baize-ai/baize-core');
|
|
99
101
|
if (result.error) {
|