uni-harness 0.2.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/hooks/guard-pre-bash.sh +39 -4
- package/.claude/hooks/guard-pre-edit.sh +122 -0
- package/.claude/hooks/observe-log.sh +21 -10
- package/.claude/hooks/pre-compact.sh +53 -0
- package/.claude/hooks/session-end.sh +103 -0
- package/.claude/hooks/session-start.sh +43 -2
- package/.claude/hooks/stop-gate.sh +84 -0
- package/.claude/settings.json +33 -0
- package/.claude/skills/distill/SKILL.md +49 -0
- package/.claude/skills/guide-audit/SKILL.md +14 -0
- package/.claude/skills/harness-init/SKILL.md +5 -1
- package/.claude/skills/ratchet/SKILL.md +31 -0
- package/AGENTS.md +26 -0
- package/README.ja.md +221 -0
- package/README.ko.md +215 -0
- package/README.md +16 -5
- package/README.zh-CN.md +209 -0
- package/bin/cli.js +5 -2
- package/harness/harness_report.py +16 -1
- package/harness/tests/test_hooks.sh +118 -12
- package/harness/tests/test_installer.sh +17 -0
- package/package.json +2 -1
package/README.zh-CN.md
ADDED
|
@@ -0,0 +1,209 @@
|
|
|
1
|
+
# uni-harness
|
|
2
|
+
|
|
3
|
+
🇺🇸 [English](README.md) | 🇰🇷 [한국어](README.ko.md) | 🇨🇳 **简体中文** | 🇯🇵 [日本語](README.ja.md)
|
|
4
|
+
|
|
5
|
+

|
|
6
|
+

|
|
7
|
+

|
|
8
|
+

|
|
9
|
+
|
|
10
|
+
一个面向 Claude Code 的智能体挽具(harness)套件。它为编码智能体包上
|
|
11
|
+
自动验证(传感器)、破坏性命令拦截(守卫)、基于检查点的会话恢复、
|
|
12
|
+
带绊线告警的全量工具调用日志,以及把每次失败都转化为永久结构的
|
|
13
|
+
棘轮(ratchet)工作流。
|
|
14
|
+
|
|
15
|
+
> 公式:**Agent = Model + Harness。** 模型带来推理能力;
|
|
16
|
+
> 其余的一切 — 规则、传感器、循环上限、记忆、可观测性 —
|
|
17
|
+
> 由这个套件提供。
|
|
18
|
+
|
|
19
|
+
## 安装
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
npx uni-harness init # 在项目根目录执行(或:init <路径>)
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
然后在项目中打开 Claude Code — Claude 会发现挽具尚未配置,并
|
|
26
|
+
**主动提议运行 `/harness-init`**(也可以自己运行)。它会扫描仓库、
|
|
27
|
+
检测构建/测试/lint 命令、**通过实际运行来验证它们**,并在你批准后
|
|
28
|
+
填入 `CLAUDE.md` 和 `.harness/commands.env`。在此步骤完成前,
|
|
29
|
+
验证传感器保持待机。
|
|
30
|
+
|
|
31
|
+
其他安装器命令:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
npx uni-harness update # 刷新套件机件(绝不触碰你的文件)
|
|
35
|
+
npx uni-harness doctor # 诊断安装状态
|
|
36
|
+
npx uni-harness uninstall --yes # 移除套件机件,保留你的文件
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
依赖要求:bash、python3(仅标准库 — 无需任何包)、安装器本身需要
|
|
40
|
+
node ≥16。
|
|
41
|
+
|
|
42
|
+
**对进行中的项目同样安全。** `init` 绝不覆盖你拥有的任何东西:
|
|
43
|
+
已有的 `CLAUDE.md` 会被保留(运行 `/harness-init` 后它会以追加方式
|
|
44
|
+
提议补充挽具章节),已有 `settings.json` 中的钩子和权限会被保留
|
|
45
|
+
(套件钩子以合并方式加入),你的 `.gitignore` 只会被追加而不是替换。
|
|
46
|
+
`update` 只刷新未被修改的套件文件 — 你自定义过的一律跳过
|
|
47
|
+
(会列出,可用 `--force` 覆盖)。
|
|
48
|
+
|
|
49
|
+
## 内含组件
|
|
50
|
+
|
|
51
|
+
| 文件 | 作用 |
|
|
52
|
+
|---|---|
|
|
53
|
+
| `CLAUDE.md` | 项目命令、规则、反模式、工作循环与检查点协议 |
|
|
54
|
+
| `AGENTS.md` | 规则的跨工具镜像 — Codex、Cursor、Aider、Gemini CLI 等 20+ 工具都会读取 |
|
|
55
|
+
| `.claude/settings.json` | 钩子注册 |
|
|
56
|
+
| `.claude/hooks/sensor-post-edit.sh` | 每次代码编辑立即运行 lint,将失败反馈回去 |
|
|
57
|
+
| `.claude/hooks/stop-gate.sh` | 回合结束时批量运行测试;失败状态下阻止收工(每会话 3 次上限,超过则强制升级上报);执行可选的轨迹规则(`.harness/trace.rules`:"回合结束前必须发生过 X") |
|
|
58
|
+
| `.claude/hooks/guard-pre-bash.sh` | 在执行前拦截破坏性命令与验证绕过(`--no-verify`);当要提交从未通过测试的编辑时先向用户确认 |
|
|
59
|
+
| `.claude/hooks/guard-pre-edit.sh` | 反作弊关卡:削弱测试的编辑(skip/only 标记、删除测试、掏空断言)以及配置/挽具文件的更改都需要用户批准 |
|
|
60
|
+
| `.claude/hooks/pre-compact.sh` | 在上下文压缩前对检查点做快照;恢复时的检查点刷新指令由 `session-start.sh` 负责 |
|
|
61
|
+
| `.claude/hooks/session-start.sh` | 会话启动/恢复/压缩时重新注入进行中的检查点;失败堆积时提示 `/ratchet` |
|
|
62
|
+
| `.claude/hooks/session-end.sh` | 每个会话写入一行台账(时长、调用/失败/编辑数、收工拦截次数、测试状态)→ `sessions.jsonl` |
|
|
63
|
+
| `.claude/hooks/observe-log.sh` | 以 JSONL 记录每次工具调用 + 绊线(同一失败 3 次、调用激增) |
|
|
64
|
+
| `harness/harness_report.py` | 基于日志的健康记分卡 |
|
|
65
|
+
| `harness/tests/` | 钩子与安装器的自测 |
|
|
66
|
+
| `bin/cli.js` | 安装器(init / update / doctor / uninstall) |
|
|
67
|
+
|
|
68
|
+
技能:
|
|
69
|
+
|
|
70
|
+
| 技能 | 作用 |
|
|
71
|
+
|---|---|
|
|
72
|
+
| `/harness-init` | 扫描仓库 → 填写 PROJECT 章节 & commands.env(安装时一次) |
|
|
73
|
+
| `/checkpoint` | 将状态保存到 `.harness/state/`(plan.md / decisions.jsonl / progress.json) |
|
|
74
|
+
| `/ratchet [失误]` | 复现 → 归类 → 提议规则/传感器/权限 → 验证(无参数:诊断日志)。每个提议附带可证伪的预测(`predictions.jsonl`) |
|
|
75
|
+
| `/guide-audit` | 审计 CLAUDE.md 规则 — 保留 / 删除 / 转为传感器(每月);将过往预测与日志比对验证,没起作用的规则凭证据删除 |
|
|
76
|
+
| `/distill [成功]` | 成功侧的棘轮:从日志中挖掘已完成的任务,提议为可复用的项目技能 |
|
|
77
|
+
|
|
78
|
+
## 推荐权限配置(可选)
|
|
79
|
+
|
|
80
|
+
套件不强加权限策略。若用于无人值守或高自治场景,可考虑在项目的
|
|
81
|
+
`.claude/settings.json` 中加入类似配置 — 尤其是阻止智能体修改
|
|
82
|
+
自身挽具的那些条目:
|
|
83
|
+
|
|
84
|
+
```json
|
|
85
|
+
{
|
|
86
|
+
"permissions": {
|
|
87
|
+
"ask": [
|
|
88
|
+
"Edit(.claude/**)", "Write(.claude/**)",
|
|
89
|
+
"Edit(.harness/**)", "Write(.harness/**)",
|
|
90
|
+
"Edit(.env*)", "Edit(*.config.*)", "Edit(.github/**)"
|
|
91
|
+
],
|
|
92
|
+
"deny": [
|
|
93
|
+
"Read(.env)", "Read(.env.*)", "Read(**/secrets/**)",
|
|
94
|
+
"Read(**/*.pem)", "Bash(sudo*)"
|
|
95
|
+
]
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
## 运营例程
|
|
101
|
+
|
|
102
|
+
- **每天 / 每个任务:** 正常干活即可。传感器和守卫是自动的。
|
|
103
|
+
长任务用 `/checkpoint` 保存状态。
|
|
104
|
+
- **发现失误时:** 不要只在对话里修掉 — 运行 `/ratchet [发生了什么]`
|
|
105
|
+
把它结构化。同一审查意见出现 3 次 → 变成规则;同一规则被违反
|
|
106
|
+
3 次 → 升级为守卫/权限。
|
|
107
|
+
- **每周:** 用 `python3 harness/harness_report.py` 查看记分卡;
|
|
108
|
+
出现重复失败就运行 `/ratchet`。
|
|
109
|
+
- **一次有意义的成功之后:** 若流程还会重复,用 `/distill` 把它
|
|
110
|
+
变成项目技能。
|
|
111
|
+
- **每月:** 用 `/guide-audit` 审计 CLAUDE.md — 删除传感器已自动
|
|
112
|
+
强制执行的规则,合并互相矛盾的规则,并将每条规则的预测与日志
|
|
113
|
+
比对(同一失败仍在发生的规则,凭数字而非感觉删除或升级)。
|
|
114
|
+
|
|
115
|
+
## 自定义
|
|
116
|
+
|
|
117
|
+
- 危险命令模式:`guard-pre-bash.sh` 中的 `DENY_PATTERNS`
|
|
118
|
+
- 绊线阈值:`observe-log.sh`(默认:同一失败 3 次、300 次调用)
|
|
119
|
+
- 传感器文件扩展名:`sensor-post-edit.sh` 中的 case 语句
|
|
120
|
+
- 收工拦截上限:`stop-gate.sh`(默认:每会话 3 次)
|
|
121
|
+
- 修改了钩子就在 `harness/tests/test_hooks.sh` 加一个用例并运行
|
|
122
|
+
|
|
123
|
+
## 什么时候不该用
|
|
124
|
+
|
|
125
|
+
对一次性提问、探索性头脑风暴、无法验证的创意工作来说它是杀鸡用牛刀。
|
|
126
|
+
它的价值体现在:反复运行的工作、失败有真实代价的工作、无人值守运行
|
|
127
|
+
的工作、以及必须跨会话保持状态的工作。
|
|
128
|
+
|
|
129
|
+
## 设计背景
|
|
130
|
+
|
|
131
|
+
本套件是两份材料的可运行实现。完整分析在 [`docs/`](docs/) 中 —
|
|
132
|
+
它们不随 npm 包分发(运行中的智能体不需要理论),但解释了这里的
|
|
133
|
+
每一个设计决策。
|
|
134
|
+
|
|
135
|
+
### 📄 EnvHarness: Awakening Static Worlds for Agent Learning
|
|
136
|
+
|
|
137
|
+
> **[arXiv:2608.19880](https://arxiv.org/abs/2608.19880)** · Chengsong Huang, Zifeng Wang, Rujun Han, Chen-Yu Lee 等 (2026)
|
|
138
|
+
> 深度解读:[docs/analysis-01-envharness.md](docs/analysis-01-envharness.md)
|
|
139
|
+
|
|
140
|
+
**论文主张。** 正如给*智能体*装上挽具(工具、记忆、技能)可以在不动
|
|
141
|
+
权重的情况下扩展它,你也可以给*环境*装上挽具,在不动其代码的情况下
|
|
142
|
+
定制学习信号:`Static Env + EnvHarness = Customized Env`。环境只在
|
|
143
|
+
接口层通过三个可组合的组件被变换 — **Stage**(重塑初始状态)、
|
|
144
|
+
**Contract**(过滤动作、变换观测、包装转移)、**Chain**(组合环境)—
|
|
145
|
+
因此原始的真值验证器始终得到保留,这正是它相对 LLM 生成环境的核心
|
|
146
|
+
优势。一个名为 **EnvRigger** 的自动化循环驱动设计:*观察*失败轨迹 →
|
|
147
|
+
*诊断*根因 → *编写*变换 → 对原始失败进行*验证*。在五个基准上,
|
|
148
|
+
包装后的环境提升了智能体表现(如 ALFWorld 分布外 +9.0 分、SWE-bench
|
|
149
|
+
Verified +2.7 且步数减少 9.8%)— 分布外增益最大,是迁移而非记忆的
|
|
150
|
+
证据。
|
|
151
|
+
|
|
152
|
+
**本套件从中借鉴的:**
|
|
153
|
+
|
|
154
|
+
| 论文概念 | 此处的实现 |
|
|
155
|
+
|---|---|
|
|
156
|
+
| EnvRigger 循环(观察 → 诊断 → 编写 → 验证) | `/ratchet` 技能 — 自动接受被替换为**用户批准** |
|
|
157
|
+
| Contract:动作过滤 | `guard-pre-bash.sh`(破坏性命令执行前拦截) |
|
|
158
|
+
| Contract:结构化反馈 | `sensor-post-edit.sh` / `stop-gate.sh`(验证结果反馈) |
|
|
159
|
+
| Stage:备妥的初始状态 | 会话启动时的检查点恢复 |
|
|
160
|
+
| **保留原始验证器** | 不变量:挽具包装你的测试套件,但绝不修改或绕过它 |
|
|
161
|
+
|
|
162
|
+
### 📘 挽具工程:6 层生产实战手册
|
|
163
|
+
|
|
164
|
+
> 公开实践者材料的综合 — [Mitchell Hashimoto](https://mitchellh.com/)
|
|
165
|
+
> 的棘轮方法论、[OpenAI Codex 实战报告](https://openai.com/index/harness-engineering)、
|
|
166
|
+
> [Martin Fowler](https://martinfowler.com/) 的向导与传感器分类法,以及
|
|
167
|
+
> Anthropic / LangChain / Cursor 的资料。
|
|
168
|
+
> 深度解读:[docs/analysis-02-harness-engineering.md](docs/analysis-02-harness-engineering.md)
|
|
169
|
+
|
|
170
|
+
**手册主张。** 提示词工程(模型*说*什么)和上下文工程(模型*看到*
|
|
171
|
+
什么)都被挽具工程所涵盖:模型*能做*什么、什么能在失败中幸存、什么被
|
|
172
|
+
允许、什么算完成。**Agent = Model + Harness** 这一主张有自报但一致的
|
|
173
|
+
证据支撑 — 同一个模型仅靠更换挽具在 GAIA 上从 30.91% 跃升至 74.55%,
|
|
174
|
+
固定模型仅靠挽具优化在 Terminal Bench 上从第 30 名爬到第 5 名。
|
|
175
|
+
架构分为六层,与本套件一一对应:
|
|
176
|
+
|
|
177
|
+
| # | 层 | 原则 | 在本套件中 |
|
|
178
|
+
|---|---|---|---|
|
|
179
|
+
| 1 | **向导** | 每一行 = 对一次过往失败的永久预防 | `CLAUDE.md` |
|
|
180
|
+
| 2 | **传感器** | 外部确定性检查,绝不自我评判 | `sensor-post-edit.sh`、`stop-gate.sh` |
|
|
181
|
+
| 3 | **智能体循环** | 每项预算都有上限,耗尽即升级上报 | 工作循环协议 + 绊线 + 3 次收工拦截上限 |
|
|
182
|
+
| 4 | **记忆** | 文件系统即记忆;必须通过恢复测试 | `/checkpoint` + `session-start.sh` |
|
|
183
|
+
| 5 | **权限** | 模型无法限制它自己 | `guard-pre-bash.sh` + 推荐权限 |
|
|
184
|
+
| 6 | **可观测性** | 记录一切,对漂移告警 | `observe-log.sh` + `harness_report.py` |
|
|
185
|
+
|
|
186
|
+
运营规则也来自于此 — Hashimoto 的**棘轮原则**("智能体每犯一次错,
|
|
187
|
+
就工程化一个让该错误不可能重演的方案")以及 Cursor 的晋升阶梯:
|
|
188
|
+
|
|
189
|
+
```mermaid
|
|
190
|
+
flowchart LR
|
|
191
|
+
F[观察到失败] --> R["/ratchet: 复现 + 诊断"]
|
|
192
|
+
R --> P{最强的层}
|
|
193
|
+
P -->|缺的是上下文| G["向导规则 (CLAUDE.md)"]
|
|
194
|
+
P -->|检查可以捕获| S[传感器 / lint 规则]
|
|
195
|
+
P -->|应当彻底不可能| H[守卫 / 权限]
|
|
196
|
+
G & S & H --> V[针对原始失败验证]
|
|
197
|
+
V --> N[同一失败无法重演]
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
同一审查意见出现 **3 次** → 变成规则。同一规则被违反 **3 次** →
|
|
201
|
+
变成关卡。而成熟的信号是规则增速的*下降* — 只会堆积的挽具就是债务,
|
|
202
|
+
这正是 `/guide-audit` 存在的意义。
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
206
|
+
想扩展套件本身?从
|
|
207
|
+
[docs/AGENT_BRIEFING.md](docs/AGENT_BRIEFING.md) 开始 — 那里有
|
|
208
|
+
不变量(不得削弱验证、未经批准不得自动应用、最小基础设施)和经过
|
|
209
|
+
审核的待办清单。
|
package/bin/cli.js
CHANGED
|
@@ -229,7 +229,7 @@ function init(target, force) {
|
|
|
229
229
|
}
|
|
230
230
|
|
|
231
231
|
let claudeMdExisted = false;
|
|
232
|
-
for (const rel of ['CLAUDE.md', '.harness/commands.env']) {
|
|
232
|
+
for (const rel of ['CLAUDE.md', 'AGENTS.md', '.harness/commands.env']) {
|
|
233
233
|
const dst = path.join(target, rel);
|
|
234
234
|
if (fs.existsSync(dst)) {
|
|
235
235
|
log(' = ' + rel + ' (existing file kept)');
|
|
@@ -304,6 +304,9 @@ function update(target, force) {
|
|
|
304
304
|
writeManifest(target, files);
|
|
305
305
|
ensureRuntimeDirs(target);
|
|
306
306
|
migrateLegacyState(target);
|
|
307
|
+
// hooks added in newer kit versions need registering too — mergeSettings
|
|
308
|
+
// is append-only and idempotent, so this never disturbs user settings
|
|
309
|
+
mergeSettings(target);
|
|
307
310
|
log(`\n${changed} file(s) updated (v${manifest.kitVersion} -> v${KIT_VERSION}). ` +
|
|
308
311
|
'CLAUDE.md / commands.env / settings.json were not touched.');
|
|
309
312
|
|
|
@@ -360,7 +363,7 @@ function doctor(target) {
|
|
|
360
363
|
const body = fs.readFileSync(sp, 'utf8');
|
|
361
364
|
JSON.parse(body);
|
|
362
365
|
check(true, '.claude/settings.json parses');
|
|
363
|
-
for (const h of ['guard-pre-bash.sh', 'sensor-post-edit.sh', 'stop-gate.sh', 'observe-log.sh', 'session-start.sh']) {
|
|
366
|
+
for (const h of ['guard-pre-bash.sh', 'guard-pre-edit.sh', 'sensor-post-edit.sh', 'stop-gate.sh', 'observe-log.sh', 'session-start.sh', 'pre-compact.sh', 'session-end.sh']) {
|
|
364
367
|
if (!body.includes(h)) check(false, `hook registered: ${h}`, 'not in settings.json');
|
|
365
368
|
}
|
|
366
369
|
} catch { check(false, '.claude/settings.json', 'missing or failed to parse'); }
|
|
@@ -1,7 +1,9 @@
|
|
|
1
1
|
#!/usr/bin/env python3
|
|
2
2
|
"""Harness health scorecard.
|
|
3
3
|
|
|
4
|
-
Reads .harness/logs/tool_calls.jsonl and
|
|
4
|
+
Reads .harness/logs/tool_calls.jsonl (raw tool calls) and, if present,
|
|
5
|
+
.harness/logs/sessions.jsonl (the per-session ledger written by
|
|
6
|
+
session-end.sh) and prints health metrics.
|
|
5
7
|
|
|
6
8
|
Usage:
|
|
7
9
|
python3 harness/harness_report.py [--days 7]
|
|
@@ -79,6 +81,19 @@ def main() -> int:
|
|
|
79
81
|
print(f" [{n}x] {tool}: {err}")
|
|
80
82
|
print("\n -> For failures repeated 2+ times, consider /ratchet to turn them into rules.")
|
|
81
83
|
|
|
84
|
+
ledger = load(root / ".harness" / "logs" / "sessions.jsonl", since)
|
|
85
|
+
if ledger:
|
|
86
|
+
clean = sum(1 for e in ledger if e.get("tests_clean"))
|
|
87
|
+
blocked = sum(e.get("stop_blocks") or 0 for e in ledger)
|
|
88
|
+
durations = sorted(e.get("duration_s") or 0 for e in ledger)
|
|
89
|
+
print("\n─ Session ledger ─")
|
|
90
|
+
print(f" Ended sessions: {len(ledger)}")
|
|
91
|
+
print(f" Tests clean at end: {clean}/{len(ledger)}")
|
|
92
|
+
print(f" Stop-gate blocks: {blocked}")
|
|
93
|
+
print(f" Duration (median): {durations[len(durations) // 2]}s")
|
|
94
|
+
if clean < len(ledger):
|
|
95
|
+
print(" -> Sessions ending with untested edits are quality-drift risk; check why.")
|
|
96
|
+
|
|
82
97
|
print("\n─ How to read this ─")
|
|
83
98
|
print(" · Is the failure rate trending down week over week?")
|
|
84
99
|
print(" · The same failure signature next week means the ratchet isn't working")
|
|
@@ -44,6 +44,55 @@ OUT=$(echo '{"tool_name":"Bash","tool_input":{"command":"git push --force-with-l
|
|
|
44
44
|
check "allows force-with-lease" "EMPTY" "$OUT"
|
|
45
45
|
OUT=$(echo '{"tool_name":"Bash","tool_input":{"command":"git status"}}' | bash "$HOOKS/guard-pre-bash.sh")
|
|
46
46
|
check "passes safe commands" "EMPTY" "$OUT"
|
|
47
|
+
# state-aware commit gate: pending_test marker + configured TEST_CMD -> ask
|
|
48
|
+
printf 'LINT_CMD=""\nTEST_CMD="true"\n' > "$TMP/.harness/commands.env"
|
|
49
|
+
echo "a.py" > "$TMP/.harness/logs/pending_test.s1"
|
|
50
|
+
OUT=$(echo '{"tool_name":"Bash","session_id":"s1","tool_input":{"command":"git commit -m wip"}}' | bash "$HOOKS/guard-pre-bash.sh")
|
|
51
|
+
check "asks on commit with untested edits" '"permissionDecision": "ask"' "$OUT"
|
|
52
|
+
OUT=$(echo '{"tool_name":"Bash","session_id":"s2","tool_input":{"command":"git commit -m ok"}}' | bash "$HOOKS/guard-pre-bash.sh")
|
|
53
|
+
check "allows commit in a session without pending edits" "EMPTY" "$OUT"
|
|
54
|
+
printf 'LINT_CMD=""\nTEST_CMD=""\n' > "$TMP/.harness/commands.env"
|
|
55
|
+
OUT=$(echo '{"tool_name":"Bash","session_id":"s1","tool_input":{"command":"git commit -m x"}}' | bash "$HOOKS/guard-pre-bash.sh")
|
|
56
|
+
check "commit gate silent when TEST_CMD unconfigured" "EMPTY" "$OUT"
|
|
57
|
+
rm -f "$TMP/.harness/logs/pending_test.s1"
|
|
58
|
+
|
|
59
|
+
# ── guard-pre-edit.sh ────────────────────────────────────────────
|
|
60
|
+
echo "guard-pre-edit.sh"
|
|
61
|
+
fresh_project
|
|
62
|
+
OUT=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"'"$TMP"'/.claude/settings.json","old_string":"a","new_string":"b"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
63
|
+
check "asks on harness machinery edit" '"permissionDecision": "ask"' "$OUT"
|
|
64
|
+
OUT=$(echo '{"tool_name":"Write","tool_input":{"file_path":"'"$TMP"'/.harness/state/progress.json","content":"{}"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
65
|
+
check "allows checkpoint writes" "EMPTY" "$OUT"
|
|
66
|
+
OUT=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"'"$TMP"'/src/app.py","old_string":"a","new_string":"b"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
67
|
+
check "allows normal source edits" "EMPTY" "$OUT"
|
|
68
|
+
OUT=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"'"$TMP"'/.env","old_string":"a","new_string":"b"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
69
|
+
check "asks on env file edit" '"permissionDecision": "ask"' "$OUT"
|
|
70
|
+
OUT=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"'"$TMP"'/vite.config.ts","old_string":"a","new_string":"b"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
71
|
+
check "asks on tool config edit" '"permissionDecision": "ask"' "$OUT"
|
|
72
|
+
OUT=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"'"$TMP"'/tests/test_app.py","old_string":"def test_a():","new_string":"@pytest.mark.skip\ndef test_a():"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
73
|
+
check "asks on added skip marker" "skip" "$OUT"
|
|
74
|
+
OUT=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"'"$TMP"'/src/api.test.ts","old_string":"it(\"works\", () => {","new_string":"it.skip(\"works\", () => {"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
75
|
+
check "asks on it.skip in js tests" '"permissionDecision": "ask"' "$OUT"
|
|
76
|
+
mkdir -p "$TMP/tests"
|
|
77
|
+
printf 'def test_a():\n pass\ndef test_b():\n pass\n' > "$TMP/tests/test_app.py"
|
|
78
|
+
OUT=$(echo '{"tool_name":"Write","tool_input":{"file_path":"'"$TMP"'/tests/test_app.py","content":"def test_a():\n pass\n"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
79
|
+
check "asks on removed test function" "removes 1 test function" "$OUT"
|
|
80
|
+
OUT=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"'"$TMP"'/tests/test_app.py","old_string":"def test_a():","new_string":"def test_a():\ndef test_c():"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
81
|
+
check "allows adding tests" "EMPTY" "$OUT"
|
|
82
|
+
OUT=$(echo '{"tool_name":"Write","tool_input":{"file_path":"'"$TMP"'/tests/test_new.py","content":"def test_x():\n pass\n"}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
83
|
+
check "allows brand-new test file" "EMPTY" "$OUT"
|
|
84
|
+
OUT=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"/somewhere/else/test_x.py","old_string":"def test_a():","new_string":""}}' | bash "$HOOKS/guard-pre-edit.sh")
|
|
85
|
+
check "ignores files outside the project" "EMPTY" "$OUT"
|
|
86
|
+
|
|
87
|
+
# ── pre-compact.sh ───────────────────────────────────────────────
|
|
88
|
+
echo "pre-compact.sh"
|
|
89
|
+
fresh_project
|
|
90
|
+
mkdir -p "$TMP/.harness/state"
|
|
91
|
+
echo '{"task_id":"t1"}' > "$TMP/.harness/state/progress.json"
|
|
92
|
+
OUT=$(echo '{"hook_event_name":"PreCompact","session_id":"s1","trigger":"auto"}' | bash "$HOOKS/pre-compact.sh")
|
|
93
|
+
check "emits no JSON (PreCompact schema has no context channel)" "EMPTY" "$OUT"
|
|
94
|
+
check "snapshots checkpoint" "yes" "$([ -f "$TMP/.harness/state/.pre-compact-backup/progress.json" ] && echo yes)"
|
|
95
|
+
check "logs the compaction event" "PreCompact" "$(cat "$TMP/.harness/logs/tool_calls.jsonl")"
|
|
47
96
|
|
|
48
97
|
# ── observe-log.sh ───────────────────────────────────────────────
|
|
49
98
|
echo "observe-log.sh"
|
|
@@ -53,11 +102,16 @@ LINES=$(wc -l < "$TMP/.harness/logs/tool_calls.jsonl" | tr -d ' ')
|
|
|
53
102
|
check "logs one call" "1" "$LINES"
|
|
54
103
|
echo '{"hook_event_name":"PostToolUse","session_id":"s1","tool_name":"Edit","tool_input":{"file_path":"a.py"},"agent_id":"x1","agent_type":"code-reviewer"}' | bash "$HOOKS/observe-log.sh" > /dev/null
|
|
55
104
|
check "records subagent fields" "code-reviewer" "$(tail -1 "$TMP/.harness/logs/tool_calls.jsonl")"
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
105
|
+
# same error 3x — line numbers differ (normalization) and a success is
|
|
106
|
+
# interleaved (no longer requires "in a row")
|
|
107
|
+
OUT=$(echo '{"hook_event_name":"PostToolUseFailure","session_id":"s1","tool_name":"Bash","tool_input":{"command":"pytest"},"tool_response":"E: boom at line 12"}' | bash "$HOOKS/observe-log.sh")
|
|
108
|
+
check "no tripwire on first failure" "EMPTY" "$OUT"
|
|
109
|
+
echo '{"hook_event_name":"PostToolUse","session_id":"s1","tool_name":"Read","tool_input":{"file_path":"a.py"}}' | bash "$HOOKS/observe-log.sh" > /dev/null
|
|
110
|
+
OUT=$(echo '{"hook_event_name":"PostToolUseFailure","session_id":"s1","tool_name":"Bash","tool_input":{"command":"pytest"},"tool_response":"E: boom at line 34"}' | bash "$HOOKS/observe-log.sh")
|
|
111
|
+
OUT=$(echo '{"hook_event_name":"PostToolUseFailure","session_id":"s1","tool_name":"Bash","tool_input":{"command":"pytest"},"tool_response":"E: boom at line 56"}' | bash "$HOOKS/observe-log.sh")
|
|
112
|
+
check "tripwire on 3 normalized repeats" "escalation packet" "$OUT"
|
|
113
|
+
OUT=$(echo '{"hook_event_name":"PostToolUseFailure","session_id":"s1","tool_name":"Bash","tool_input":{"command":"npm test"},"tool_response":"E: different failure"}' | bash "$HOOKS/observe-log.sh")
|
|
114
|
+
check "no tripwire on a different error" "EMPTY" "$OUT"
|
|
61
115
|
|
|
62
116
|
# ── sensor-post-edit.sh ──────────────────────────────────────────
|
|
63
117
|
echo "sensor-post-edit.sh"
|
|
@@ -96,38 +150,90 @@ OUT=$(echo '{"session_id":"s1"}' | bash "$HOOKS/stop-gate.sh")
|
|
|
96
150
|
check "silent when tests pass" "EMPTY" "$OUT"
|
|
97
151
|
[ -f "$TMP/.harness/logs/pending_test.s1" ] && MARKER_LEFT="yes" || MARKER_LEFT=""
|
|
98
152
|
check "clears marker on pass" "EMPTY" "$MARKER_LEFT"
|
|
153
|
+
# trace rules: require-before-stop blocks once, then lets the stop pass
|
|
154
|
+
fresh_project
|
|
155
|
+
printf 'LINT_CMD=""\nTEST_CMD="true"\n' > "$TMP/.harness/commands.env"
|
|
156
|
+
printf '# comment line\nrequire-before-stop npm.run.build Run the build before ending the turn.\n' > "$TMP/.harness/trace.rules"
|
|
157
|
+
echo "a.py" > "$TMP/.harness/logs/pending_test.s1"
|
|
158
|
+
echo '{"ts":"2026-01-01T00:00:00","session_id":"s1","event":"PostToolUse","tool":"Edit","summary":"a.py"}' > "$TMP/.harness/logs/tool_calls.jsonl"
|
|
159
|
+
OUT=$(echo '{"session_id":"s1"}' | bash "$HOOKS/stop-gate.sh")
|
|
160
|
+
check "trace rule blocks on missing required step" "trace rule" "$OUT"
|
|
161
|
+
OUT=$(echo '{"session_id":"s1"}' | bash "$HOOKS/stop-gate.sh")
|
|
162
|
+
check "trace rule blocks only once per session" "EMPTY" "$OUT"
|
|
163
|
+
echo "a.py" > "$TMP/.harness/logs/pending_test.s2"
|
|
164
|
+
echo '{"ts":"2026-01-01T00:01:00","session_id":"s2","event":"PostToolUse","tool":"Bash","summary":"npm run build"}' >> "$TMP/.harness/logs/tool_calls.jsonl"
|
|
165
|
+
OUT=$(echo '{"session_id":"s2"}' | bash "$HOOKS/stop-gate.sh")
|
|
166
|
+
check "trace rule satisfied by matching log entry" "EMPTY" "$OUT"
|
|
167
|
+
rm -f "$TMP/.harness/trace.rules"
|
|
168
|
+
|
|
169
|
+
# ── session-end.sh ───────────────────────────────────────────────
|
|
170
|
+
echo "session-end.sh"
|
|
171
|
+
fresh_project
|
|
172
|
+
OUT=$(echo '{"hook_event_name":"SessionEnd","session_id":"s9","reason":"exit"}' | bash "$HOOKS/session-end.sh")
|
|
173
|
+
check "silent with no session activity" "EMPTY" "$([ -f "$TMP/.harness/logs/sessions.jsonl" ] && echo present)"
|
|
174
|
+
cat > "$TMP/.harness/logs/tool_calls.jsonl" <<'EOF'
|
|
175
|
+
{"ts":"2026-01-01T10:00:00","session_id":"s9","event":"PostToolUse","tool":"Bash","summary":"ls"}
|
|
176
|
+
{"ts":"2026-01-01T10:05:00","session_id":"s9","event":"PostToolUse","tool":"Edit","summary":"a.py"}
|
|
177
|
+
{"ts":"2026-01-01T10:06:00","session_id":"s9","event":"PostToolUseFailure","tool":"Bash","error":"boom"}
|
|
178
|
+
{"ts":"2026-01-01T10:07:00","session_id":"other","event":"PostToolUse","tool":"Bash","summary":"ls"}
|
|
179
|
+
EOF
|
|
180
|
+
echo "a.py" > "$TMP/.harness/logs/pending_test.s9"
|
|
181
|
+
echo "2" > "$TMP/.harness/logs/stop_blocks.s9"
|
|
182
|
+
OUT=$(echo '{"hook_event_name":"SessionEnd","session_id":"s9","reason":"exit"}' | bash "$HOOKS/session-end.sh")
|
|
183
|
+
ROW=$(cat "$TMP/.harness/logs/sessions.jsonl" 2>/dev/null)
|
|
184
|
+
check "writes one ledger row" '"session_id": "s9"' "$ROW"
|
|
185
|
+
check "counts calls for this session only" '"calls": 3' "$ROW"
|
|
186
|
+
check "counts failures and edits" '"failures": 1' "$ROW"
|
|
187
|
+
check "records duration" '"duration_s": 360' "$ROW"
|
|
188
|
+
check "records unclean test state" '"tests_clean": false' "$ROW"
|
|
189
|
+
check "records stop blocks" '"stop_blocks": 2' "$ROW"
|
|
190
|
+
[ -f "$TMP/.harness/logs/pending_test.s9" ] && LEFT="yes" || LEFT=""
|
|
191
|
+
check "cleans session marker files" "EMPTY" "$LEFT"
|
|
99
192
|
|
|
100
193
|
# ── session-start.sh ─────────────────────────────────────────────
|
|
101
194
|
echo "session-start.sh"
|
|
102
195
|
fresh_project
|
|
103
|
-
OUT=$(bash "$HOOKS/session-start.sh")
|
|
196
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
104
197
|
check "silent without checkpoint" "EMPTY" "$OUT"
|
|
105
198
|
mkdir -p "$TMP/.harness/state"
|
|
106
199
|
echo '{"task_id":"t1","status":"in_progress","next_step":"do X"}' > "$TMP/.harness/state/progress.json"
|
|
107
|
-
OUT=$(bash "$HOOKS/session-start.sh")
|
|
200
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
108
201
|
check "injects in-progress checkpoint" ".harness/state" "$OUT"
|
|
109
202
|
echo '{"task_id":"t1","status":"completed"}' > "$TMP/.harness/state/progress.json"
|
|
110
|
-
OUT=$(bash "$HOOKS/session-start.sh")
|
|
203
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
111
204
|
check "skips completed checkpoint" "EMPTY" "$OUT"
|
|
112
205
|
rm "$TMP/.harness/state/progress.json"
|
|
113
206
|
# legacy layout (pre-v0.2): checkpoint at the project root still read
|
|
114
207
|
echo '{"task_id":"t1","status":"in_progress","next_step":"do X"}' > "$TMP/progress.json"
|
|
115
|
-
OUT=$(bash "$HOOKS/session-start.sh")
|
|
208
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
116
209
|
check "reads legacy root checkpoint" "progress.json" "$OUT"
|
|
117
210
|
rm "$TMP/progress.json"
|
|
211
|
+
# CLAUDE.md still pointing checkpoints at the root -> propose-a-diff nudge
|
|
212
|
+
printf '# P\n- write the plan to plan.md\n' > "$TMP/CLAUDE.md"
|
|
213
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
214
|
+
check "nudges CLAUDE.md path migration" "user's approval" "$OUT"
|
|
215
|
+
printf '# P\n- write the plan to .harness/state/plan.md\n' > "$TMP/CLAUDE.md"
|
|
216
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
217
|
+
check "silent when CLAUDE.md paths are current" "EMPTY" "$OUT"
|
|
218
|
+
rm "$TMP/CLAUDE.md"
|
|
118
219
|
NOW=$(python3 -c 'import datetime; print(datetime.datetime.now().isoformat(timespec="seconds"))')
|
|
119
220
|
for _ in 1 2 3; do
|
|
120
221
|
echo "{\"ts\":\"$NOW\",\"session_id\":\"old\",\"event\":\"PostToolUseFailure\",\"tool\":\"Bash\",\"error\":\"x\"}" >> "$TMP/.harness/logs/tool_calls.jsonl"
|
|
121
222
|
done
|
|
122
|
-
OUT=$(bash "$HOOKS/session-start.sh")
|
|
223
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
123
224
|
check "nudges /ratchet on accumulated failures" "/ratchet" "$OUT"
|
|
124
225
|
fresh_project
|
|
125
226
|
printf 'LINT_CMD=""\nTEST_CMD=""\n' > "$TMP/.harness/commands.env"
|
|
126
|
-
OUT=$(bash "$HOOKS/session-start.sh")
|
|
227
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
127
228
|
check "offers /harness-init when unconfigured" "/harness-init" "$OUT"
|
|
128
229
|
printf 'LINT_CMD="true"\nTEST_CMD=""\n' > "$TMP/.harness/commands.env"
|
|
129
|
-
OUT=$(bash "$HOOKS/session-start.sh")
|
|
230
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"startup"}' | bash "$HOOKS/session-start.sh")
|
|
130
231
|
check "no setup nudge once configured" "EMPTY" "$OUT"
|
|
232
|
+
# resuming right after a compaction -> checkpoint-refresh instruction
|
|
233
|
+
# (pre-compact.sh cannot inject context; this is the supported channel)
|
|
234
|
+
mkdir -p "$TMP/.harness/state/.pre-compact-backup"
|
|
235
|
+
OUT=$(echo '{"hook_event_name":"SessionStart","source":"compact"}' | bash "$HOOKS/session-start.sh")
|
|
236
|
+
check "instructs checkpoint refresh after compaction" "pre-compact-backup" "$OUT"
|
|
131
237
|
|
|
132
238
|
# ── result ───────────────────────────────────────────────────────
|
|
133
239
|
echo
|
|
@@ -35,6 +35,7 @@ check "installs hooks" "yes" "$([ -f "$TGT/.claude/hooks/stop-gate.sh" ] && echo
|
|
|
35
35
|
check "hooks are executable" "yes" "$([ -x "$TGT/.claude/hooks/stop-gate.sh" ] && echo yes)"
|
|
36
36
|
check "installs skills" "yes" "$([ -f "$TGT/.claude/skills/ratchet/SKILL.md" ] && echo yes)"
|
|
37
37
|
check "installs CLAUDE.md" "yes" "$([ -f "$TGT/CLAUDE.md" ] && echo yes)"
|
|
38
|
+
check "installs AGENTS.md" "yes" "$([ -f "$TGT/AGENTS.md" ] && echo yes)"
|
|
38
39
|
check "installs settings.json" "yes" "$([ -f "$TGT/.claude/settings.json" ] && echo yes)"
|
|
39
40
|
check "writes manifest" "yes" "$([ -f "$TGT/.harness/kit-manifest.json" ] && echo yes)"
|
|
40
41
|
check "gitignores logs" ".harness/logs/" "$(cat "$TGT/.gitignore")"
|
|
@@ -50,6 +51,7 @@ check "does not ship test_installer.sh" "EMPTY" "$([ -f "$TGT/harness/tests/test
|
|
|
50
51
|
echo "init (brownfield)"
|
|
51
52
|
BF=$(mktemp -d)
|
|
52
53
|
echo "MY PROJECT RULES" > "$BF/CLAUDE.md"
|
|
54
|
+
echo "MY OWN AGENTS FILE" > "$BF/AGENTS.md"
|
|
53
55
|
echo "node_modules/" > "$BF/.gitignore"
|
|
54
56
|
mkdir -p "$BF/.claude"
|
|
55
57
|
cat > "$BF/.claude/settings.json" <<'EOF'
|
|
@@ -64,6 +66,7 @@ cat > "$BF/.claude/settings.json" <<'EOF'
|
|
|
64
66
|
EOF
|
|
65
67
|
OUT=$(node "$CLI" init "$BF" 2>&1)
|
|
66
68
|
check "keeps existing CLAUDE.md" "MY PROJECT RULES" "$(cat "$BF/CLAUDE.md")"
|
|
69
|
+
check "keeps existing AGENTS.md" "MY OWN AGENTS FILE" "$(cat "$BF/AGENTS.md")"
|
|
67
70
|
check "notes CLAUDE.md was kept" "existing CLAUDE.md was kept" "$OUT"
|
|
68
71
|
check "keeps user permissions" '"Read"' "$(cat "$BF/.claude/settings.json")"
|
|
69
72
|
check "keeps user's own hook" "my-own-hook.sh" "$(cat "$BF/.claude/settings.json")"
|
|
@@ -108,6 +111,20 @@ sed -i.bak '/.harness\/state/d' "$TGT/.gitignore" && rm -f "$TGT/.gitignore.bak"
|
|
|
108
111
|
OUT=$(node "$CLI" update "$TGT" 2>&1)
|
|
109
112
|
check "update re-adds state gitignore entry" ".harness/state/" "$(cat "$TGT/.gitignore")"
|
|
110
113
|
|
|
114
|
+
# update on an install from before a hook existed must register the new
|
|
115
|
+
# hook in settings.json (mergeSettings runs on update too, append-only)
|
|
116
|
+
python3 - "$TGT" <<'EOF'
|
|
117
|
+
import json, sys, os
|
|
118
|
+
sp = os.path.join(sys.argv[1], '.claude/settings.json')
|
|
119
|
+
s = json.load(open(sp))
|
|
120
|
+
s['hooks'].pop('SessionEnd', None)
|
|
121
|
+
s['user_marker'] = 'kept'
|
|
122
|
+
json.dump(s, open(sp, 'w'), indent=2)
|
|
123
|
+
EOF
|
|
124
|
+
OUT=$(node "$CLI" update "$TGT" 2>&1)
|
|
125
|
+
check "update registers newly shipped hooks" "session-end.sh" "$(cat "$TGT/.claude/settings.json")"
|
|
126
|
+
check "update merge keeps user settings" '"user_marker"' "$(cat "$TGT/.claude/settings.json")"
|
|
127
|
+
|
|
111
128
|
# pre-v0.2 CLAUDE.md pointing checkpoints at the project root -> migration
|
|
112
129
|
# note (told, never touched); the current template must NOT trigger it
|
|
113
130
|
printf '# old project\n- read progress.json first\n' > "$TGT/CLAUDE.md"
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "uni-harness",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Agent harness kit for Claude Code — installs verification sensors, destructive-command guards, checkpoint recovery, observability logs, and ratchet skills into your project",
|
|
5
5
|
"bin": {
|
|
6
6
|
"uni-harness": "bin/cli.js"
|
|
@@ -8,6 +8,7 @@
|
|
|
8
8
|
"files": [
|
|
9
9
|
"bin",
|
|
10
10
|
"CLAUDE.md",
|
|
11
|
+
"AGENTS.md",
|
|
11
12
|
".claude",
|
|
12
13
|
".harness/commands.env",
|
|
13
14
|
"harness",
|