braintrust-lite 0.1.6 → 0.1.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +73 -102
- package/bin/braintrust +12 -0
- package/package.json +21 -13
- package/skills/consult/SKILL.md +213 -0
- package/src/config.js +60 -0
- package/src/doctor.js +120 -0
- package/src/format.js +25 -37
- package/src/judge.js +87 -0
- package/src/main.js +332 -0
- package/src/memory/db.js +183 -0
- package/src/memory/index.js +31 -0
- package/src/normalize.js +172 -0
- package/src/normalize.test.js +125 -0
- package/src/prompts/architecture.md +21 -0
- package/src/prompts/code.md +21 -0
- package/src/prompts/general.md +22 -0
- package/src/prompts/index.js +49 -0
- package/src/prompts/writing.md +21 -0
- package/src/providers/claude.js +45 -0
- package/src/providers/codex.js +69 -0
- package/src/providers/gemini.js +81 -0
- package/src/providers/index.js +22 -0
- package/src/reflector.js +244 -0
- package/src/save.js +93 -0
- package/src/server.js +225 -52
- package/LICENSE +0 -21
- package/bin/consult +0 -79
- package/src/consult.js +0 -112
- package/src/providers.js +0 -88
package/README.md
CHANGED
|
@@ -1,149 +1,120 @@
|
|
|
1
|
-
#
|
|
1
|
+
# brantrust
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
同题多模型融合器 — 把同一个问题同时发给 Claude、Codex、Gemini,然后用一个 Judge 综合出"集大成方案"。
|
|
4
4
|
|
|
5
5
|
```
|
|
6
|
-
|
|
7
|
-
├─ Task(subagent_type=Plan, prompt=X) ← 正常子 agent
|
|
8
|
-
└─ mcp__braintrust_lite__consult(prompt=X) ← Codex + Gemini 旁路咨询
|
|
9
|
-
→ 主 Claude 融合三方视角 → 最终方案
|
|
6
|
+
输入 → 并发生成(3) → 清洗归一化 → Judge 融合(1) → 输出 + 落盘
|
|
10
7
|
```
|
|
11
8
|
|
|
12
|
-
|
|
9
|
+
4 次 API 调用,低成本,天天能用。
|
|
13
10
|
|
|
14
11
|
---
|
|
15
12
|
|
|
16
13
|
## 安装
|
|
17
14
|
|
|
18
|
-
**前置条件**:`codex` 和 `gemini` CLI 均已登录。
|
|
19
|
-
|
|
20
15
|
```bash
|
|
21
16
|
# 克隆
|
|
22
|
-
git clone https://github.com/HongjieRen/
|
|
23
|
-
cd
|
|
24
|
-
|
|
25
|
-
# 安装依赖
|
|
26
|
-
npm install
|
|
27
|
-
|
|
28
|
-
# 可选:把 CLI 软链到 PATH
|
|
29
|
-
ln -sf "$(pwd)/bin/consult" ~/.local/bin/consult
|
|
30
|
-
chmod +x bin/consult
|
|
31
|
-
```
|
|
17
|
+
git clone https://github.com/HongjieRen/brantrust.git
|
|
18
|
+
cd brantrust
|
|
32
19
|
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
```bash
|
|
38
|
-
claude mcp add braintrust-lite node "$(pwd)/src/server.js"
|
|
20
|
+
# 软链接到 PATH
|
|
21
|
+
ln -sf "$(pwd)/brantrust" ~/.local/bin/brantrust
|
|
22
|
+
chmod +x brantrust
|
|
39
23
|
```
|
|
40
24
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
重启 Claude Code 后生效。
|
|
44
|
-
|
|
45
|
-
---
|
|
46
|
-
|
|
47
|
-
## 安装 Skill 引导
|
|
48
|
-
|
|
49
|
-
把 skill 软链到 Claude Code 全局 skill 目录,让主 Claude 知道何时该主动使用 consult:
|
|
50
|
-
|
|
51
|
-
```bash
|
|
52
|
-
ln -sf "$(pwd)/skills/consult" ~/.claude/skills/consult
|
|
53
|
-
```
|
|
25
|
+
**前置依赖**(三个 CLI 均需已登录):
|
|
54
26
|
|
|
55
|
-
|
|
27
|
+
| Provider | CLI | 验证命令 |
|
|
28
|
+
|----------|-----|---------|
|
|
29
|
+
| Claude | `claude` | `claude -p "hi" --output-format json` |
|
|
30
|
+
| OpenAI Codex | `codex` | `codex exec "hi" --json --skip-git-repo-check --ephemeral` |
|
|
31
|
+
| Google Gemini | `gemini` | `gemini -p "hi" -o json` |
|
|
56
32
|
|
|
57
33
|
---
|
|
58
34
|
|
|
59
|
-
##
|
|
60
|
-
|
|
61
|
-
### 在 Claude Code 里(推荐)
|
|
62
|
-
|
|
63
|
-
Claude 会在处理规划/设计类任务时自动(或在 `/consult` 引导下)并发调用:
|
|
64
|
-
|
|
65
|
-
```
|
|
66
|
-
你处理一个架构选型任务时,Claude 会同时:
|
|
67
|
-
1. 启动 Plan sub-agent 做深度分析
|
|
68
|
-
2. 调用 mcp__braintrust_lite__consult 获取 Codex + Gemini 的独立视角
|
|
69
|
-
3. 融合三方输出给你最终方案
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
### 终端 CLI(fallback / 调试)
|
|
35
|
+
## 用法
|
|
73
36
|
|
|
74
37
|
```bash
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
38
|
+
brantrust "解释 CAP 定理" # 默认:3 generator + 1 judge
|
|
39
|
+
brantrust --no-judge "React vs Vue" # 只并发收集,不 judge
|
|
40
|
+
brantrust --judge-model gemini "数据库选型" # 切换 Judge 模型
|
|
41
|
+
brantrust --skip codex "量子计算" # 跳过某个模型(可多次)
|
|
42
|
+
cat app.ts | brantrust "review 这段代码" # stdin 管道
|
|
43
|
+
brantrust --dir ~/project "项目分析" # 指定工作目录
|
|
44
|
+
brantrust --context-file design.md "实现方案" # 附加上下文文件
|
|
45
|
+
brantrust --timeout 60 "快速问题" # 超时秒数
|
|
46
|
+
brantrust --no-save "临时问答" # 不保存到磁盘
|
|
47
|
+
brantrust --json "问题" # 输出完整 JSON
|
|
48
|
+
brantrust --list # 查看历史运行
|
|
49
|
+
brantrust --strict "关键决策" # [v2] 完整 Judge 流水线
|
|
82
50
|
```
|
|
83
51
|
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
## 参数
|
|
52
|
+
### 参数一览
|
|
87
53
|
|
|
88
54
|
| 参数 | 默认 | 说明 |
|
|
89
|
-
|
|
90
|
-
| `prompt` | 必须 |
|
|
91
|
-
|
|
|
92
|
-
| `
|
|
93
|
-
| `
|
|
94
|
-
|
|
|
95
|
-
| `--
|
|
55
|
+
|------|------|------|
|
|
56
|
+
| `"prompt"` | 必须 | 问题文本 |
|
|
57
|
+
| `--skip <model>` | — | 跳过模型:claude / codex / gemini,可多次使用 |
|
|
58
|
+
| `--judge-model <model>` | `claude` | Judge 使用的模型 |
|
|
59
|
+
| `--no-judge` | false | 关闭 Judge,只展示各模型原始回答 |
|
|
60
|
+
| `--timeout <sec>` | `120` | 每个模型的超时秒数 |
|
|
61
|
+
| `--dir <path>` | cwd | CLI 工具的工作目录 |
|
|
62
|
+
| `--context-file <file>` | — | 附加文件内容作为上下文(最多 8000 字符)|
|
|
63
|
+
| `--no-save` | false | 不保存结果到磁盘 |
|
|
64
|
+
| `--json` | false | 将完整结果以 JSON 格式输出到 stdout |
|
|
65
|
+
| `--list` | — | 列出最近 20 条历史运行 |
|
|
66
|
+
| `--strict` | — | [v2 占位] 两阶段 Judge + swap-compare |
|
|
96
67
|
|
|
97
68
|
---
|
|
98
69
|
|
|
99
|
-
##
|
|
70
|
+
## 输出
|
|
100
71
|
|
|
101
|
-
|
|
102
|
-
## CODEX (8.2s)
|
|
103
|
-
|
|
104
|
-
<codex 完整回答>
|
|
105
|
-
|
|
106
|
-
---
|
|
72
|
+
**终端**:各模型回答 + Judge 融合报告(Markdown 格式)
|
|
107
73
|
|
|
108
|
-
|
|
74
|
+
**落盘**(`~/ai-outputs/<timestamp>/`):
|
|
109
75
|
|
|
110
|
-
<gemini 完整回答>
|
|
111
76
|
```
|
|
112
|
-
|
|
113
|
-
|
|
77
|
+
~/ai-outputs/2026-04-09T11-23-45-678/
|
|
78
|
+
├── raw/
|
|
79
|
+
│ ├── claude.txt
|
|
80
|
+
│ ├── codex.txt
|
|
81
|
+
│ └── gemini.txt
|
|
82
|
+
├── normalized.json # 三个模型的结构化摘要
|
|
83
|
+
└── report.md # 最终融合报告
|
|
84
|
+
```
|
|
114
85
|
|
|
115
86
|
---
|
|
116
87
|
|
|
117
88
|
## 架构
|
|
118
89
|
|
|
119
90
|
```
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
│ └── format.js Markdown / JSON 渲染
|
|
126
|
-
├── bin/
|
|
127
|
-
│ └── consult CLI 入口
|
|
128
|
-
├── skills/
|
|
129
|
-
│ └── consult/
|
|
130
|
-
│ └── SKILL.md Claude Code skill 引导
|
|
131
|
-
└── docs/
|
|
132
|
-
└── spec.md 设计文档
|
|
91
|
+
runGenerators() # 并发调用三个 CLI,AbortController 超时,Promise.allSettled 容错
|
|
92
|
+
normalizeResults() # 各适配器提取 content / key_claims / assumptions / risks
|
|
93
|
+
runSimpleJudge() # 单次 Judge 调用,只传归一化摘要(非全文),控制 token
|
|
94
|
+
writeRunArtifacts() # 落盘 raw/ + normalized.json + report.md
|
|
95
|
+
runFullJudgePipeline() # [v2 占位] 两阶段 Judge + swap-compare + 抗偏置
|
|
133
96
|
```
|
|
134
97
|
|
|
98
|
+
**Judge prompt 匿名化**:候选标签只用 A / B / C,不暴露 provider 名称,避免模型偏置。
|
|
99
|
+
|
|
135
100
|
---
|
|
136
101
|
|
|
137
|
-
##
|
|
102
|
+
## 成本估算
|
|
103
|
+
|
|
104
|
+
每次运行 = 4 次 API 调用(3 generator + 1 judge):
|
|
138
105
|
|
|
139
|
-
|
|
|
140
|
-
|
|
141
|
-
|
|
|
142
|
-
|
|
|
143
|
-
|
|
|
106
|
+
| 问题复杂度 | 估算成本 |
|
|
107
|
+
|-----------|---------|
|
|
108
|
+
| 简单 | $0.20 – 0.50 |
|
|
109
|
+
| 中等 | $0.50 – 1.00 |
|
|
110
|
+
| 复杂 | $1.00 – 2.00 |
|
|
144
111
|
|
|
145
112
|
---
|
|
146
113
|
|
|
147
|
-
##
|
|
114
|
+
## V2 路线图
|
|
148
115
|
|
|
149
|
-
|
|
116
|
+
1. `--strict`:两阶段 Judge (A+B) + swap-compare + 抗偏置
|
|
117
|
+
2. `--continue`:线程续聊
|
|
118
|
+
3. `--context-file` 智能截断 + git diff 注入
|
|
119
|
+
4. 成本 / token 预算控制器
|
|
120
|
+
5. 更多 provider(Goose、本地模型等)
|
package/bin/braintrust
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
'use strict';
|
|
3
|
+
|
|
4
|
+
// Shim: delegates to src/main.js
|
|
5
|
+
// The symlink ~/.local/bin/braintrust → this file remains unchanged.
|
|
6
|
+
|
|
7
|
+
const { main } = require('../src/main.js');
|
|
8
|
+
|
|
9
|
+
main(process.argv.slice(2)).catch(e => {
|
|
10
|
+
process.stderr.write(`[braintrust error] ${e.message}\n`);
|
|
11
|
+
process.exit(1);
|
|
12
|
+
});
|
package/package.json
CHANGED
|
@@ -1,32 +1,40 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "braintrust-lite",
|
|
3
|
-
"version": "0.1.
|
|
4
|
-
"description": "
|
|
5
|
-
"type": "module",
|
|
3
|
+
"version": "0.1.8",
|
|
4
|
+
"description": "Multi-model AI consultation MCP for Claude Code — runs Claude, Codex, and Gemini in parallel for Judge-style synthesis",
|
|
6
5
|
"bin": {
|
|
7
|
-
"
|
|
8
|
-
"braintrust
|
|
6
|
+
"braintrust-lite": "./src/server.js",
|
|
7
|
+
"braintrust": "./bin/braintrust",
|
|
8
|
+
"braintrust-doctor": "./src/doctor.js"
|
|
9
9
|
},
|
|
10
|
+
"main": "src/main.js",
|
|
11
|
+
"files": [
|
|
12
|
+
"src/",
|
|
13
|
+
"skills/",
|
|
14
|
+
"bin/",
|
|
15
|
+
"README.md"
|
|
16
|
+
],
|
|
10
17
|
"scripts": {
|
|
11
|
-
"
|
|
18
|
+
"test": "node --test src/normalize.test.js"
|
|
12
19
|
},
|
|
13
20
|
"dependencies": {
|
|
14
|
-
"
|
|
21
|
+
"better-sqlite3": "^11.0.0"
|
|
15
22
|
},
|
|
16
23
|
"engines": {
|
|
17
|
-
"node": ">=18"
|
|
24
|
+
"node": ">=18.0.0"
|
|
18
25
|
},
|
|
19
26
|
"keywords": [
|
|
20
27
|
"mcp",
|
|
21
28
|
"claude-code",
|
|
29
|
+
"multi-model",
|
|
30
|
+
"ai",
|
|
22
31
|
"codex",
|
|
23
32
|
"gemini",
|
|
24
|
-
"
|
|
25
|
-
"ai"
|
|
33
|
+
"braintrust"
|
|
26
34
|
],
|
|
27
|
-
"license": "MIT",
|
|
28
35
|
"repository": {
|
|
29
36
|
"type": "git",
|
|
30
|
-
"url": "
|
|
31
|
-
}
|
|
37
|
+
"url": "https://github.com/HongjieRen/braintrust.git"
|
|
38
|
+
},
|
|
39
|
+
"license": "MIT"
|
|
32
40
|
}
|
|
@@ -0,0 +1,213 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: consult
|
|
3
|
+
version: 1.4.0
|
|
4
|
+
description: 在处理规划/设计/架构/调研类任务时,并发调用 codex + gemini + claude 获取多视角,主 Claude 担任 Judge 盲评综合输出。支持渐进式多轮对话和自动更新。
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# consult — 多模型旁路军师
|
|
8
|
+
|
|
9
|
+
## Preamble(每次调用前先执行)
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
_REMOTE_VER=$(curl -sf --max-time 3 "https://raw.githubusercontent.com/HongjieRen/braintrust/main/skills/consult/SKILL.md" | awk '/^version:/ {print $2; exit}')
|
|
13
|
+
_LOCAL_VER=$(awk '/^version:/ {print $2; exit}' ~/.claude/skills/consult/SKILL.md 2>/dev/null)
|
|
14
|
+
[ -n "$_REMOTE_VER" ] && [ "$_REMOTE_VER" != "$_LOCAL_VER" ] && echo "UPGRADE_AVAILABLE $_LOCAL_VER $_REMOTE_VER" || echo "UP_TO_DATE $_LOCAL_VER"
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
**如果输出 `UPGRADE_AVAILABLE <old> <new>`**:静默自动更新,不打断用户:
|
|
18
|
+
```bash
|
|
19
|
+
cp ~/.claude/skills/consult/SKILL.md ~/.claude/skills/consult/SKILL.md.bak 2>/dev/null || true
|
|
20
|
+
curl -fsSL "https://raw.githubusercontent.com/HongjieRen/braintrust/main/skills/consult/SKILL.md" \
|
|
21
|
+
-o ~/.claude/skills/consult/SKILL.md && echo "Updated consult skill $_old → $_new"
|
|
22
|
+
```
|
|
23
|
+
更新完成后继续执行本次任务,在最终回复末尾附一行:`*(consult skill 已自动更新 v{old} → v{new})*`
|
|
24
|
+
|
|
25
|
+
**如果输出 `UP_TO_DATE`**:直接继续,无需提示。
|
|
26
|
+
**如果 curl 失败**:忽略,继续执行,不提示用户。
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## 这是什么
|
|
31
|
+
|
|
32
|
+
`braintrust-lite` 提供的 MCP tool `mcp__braintrust_lite__consult` 会在后台并发调用 **Codex CLI**、**Gemini CLI** 和 **Claude CLI**,把三个顶尖模型的独立回答以匿名形式(Model A/B/C)交回给你。
|
|
33
|
+
|
|
34
|
+
你(主 Claude)负责担任 Judge——盲评内容,提炼共识、标注独特洞见、裁决分歧,输出集大成方案。
|
|
35
|
+
|
|
36
|
+
## 何时使用
|
|
37
|
+
|
|
38
|
+
对以下类型的任务,**在启动子 agent 的同时并行调用 consult**:
|
|
39
|
+
|
|
40
|
+
- 架构选型、技术选型、框架比较
|
|
41
|
+
- 方案设计(新功能、重大重构、系统集成)
|
|
42
|
+
- 复杂 bug 根因分析(多种假设并存时)
|
|
43
|
+
- 非显而易见的技术决策(有明显 trade-off 的场景)
|
|
44
|
+
- 安全或性能评审
|
|
45
|
+
|
|
46
|
+
## 何时跳过
|
|
47
|
+
|
|
48
|
+
- typo 修复、单行改动、简单 rename
|
|
49
|
+
- 只读信息查询(用 Grep / Read 就够)
|
|
50
|
+
- 用户已经明确指定方案,不需要二次意见
|
|
51
|
+
- 已知有标准答案的操作性任务
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## 工作流:单轮
|
|
56
|
+
|
|
57
|
+
```
|
|
58
|
+
1. 发一条 message,同时 parallel call:
|
|
59
|
+
├─ Task(subagent_type=Plan/Explore/..., prompt=X)
|
|
60
|
+
└─ mcp__braintrust_lite__consult(prompt=X, timeout_sec=<见下表>)
|
|
61
|
+
|
|
62
|
+
2. 等两者都返回后,你亲自担任 Judge(盲评流程):
|
|
63
|
+
|
|
64
|
+
步骤一:只看 Model A/B/C 内容,按结构完成评估(见下方 Judge 输出格式)
|
|
65
|
+
|
|
66
|
+
步骤二:读 REVEAL 映射表
|
|
67
|
+
|
|
68
|
+
步骤三:在回复末尾揭晓:
|
|
69
|
+
"揭晓:Model A = Gemini,Model B = Claude,Model C = Codex"
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### Judge 输出格式(必须分节,供多轮渐进加载)
|
|
73
|
+
|
|
74
|
+
每轮 Judge 输出**强制使用以下四节**,不可合并:
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
### VERDICT
|
|
78
|
+
<核心结论,1-3句,永远保留进历史>
|
|
79
|
+
|
|
80
|
+
### REASONING
|
|
81
|
+
<关键推理和裁决依据,有追问才加载>
|
|
82
|
+
|
|
83
|
+
### TRADEOFFS
|
|
84
|
+
<权衡分析、已排除方案及理由,用户问"有没有其他方案"时加载>
|
|
85
|
+
|
|
86
|
+
### OPEN_QUESTIONS
|
|
87
|
+
<未解决的分歧或待确认的假设,用户问"还有什么不确定"时加载>
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## 工作流:多轮对话(会话模式)
|
|
93
|
+
|
|
94
|
+
### 进入信号
|
|
95
|
+
|
|
96
|
+
`/consult` 触发后,第一轮回复顶部显示:
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
┌─ Consult 会话已启动 ──────────────────────────┐
|
|
100
|
+
│ 模型:Codex · Gemini · Claude CLI │
|
|
101
|
+
│ 记忆:Balanced(可用 !brief / !deep 切换) │
|
|
102
|
+
│ 输入问题继续追问,或输入 /done 退出 │
|
|
103
|
+
└────────────────────────────────────────────────┘
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### 每轮状态栏(**每轮回复第一行**,始终显示)
|
|
107
|
+
|
|
108
|
+
每轮 Judge 输出**最开始**,必须先输出这一行状态栏,再输出任何正文:
|
|
109
|
+
|
|
110
|
+
```
|
|
111
|
+
[Consult·R{N} | 3 models | Consensus: {High/Split}]
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
- `R{N}` = 第几轮,帮助用户感知多轮积累
|
|
115
|
+
- `Consensus: Split` 时额外显示一行分歧摘要:`Note: split on <主要分歧点>`
|
|
116
|
+
- 平时无分歧则只显示 `High`,不展开
|
|
117
|
+
- 若模型降级(实际跑了少于 3 个),显示 `⚠ 2/3 models` 代替 `3 models`
|
|
118
|
+
|
|
119
|
+
### 多轮上下文:渐进式加载(核心设计)
|
|
120
|
+
|
|
121
|
+
**设计原则:不预先决定压缩多少,而是根据 follow-up 意图决定加载什么。**
|
|
122
|
+
|
|
123
|
+
每轮结束后维护一个**会话状态对象**(始终随 prompt 携带,~100 token):
|
|
124
|
+
|
|
125
|
+
```
|
|
126
|
+
[Session State]
|
|
127
|
+
Goal: <用户核心目标>
|
|
128
|
+
Constraints: <已确认约束>
|
|
129
|
+
Decisions: <已做决策及理由>
|
|
130
|
+
Rejected: <已排除选项>
|
|
131
|
+
Open: <未解决问题>
|
|
132
|
+
Current best: <当前推荐方案一句话>
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
历史内容**按意图懒加载**,不机械按轮次:
|
|
136
|
+
|
|
137
|
+
| follow-up 意图 | 加载的历史内容 |
|
|
138
|
+
|---------------|--------------|
|
|
139
|
+
| 普通追问、深入某方向 | Session State + 所有历史 VERDICT |
|
|
140
|
+
| "为什么这样判断" | + 最近1轮 REASONING |
|
|
141
|
+
| "有没有其他方案" | + 最近1轮 TRADEOFFS |
|
|
142
|
+
| "还有什么不确定" | + 最近1轮 OPEN_QUESTIONS |
|
|
143
|
+
| "刚才某模型说的那个点" | + 按需检索原文片段(Model A/B/C 原始回答存档备查) |
|
|
144
|
+
|
|
145
|
+
历史 VERDICT 全部保留(每条 ~50 token),其余节只保留最近1-2轮,更老的丢弃。
|
|
146
|
+
|
|
147
|
+
### 自动降级
|
|
148
|
+
|
|
149
|
+
用户回复是简单确认时("好的"、"谢谢"、"明白了"等),**不触发三模型并发**,由主 Claude 直接响应,节省成本和延迟。
|
|
150
|
+
|
|
151
|
+
判断标准:用户回复 < 20 字且不含实质性新问题。
|
|
152
|
+
|
|
153
|
+
### 多轮终止条件
|
|
154
|
+
|
|
155
|
+
- 用户输入 `/done`(或 `!stop`)
|
|
156
|
+
- 用户明确表示满意("好了"、"没问题了")
|
|
157
|
+
- 用户切换到无关新话题
|
|
158
|
+
- 已进行 5 轮(自动退出,告知用户可重新 `/consult`)
|
|
159
|
+
|
|
160
|
+
退出时显示:`── Consult 会话结束(共 {N} 轮)──`
|
|
161
|
+
|
|
162
|
+
### 用户控制命令
|
|
163
|
+
|
|
164
|
+
```
|
|
165
|
+
!brief 切换到精简记忆(只带 VERDICT,适合快速迭代)
|
|
166
|
+
!deep 切换到完整记忆(带最近1轮 REASONING + TRADEOFFS,适合复杂设计)
|
|
167
|
+
/done 退出 Consult 会话模式(!stop 同效)
|
|
168
|
+
!deltas 展开本轮三模型核心主张各一句(不显示原文全文)
|
|
169
|
+
!raw 展开本轮三模型完整原始回答
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
---
|
|
173
|
+
|
|
174
|
+
## consult tool 参数
|
|
175
|
+
|
|
176
|
+
```
|
|
177
|
+
prompt (必须) 问题,建议精炼、自包含,含必要上下文
|
|
178
|
+
only (可选) "codex" | "gemini" | "claude" — 只调用一个
|
|
179
|
+
skip (可选) ["codex"] | ["gemini"] | ["claude"] — 跳过某个
|
|
180
|
+
timeout_sec (可选) 每个模型超时秒数,默认 90;传 0 = 不限时等待完成
|
|
181
|
+
blind (可选) 默认 true;传 false 可直接看真实模型名称
|
|
182
|
+
cwd (可选) 子进程工作目录
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
## timeout 选择策略
|
|
186
|
+
|
|
187
|
+
**你(主 Claude)负责决定 timeout_sec:**
|
|
188
|
+
|
|
189
|
+
| 任务类型 | timeout_sec |
|
|
190
|
+
|---------|------------|
|
|
191
|
+
| 深度调研、市场分析、可行性研究 | **0**(不限时) |
|
|
192
|
+
| 架构设计、复杂方案对比 | **0**(不限时) |
|
|
193
|
+
| 代码审查、技术选型 | 180 |
|
|
194
|
+
| 简单问答、快速决策 | 90(默认) |
|
|
195
|
+
|
|
196
|
+
调研类任务一律传 `timeout_sec: 0`。
|
|
197
|
+
|
|
198
|
+
## 成本与延迟
|
|
199
|
+
|
|
200
|
+
- 每次 consult = 3 次 API 调用(codex + gemini + claude)
|
|
201
|
+
- 延迟 = `max(三者响应时间)`(并发)
|
|
202
|
+
- 简单问题 ~$0.05–0.20,中等 ~$0.20–0.50
|
|
203
|
+
- 自动降级(简单确认)= 0 次额外 API 调用
|
|
204
|
+
|
|
205
|
+
## 终端 fallback
|
|
206
|
+
|
|
207
|
+
```bash
|
|
208
|
+
consult "你的问题"
|
|
209
|
+
consult --only codex "快速问题"
|
|
210
|
+
consult --timeout 0 "深度调研问题"
|
|
211
|
+
consult --dir /your/project "review this project"
|
|
212
|
+
cat file.ts | consult "review this code"
|
|
213
|
+
```
|
package/src/config.js
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
const { join } = require('path');
|
|
4
|
+
|
|
5
|
+
const PROJECT_ROOT = join(__dirname, '..');
|
|
6
|
+
const OUTPUT_DIR = join(PROJECT_ROOT, 'ai-outputs');
|
|
7
|
+
const STATE_DIR = join(OUTPUT_DIR, '.state');
|
|
8
|
+
const DB_PATH = join(STATE_DIR, 'braintrust.sqlite');
|
|
9
|
+
const POLICY_PATH = join(STATE_DIR, 'policy.json');
|
|
10
|
+
const REFLECTOR_LOG = join(STATE_DIR, 'reflector.log');
|
|
11
|
+
|
|
12
|
+
const DEFAULT_TIMEOUT_S = 120;
|
|
13
|
+
const DEFAULT_JUDGE_MODEL = 'claude';
|
|
14
|
+
const DEFAULT_MEMORY_K = 3;
|
|
15
|
+
const MAX_CONTEXT_CHARS = 30000;
|
|
16
|
+
const CONTEXT_FILE_MAX = 8000;
|
|
17
|
+
|
|
18
|
+
// Memory injection hard limits (chars)
|
|
19
|
+
const MEMORY_INJECT_LIMIT = 1500;
|
|
20
|
+
const LESSONS_INJECT_LIMIT = 600;
|
|
21
|
+
const SKILLS_INJECT_LIMIT = 800;
|
|
22
|
+
|
|
23
|
+
// Novelty check threshold: cosine similarity above this → prompt reuse
|
|
24
|
+
const NOVELTY_THRESHOLD = 0.9;
|
|
25
|
+
|
|
26
|
+
// Critique-revise disagreement threshold
|
|
27
|
+
const DISAGREE_THRESHOLD = 0.5;
|
|
28
|
+
|
|
29
|
+
// Economy mode: disable all extra LLM calls
|
|
30
|
+
const ECONOMY = process.env.BRAINTRUST_ECONOMY === '1';
|
|
31
|
+
|
|
32
|
+
// Reflector model: codex with gpt-5.4-mini.
|
|
33
|
+
// Chosen over haiku/flash for better Chinese text quality.
|
|
34
|
+
// Must differ from the default judge model (claude) to avoid self-evaluation bias.
|
|
35
|
+
const REFLECTOR_MODEL = 'gpt-5.4-mini';
|
|
36
|
+
const REFLECTOR_CMD = 'codex';
|
|
37
|
+
const REFLECTOR_ARGS_PREFIX = ['exec', '--json', '--skip-git-repo-check', '--ephemeral', '-m', REFLECTOR_MODEL];
|
|
38
|
+
|
|
39
|
+
module.exports = {
|
|
40
|
+
PROJECT_ROOT,
|
|
41
|
+
OUTPUT_DIR,
|
|
42
|
+
STATE_DIR,
|
|
43
|
+
DB_PATH,
|
|
44
|
+
POLICY_PATH,
|
|
45
|
+
REFLECTOR_LOG,
|
|
46
|
+
DEFAULT_TIMEOUT_S,
|
|
47
|
+
DEFAULT_JUDGE_MODEL,
|
|
48
|
+
DEFAULT_MEMORY_K,
|
|
49
|
+
MAX_CONTEXT_CHARS,
|
|
50
|
+
CONTEXT_FILE_MAX,
|
|
51
|
+
MEMORY_INJECT_LIMIT,
|
|
52
|
+
LESSONS_INJECT_LIMIT,
|
|
53
|
+
SKILLS_INJECT_LIMIT,
|
|
54
|
+
NOVELTY_THRESHOLD,
|
|
55
|
+
DISAGREE_THRESHOLD,
|
|
56
|
+
ECONOMY,
|
|
57
|
+
REFLECTOR_MODEL,
|
|
58
|
+
REFLECTOR_CMD,
|
|
59
|
+
REFLECTOR_ARGS_PREFIX,
|
|
60
|
+
};
|
package/src/doctor.js
ADDED
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
'use strict';
|
|
3
|
+
|
|
4
|
+
const { execFileSync, spawnSync } = require('child_process');
|
|
5
|
+
const { existsSync, readFileSync } = require('fs');
|
|
6
|
+
const { join } = require('path');
|
|
7
|
+
const { version: PKG_VERSION } = require('../package.json');
|
|
8
|
+
|
|
9
|
+
const GREEN = '\x1b[32m✓\x1b[0m';
|
|
10
|
+
const RED = '\x1b[31m✗\x1b[0m';
|
|
11
|
+
const WARN = '\x1b[33m!\x1b[0m';
|
|
12
|
+
|
|
13
|
+
function check(label, ok, detail) {
|
|
14
|
+
const icon = ok === true ? GREEN : ok === 'warn' ? WARN : RED;
|
|
15
|
+
const line = ` ${icon} ${label.padEnd(28)} ${detail || ''}`;
|
|
16
|
+
console.log(line);
|
|
17
|
+
return ok === true;
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
function getVersion(cmd, args) {
|
|
21
|
+
try {
|
|
22
|
+
const result = spawnSync(cmd, args, { timeout: 5000, encoding: 'utf8' });
|
|
23
|
+
if (result.status === 0) {
|
|
24
|
+
return (result.stdout || result.stderr || '').split('\n')[0].trim().slice(0, 40);
|
|
25
|
+
}
|
|
26
|
+
return null;
|
|
27
|
+
} catch {
|
|
28
|
+
return null;
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
function getSkillVersion(skillPath) {
|
|
33
|
+
try {
|
|
34
|
+
const content = readFileSync(skillPath, 'utf8');
|
|
35
|
+
const m = content.match(/^version:\s*(.+)$/m);
|
|
36
|
+
return m ? m[1].trim() : 'unknown';
|
|
37
|
+
} catch {
|
|
38
|
+
return null;
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
function checkMcpServer() {
|
|
43
|
+
// Probe MCP server: send initialize, expect a valid JSON-RPC response
|
|
44
|
+
const serverPath = join(__dirname, 'server.js');
|
|
45
|
+
if (!existsSync(serverPath)) return { ok: false, detail: 'src/server.js not found' };
|
|
46
|
+
|
|
47
|
+
try {
|
|
48
|
+
const msg = JSON.stringify({
|
|
49
|
+
jsonrpc: '2.0', id: 1, method: 'initialize',
|
|
50
|
+
params: { protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 'doctor', version: '0' } },
|
|
51
|
+
});
|
|
52
|
+
const result = spawnSync(process.execPath, [serverPath], {
|
|
53
|
+
input: msg + '\n',
|
|
54
|
+
timeout: 5000,
|
|
55
|
+
encoding: 'utf8',
|
|
56
|
+
});
|
|
57
|
+
const line = (result.stdout || '').split('\n').find(l => l.trim().startsWith('{'));
|
|
58
|
+
if (!line) return { ok: false, detail: 'no JSON response from server' };
|
|
59
|
+
const resp = JSON.parse(line);
|
|
60
|
+
if (resp.result && resp.result.serverInfo) {
|
|
61
|
+
return { ok: true, detail: `v${resp.result.serverInfo.version}` };
|
|
62
|
+
}
|
|
63
|
+
return { ok: false, detail: 'unexpected response shape' };
|
|
64
|
+
} catch (err) {
|
|
65
|
+
return { ok: false, detail: err.message.slice(0, 60) };
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
|
|
69
|
+
function main() {
|
|
70
|
+
console.log(`\nbraintrust doctor (package v${PKG_VERSION})\n`);
|
|
71
|
+
|
|
72
|
+
let allOk = true;
|
|
73
|
+
|
|
74
|
+
// ── CLI tools ──────────────────────────────────────────────────────────────
|
|
75
|
+
console.log('CLI tools:');
|
|
76
|
+
for (const [cmd, vArgs, installHint] of [
|
|
77
|
+
['claude', ['--version'], 'https://claude.ai/download'],
|
|
78
|
+
['codex', ['--version'], 'npm i -g @openai/codex'],
|
|
79
|
+
['gemini', ['--version'], 'npm i -g @google/gemini-cli'],
|
|
80
|
+
]) {
|
|
81
|
+
const ver = getVersion(cmd, vArgs);
|
|
82
|
+
if (ver) {
|
|
83
|
+
check(cmd, true, ver);
|
|
84
|
+
} else {
|
|
85
|
+
check(cmd, false, `not found — ${installHint}`);
|
|
86
|
+
allOk = false;
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
// ── MCP server ─────────────────────────────────────────────────────────────
|
|
91
|
+
console.log('\nMCP server:');
|
|
92
|
+
const mcp = checkMcpServer();
|
|
93
|
+
if (!check('braintrust-lite server', mcp.ok, mcp.detail)) allOk = false;
|
|
94
|
+
|
|
95
|
+
// ── Skill ──────────────────────────────────────────────────────────────────
|
|
96
|
+
console.log('\nConsult skill:');
|
|
97
|
+
const skillPath = join(process.env.HOME || '~', '.claude', 'skills', 'consult', 'SKILL.md');
|
|
98
|
+
const skillVer = getSkillVersion(skillPath);
|
|
99
|
+
if (skillVer) {
|
|
100
|
+
check('SKILL.md installed', true, `v${skillVer} at ${skillPath}`);
|
|
101
|
+
} else {
|
|
102
|
+
check('SKILL.md installed', false, `not found at ${skillPath}`);
|
|
103
|
+
allOk = false;
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
const bakPath = skillPath + '.bak';
|
|
107
|
+
check('SKILL.md.bak exists', existsSync(bakPath) ? 'warn' : 'warn',
|
|
108
|
+
existsSync(bakPath) ? 'backup present' : 'no backup yet (created on first auto-update)');
|
|
109
|
+
|
|
110
|
+
// ── Summary ────────────────────────────────────────────────────────────────
|
|
111
|
+
console.log();
|
|
112
|
+
if (allOk) {
|
|
113
|
+
console.log(' \x1b[32mAll checks passed — braintrust is ready.\x1b[0m\n');
|
|
114
|
+
} else {
|
|
115
|
+
console.log(' \x1b[31mSome checks failed — fix the issues above before using braintrust.\x1b[0m\n');
|
|
116
|
+
process.exit(1);
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
main();
|