oh-langfuse 0.1.19 → 0.1.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +70 -119
- package/langfuse_hook.py +72 -51
- package/package.json +1 -1
- package/scripts/langfuse-check.mjs +161 -90
- package/scripts/langfuse-setup.mjs +16 -15
- package/scripts/real-self-verify.mjs +414 -0
package/README.md
CHANGED
|
@@ -1,175 +1,126 @@
|
|
|
1
1
|
# oh-langfuse
|
|
2
2
|
|
|
3
|
-
`oh-langfuse`
|
|
4
|
-
Claude Code、OpenCode 和 Codex。它提供终端安装向导、直接 setup/check
|
|
5
|
-
命令,以及不同工具所需的安装脚本和校验脚本。
|
|
3
|
+
`oh-langfuse` 是用于给 Claude Code、OpenCode 和 Codex 配置 Langfuse 追踪的命令行工具。它提供交互式安装向导,也支持 `setup` / `check` 直接命令,方便在用户机器上安装、修复和校验配置。
|
|
6
4
|
|
|
7
|
-
|
|
5
|
+
当前 npm 版本:`0.1.20`
|
|
8
6
|
|
|
9
|
-
|
|
7
|
+
## 能做什么
|
|
10
8
|
|
|
11
|
-
-
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
-
|
|
9
|
+
- 给 Claude Code 安装 `Stop` hook,把会话事件写入 Langfuse。
|
|
10
|
+
- 给 OpenCode 安装并 patch `opencode-plugin-langfuse`,开启 OpenTelemetry。
|
|
11
|
+
- 给 Codex 安装 `notify` hook,增量读取 session JSONL 并写入 Langfuse。
|
|
12
|
+
- 安装完成后自动执行对应 `check`,尽早发现配置缺失。
|
|
13
|
+
- 校验员工号格式,必须匹配 `^[a-z](?:\d{8}|wx\d{7})$`,例如 `h00613222` 或 `hwx1234567`。
|
|
15
14
|
|
|
16
|
-
|
|
17
|
-
`oh-aicoding-tool` 调用。
|
|
15
|
+
## 快速使用
|
|
18
16
|
|
|
19
|
-
|
|
17
|
+
建议始终带 `@latest`,避免 npx 使用本地旧缓存:
|
|
20
18
|
|
|
21
|
-
|
|
19
|
+
```bash
|
|
20
|
+
npx oh-langfuse@latest
|
|
21
|
+
npx oh-langfuse@latest setup
|
|
22
|
+
npx oh-langfuse@latest check
|
|
23
|
+
```
|
|
22
24
|
|
|
23
|
-
|
|
24
|
-
| --- | --- |
|
|
25
|
-
| Claude Code | Node.js、npm、Python、pip |
|
|
26
|
-
| OpenCode | Node.js、npm、OpenCode CLI |
|
|
27
|
-
| Codex | Node.js、npm、Python、pip |
|
|
25
|
+
也可以指定目标:
|
|
28
26
|
|
|
29
|
-
|
|
27
|
+
```bash
|
|
28
|
+
npx oh-langfuse@latest setup claude
|
|
29
|
+
npx oh-langfuse@latest setup opencode
|
|
30
|
+
npx oh-langfuse@latest setup codex
|
|
30
31
|
|
|
31
|
-
|
|
32
|
+
npx oh-langfuse@latest check claude
|
|
33
|
+
npx oh-langfuse@latest check opencode
|
|
34
|
+
npx oh-langfuse@latest check codex
|
|
35
|
+
```
|
|
32
36
|
|
|
33
|
-
|
|
37
|
+
本地开发运行:
|
|
34
38
|
|
|
35
39
|
```bash
|
|
36
40
|
npm install
|
|
37
41
|
npm start
|
|
38
42
|
```
|
|
39
43
|
|
|
40
|
-
|
|
44
|
+
## 交互式菜单
|
|
41
45
|
|
|
42
|
-
|
|
43
|
-
npx oh-langfuse@latest
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
建议始终带 `@latest` 运行,避免 npx 使用本地缓存里的旧版本。
|
|
46
|
+
运行 `npx oh-langfuse@latest` 会打开交互式菜单:
|
|
47
47
|
|
|
48
|
-
|
|
48
|
+
- `Up` / `Down` 移动选项。
|
|
49
|
+
- 数字键快速定位。
|
|
50
|
+
- `Space` 勾选或取消。
|
|
51
|
+
- `Enter` 确认。
|
|
52
|
+
- `q` 或 `Esc` 退出。
|
|
49
53
|
|
|
50
|
-
|
|
51
|
-
- 数字键:快速选择;
|
|
52
|
-
- `Space`:多选切换;
|
|
53
|
-
- `Enter`:确认;
|
|
54
|
-
- `q` 或 `Esc`:退出。
|
|
54
|
+
多选菜单默认不勾选任何目标,避免误装。终端不支持原始按键输入时,会自动降级为数字输入模式。
|
|
55
55
|
|
|
56
|
-
|
|
56
|
+
## OpenCode 说明
|
|
57
57
|
|
|
58
|
-
|
|
58
|
+
OpenCode 安装会写入:
|
|
59
59
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
oh-langfuse setup codex
|
|
66
|
-
oh-langfuse check
|
|
67
|
-
oh-langfuse check environment
|
|
68
|
-
oh-langfuse check claude
|
|
69
|
-
oh-langfuse check opencode
|
|
70
|
-
oh-langfuse check codex
|
|
71
|
-
```
|
|
60
|
+
- `~/.config/opencode/opencode.json`
|
|
61
|
+
- `~/.config/opencode/plugins/opencode-plugin-langfuse`
|
|
62
|
+
- `~/.config/opencode-plugin-langfuse/config.json`
|
|
63
|
+
- Windows 下的 `~/.config/opencode/launch-opencode-langfuse.cmd`
|
|
64
|
+
- Linux/macOS 下的 `~/.config/opencode/launch-opencode-langfuse.sh`
|
|
72
65
|
|
|
73
|
-
|
|
66
|
+
Windows 下安装器会尝试写入用户级 `LANGFUSE_PUBLIC_KEY`、`LANGFUSE_SECRET_KEY`、`LANGFUSE_BASEURL`。这些变量对当前已经打开的终端不会立即生效,需要新开终端,或者使用生成的 launcher 启动 OpenCode。
|
|
74
67
|
|
|
75
|
-
|
|
76
|
-
oh-langfuse setup --dry-run
|
|
77
|
-
```
|
|
68
|
+
`check opencode` 已按实际情况处理 Windows:如果当前终端还看不到 `LANGFUSE_*`,但用户级环境变量已写入,或 launcher 已生成,会显示 OpenCode 仍有可用的 Langfuse 环境变量来源,不再把这种情况误判为安装失败。
|
|
78
69
|
|
|
79
|
-
|
|
70
|
+
WSL 用户请直接在 WSL shell 中安装和检查:
|
|
80
71
|
|
|
81
72
|
```bash
|
|
82
|
-
|
|
83
|
-
npm run opencode:setup
|
|
84
|
-
npm run codex:setup
|
|
85
|
-
npx oh-langfuse@latest check claude
|
|
73
|
+
npx oh-langfuse@latest setup opencode --userId=h00613222 --yes
|
|
86
74
|
npx oh-langfuse@latest check opencode
|
|
87
|
-
npx oh-langfuse@latest check codex
|
|
88
75
|
```
|
|
89
76
|
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
### Claude Code
|
|
77
|
+
不会做 Windows 到 WSL 的转发;在 WSL 中运行时,配置写入 WSL 用户自己的 `$HOME/.config/opencode`。
|
|
93
78
|
|
|
94
|
-
|
|
95
|
-
`langfuse`,并把 Langfuse 环境变量和 `Stop` hook 合并写入
|
|
96
|
-
`~/.claude/settings.json`。
|
|
79
|
+
## Claude Code 说明
|
|
97
80
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
安装并 patch `opencode-plugin-langfuse`,在
|
|
101
|
-
`~/.config/opencode/opencode.json` 中启用 OpenTelemetry,写入插件用户配置,
|
|
102
|
-
并可写入用户级 `LANGFUSE_*` 环境变量。Windows 下会生成
|
|
103
|
-
`launch-opencode-langfuse.cmd`,Linux/macOS 下会生成
|
|
104
|
-
`launch-opencode-langfuse.sh`,用于在 shell 环境变量未生效时显式带
|
|
105
|
-
Langfuse 配置启动 OpenCode。安装器可以读取带注释或尾随逗号的
|
|
106
|
-
`opencode.json`,合并配置后会写回标准 JSON。
|
|
107
|
-
|
|
108
|
-
如果你在 WSL 里运行 OpenCode,就在 WSL shell 里直接执行安装和检查命令。
|
|
109
|
-
此时配置会写入 WSL 用户自己的 `$HOME/.config/opencode`:
|
|
110
|
-
|
|
111
|
-
```bash
|
|
112
|
-
npx oh-langfuse@latest setup opencode --userId=h00613222 --yes
|
|
113
|
-
npx oh-langfuse@latest check opencode
|
|
114
|
-
```
|
|
81
|
+
Claude Code 安装会:
|
|
115
82
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
`~/.
|
|
83
|
+
- 安装 `langfuse_hook.py`
|
|
84
|
+
- 创建 `~/.claude/langfuse-venv`
|
|
85
|
+
- 安装 Python 包 `langfuse`
|
|
86
|
+
- 合并更新 `~/.claude/settings.json`
|
|
87
|
+
- 配置 `Stop` hook
|
|
120
88
|
|
|
121
|
-
|
|
122
|
-
再写回标准 JSON。
|
|
89
|
+
Linux 如果缺少 venv 支持,请根据提示安装 `python3-venv` 或对应 Python 版本的 venv 包。
|
|
123
90
|
|
|
124
|
-
|
|
91
|
+
## Codex 说明
|
|
125
92
|
|
|
126
|
-
|
|
127
|
-
包 `langfuse`,把 Langfuse 凭据写入 `~/.codex/langfuse/config.json`,并更新
|
|
128
|
-
`~/.codex/config.toml` 顶层 `notify` 命令。安装后需要重启 Codex。
|
|
93
|
+
Codex 安装会:
|
|
129
94
|
|
|
130
|
-
|
|
131
|
-
`~/.codex/
|
|
132
|
-
|
|
95
|
+
- 安装 `codex_langfuse_notify.py`
|
|
96
|
+
- 创建 `~/.codex/langfuse-venv`
|
|
97
|
+
- 写入 `~/.codex/langfuse/config.json`
|
|
98
|
+
- 更新 `~/.codex/config.toml` 顶层 `notify`
|
|
133
99
|
|
|
134
|
-
|
|
100
|
+
配置后需要重启 Codex。notify hook 会增量读取 `~/.codex/sessions/**/*.jsonl`,把新的 user、assistant、tool、token 事件转换为 Langfuse observation,并把读取偏移记录到 `~/.codex/langfuse/state.json`。
|
|
135
101
|
|
|
136
|
-
|
|
102
|
+
## 环境变量
|
|
137
103
|
|
|
138
104
|
| 变量 | 作用 |
|
|
139
105
|
| --- | --- |
|
|
140
106
|
| `LANGFUSE_BASEURL` / `LANGFUSE_HOST` | Langfuse 服务地址 |
|
|
141
107
|
| `LANGFUSE_PUBLIC_KEY` | Langfuse public key |
|
|
142
108
|
| `LANGFUSE_SECRET_KEY` | Langfuse secret key |
|
|
143
|
-
| `LANGFUSE_USER_ID` / `CC_USER_ID` |
|
|
109
|
+
| `LANGFUSE_USER_ID` / `CC_USER_ID` | 用户标识,必须匹配员工号规则 |
|
|
144
110
|
| `CODEX_HOME` | 自定义 Codex home 目录 |
|
|
145
|
-
| `OPENCODE_SKIP_PLUGIN_INSTALL` | OpenCode
|
|
146
|
-
| `OPENCODE_NPM_REGISTRY` | OpenCode 插件 npm
|
|
147
|
-
|
|
148
|
-
交互界面不会明文打印 secret key。
|
|
111
|
+
| `OPENCODE_SKIP_PLUGIN_INSTALL` | 跳过 OpenCode npm 插件安装 |
|
|
112
|
+
| `OPENCODE_NPM_REGISTRY` | OpenCode 插件 npm 安装源 |
|
|
149
113
|
|
|
150
|
-
|
|
114
|
+
Secret key 不会在交互式界面中明文展示。
|
|
151
115
|
|
|
152
|
-
|
|
153
|
-
| --- | --- |
|
|
154
|
-
| `bin/cli.js` | 交互式和直接命令入口 |
|
|
155
|
-
| `scripts/langfuse-setup.mjs` | Claude Code Langfuse 安装 |
|
|
156
|
-
| `scripts/opencode-langfuse-setup.mjs` | OpenCode Langfuse 安装 |
|
|
157
|
-
| `scripts/codex-langfuse-setup.mjs` | Codex Langfuse 安装 |
|
|
158
|
-
| `scripts/*-check.mjs` | 配置校验脚本 |
|
|
159
|
-
| `langfuse_hook.py` | Claude Code hook exporter |
|
|
160
|
-
| `codex_langfuse_notify.py` | Codex notify exporter |
|
|
161
|
-
|
|
162
|
-
## 发布与维护
|
|
163
|
-
|
|
164
|
-
- npm 包名:`oh-langfuse`
|
|
165
|
-
- 主命令:`oh-langfuse`
|
|
166
|
-
- 兼容命令:`code-tool-langfuse`
|
|
167
|
-
- 发布前建议运行:
|
|
116
|
+
## 常用维护命令
|
|
168
117
|
|
|
169
118
|
```bash
|
|
170
119
|
npm run check
|
|
171
120
|
npm pack --dry-run
|
|
172
121
|
```
|
|
173
122
|
|
|
174
|
-
|
|
175
|
-
|
|
123
|
+
发布包暴露两个命令:
|
|
124
|
+
|
|
125
|
+
- `oh-langfuse`
|
|
126
|
+
- `code-tool-langfuse`,兼容旧入口
|
package/langfuse_hook.py
CHANGED
|
@@ -15,10 +15,18 @@ from pathlib import Path
|
|
|
15
15
|
from typing import Any, Dict, List, Optional, Tuple
|
|
16
16
|
|
|
17
17
|
# --- Langfuse import (fail-open) ---
|
|
18
|
-
try:
|
|
19
|
-
from langfuse import Langfuse, propagate_attributes
|
|
20
|
-
except Exception:
|
|
21
|
-
|
|
18
|
+
try:
|
|
19
|
+
from langfuse import Langfuse, propagate_attributes
|
|
20
|
+
except Exception as e:
|
|
21
|
+
try:
|
|
22
|
+
state_dir = Path.home() / ".claude" / "state"
|
|
23
|
+
state_dir.mkdir(parents=True, exist_ok=True)
|
|
24
|
+
with open(state_dir / "langfuse_hook.log", "a", encoding="utf-8") as f:
|
|
25
|
+
ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
|
26
|
+
f.write(f"{ts} [ERROR] Failed to import langfuse package: {e}\n")
|
|
27
|
+
except Exception:
|
|
28
|
+
pass
|
|
29
|
+
sys.exit(0)
|
|
22
30
|
|
|
23
31
|
# --- Paths ---
|
|
24
32
|
STATE_DIR = Path.home() / ".claude" / "state"
|
|
@@ -325,22 +333,32 @@ def read_new_jsonl(transcript_path: Path, ss: SessionState) -> Tuple[List[Dict[s
|
|
|
325
333
|
text = chunk.decode(errors="replace")
|
|
326
334
|
|
|
327
335
|
combined = ss.buffer + text
|
|
328
|
-
lines = combined.split("\n")
|
|
329
|
-
|
|
330
|
-
ss.
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
line = line.strip()
|
|
336
|
+
lines = combined.split("\n")
|
|
337
|
+
tail = lines[-1]
|
|
338
|
+
ss.offset = new_offset
|
|
339
|
+
|
|
340
|
+
msgs: List[Dict[str, Any]] = []
|
|
341
|
+
for line in lines[:-1]:
|
|
342
|
+
line = line.strip()
|
|
336
343
|
if not line:
|
|
337
344
|
continue
|
|
338
345
|
try:
|
|
339
346
|
msgs.append(json.loads(line))
|
|
340
|
-
except Exception:
|
|
341
|
-
continue
|
|
342
|
-
|
|
343
|
-
|
|
347
|
+
except Exception:
|
|
348
|
+
continue
|
|
349
|
+
|
|
350
|
+
tail = tail.strip()
|
|
351
|
+
if tail:
|
|
352
|
+
try:
|
|
353
|
+
msgs.append(json.loads(tail))
|
|
354
|
+
ss.buffer = ""
|
|
355
|
+
except Exception:
|
|
356
|
+
# Keep a genuinely partial final line for the next hook run.
|
|
357
|
+
ss.buffer = tail
|
|
358
|
+
else:
|
|
359
|
+
ss.buffer = ""
|
|
360
|
+
|
|
361
|
+
return msgs, ss
|
|
344
362
|
|
|
345
363
|
# ----------------- Turn assembly -----------------
|
|
346
364
|
@dataclass
|
|
@@ -520,32 +538,34 @@ def main() -> int:
|
|
|
520
538
|
start = time.time()
|
|
521
539
|
debug("Hook started")
|
|
522
540
|
|
|
523
|
-
if os.environ.get("TRACE_TO_LANGFUSE", "").lower() != "true":
|
|
524
|
-
return 0
|
|
525
|
-
|
|
526
|
-
public_key = os.environ.get("CC_LANGFUSE_PUBLIC_KEY") or os.environ.get("LANGFUSE_PUBLIC_KEY")
|
|
527
|
-
secret_key = os.environ.get("CC_LANGFUSE_SECRET_KEY") or os.environ.get("LANGFUSE_SECRET_KEY")
|
|
528
|
-
host = os.environ.get("CC_LANGFUSE_BASE_URL") or os.environ.get("LANGFUSE_BASEURL") or "https://cloud.langfuse.com"
|
|
529
|
-
|
|
530
|
-
if not public_key or not secret_key:
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
541
|
+
if os.environ.get("TRACE_TO_LANGFUSE", "").lower() != "true":
|
|
542
|
+
return 0
|
|
543
|
+
|
|
544
|
+
public_key = os.environ.get("CC_LANGFUSE_PUBLIC_KEY") or os.environ.get("LANGFUSE_PUBLIC_KEY")
|
|
545
|
+
secret_key = os.environ.get("CC_LANGFUSE_SECRET_KEY") or os.environ.get("LANGFUSE_SECRET_KEY")
|
|
546
|
+
host = os.environ.get("CC_LANGFUSE_BASE_URL") or os.environ.get("LANGFUSE_BASEURL") or "https://cloud.langfuse.com"
|
|
547
|
+
|
|
548
|
+
if not public_key or not secret_key:
|
|
549
|
+
warn("Missing Langfuse public/secret key in hook environment; exiting.")
|
|
550
|
+
return 0
|
|
551
|
+
|
|
552
|
+
payload = read_hook_payload()
|
|
553
|
+
session_id, transcript_path, user_id = extract_session_transcript_and_user(payload)
|
|
554
|
+
|
|
555
|
+
if not session_id or not transcript_path:
|
|
556
|
+
# No structured payload; fail open (do not guess)
|
|
557
|
+
warn("Missing session_id or transcript_path from hook payload; exiting.")
|
|
558
|
+
return 0
|
|
559
|
+
|
|
560
|
+
if not transcript_path.exists():
|
|
561
|
+
warn(f"Transcript path does not exist: {transcript_path}")
|
|
562
|
+
return 0
|
|
563
|
+
|
|
564
|
+
try:
|
|
565
|
+
langfuse = Langfuse(public_key=public_key, secret_key=secret_key, host=host)
|
|
566
|
+
except Exception as e:
|
|
567
|
+
warn(f"Langfuse init failed: {e}")
|
|
568
|
+
return 0
|
|
549
569
|
|
|
550
570
|
try:
|
|
551
571
|
with FileLock(LOCK_FILE):
|
|
@@ -570,20 +590,21 @@ def main() -> int:
|
|
|
570
590
|
for t in turns:
|
|
571
591
|
emitted += 1
|
|
572
592
|
turn_num = ss.turn_count + emitted
|
|
573
|
-
try:
|
|
574
|
-
emit_turn(langfuse, session_id, user_id, turn_num, t, transcript_path)
|
|
575
|
-
except Exception as e:
|
|
576
|
-
|
|
577
|
-
# continue emitting other turns
|
|
593
|
+
try:
|
|
594
|
+
emit_turn(langfuse, session_id, user_id, turn_num, t, transcript_path)
|
|
595
|
+
except Exception as e:
|
|
596
|
+
warn(f"emit_turn failed: {e}")
|
|
597
|
+
# continue emitting other turns
|
|
578
598
|
|
|
579
599
|
ss.turn_count += emitted
|
|
580
600
|
write_session_state(state, key, ss)
|
|
581
601
|
save_state(state)
|
|
582
602
|
|
|
583
|
-
try:
|
|
584
|
-
langfuse.flush()
|
|
585
|
-
except Exception:
|
|
586
|
-
|
|
603
|
+
try:
|
|
604
|
+
langfuse.flush()
|
|
605
|
+
except Exception as e:
|
|
606
|
+
warn(f"Langfuse flush failed: {e}")
|
|
607
|
+
pass
|
|
587
608
|
|
|
588
609
|
dur = time.time() - start
|
|
589
610
|
info(f"Processed {emitted} turns in {dur:.2f}s (session={session_id})")
|
package/package.json
CHANGED
|
@@ -1,90 +1,161 @@
|
|
|
1
|
-
import fs from "node:fs";
|
|
2
|
-
import path from "node:path";
|
|
3
|
-
import os from "node:os";
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
const
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
"
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
const
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
const
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
}
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
1
|
+
import fs from "node:fs";
|
|
2
|
+
import path from "node:path";
|
|
3
|
+
import os from "node:os";
|
|
4
|
+
import { spawnSync } from "node:child_process";
|
|
5
|
+
|
|
6
|
+
function readJsonIfExists(p) {
|
|
7
|
+
if (!fs.existsSync(p)) return null;
|
|
8
|
+
const txt = fs.readFileSync(p, "utf8");
|
|
9
|
+
if (!txt.trim()) return null;
|
|
10
|
+
return JSON.parse(txt);
|
|
11
|
+
}
|
|
12
|
+
|
|
13
|
+
function addResult(results, item, ok, detail, fix = "") {
|
|
14
|
+
results.push({ item, ok, detail, fix });
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
function findLangfuseHook(settings) {
|
|
18
|
+
const stopHooks = settings?.hooks?.Stop;
|
|
19
|
+
if (!Array.isArray(stopHooks)) return null;
|
|
20
|
+
for (const entry of stopHooks) {
|
|
21
|
+
const hooks = entry?.hooks;
|
|
22
|
+
if (!Array.isArray(hooks)) continue;
|
|
23
|
+
for (const h of hooks) {
|
|
24
|
+
if (h?.type !== "command") continue;
|
|
25
|
+
const command = typeof h.command === "string" ? h.command : "";
|
|
26
|
+
const args = Array.isArray(h.args) ? h.args : [];
|
|
27
|
+
const joined = [command, ...args].join(" ");
|
|
28
|
+
if (joined.includes("langfuse_hook.py")) return { command, args };
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
return null;
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
function hookUsesExecForm(hook) {
|
|
35
|
+
return !!(hook?.command && Array.isArray(hook.args) && hook.args.some((arg) => String(arg).includes("langfuse_hook.py")));
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
function pathExists(p) {
|
|
39
|
+
return typeof p === "string" && p.length > 0 && fs.existsSync(p);
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
function checkPythonLangfuseImport(pythonPath) {
|
|
43
|
+
if (!pathExists(pythonPath)) return { ok: false, detail: "python executable missing" };
|
|
44
|
+
const result = spawnSync(pythonPath, ["-c", "import langfuse; print('OK')"], {
|
|
45
|
+
encoding: "utf8",
|
|
46
|
+
timeout: 15000,
|
|
47
|
+
windowsHide: true
|
|
48
|
+
});
|
|
49
|
+
if (result.status === 0) return { ok: true, detail: "OK" };
|
|
50
|
+
const err = `${result.stderr || result.stdout || result.error?.message || "unknown error"}`.trim();
|
|
51
|
+
return { ok: false, detail: err || "import failed" };
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
function main() {
|
|
55
|
+
const userHome = os.homedir();
|
|
56
|
+
const claudeDir = path.join(userHome, ".claude");
|
|
57
|
+
const hooksDir = path.join(claudeDir, "hooks");
|
|
58
|
+
const settingsPath = path.join(claudeDir, "settings.json");
|
|
59
|
+
const logPath = path.join(claudeDir, "state", "langfuse_hook.log");
|
|
60
|
+
|
|
61
|
+
const results = [];
|
|
62
|
+
|
|
63
|
+
addResult(results, "Claude config dir", fs.existsSync(claudeDir), claudeDir, "Run: npx oh-langfuse@latest setup claude");
|
|
64
|
+
addResult(results, "hooks dir", fs.existsSync(hooksDir), hooksDir, "Run setup again to install the hook script.");
|
|
65
|
+
|
|
66
|
+
const pyPath = path.join(hooksDir, "langfuse_hook.py");
|
|
67
|
+
addResult(results, "hook script", fs.existsSync(pyPath), pyPath, "Run setup again to copy langfuse_hook.py.");
|
|
68
|
+
|
|
69
|
+
const settings = readJsonIfExists(settingsPath);
|
|
70
|
+
addResult(results, "settings.json", !!settings, settingsPath, "Run setup again to write Claude settings.");
|
|
71
|
+
|
|
72
|
+
const env = settings?.env;
|
|
73
|
+
const neededEnv = [
|
|
74
|
+
"TRACE_TO_LANGFUSE",
|
|
75
|
+
"LANGFUSE_PUBLIC_KEY",
|
|
76
|
+
"LANGFUSE_SECRET_KEY",
|
|
77
|
+
"LANGFUSE_HOST",
|
|
78
|
+
"CC_LANGFUSE_BASE_URL",
|
|
79
|
+
"LANGFUSE_BASEURL"
|
|
80
|
+
];
|
|
81
|
+
|
|
82
|
+
const missingEnv = neededEnv.filter((k) => !env || typeof env[k] === "undefined" || env[k] === "");
|
|
83
|
+
addResult(
|
|
84
|
+
results,
|
|
85
|
+
"Langfuse env fields",
|
|
86
|
+
missingEnv.length === 0,
|
|
87
|
+
missingEnv.length ? `missing: ${missingEnv.join(", ")}` : "OK",
|
|
88
|
+
"Run setup again so settings.json contains the Langfuse environment fields."
|
|
89
|
+
);
|
|
90
|
+
|
|
91
|
+
const userEnvOk = !!(
|
|
92
|
+
env?.CC_LANGFUSE_USER_ID ||
|
|
93
|
+
env?.CLAUDE_USER_ID ||
|
|
94
|
+
env?.CC_USER_ID ||
|
|
95
|
+
env?.LANGFUSE_USER_ID
|
|
96
|
+
);
|
|
97
|
+
addResult(results, "user id env field", userEnvOk, userEnvOk ? "OK" : "missing", "Run setup again and enter a valid employee id.");
|
|
98
|
+
|
|
99
|
+
const hook = findLangfuseHook(settings);
|
|
100
|
+
addResult(results, "hooks.Stop contains Langfuse", !!hook, hook ? "OK" : "missing", "Run setup again to register the Stop hook.");
|
|
101
|
+
|
|
102
|
+
const execFormOk = hookUsesExecForm(hook);
|
|
103
|
+
addResult(
|
|
104
|
+
results,
|
|
105
|
+
"hook command form",
|
|
106
|
+
process.platform === "win32" ? execFormOk : !!hook,
|
|
107
|
+
execFormOk ? "exec form command + args" : hook ? "legacy shell-form command" : "missing",
|
|
108
|
+
process.platform === "win32"
|
|
109
|
+
? "Run setup again. Windows should use command plus args so paths are not parsed by Git Bash/PowerShell."
|
|
110
|
+
: "Run setup again to refresh the hook command."
|
|
111
|
+
);
|
|
112
|
+
|
|
113
|
+
if (hook) {
|
|
114
|
+
addResult(
|
|
115
|
+
results,
|
|
116
|
+
"hook python executable",
|
|
117
|
+
pathExists(hook.command),
|
|
118
|
+
hook.command || "missing",
|
|
119
|
+
"Run setup again to recreate the Python venv and write an absolute python path."
|
|
120
|
+
);
|
|
121
|
+
const hookScriptPath = execFormOk ? String(hook.args.find((arg) => String(arg).includes("langfuse_hook.py")) || "") : pyPath;
|
|
122
|
+
addResult(
|
|
123
|
+
results,
|
|
124
|
+
"hook args script",
|
|
125
|
+
pathExists(hookScriptPath),
|
|
126
|
+
hookScriptPath || "missing",
|
|
127
|
+
"Run setup again so the hook args point at langfuse_hook.py."
|
|
128
|
+
);
|
|
129
|
+
const importCheck = checkPythonLangfuseImport(hook.command);
|
|
130
|
+
addResult(
|
|
131
|
+
results,
|
|
132
|
+
"python package langfuse",
|
|
133
|
+
importCheck.ok,
|
|
134
|
+
importCheck.detail,
|
|
135
|
+
`Run: ${hook.command} -m pip install -U langfuse`
|
|
136
|
+
);
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
addResult(results, "hook log path", true, logPath);
|
|
140
|
+
|
|
141
|
+
const pad = (s, n) => (s.length >= n ? s : s + " ".repeat(n - s.length));
|
|
142
|
+
const w = Math.max(...results.map((r) => r.item.length)) + 2;
|
|
143
|
+
const failed = [];
|
|
144
|
+
for (const r of results) {
|
|
145
|
+
const status = r.ok ? "OK " : "BAD";
|
|
146
|
+
console.log(`${status} ${pad(r.item, w)} ${r.detail}`);
|
|
147
|
+
if (!r.ok) failed.push(r);
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
if (failed.length) {
|
|
151
|
+
console.log("");
|
|
152
|
+
console.log("Fix suggestions:");
|
|
153
|
+
for (const r of failed) {
|
|
154
|
+
if (r.fix) console.log(`- ${r.item}: ${r.fix}`);
|
|
155
|
+
}
|
|
156
|
+
}
|
|
157
|
+
|
|
158
|
+
process.exit(failed.length ? 2 : 0);
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
main();
|
|
@@ -177,11 +177,13 @@ async function main() {
|
|
|
177
177
|
}
|
|
178
178
|
assertValidUserId(userId);
|
|
179
179
|
|
|
180
|
-
const langfuseHost =
|
|
181
|
-
args.
|
|
182
|
-
args.
|
|
183
|
-
|
|
184
|
-
|
|
180
|
+
const langfuseHost =
|
|
181
|
+
args.langfuseBaseUrl ||
|
|
182
|
+
args.langfuseHost ||
|
|
183
|
+
args.host ||
|
|
184
|
+
process.env.LANGFUSE_BASEURL ||
|
|
185
|
+
process.env.LANGFUSE_HOST ||
|
|
186
|
+
"http://120.46.221.227:3000";
|
|
185
187
|
|
|
186
188
|
const publicKey =
|
|
187
189
|
args.publicKey || process.env.LANGFUSE_PUBLIC_KEY || "pk-lf-da0c90a7-6e93-4eb7-bb86-c1047c8d187d";
|
|
@@ -246,10 +248,8 @@ async function main() {
|
|
|
246
248
|
const settingsPath = path.join(claudeDir, "settings.json");
|
|
247
249
|
const existing = readJsonIfExists(settingsPath) ?? {};
|
|
248
250
|
|
|
249
|
-
const
|
|
250
|
-
|
|
251
|
-
const desired = {
|
|
252
|
-
env: {
|
|
251
|
+
const desired = {
|
|
252
|
+
env: {
|
|
253
253
|
TRACE_TO_LANGFUSE: "true",
|
|
254
254
|
LANGFUSE_PUBLIC_KEY: publicKey,
|
|
255
255
|
LANGFUSE_SECRET_KEY: secretKey,
|
|
@@ -265,12 +265,13 @@ async function main() {
|
|
|
265
265
|
Stop: [
|
|
266
266
|
{
|
|
267
267
|
hooks: [
|
|
268
|
-
{
|
|
269
|
-
type: "command",
|
|
270
|
-
command:
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
268
|
+
{
|
|
269
|
+
type: "command",
|
|
270
|
+
command: hookPython,
|
|
271
|
+
args: [pyPath]
|
|
272
|
+
}
|
|
273
|
+
]
|
|
274
|
+
}
|
|
274
275
|
]
|
|
275
276
|
}
|
|
276
277
|
};
|
|
@@ -0,0 +1,414 @@
|
|
|
1
|
+
import fs from "node:fs";
|
|
2
|
+
import os from "node:os";
|
|
3
|
+
import path from "node:path";
|
|
4
|
+
import crypto from "node:crypto";
|
|
5
|
+
import { spawnSync } from "node:child_process";
|
|
6
|
+
import { fileURLToPath } from "node:url";
|
|
7
|
+
import { resolveOpencodeCli } from "./resolve-opencode-cli.mjs";
|
|
8
|
+
|
|
9
|
+
const rootDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
|
10
|
+
const packageJson = JSON.parse(fs.readFileSync(path.join(rootDir, "package.json"), "utf8"));
|
|
11
|
+
|
|
12
|
+
const DEFAULT_LANGFUSE_BASE_URL = "http://120.46.221.227:3000";
|
|
13
|
+
const DEFAULT_LANGFUSE_PUBLIC_KEY = "pk-lf-da0c90a7-6e93-4eb7-bb86-c1047c8d187d";
|
|
14
|
+
const DEFAULT_LANGFUSE_SECRET_KEY = "sk-lf-0269b85d-bfdc-442c-bfa3-e737954e3315";
|
|
15
|
+
|
|
16
|
+
function parseArgs(argv) {
|
|
17
|
+
const args = { _: [] };
|
|
18
|
+
for (let i = 0; i < argv.length; i += 1) {
|
|
19
|
+
const raw = argv[i];
|
|
20
|
+
if (!raw.startsWith("--")) {
|
|
21
|
+
args._.push(raw);
|
|
22
|
+
continue;
|
|
23
|
+
}
|
|
24
|
+
const eq = raw.indexOf("=");
|
|
25
|
+
if (eq !== -1) {
|
|
26
|
+
args[raw.slice(2, eq)] = raw.slice(eq + 1);
|
|
27
|
+
continue;
|
|
28
|
+
}
|
|
29
|
+
const key = raw.slice(2);
|
|
30
|
+
const next = argv[i + 1];
|
|
31
|
+
if (next && !next.startsWith("--")) {
|
|
32
|
+
args[key] = next;
|
|
33
|
+
i += 1;
|
|
34
|
+
} else {
|
|
35
|
+
args[key] = true;
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
return args;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
function printHelp() {
|
|
42
|
+
console.log(`${packageJson.name} real self verification`);
|
|
43
|
+
console.log("");
|
|
44
|
+
console.log("Usage:");
|
|
45
|
+
console.log(" npm run self:verify -- --targets=opencode --userId=h00613222");
|
|
46
|
+
console.log(" node scripts/real-self-verify.mjs --targets=claude,opencode,codex --userId=h00613222");
|
|
47
|
+
console.log("");
|
|
48
|
+
console.log("What it does:");
|
|
49
|
+
console.log(" 1. Runs the real setup command for each target unless --skip-install is set.");
|
|
50
|
+
console.log(" 2. Starts the actual Claude/OpenCode/Codex CLI with a unique validation marker.");
|
|
51
|
+
console.log(" 3. Polls Langfuse Public API until that marker appears in traces/observations.");
|
|
52
|
+
console.log("");
|
|
53
|
+
console.log("Options:");
|
|
54
|
+
console.log(" --targets=opencode,claude,codex");
|
|
55
|
+
console.log(" --userId=<id> Required unless LANGFUSE_USER_ID is set.");
|
|
56
|
+
console.log(" --langfuseBaseUrl=<url> Defaults to package default or LANGFUSE_BASEURL.");
|
|
57
|
+
console.log(" --publicKey=<key> Defaults to package default or LANGFUSE_PUBLIC_KEY.");
|
|
58
|
+
console.log(" --secretKey=<key> Defaults to package default or LANGFUSE_SECRET_KEY.");
|
|
59
|
+
console.log(" --marker=<text> Override generated validation marker.");
|
|
60
|
+
console.log(" --timeoutMs=180000 Langfuse polling timeout.");
|
|
61
|
+
console.log(" --triggerTimeoutMs=300000 Per CLI invocation timeout.");
|
|
62
|
+
console.log(" --skip-install Do not run setup first.");
|
|
63
|
+
console.log(" --skip-trigger Only install and poll for a pre-existing marker.");
|
|
64
|
+
console.log(" --allow-trigger-failure Poll Langfuse even if the CLI command exits non-zero.");
|
|
65
|
+
console.log(" --opencodeCmd=<path> Override OpenCode CLI path.");
|
|
66
|
+
console.log(" --claudeCmd=<path> Override Claude CLI path.");
|
|
67
|
+
console.log(" --codexCmd=<path> Override Codex CLI path.");
|
|
68
|
+
console.log(" --prompt=<text> Override the validation prompt.");
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
function splitTargets(raw) {
|
|
72
|
+
const value = String(raw || "opencode").trim();
|
|
73
|
+
const out = value
|
|
74
|
+
.split(/[, ]+/)
|
|
75
|
+
.map((x) => x.trim().toLowerCase())
|
|
76
|
+
.filter(Boolean);
|
|
77
|
+
const allowed = new Set(["claude", "opencode", "codex"]);
|
|
78
|
+
for (const target of out) {
|
|
79
|
+
if (!allowed.has(target)) throw new Error(`Unsupported target: ${target}`);
|
|
80
|
+
}
|
|
81
|
+
return out.length ? [...new Set(out)] : ["opencode"];
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
function makeMarker() {
|
|
85
|
+
return `real-self-verify-${packageJson.name}-${Date.now()}-${crypto.randomBytes(4).toString("hex")}`;
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
function shellNeeded(command) {
|
|
89
|
+
return process.platform === "win32" && /\.(cmd|bat)$/i.test(command);
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
function run(command, args, options = {}) {
|
|
93
|
+
const result = spawnSync(command, args, {
|
|
94
|
+
cwd: options.cwd || rootDir,
|
|
95
|
+
env: { ...process.env, ...(options.env || {}) },
|
|
96
|
+
encoding: "utf8",
|
|
97
|
+
stdio: options.stdio || "pipe",
|
|
98
|
+
timeout: options.timeoutMs,
|
|
99
|
+
windowsHide: true,
|
|
100
|
+
shell: shellNeeded(command),
|
|
101
|
+
});
|
|
102
|
+
return {
|
|
103
|
+
command,
|
|
104
|
+
args,
|
|
105
|
+
status: result.status ?? (result.error ? 1 : 0),
|
|
106
|
+
stdout: result.stdout || "",
|
|
107
|
+
stderr: result.stderr || "",
|
|
108
|
+
error: result.error,
|
|
109
|
+
signal: result.signal,
|
|
110
|
+
};
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
function printRunResult(label, result) {
|
|
114
|
+
const status = result.status === 0 ? "OK" : "FAIL";
|
|
115
|
+
console.log(`[${status}] ${label}: ${result.command} ${result.args.join(" ")}`);
|
|
116
|
+
const text = `${result.stdout}\n${result.stderr}`.trim();
|
|
117
|
+
if (text) {
|
|
118
|
+
const clipped = text.length > 5000 ? `${text.slice(0, 5000)}\n...<clipped>` : text;
|
|
119
|
+
console.log(clipped);
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
function findOnPath(names) {
|
|
124
|
+
for (const name of names) {
|
|
125
|
+
const probe = run(name, ["--version"], { timeoutMs: 15000 });
|
|
126
|
+
if (probe.status === 0) return name;
|
|
127
|
+
}
|
|
128
|
+
return "";
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
function configFromArgs(args) {
|
|
132
|
+
return {
|
|
133
|
+
baseUrl:
|
|
134
|
+
String(args.langfuseBaseUrl || args.langfuseHost || args.host || process.env.LANGFUSE_BASEURL || process.env.LANGFUSE_HOST || DEFAULT_LANGFUSE_BASE_URL),
|
|
135
|
+
publicKey: String(args.publicKey || process.env.LANGFUSE_PUBLIC_KEY || DEFAULT_LANGFUSE_PUBLIC_KEY),
|
|
136
|
+
secretKey: String(args.secretKey || process.env.LANGFUSE_SECRET_KEY || DEFAULT_LANGFUSE_SECRET_KEY),
|
|
137
|
+
userId: String(args.userId || args.userid || process.env.LANGFUSE_USER_ID || process.env.CC_USER_ID || "").trim(),
|
|
138
|
+
};
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
function setupTarget(target, config, args) {
|
|
142
|
+
const cli = path.join(rootDir, "bin", "cli.js");
|
|
143
|
+
const setupArgs = [
|
|
144
|
+
cli,
|
|
145
|
+
"setup",
|
|
146
|
+
target,
|
|
147
|
+
"--yes",
|
|
148
|
+
`--userId=${config.userId}`,
|
|
149
|
+
`--langfuseBaseUrl=${config.baseUrl}`,
|
|
150
|
+
`--publicKey=${config.publicKey}`,
|
|
151
|
+
`--secretKey=${config.secretKey}`,
|
|
152
|
+
];
|
|
153
|
+
if (args.npmRegistry) setupArgs.push(`--npmRegistry=${args.npmRegistry}`);
|
|
154
|
+
if (args.pipIndexUrl) setupArgs.push(`--pipIndexUrl=${args.pipIndexUrl}`);
|
|
155
|
+
if (args["skip-plugin-install"]) setupArgs.push("--skip-plugin-install");
|
|
156
|
+
if (args["no-set-env"]) setupArgs.push("--no-set-env");
|
|
157
|
+
if (args.cmd) setupArgs.push(`--cmd=${args.cmd}`);
|
|
158
|
+
const result = run(process.execPath, setupArgs, { stdio: "inherit", timeoutMs: Number(args.setupTimeoutMs || 600000) });
|
|
159
|
+
return result.status;
|
|
160
|
+
}
|
|
161
|
+
|
|
162
|
+
function validationPrompt(marker, target, customPrompt) {
|
|
163
|
+
if (customPrompt) return String(customPrompt).replaceAll("{marker}", marker).replaceAll("{target}", target);
|
|
164
|
+
return [
|
|
165
|
+
`This is an automated real self-verification run for ${packageJson.name}.`,
|
|
166
|
+
`Validation marker: ${marker}`,
|
|
167
|
+
"Reply with the validation marker exactly once and do not run tools.",
|
|
168
|
+
].join("\n");
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
function triggerClaude(prompt, args, env) {
|
|
172
|
+
const cmd = String(args.claudeCmd || findOnPath(process.platform === "win32" ? ["claude.cmd", "claude"] : ["claude"]));
|
|
173
|
+
if (!cmd) return { status: 127, stdout: "", stderr: "Claude CLI not found. Set --claudeCmd=<path>.", command: "claude", args: [] };
|
|
174
|
+
const candidates = [
|
|
175
|
+
["-p", prompt],
|
|
176
|
+
["--print", prompt],
|
|
177
|
+
];
|
|
178
|
+
let last = null;
|
|
179
|
+
for (const candidate of candidates) {
|
|
180
|
+
last = run(cmd, candidate, { env, timeoutMs: Number(args.triggerTimeoutMs || 300000) });
|
|
181
|
+
if (last.status === 0) return last;
|
|
182
|
+
}
|
|
183
|
+
return last;
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
function triggerOpenCode(prompt, args, env) {
|
|
187
|
+
const cmd = resolveOpencodeCli(args.opencodeCmd || args.cmd);
|
|
188
|
+
if (!cmd) return { status: 127, stdout: "", stderr: "OpenCode CLI not found. Set --opencodeCmd=<path>.", command: "opencode", args: [] };
|
|
189
|
+
const candidates = [
|
|
190
|
+
["run", prompt],
|
|
191
|
+
["run", "--print", prompt],
|
|
192
|
+
[prompt],
|
|
193
|
+
];
|
|
194
|
+
let last = null;
|
|
195
|
+
for (const candidate of candidates) {
|
|
196
|
+
last = run(cmd, candidate, { env, timeoutMs: Number(args.triggerTimeoutMs || 300000) });
|
|
197
|
+
if (last.status === 0) return last;
|
|
198
|
+
}
|
|
199
|
+
return last;
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
function triggerCodex(prompt, args, env) {
|
|
203
|
+
const cmd = String(args.codexCmd || findOnPath(process.platform === "win32" ? ["codex.cmd", "codex"] : ["codex"]));
|
|
204
|
+
if (!cmd) return { status: 127, stdout: "", stderr: "Codex CLI not found. Set --codexCmd=<path>.", command: "codex", args: [] };
|
|
205
|
+
const candidates = [
|
|
206
|
+
["exec", prompt],
|
|
207
|
+
["exec", "--skip-git-repo-check", prompt],
|
|
208
|
+
];
|
|
209
|
+
let last = null;
|
|
210
|
+
for (const candidate of candidates) {
|
|
211
|
+
last = run(cmd, candidate, { env, timeoutMs: Number(args.triggerTimeoutMs || 300000) });
|
|
212
|
+
if (last.status === 0) return last;
|
|
213
|
+
}
|
|
214
|
+
return last;
|
|
215
|
+
}
|
|
216
|
+
|
|
217
|
+
function apiBase(baseUrl) {
|
|
218
|
+
const trimmed = String(baseUrl || "").replace(/\/+$/, "");
|
|
219
|
+
return trimmed.endsWith("/api/public") ? trimmed : `${trimmed}/api/public`;
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
function basicAuth(publicKey, secretKey) {
|
|
223
|
+
return `Basic ${Buffer.from(`${publicKey}:${secretKey}`, "utf8").toString("base64")}`;
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
async function langfuseGet(config, pathname, params = {}) {
|
|
227
|
+
const url = new URL(`${apiBase(config.baseUrl)}${pathname}`);
|
|
228
|
+
for (const [key, value] of Object.entries(params)) {
|
|
229
|
+
if (value !== undefined && value !== null && String(value) !== "") url.searchParams.set(key, String(value));
|
|
230
|
+
}
|
|
231
|
+
const response = await fetch(url, {
|
|
232
|
+
headers: {
|
|
233
|
+
accept: "application/json",
|
|
234
|
+
authorization: basicAuth(config.publicKey, config.secretKey),
|
|
235
|
+
},
|
|
236
|
+
});
|
|
237
|
+
if (!response.ok) {
|
|
238
|
+
const body = await response.text().catch(() => "");
|
|
239
|
+
const error = new Error(`Langfuse API ${response.status} ${response.statusText}: ${body.slice(0, 500)}`);
|
|
240
|
+
error.status = response.status;
|
|
241
|
+
throw error;
|
|
242
|
+
}
|
|
243
|
+
return await response.json();
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
function dataArray(value) {
|
|
247
|
+
if (Array.isArray(value)) return value;
|
|
248
|
+
if (Array.isArray(value?.data)) return value.data;
|
|
249
|
+
if (Array.isArray(value?.items)) return value.items;
|
|
250
|
+
return [];
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
function containsMarker(value, marker) {
|
|
254
|
+
try {
|
|
255
|
+
return JSON.stringify(value).includes(marker);
|
|
256
|
+
} catch {
|
|
257
|
+
return false;
|
|
258
|
+
}
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
function idOf(value) {
|
|
262
|
+
return value?.id || value?.traceId || value?.trace_id || "";
|
|
263
|
+
}
|
|
264
|
+
|
|
265
|
+
async function findLangfuseMarker(config, marker, { since, target }) {
|
|
266
|
+
const baseParams = {
|
|
267
|
+
limit: 100,
|
|
268
|
+
fromTimestamp: since.toISOString(),
|
|
269
|
+
};
|
|
270
|
+
const traceQueries = [
|
|
271
|
+
{ ...baseParams, userId: config.userId },
|
|
272
|
+
baseParams,
|
|
273
|
+
];
|
|
274
|
+
|
|
275
|
+
for (const params of traceQueries) {
|
|
276
|
+
let traces;
|
|
277
|
+
try {
|
|
278
|
+
traces = await langfuseGet(config, "/traces", params);
|
|
279
|
+
} catch (error) {
|
|
280
|
+
if (error.status === 404) continue;
|
|
281
|
+
throw error;
|
|
282
|
+
}
|
|
283
|
+
for (const trace of dataArray(traces)) {
|
|
284
|
+
if (containsMarker(trace, marker)) return { kind: "trace-list", target, id: idOf(trace), item: trace };
|
|
285
|
+
}
|
|
286
|
+
for (const trace of dataArray(traces).slice(0, 25)) {
|
|
287
|
+
const id = idOf(trace);
|
|
288
|
+
if (!id) continue;
|
|
289
|
+
try {
|
|
290
|
+
const detail = await langfuseGet(config, `/traces/${encodeURIComponent(id)}`);
|
|
291
|
+
if (containsMarker(detail, marker)) return { kind: "trace-detail", target, id, item: detail };
|
|
292
|
+
} catch (error) {
|
|
293
|
+
if (error.status !== 404) throw error;
|
|
294
|
+
}
|
|
295
|
+
}
|
|
296
|
+
}
|
|
297
|
+
|
|
298
|
+
const observationQueries = [
|
|
299
|
+
["/observations", { ...baseParams, userId: config.userId }],
|
|
300
|
+
["/observations", baseParams],
|
|
301
|
+
["/v2/observations", { ...baseParams, fields: "core,basic,usage", userId: config.userId }],
|
|
302
|
+
["/v2/observations", { ...baseParams, fields: "core,basic,usage" }],
|
|
303
|
+
];
|
|
304
|
+
for (const [pathname, params] of observationQueries) {
|
|
305
|
+
try {
|
|
306
|
+
const observations = await langfuseGet(config, pathname, params);
|
|
307
|
+
for (const observation of dataArray(observations)) {
|
|
308
|
+
if (containsMarker(observation, marker)) return { kind: pathname, target, id: idOf(observation), item: observation };
|
|
309
|
+
}
|
|
310
|
+
} catch (error) {
|
|
311
|
+
if (error.status === 404 || error.status === 400) continue;
|
|
312
|
+
throw error;
|
|
313
|
+
}
|
|
314
|
+
}
|
|
315
|
+
|
|
316
|
+
return null;
|
|
317
|
+
}
|
|
318
|
+
|
|
319
|
+
async function pollLangfuse(config, marker, options) {
|
|
320
|
+
const timeoutMs = Number(options.timeoutMs || 180000);
|
|
321
|
+
const pollMs = Number(options.pollMs || 5000);
|
|
322
|
+
const deadline = Date.now() + timeoutMs;
|
|
323
|
+
const since = options.since || new Date(Date.now() - 10 * 60 * 1000);
|
|
324
|
+
let lastError = null;
|
|
325
|
+
|
|
326
|
+
while (Date.now() <= deadline) {
|
|
327
|
+
try {
|
|
328
|
+
const found = await findLangfuseMarker(config, marker, { since, target: options.target });
|
|
329
|
+
if (found) return found;
|
|
330
|
+
console.log(`[WAIT] Langfuse has not returned marker yet (${marker}).`);
|
|
331
|
+
} catch (error) {
|
|
332
|
+
lastError = error;
|
|
333
|
+
console.log(`[WAIT] Langfuse query failed: ${error.message}`);
|
|
334
|
+
}
|
|
335
|
+
await new Promise((resolve) => setTimeout(resolve, pollMs));
|
|
336
|
+
}
|
|
337
|
+
|
|
338
|
+
if (lastError) throw new Error(`Timed out waiting for marker. Last Langfuse error: ${lastError.message}`);
|
|
339
|
+
throw new Error(`Timed out waiting for marker in Langfuse: ${marker}`);
|
|
340
|
+
}
|
|
341
|
+
|
|
342
|
+
async function main() {
|
|
343
|
+
const args = parseArgs(process.argv.slice(2));
|
|
344
|
+
if (args.help || args.h) {
|
|
345
|
+
printHelp();
|
|
346
|
+
return 0;
|
|
347
|
+
}
|
|
348
|
+
|
|
349
|
+
const config = configFromArgs(args);
|
|
350
|
+
if (!config.userId) throw new Error("Missing --userId or LANGFUSE_USER_ID. Real verification needs a user id to install and filter traces.");
|
|
351
|
+
const targets = splitTargets(args.targets || args.target || args._[0]);
|
|
352
|
+
const marker = String(args.marker || makeMarker());
|
|
353
|
+
const since = new Date();
|
|
354
|
+
const results = [];
|
|
355
|
+
|
|
356
|
+
console.log(`[INFO] Package: ${packageJson.name}@${packageJson.version}`);
|
|
357
|
+
console.log(`[INFO] Targets: ${targets.join(", ")}`);
|
|
358
|
+
console.log(`[INFO] Langfuse: ${config.baseUrl}`);
|
|
359
|
+
console.log(`[INFO] User ID: ${config.userId}`);
|
|
360
|
+
console.log(`[INFO] Marker: ${marker}`);
|
|
361
|
+
console.log("");
|
|
362
|
+
|
|
363
|
+
for (const target of targets) {
|
|
364
|
+
if (!args["skip-install"]) {
|
|
365
|
+
const code = setupTarget(target, config, args);
|
|
366
|
+
if (code !== 0) throw new Error(`Setup failed for ${target} with exit code ${code}.`);
|
|
367
|
+
}
|
|
368
|
+
|
|
369
|
+
const prompt = validationPrompt(marker, target, args.prompt);
|
|
370
|
+
const env = {
|
|
371
|
+
TRACE_TO_LANGFUSE: "true",
|
|
372
|
+
LANGFUSE_PUBLIC_KEY: config.publicKey,
|
|
373
|
+
LANGFUSE_SECRET_KEY: config.secretKey,
|
|
374
|
+
LANGFUSE_BASEURL: config.baseUrl,
|
|
375
|
+
LANGFUSE_HOST: config.baseUrl,
|
|
376
|
+
LANGFUSE_USER_ID: config.userId,
|
|
377
|
+
CC_USER_ID: config.userId,
|
|
378
|
+
CC_LANGFUSE_USER_ID: config.userId,
|
|
379
|
+
CODEX_LANGFUSE_USER_ID: config.userId,
|
|
380
|
+
OHAI_REAL_SELF_VERIFY_MARKER: marker,
|
|
381
|
+
OHAI_REAL_SELF_VERIFY_TARGET: target,
|
|
382
|
+
};
|
|
383
|
+
|
|
384
|
+
if (!args["skip-trigger"]) {
|
|
385
|
+
const trigger =
|
|
386
|
+
target === "claude"
|
|
387
|
+
? triggerClaude(prompt, args, env)
|
|
388
|
+
: target === "opencode"
|
|
389
|
+
? triggerOpenCode(prompt, args, env)
|
|
390
|
+
: triggerCodex(prompt, args, env);
|
|
391
|
+
printRunResult(`${target} trigger`, trigger);
|
|
392
|
+
if (trigger.status !== 0 && !args["allow-trigger-failure"]) {
|
|
393
|
+
throw new Error(`${target} trigger failed before Langfuse polling. Use --allow-trigger-failure to poll anyway.`);
|
|
394
|
+
}
|
|
395
|
+
}
|
|
396
|
+
|
|
397
|
+
const found = await pollLangfuse(config, marker, { ...args, since, target });
|
|
398
|
+
console.log(`[OK] Langfuse marker found for ${target}: ${found.kind} ${found.id || ""}`.trim());
|
|
399
|
+
results.push({ target, marker, langfuse: { kind: found.kind, id: found.id || "" } });
|
|
400
|
+
}
|
|
401
|
+
|
|
402
|
+
console.log("");
|
|
403
|
+
console.log(JSON.stringify({ ok: true, package: packageJson.name, marker, results }, null, 2));
|
|
404
|
+
return 0;
|
|
405
|
+
}
|
|
406
|
+
|
|
407
|
+
main()
|
|
408
|
+
.then((code) => {
|
|
409
|
+
process.exitCode = code;
|
|
410
|
+
})
|
|
411
|
+
.catch((error) => {
|
|
412
|
+
console.error(`[FAIL] ${error?.message || String(error)}`);
|
|
413
|
+
process.exitCode = 1;
|
|
414
|
+
});
|