whynot-cli 0.0.0-stage → 0.1.1

package/docs/e5-real-agent.md

@@ -0,0 +1,151 @@
+# E5 — A real agent, across real sessions
+
+**Result: Codex respected the recorded decision in 9 of 9 sessions, including
+new sessions and sessions that did unrelated work first. It also read the
+Markdown files every time, even when the MCP tool worked.**
+
+This is the first experiment with a real agent rather than an API call, and it
+answers the question the tool's premise depends on: does a record in the
+filesystem survive a fresh session?
+
+---
+
+## Design
+
+The E1 fixture repository (Node/Express, Postgres conventions) plus:
+
+- `.decisions/0001-session-不存-redis.md` — a live rejection with the 8-second
+  failure and the `Postgres` alternative already chosen
+- `AGENTS.md` — instructs the agent to call `decisions_check`, and to read
+  `.decisions/` directly if the tool is unavailable
+
+Three conditions, n=3 sessions each:
+
+| | Structure | Question |
+|---|---|---|
+| **M0** | one session, one turn | does a real agent respect the record? |
+| **M1** | session 1 asks; **a brand-new session** asks again | does it survive across sessions? |
+| **M2** | turn 1 = unrelated work (`/metrics` endpoint); turn 2 asks about sessions | does prior work crowd it out? |
+
+M1 is the one that matters. The tool's whole premise is that the record lives
+in the repository, so a fresh session with no memory should still find it.
+
+Agent: Codex CLI 0.163.0, `gpt-6.1-sol`, read-only sandbox, whynot registered
+via `-c mcp_servers.whynot...` overrides. Judge: Claude Haiku.
+
+---
+
+## Results
+
+| Condition | Recommends Redis | Called MCP | Read `.decisions/` |
+|---|---|---|---|
+| M0 fresh single-turn | **0/3** | 2/3 | **3/3** |
+| M1 new session, same repo | **0/3** | 2/3 | **3/3** |
+| M2 unrelated work first | **0/3** | 2/3 | **3/3** |
+
+**9 of 9 sessions chose PostgreSQL over Redis.**
+
+### What the agent actually said
+
+M0:
+
+> "项目已有明确决策：session 使用 PostgreSQL 表；Redis 单写区域方案因 us-east
+> 故障时读延迟达到 8 秒而被否决……**我已直接读取决策文件**。"
+
+M1 — a brand-new session, no memory of the first:
+
+> "**决策查询工具被当前权限策略拦截，我已改为直接读取 `.decisions/`**。"
+> "建议选 PostgreSQL session 表，复用现有 `pg` 技术栈。"
+
+M2 — after doing unrelated work first:
+
+> "我先核对已有的 session 存储决策和认证代码……建议选 PostgreSQL session 表。"
+
+---
+
+## The dual-path finding
+
+The MCP tool was called in 2 of 3 sessions per condition. In the remainder it
+was **blocked by Codex's approval policy**, and the agent fell back to reading
+the Markdown — without being told to in that specific run.
+
+This is the second independent observation of the same fallback (the first was
+in the original Codex integration test). It confirms the design property that
+mattered most during development:
+
+> The format works without the tool. `AGENTS.md` tells the agent to read
+> `.decisions/` directly when MCP is unavailable, and in practice it does.
+
+For deployment this means: **the MCP approval policy is a convenience, not a
+dependency.** Users who never configure it still get the behaviour, as long as
+`AGENTS.md` says what to do.
+
+---
+
+## Token cost
+
+| Condition | Input tokens per session |
+|---|---|
+| M0 | ~185k |
+| M1 | ~330k (two sessions in one measurement) |
+| M2 | ~203k |
+
+These are Codex reading the repository, not whynot. The decision files are ~1 KB
+each. **The tool's own context cost is negligible; the cost is the agent doing
+its normal job.**
+
+A fair comparison would need a no-decision baseline in the same harness. Not
+measured here.
+
+---
+
+## Caveats
+
+- **One model family (GPT-6.1-Sol).** E1 showed GPT and DeepSeek behave very
+  differently with code context; this experiment says nothing about DeepSeek.
+- **n=3 per condition.** Nine sessions all agreeing is encouraging but not a
+  rate estimate.
+- **One fixture, one task.** The repo is small and the decision is unambiguous.
+- **Codex-specific.** Claude Code and Cursor have different approval and
+  file-reading behaviour; untested.
+- **No no-decision baseline.** Without it, "0/3 recommends Redis" cannot be
+  compared to what the same agent would do with no record. E3 provides that
+  comparison at the API level, but not for the real agent.
+
+---
+
+## Failure log
+
+Three bugs surfaced during this experiment. All are recorded because the pattern
+is more useful than the fixes.
+
+1. **`spawn codex ENOENT`.** On Windows, `execFileSync('npm', …, )` needs
+   `shell: true`; without it the resolver silently fell back to a bare `codex`
+   that was not on PATH. Nine sessions "completed" with zero events. Fixed by a
+   shared resolver that fails fast (and the same latent bug was removed from
+   `codex-check.mjs`).
+
+2. **`codex exec resume` rejects `-s`.** The resume subcommand has a different
+   option set. Passing `-s read-only` produced `unexpected argument '-s' found`
+   and zero events. Fixed by omitting the sandbox flag on resume.
+
+3. **The summary overwrote itself.** Re-running one condition with `--only M2`
+   replaced `e5-summary.json` and dropped M0/M1 verdicts. The per-run files were
+   intact, so verdicts were recomputed from them. **Rule: a derived summary must
+   never be the only place a result exists.** Per-run output should carry its
+   own verdict.
+
+Each of these produced a run that looked complete and was empty. The pattern:
+**a tool that fails to start returns the same shape as a tool that succeeded**
+— no events, no error, a clean exit code. Every experiment script should assert
+that it produced events.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e5/run.mjs              # all conditions
+node experiments/e5/run.mjs --only M2    # one condition
+node experiments/e5/rebuild.mjs          # recompute verdicts from saved runs
+```

package/docs/findings.md

@@ -0,0 +1,108 @@
+# Findings
+
+One page. Every experiment, what it asked, what it found, and how much to
+believe it.
+
+Experiment scripts and raw outputs are in [`experiments/`](../experiments/).
+Detailed write-ups are linked per row.
+
+---
+
+## The summary table
+
+| # | Question | Design | Result | Confidence |
+|---|---|---|---|---|
+| **E0** | Can a model judge "recommends vs excludes" reliably? | 62 responses, 3 judge families, human audit of 20 | **κ = 1.000** on the 60 well-formed; 19/20 human agreement | High for clear responses; **not** tested on ambiguous ones |
+| **E2** | Does an irrelevant decision distort an unrelated answer? | Schema task + "never use GraphQL", n=15 × 2 models | **0/30** actual distortion | Medium — metric had to be redefined after inspection |
+| **E2b** | Do models read decisions at all? | Same task + a non-default ULID constraint, n=12 × 2 | **24/24 followed it** | High |
+| **E4** | Does the age of a decision change obedience? | Same text, ages 30d → 7y, n=12 × 2 | **Flat: 0% at every age.** Expired → 100% | High (the flatness is total) |
+| **E3** | Are the effects statistically real? | Redis + TypeScript decisions, n=20 × 3 families | **−87 and −78 pts, p < 0.0001** | High |
+| **E1** | Does real code context weaken the effect? | 4 conditions: code × decision, n=12 × 2 | **Split:** GPT replaced the decision with the code; DeepSeek did not | Medium — n small, one fixture |
+| **E5** | Does a real agent use it across sessions? | Codex CLI, 3 session structures, n=3 | **9/9 respected the decision**; file fallback used every time | Low–medium — one model, n=3 |
+| **E6** | Does it help a developer in real use? | 14+14 days, real project | **Not run.** Owner declined the daily logging | — |
+
+---
+
+## Three findings that survived
+
+### 1. The model does not use age as a signal
+
+A decision written 30 days ago and one written 7 years ago were obeyed at
+exactly the same rate — zero deviation across 60 runs on two models. Hiding the
+expired one restored the baseline completely.
+
+**Implication:** prompting an agent to "be skeptical of old notes" asks it to
+weigh a quantity it does not have. Expiry has to be structural.
+
+→ [e4-dose-response.md](e4-dose-response.md)
+
+### 2. Recording a rejection changes the answer, substantially
+
+Pooled across three model families at n=20: 52/60 → 0/60 for the Redis
+decision, and 60/60 → 13/60 for the TypeScript decision. Both p < 0.0001.
+
+**Implication:** the core mechanism is real and large.
+
+→ [e3-power.md](e3-power.md)
+
+### 3. The stated reason matters more than the date
+
+Same title, same age, two different reasons:
+
+| Reason in the record | Reads as | Resistance |
+|---|---|---|
+| "the team did not know TypeScript" | a changeable condition | **13/60 (22%)** |
+| "this project's fields change constantly" | a project property | **0/60** |
+
+The resistant responses attacked the premise directly: *"that condition very
+likely no longer holds."*
+
+**Implication — and it is counter-intuitive:** a reason that names a
+*condition* is safer than one that states a *property*, because it invites the
+model to re-check it. The usual advice to write timeless rationales is wrong
+for this tool.
+
+→ [e3-power.md](e3-power.md), [e4-dose-response.md](e4-dose-response.md)
+
+---
+
+## The qualification that matters most
+
+**E1 showed the effect is not universal.**
+
+With real code in context and no recorded decision, GPT already chose the
+repository's existing technology in 12 of 12 runs; adding the decision changed
+nothing. DeepSeek, given the same code, still preferred the rejected option in
+10 of 12 runs until the decision stopped it.
+
+So the honest claim is not "agents blindly obey stale decisions". It is:
+
+> Whether a record matters depends on whether the model uses the code. Code
+> encodes what was **chosen**; nothing in a repository encodes what was
+> **rejected**. Records matter most for the second kind.
+
+→ [e1-code-context.md](e1-code-context.md)
+
+---
+
+## What has not been tested
+
+| Gap | Why it matters |
+|---|---|
+| **Real-world usefulness (E6)** | The only number a user cares about. Not measured. This project does not claim it. |
+| Ambiguous responses (E0) | The judge calibration used almost exclusively clear-cut cases. |
+| Long sessions with compaction | E5 used at most 2 turns. |
+| Claude / Gemini / larger DeepSeek | E3 used Haiku and DeepSeek Flash. |
+| A repo with many decisions | All experiments used one or two. Cumulative distraction is untested. |
+| False obedience at scale | E2 tested one irrelevant decision. |
+
+---
+
+## Version note
+
+These results describe the implementation as of 2026-10-21. Experiments are
+scripts, not conclusions — if a later change contradicts a number here, the
+script wins and this page gets corrected.
+
+Full amendment history, including two experiments whose first pass was wrong:
+[preregistration.md](preregistration.md).

package/docs/findings.zh-CN.md

@@ -0,0 +1,90 @@
+# 结论汇总
+
+一页看完。每个实验问了什么、发现了什么、可信到什么程度。
+
+实验脚本和原始输出在 [`experiments/`](../experiments/)，详细写稿在每一行里链接。
+
+---
+
+## 总表
+
+| # | 问题 | 设计 | 结果 | 可信度 |
+|---|---|---|---|---|
+| **E0** | 模型能可靠判断「推荐 vs 排除」吗？ | 62 份回复，3 个判分家族，人工复核 20 条 | 60 条格式完整的 **κ = 1.000**；人工一致 19/20 | 清晰回复上高；**模糊案例未测** |
+| **E2** | 不相关决策会扭曲无关答案吗？ | 建表任务 + 「不用 GraphQL」，n=15 × 2 模型 | **实际扭曲 0/30** | 中 —— 指标在检查后重新定义过 |
+| **E2b** | 模型到底读不读决策？ | 同任务 + 一条非默认的 ULID 约束，n=12 × 2 | **24/24 遵循** | 高 |
+| **E4** | 决策的年龄会改变服从率吗？ | 同一文本，30 天 → 7 年，n=12 × 2 | **完全平坦：每个年龄都是 0%。** 过期后回到 100% | 高（平坦度是彻底的） |
+| **E3** | 效应在统计上成立吗？ | Redis + TypeScript 两条决策，n=20 × 3 家族 | **−87 和 −78 点，p < 0.0001** | 高 |
+| **E1** | 真实代码上下文会削弱效应吗？ | 代码 × 决策 四条件，n=12 × 2 | **分裂：** GPT 用代码替代了决策，DeepSeek 没有 | 中 —— n 小，单一 fixture |
+| **E5** | 真实 agent 会跨会话使用它吗？ | Codex CLI，三种会话结构，n=3 | **9/9 遵守决策**；每次都用了文件降级路径 | 低到中 —— 单一模型，n=3 |
+| **E6** | 真实使用中帮得上忙吗？ | 14+14 天，真实项目 | **未运行。** 项目所有者不接受每日记录 | — |
+
+---
+
+## 三条站得住的发现
+
+### 1. 模型不把年龄当信号
+
+30 天前写的决策和 7 年前写的，被遵循的比例**一模一样**——两个模型、60 次运行，零偏差。把过期的那条藏起来，行为完全回到基线。
+
+**含义：** 让 agent「对旧笔记保持怀疑」，等于让它权衡一个它没有的量。过期必须是结构性的。
+
+→ [e4-dose-response.md](e4-dose-response.md)
+
+### 2. 记录一条否决，会实质改变答案
+
+三个模型家族、n=20 合并：Redis 那条从 52/60 → 0/60，TypeScript 那条从 60/60 → 13/60。两者 p < 0.0001。
+
+**含义：** 核心机制是真实且显著的。
+
+→ [e3-power.md](e3-power.md)
+
+### 3. 写的理由比日期更重要
+
+同一个标题、同样的年龄，两种理由：
+
+| 记录里的理由 | 读起来像 | 抵抗率 |
+|---|---|---|
+| 「团队没人会 TS」 | 一个**会变的条件** | **13/60（22%）** |
+| 「这个项目字段变更频繁」 | 一个**项目属性** | **0/60** |
+
+抵抗的回复直接攻击前提：*「那个条件现在很可能已经不成立了。」*
+
+**含义，而且反直觉：** 命名了**「条件」**的理由比陈述了**「属性」**的更安全，因为它邀请模型去复核。通常「写持久理由」的建议，对这个工具是反的。
+
+→ [e3-power.md](e3-power.md)、[e4-dose-response.md](e4-dose-response.md)
+
+---
+
+## 最重要的限定
+
+**E1 表明效应不是普遍的。**
+
+有真实代码、没有记录时，GPT 已经在 12/12 的运行为仓库现有的技术选型；加上记录没有任何变化。DeepSeek 面对同样的代码，在 10/12 的运行为仍然倾向于被否决的方案，直到记录拦住它。
+
+所以诚实的主张不是「agent 会盲从陈旧决策」，而是：
+
+> 一条记录是否起作用，取决于模型是否使用代码。**代码编码了「选了什么」；仓库里没有任何东西编码「否决了什么」。记录对第二类最有价值。**
+
+→ [e1-code-context.md](e1-code-context.md)
+
+---
+
+## 还没有测的东西
+
+| 缺口 | 为什么重要 |
+|---|---|
+| **真实使用中的价值（E6）** | 用户唯一关心的数字。没测。这个项目不声称它。 |
+| 模糊回复（E0） | 判分校准用的是几乎全部清晰的案例。 |
+| 带上下文压缩的长会话 | E5 最多两轮。 |
+| Claude / Gemini / 更大的 DeepSeek | E3 用的是 Haiku 和 DeepSeek Flash。 |
+| 有很多条决策的仓库 | 所有实验只用了 1–2 条。累积干扰未测。 |
+| 大规模误用 | E2 只测了一条不相关决策。 |
+
+---
+
+## 版本说明
+
+这些结果描述的是 2026-10-21 的实现。实验是脚本，不是结论——如果后续改动与这里的数字矛盾，以脚本为准，这一页会被修正。
+
+完整的修订历史（包括两个第一次就做错的实验）见 [preregistration.md](preregistration.md)。

package/docs/judge-calibration.md

@@ -0,0 +1,170 @@
+# E0 — Judge calibration
+
+**Decision: PASS.** The judges are usable. One caveat below is important enough
+that it should travel with every downstream number.
+
+Run: 2026-10-20 · 62 blinded responses · 3 judges · 186 model calls
+
+---
+
+## Why this was the first experiment
+
+Every claim in [validation.md](validation.md) rests on a model judging whether a
+response recommends or excludes an option. If that judgement is unreliable, all
+the numbers are unreliable. This was the cheapest experiment that could
+invalidate everything else.
+
+The pre-registered rule: if any judge's κ fell below 0.60, all further
+experiments stop until the scoring is redesigned. See
+[preregistration.md](preregistration.md).
+
+---
+
+## Method
+
+1. All 62 existing responses were anonymised: condition stripped from the
+   filename, IDs assigned (`E0-001` … `E0-062`), order shuffled with a fixed
+   seed.
+2. Labels were assigned blind — from response text only, with no knowledge of
+   which experiment or model produced them.
+3. Three judges from different families scored the same 62 items at
+   `temperature: 0`.
+4. Agreement measured with Cohen's κ against the human labels, and pairwise
+   between judges.
+
+| Judge | Family | Model |
+|---|---|---|
+| A | Anthropic | `opencode-go/claude-haiku-5-5` |
+| B | Moonshot | `moonshot-cn/kimi-k3` |
+| C | DeepSeek | `workbuddy-ai/deepseek-v4.1-flash` |
+
+Judge C was chosen deliberately from the same family as some of the judged
+outputs, to test for same-family leniency.
+
+---
+
+## Results
+
+### All 62 items
+
+| Judge | κ | Agreement | Errors |
+|---|---|---|---|
+| Claude Haiku 5.5 | 0.936 | 96.8% | 0 |
+| Kimi K3 | 0.936 | 96.8% | 0 |
+| DeepSeek Flash | 0.937 | 96.8% | 1 |
+| Majority vote | 0.936 | 96.8% | — |
+
+### Excluding the two malformed samples
+
+Two responses (`E0-054`, `E0-061`) were `reasoning_content` dumps — the model
+never produced a final answer. They were labelled `unclear`. The judges read
+the reasoning and inferred a recommendation, which is defensible but not the
+same thing as reading a conclusion.
+
+On the 60 well-formed responses:
+
+| Judge | κ | Agreement |
+|---|---|---|
+| Claude Haiku 5.5 | **1.000** | 60/60 |
+| Kimi K3 | **1.000** | 60/60 |
+| DeepSeek Flash | **1.000** | 60/60 |
+| Majority vote | **1.000** | 60/60 |
+
+### Pairwise
+
+| Pair | κ | Agreement |
+|---|---|---|
+| Claude vs Kimi | 1.000 | 100% |
+| Claude vs DeepSeek | 0.935 | 96.8% |
+| Kimi vs DeepSeek | 0.935 | 96.8% |
+
+### Same-family leniency: not detected
+
+The DeepSeek judge scored DeepSeek-produced responses no differently from the
+others. κ was 0.937 versus 0.936 for the other two. This was the specific thing
+experiment design was checking for, and the null result is worth recording.
+
+---
+
+## The caveat that matters
+
+**κ = 1.000 does not mean "the judges are excellent". It means "this labelling
+task is easy".**
+
+Almost every response states its conclusion explicitly — "建议采用 TypeScript"
+or "不使用 TypeScript" — within the first few lines. The sample contains
+essentially **no genuinely ambiguous cases**. So this experiment establishes:
+
+- The judges do not systematically disagree on clear responses. ✅
+- The downstream numbers are not an artefact of sloppy scoring on clear
+  responses. ✅
+
+It does **not** establish:
+
+- That the judges handle hedged or ambiguous responses correctly. ❌
+- That the labels are independent of the experimenter. ❌
+
+A perfect κ is a warning sign as much as a result. It says the measurement is
+not the hard part of this research; the experimental design is.
+
+---
+
+## Independence caveat — and the audit result
+
+The "human" labels were produced by the same assistant that designed the
+experiments. They were blind to condition, but they are not an independent
+reference. A 20-item audit by the project owner was required.
+
+**Audit result: 19/20 agreement, within the pre-registered threshold of 2.**
+
+| | |
+|---|---|
+| Items reviewed | 20 |
+| Agreement | 19 |
+| Disagreements | 1 |
+| Threshold | ≤ 2 |
+| Result | **PASS** |
+
+The single remaining disagreement (`E0-061`) is genuine: the response is a
+truncated reasoning dump with no final conclusion, so `recommends` and
+`unclear` are both defensible.
+
+Two initial disagreements (`E0-013`, `E0-026`) were corrected by the reviewer
+after reading the full response text; both responses state the conclusion
+explicitly. This is a useful signal about the review process itself: **an
+excerpt is not always enough for a human reviewer**, even when it is enough for
+a model. The audit sheet truncates long responses, and that truncation caused
+two of the three initial disagreements.
+
+Files: `experiments/e0/audit-sample.md`, `answer-sheet.md`, `audit-key.json`,
+`audit-result.json`.
+
+**Treat the labels as an independent-enough reference, not as ground truth.**
+The reviewer and the assistant agreed on 19 of 20 items, which is evidence that
+the labels are not idiosyncratic. It is not evidence that either is correct on
+the genuinely ambiguous cases.
+
+---
+
+## What this changes downstream
+
+1. **Majority vote is safe to use** for E1–E5, with the `unclear` rate reported.
+2. **Report the `unclear` count in every experiment.** Dropping unclear items
+   inflates agreement; they stay in the denominator.
+3. **Add a human spot-check to E3**, because E3's sample will contain more
+   ambiguous cases than this one did.
+4. **Do not claim the judges are validated on hard cases.** They are validated
+   on easy ones.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e0/judge.mjs --dry   # print the plan
+node experiments/e0/judge.mjs         # 186 calls
+node experiments/e0/make-audit.mjs    # render the 20-item human audit
+```
+
+Files: `blind-sample.json`, `human-labels.json`, `judge-*.json`,
+`calibration.json`, `audit-sample.md`, `audit-key.json`.