whynot-cli 0.0.0-stage → 0.1.0

package/docs/judge-calibration.md

@@ -0,0 +1,170 @@
+# E0 — Judge calibration
+
+**Decision: PASS.** The judges are usable. One caveat below is important enough
+that it should travel with every downstream number.
+
+Run: 2026-10-20 · 62 blinded responses · 3 judges · 186 model calls
+
+---
+
+## Why this was the first experiment
+
+Every claim in [validation.md](validation.md) rests on a model judging whether a
+response recommends or excludes an option. If that judgement is unreliable, all
+the numbers are unreliable. This was the cheapest experiment that could
+invalidate everything else.
+
+The pre-registered rule: if any judge's κ fell below 0.60, all further
+experiments stop until the scoring is redesigned. See
+[preregistration.md](preregistration.md).
+
+---
+
+## Method
+
+1. All 62 existing responses were anonymised: condition stripped from the
+   filename, IDs assigned (`E0-001` … `E0-062`), order shuffled with a fixed
+   seed.
+2. Labels were assigned blind — from response text only, with no knowledge of
+   which experiment or model produced them.
+3. Three judges from different families scored the same 62 items at
+   `temperature: 0`.
+4. Agreement measured with Cohen's κ against the human labels, and pairwise
+   between judges.
+
+| Judge | Family | Model |
+|---|---|---|
+| A | Anthropic | `opencode-go/claude-haiku-5-5` |
+| B | Moonshot | `moonshot-cn/kimi-k3` |
+| C | DeepSeek | `workbuddy-ai/deepseek-v4.1-flash` |
+
+Judge C was chosen deliberately from the same family as some of the judged
+outputs, to test for same-family leniency.
+
+---
+
+## Results
+
+### All 62 items
+
+| Judge | κ | Agreement | Errors |
+|---|---|---|---|
+| Claude Haiku 5.5 | 0.936 | 96.8% | 0 |
+| Kimi K3 | 0.936 | 96.8% | 0 |
+| DeepSeek Flash | 0.937 | 96.8% | 1 |
+| Majority vote | 0.936 | 96.8% | — |
+
+### Excluding the two malformed samples
+
+Two responses (`E0-054`, `E0-061`) were `reasoning_content` dumps — the model
+never produced a final answer. They were labelled `unclear`. The judges read
+the reasoning and inferred a recommendation, which is defensible but not the
+same thing as reading a conclusion.
+
+On the 60 well-formed responses:
+
+| Judge | κ | Agreement |
+|---|---|---|
+| Claude Haiku 5.5 | **1.000** | 60/60 |
+| Kimi K3 | **1.000** | 60/60 |
+| DeepSeek Flash | **1.000** | 60/60 |
+| Majority vote | **1.000** | 60/60 |
+
+### Pairwise
+
+| Pair | κ | Agreement |
+|---|---|---|
+| Claude vs Kimi | 1.000 | 100% |
+| Claude vs DeepSeek | 0.935 | 96.8% |
+| Kimi vs DeepSeek | 0.935 | 96.8% |
+
+### Same-family leniency: not detected
+
+The DeepSeek judge scored DeepSeek-produced responses no differently from the
+others. κ was 0.937 versus 0.936 for the other two. This was the specific thing
+experiment design was checking for, and the null result is worth recording.
+
+---
+
+## The caveat that matters
+
+**κ = 1.000 does not mean "the judges are excellent". It means "this labelling
+task is easy".**
+
+Almost every response states its conclusion explicitly — "建议采用 TypeScript"
+or "不使用 TypeScript" — within the first few lines. The sample contains
+essentially **no genuinely ambiguous cases**. So this experiment establishes:
+
+- The judges do not systematically disagree on clear responses. ✅
+- The downstream numbers are not an artefact of sloppy scoring on clear
+  responses. ✅
+
+It does **not** establish:
+
+- That the judges handle hedged or ambiguous responses correctly. ❌
+- That the labels are independent of the experimenter. ❌
+
+A perfect κ is a warning sign as much as a result. It says the measurement is
+not the hard part of this research; the experimental design is.
+
+---
+
+## Independence caveat — and the audit result
+
+The "human" labels were produced by the same assistant that designed the
+experiments. They were blind to condition, but they are not an independent
+reference. A 20-item audit by the project owner was required.
+
+**Audit result: 19/20 agreement, within the pre-registered threshold of 2.**
+
+| | |
+|---|---|
+| Items reviewed | 20 |
+| Agreement | 19 |
+| Disagreements | 1 |
+| Threshold | ≤ 2 |
+| Result | **PASS** |
+
+The single remaining disagreement (`E0-061`) is genuine: the response is a
+truncated reasoning dump with no final conclusion, so `recommends` and
+`unclear` are both defensible.
+
+Two initial disagreements (`E0-013`, `E0-026`) were corrected by the reviewer
+after reading the full response text; both responses state the conclusion
+explicitly. This is a useful signal about the review process itself: **an
+excerpt is not always enough for a human reviewer**, even when it is enough for
+a model. The audit sheet truncates long responses, and that truncation caused
+two of the three initial disagreements.
+
+Files: `experiments/e0/audit-sample.md`, `answer-sheet.md`, `audit-key.json`,
+`audit-result.json`.
+
+**Treat the labels as an independent-enough reference, not as ground truth.**
+The reviewer and the assistant agreed on 19 of 20 items, which is evidence that
+the labels are not idiosyncratic. It is not evidence that either is correct on
+the genuinely ambiguous cases.
+
+---
+
+## What this changes downstream
+
+1. **Majority vote is safe to use** for E1–E5, with the `unclear` rate reported.
+2. **Report the `unclear` count in every experiment.** Dropping unclear items
+   inflates agreement; they stay in the denominator.
+3. **Add a human spot-check to E3**, because E3's sample will contain more
+   ambiguous cases than this one did.
+4. **Do not claim the judges are validated on hard cases.** They are validated
+   on easy ones.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e0/judge.mjs --dry   # print the plan
+node experiments/e0/judge.mjs         # 186 calls
+node experiments/e0/make-audit.mjs    # render the 20-item human audit
+```
+
+Files: `blind-sample.json`, `human-labels.json`, `judge-*.json`,
+`calibration.json`, `audit-sample.md`, `audit-key.json`.

package/docs/preregistration.md

@@ -0,0 +1,257 @@
+# Pre-registration
+
+Written before the supplementary experiments were run, so that the decision
+rules cannot be adjusted after seeing the results.
+
+Created: 2026-10-20
+Status: active. Do not edit after data collection begins.
+
+---
+
+## Why this file exists
+
+The original experiments reported numbers after the fact, with thresholds
+chosen implicitly. That makes it impossible for a reader to tell whether a
+result was predicted or fitted. This document fixes the hypotheses, sample
+sizes, and decision rules in advance.
+
+If a result contradicts what is written here, the honest move is to publish the
+contradiction, not to rewrite the rule.
+
+---
+
+## Hypotheses
+
+### H1 — stale decisions are obeyed
+An agent given a decision that no longer holds will follow it rather than
+question it.
+
+**Prediction:** in the stale condition, ≥70% of responses avoid the rejected
+option.
+
+### H2 — expiry removes the effect
+If an expired entry is not shown to the agent at all, H1's effect disappears.
+
+**Prediction:** in the expired condition, the rate of avoiding the option falls
+to within 15 percentage points of the no-record baseline.
+
+### H3 — code context reduces it (the "information vacuum" hypothesis)
+The original tasks contained no code, so the model had nothing to reason from
+except the note. With a real codebase in context, the note's influence should
+be weaker.
+
+**Prediction:** with code context, the avoidance rate in the stale condition is
+at least 20 points lower than without.
+
+**This is the hypothesis most likely to be true and least flattering to the
+project.** If it holds, the public claim must be rewritten from "agents blindly
+obey stale decisions" to "agents obey given constraints when they have no other
+information".
+
+### H4 — decisions are over-applied (false obedience)
+An agent given a strong but irrelevant decision will let it distort an unrelated
+answer.
+
+**Prediction:** false-obedience rate < 10%.
+
+### H5 — obedience increases with age
+Older decisions are obeyed more often.
+
+**Prediction:** monotonic increase across 1 month → 7 years.
+
+### H5' — obedience is age-independent
+The competing hypothesis. Models have no sense of how old a note is.
+
+**Prediction:** flat across 1 month → 7 years.
+
+H5 and H5' are mutually exclusive. **Both are publishable.** H5' is the stronger
+finding because it means the problem is structural, not a matter of degree.
+
+### H6 — prompt warnings do not work
+Adding "this is old, be skeptical" to the supplied context does not change
+behaviour; removing the entry does.
+
+**Status:** already tested once (experiment 3). Result: no reliable improvement.
+
+---
+
+## Experiments and decision rules
+
+### E0 — judge calibration
+
+**Design:** all 62 existing responses, blinded, scored by three judges from
+different model families (Claude Haiku, Kimi K3, DeepSeek Flash). Compared
+against human labels using Cohen's kappa.
+
+**Decision rule (fixed before running):**
+
+| worst judge κ | action |
+|---|---|
+| ≥ 0.80 | PASS — use majority vote downstream |
+| 0.60 – 0.80 | WARN — proceed, but disclose and add human spot-checks |
+| < 0.60 | **FAIL — stop all further experiments until scoring is redesigned** |
+
+**Known limitation:** the "human" labels were produced by the same assistant
+that designed the experiments, blind to condition but not independent. The user
+must audit a random subset. If the audit disagrees on more than 10% of items,
+the labels are not a valid reference and E0 is re-run with the user's labels.
+
+---
+
+### E1 — code context
+
+**Design:** three fixture repositories (small/medium/large), each with a planted
+decision. Conditions: no decision, decision injected, decision discovered by
+reading the repo.
+
+**Sample:** n=10 per condition per model, 2 models.
+
+**Decision rule:** H3 is supported if the code-context condition shows an
+avoidance rate at least 20 points lower than the injected condition, with a
+confidence interval excluding zero.
+
+---
+
+### E2 — false obedience
+
+**Design:** a strongly worded but irrelevant decision, plus a relevant-decision
+control.
+
+**Sample:** n=15 per condition, 2 models.
+
+**Decision rule:**
+
+| false-obedience rate | action |
+|---|---|
+| < 5% | acceptable |
+| 5 – 15% | add a constraint to the AGENTS.md instruction and retest |
+| > 15% | **design defect — do not ship until fixed** |
+
+---
+
+### E3 — statistical power
+
+**Design:** the Redis and TypeScript experiments at n=20 per condition, three
+model families.
+
+**Primary analysis:** pooled effect across models, Wilson 95% intervals, Fisher
+exact test, risk difference.
+
+**Decision rule:**
+- pooled p < 0.01 and risk difference ≥ 0.40 → main claim stands
+- p < 0.05 with a smaller effect → downgrade the wording
+- not significant → **publish the correction**
+
+---
+
+### E4 — dose-response
+
+**Design:** one decision, six ages (1 month, 6 months, 1 year, 3 years, 7 years,
+plus an expired 7-year control), n=12 per cell, 2 models.
+
+**Analysis:** Cochran-Armitage trend test.
+
+**Decision rule:**
+- D1 < 30% and D5 > 70% → H5 (dose effect)
+- D1..D5 all > 70% → H5' (age-independent). Rewrite the narrative accordingly.
+- D6 not close to D0 → the expiry mechanism is faulty; stop and fix.
+
+---
+
+### E5 — multi-turn with a real agent
+
+**Design:** Codex CLI against a fixture repo, 8–12 turns, three conditions.
+
+**Decision rule:** report the turn at which the decision stops influencing
+behaviour, and the token cost. No pass/fail threshold — this is descriptive.
+
+---
+
+### E6 — two-week field test
+
+**Status: not run.**
+
+**Consequence:** the project may claim that the mechanism works under test. It
+may **not** claim that using the tool reduces repeated mistakes.
+
+**Design (for anyone who wants to run it):** 14 days without the tool, 14 days
+with it, on a real project.
+
+**Primary metric:** repeated-mistake rate — how often the agent re-proposes
+something already rejected.
+
+**Decision rule:**
+- ≥ 30% reduction → the tool helps
+- < 10% reduction → it does not
+- increase → it hurts
+
+---
+
+## Amendments
+
+Amendments are recorded with the date and the reason. Changing a decision rule
+after seeing data is exactly what pre-registration exists to prevent, so any
+change is logged here and both the original and amended rules are reported.
+
+### 2026-10-21 — E2 metric definition
+
+**Original:** "false-obedience rate < 10%", without specifying what counted.
+
+**Problem found during analysis:** the implementation used "response mentions
+the irrelevant option at all". Manual inspection of the 9 flagged responses
+showed they all said the decision *did not apply* — correct behaviour, not
+obedience. The metric conflated "considered and dismissed" with "distorted by".
+
+**Amendment:** report both metrics.
+
+- mentions: 9/30 (30%) — the pre-registered reading, which is a FAIL
+- design distorted: 0/30 (0%) — the substantive reading
+
+Both are published in `e2-false-obedience.md`. The stricter number is not
+hidden and the redefinition is argued from the raw text, not from preference.
+
+**Future experiments:** pre-register the exact rubric or the exact string
+matched, not just the concept.
+
+### 2026-10-21 — E4 judge failure
+
+Kimi K3 returned `error` for every response from D2 onward. The original
+summary counted errors as non-events, turning a judging failure into a clean
+0/12. All 168 responses were re-judged with DeepSeek Flash.
+
+**Rule added:** an error is never a data point. Errors are excluded from
+denominators and reported. This rule is now enforced in the E3 script, which
+re-judges failures with a fallback judge.
+
+### 2026-10-21 — E3 query bug (caught before analysis)
+
+The first E3 run used the query `技术 选型 存储 语言` for a decision titled
+`session 不存 Redis`. No query term matched, so `decisions_check` returned
+"nothing found" and the treatment condition was identical to the control. The
+run was stopped before any analysis.
+
+**Rule added:** every experiment that injects a decision now asserts that the
+returned context contains the decision's title, and aborts otherwise.
+
+---
+
+## Analysis conventions
+
+- Report confidence intervals, never bare proportions.
+- Report the number of excluded or errored samples.
+- Report inter-rater reliability for any model-judged outcome.
+- Judge calls use `temperature: 0`.
+- Any response that is truncated or missing a conclusion is labelled `unclear`
+  and included in the denominator. Dropping it would inflate agreement.
+
+## What would make me abandon the project
+
+Stated in advance so it cannot be rationalised later:
+
+1. E0 fails (κ < 0.60) and the scoring cannot be redesigned.
+2. E2 shows > 15% false obedience and the instruction cannot fix it.
+3. E3 shows no significant effect at n=20.
+4. E6 shows no reduction in repeated mistakes over two weeks.
+
+Any one of these is sufficient. The project's value depends on the claims being
+true, not on the claims being published.

package/docs/validation.md

@@ -0,0 +1,30 @@
+# Validation
+
+This page is now an index. The consolidated results are in
+[findings.md](findings.md); the detailed write-ups are below.
+
+| Experiment | Question | Write-up |
+|---|---|---|
+| E0 | Can a model reliably judge "recommends vs excludes"? | [judge-calibration.md](judge-calibration.md) |
+| E1 | Does real code context weaken the effect? | [e1-code-context.md](e1-code-context.md) |
+| E2 / E2b | Does an irrelevant decision distort unrelated work? | [e2-false-obedience.md](e2-false-obedience.md) |
+| E3 | Are the effects statistically real? | [e3-power.md](e3-power.md) |
+| E4 | Does the age of a decision matter? | [e4-dose-response.md](e4-dose-response.md) |
+| E5 | Does a real agent use it across sessions? | [e5-real-agent.md](e5-real-agent.md) |
+| E6 | Does it help in real use? | **Not run** — see [preregistration.md](preregistration.md) |
+
+## A note on the earlier version of this page
+
+The first version of this document reported results from a single small
+experiment (n=3 per condition, two models). Those numbers appeared in the
+README and in the launch material.
+
+Several of them were misleading, and one was simply wrong:
+
+- The "6 of 6 obeyed" framing understated the variance; at n=20 the same
+  decision was resisted in 13 of 60 responses.
+- The first judging pass for the dose-response experiment silently counted
+  judge errors as data, producing a clean-looking result that did not exist.
+
+Both are documented in [preregistration.md](preregistration.md). The numbers
+that appear throughout the project now come from the experiments linked above.

package/docs/validation.zh-CN.md

@@ -0,0 +1,24 @@
+# 验证
+
+这一页现在是一个索引。汇总结果在 [findings.zh-CN.md](findings.zh-CN.md)（英文版 [findings.md](findings.md)），详细写稿在下面。
+
+| 实验 | 问题 | 写稿 |
+|---|---|---|
+| E0 | 模型能可靠地判断「推荐 vs 排除」吗？ | [judge-calibration.md](judge-calibration.md) |
+| E1 | 真实代码上下文会削弱效应吗？ | [e1-code-context.md](e1-code-context.md) |
+| E2 / E2b | 不相关的决策会扭曲无关工作吗？ | [e2-false-obedience.md](e2-false-obedience.md) |
+| E3 | 效应在统计上成立吗？ | [e3-power.md](e3-power.md) |
+| E4 | 决策的年龄重要吗？ | [e4-dose-response.md](e4-dose-response.md) |
+| E5 | 真实 agent 会跨会话使用它吗？ | [e5-real-agent.md](e5-real-agent.md) |
+| E6 | 真实使用中帮得上忙吗？ | **未运行** —— 见 [preregistration.md](preregistration.md) |
+
+## 关于这一页的旧版本
+
+这一页最早报告的是单个小实验的结果（每条件 n=3，两个模型）。那些数字曾经出现在 README 和发布材料里。
+
+其中有几个是误导性的，有一个是**错的**：
+
+- 「6/6 全部服从」这个说法低估了方差；n=20 时同一条决策在 60 次回复里被抵抗了 13 次。
+- 剂量-反应实验的第一次判分把判分错误静默地当成了数据，产出了一个看起来很干净、实际不存在的结果。
+
+两件事都记录在 [preregistration.md](preregistration.md) 里。现在项目中使用的数字都来自上面链接的实验。

package/package.json

@@ -1,6 +1,66 @@
 {
   "name": "whynot-cli",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "Record what you ruled out, and why, so your coding agent stops re-suggesting it.",
+  "type": "module",
+  "author": "Stakenshake",
+  "homepage": "https://github.com/Stakenshake/whynot#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Stakenshake/whynot.git"
+  },
+  "bugs": {
+    "url": "https://github.com/Stakenshake/whynot/issues"
+  },
+  "bin": {
+    "whynot": "bin/whynot.js"
+  },
+  "exports": {
+    ".": "./src/index.js",
+    "./mcp": "./src/mcp.js",
+    "./audit": "./src/audit.js"
+  },
+  "files": [
+    "bin",
+    "src",
+    "docs/design.md",
+    "docs/design.zh-CN.md",
+    "docs/findings.md",
+    "docs/findings.zh-CN.md",
+    "docs/validation.md",
+    "docs/validation.zh-CN.md",
+    "docs/judge-calibration.md",
+    "docs/e1-code-context.md",
+    "docs/e2-false-obedience.md",
+    "docs/e3-power.md",
+    "docs/e4-dose-response.md",
+    "docs/e5-real-agent.md",
+    "docs/preregistration.md",
+    "AGENTS.snippet.md",
+    "AGENTS.snippet.zh-CN.md",
+    "README.md",
+    "README.zh-CN.md",
+    "CONTRIBUTING.md",
+    "LICENSE"
+  ],
+  "engines": {
+    "node": ">=20"
+  },
+  "scripts": {
+    "test": "node --test \"test/*.test.js\"",
+    "mcp": "node bin/whynot.js mcp"
+  },
+  "keywords": [
+    "mcp",
+    "model-context-protocol",
+    "coding-agents",
+    "claude-code",
+    "codex",
+    "context-engineering",
+    "decision-log",
+    "architecture-decision-record",
+    "adr"
+  ],
+  "license": "MIT",
+  "dependencies": {}
+}