whynot-cli 0.0.0-stage → 0.1.1

package/docs/e1-code-context.md

@@ -0,0 +1,169 @@
+# E1 — Does real code context change the effect?
+
+**Result: it depends on the model, and the pooled number hides that. GPT
+replaces the decision with the code; DeepSeek does not.**
+
+This is the experiment that most threatens the project's claim, and the result
+is genuinely mixed. It also produces the sharpest statement of what the tool is
+actually for.
+
+---
+
+## Why this experiment exists
+
+Every earlier experiment gave the model a task with no code. That leaves an
+obvious alternative explanation:
+
+> The model obeyed the decision only because it had nothing else to reason from.
+
+If true, the public claim would have to change from "agents blindly obey stale
+decisions" to "agents obey any given constraint when starved of context".
+
+---
+
+## Design
+
+A real Node/Express project (`experiments/e1/fixture/`, 19 files, ~410 lines)
+with conventions that matter: Postgres for all app data, service layer takes
+`tenantId` first, `REDIS_URL` declared in config but unused, README notes that
+Redis was being evaluated.
+
+Four conditions, n=12 per model per cell:
+
+| | Code context | Decision |
+|---|---|---|
+| C0 | no | no |
+| C1 | no | yes |
+| C2 | yes | no |
+| C3 | yes | yes |
+
+Task: add session storage to this project, 100k DAU, two regions.
+
+Models: GPT-6.1-Sol, DeepSeek V4.1 Flash. Judge: Claude Haiku (third family).
+**96 responses, 0 judging errors.**
+
+The decision was injected through the real `decisions_check` tool; the script
+aborts if the returned context does not contain the entry.
+
+---
+
+## Results
+
+| Condition | GPT | DeepSeek | Pooled |
+|---|---|---|---|
+| C0 no code, no decision | 10/12 | 12/12 | **22/24 (92%)** |
+| C1 no code + decision | 0/12 | 0/12 | **0/24 (0%)** |
+| C2 code, no decision | **0/12** | **10/12** | **10/24 (42%)** |
+| C3 code + decision | 0/12 | 0/12 | **0/24 (0%)** |
+
+Derived effects:
+
+| Effect | Size |
+|---|---|
+| Decision, without code (C1 − C0) | −92 pts |
+| Decision, with code (C3 − C2) | −42 pts |
+| Code alone, no decision (C2 − C0) | −50 pts |
+
+The pooled H3 test — "does the decision still work with code?" — reads as
+**0 pts change** and the automated verdict said "H3 not supported".
+
+**That verdict is wrong, because the two models do opposite things.**
+
+---
+
+## The split
+
+### GPT: the code replaces the decision
+
+With code and no decision, GPT already refuses Redis in 12 of 12 runs. It says
+so explicitly:
+
+> "现有 `REDIS_URL` 只是历史配置，不应成为选用 Redis 的理由。"
+> *(The existing REDIS_URL is just historical config; it is not a reason to pick Redis.)*
+
+> "现有项目的数据已经全部放在 PostgreSQL，增加 session 表能直接复用连接池、备份和运维体系。暂时不引入 Redis。"
+
+Adding the recorded decision changes nothing (0/12 → 0/12). For GPT, **the code
+is sufficient**; the decision is redundant in this case.
+
+### DeepSeek: the code is not enough
+
+DeepSeek reads the same code and still recommends Redis in 10 of 12 runs. Only
+when the decision is present does it stop (0/12).
+
+For DeepSeek, the decision remains **decisive even with the code in front of it**.
+
+---
+
+## What this actually says
+
+The pooled framing ("the effect survives code context") is technically what the
+numbers say and substantively wrong. The honest statement:
+
+> Whether a recorded decision matters depends on whether the model uses the code.
+> For a model that reads the repository and infers its conventions, a decision
+> that the code already implies is redundant. For a model that does not, the
+> decision is the only thing that stops it.
+
+### And it sharpens the product thesis
+
+The fixture's code makes "use Postgres" obvious — that is what code does, it
+encodes the choices that were made. **What code cannot encode is the
+alternative that was rejected.** Nothing in the repository says "we tried Redis
+and it failed at 8 seconds", because a rejected option leaves no artifact.
+
+That is exactly the gap the tool fills, and E1 shows it concretely:
+
+- GPT inferred the chosen option from the code, so the decision added nothing.
+- Neither model could infer the **rejection** from the code, and for DeepSeek
+  that mattered at 10/12.
+
+So the correct positioning is narrower and stronger than the original:
+
+> Don't record decisions the code already implies. Record the alternatives that
+> were rejected, because nothing else in the repository remembers them.
+
+---
+
+## What to change in the public claim
+
+The README should not say "agents blindly obey stale decisions" without
+qualification. The defensible version:
+
+> A recorded rejection changes what an agent proposes by 42–92 percentage
+> points, depending on the model and on whether the repository already implies
+> the answer. Models that read the code can infer what was chosen; none of them
+> can infer what was rejected.
+
+---
+
+## Caveats
+
+- Two models. The split may look different on Claude or on a larger DeepSeek.
+- One fixture, written by us, with deliberately strong conventions. A messier
+  repo might not substitute for the decision as cleanly.
+- n=12 per cell. The GPT/DeepSeek split is stark (0/12 vs 10/12), but the exact
+  rates should not be treated as population estimates.
+- The task is still synthetic. E5 (multi-turn with a real agent) is the next
+  step for external validity.
+
+---
+
+## A reporting lesson
+
+The script printed "H3 NOT SUPPORTED — the effect survives code context" based
+on pooled rates. The per-model table shows the opposite pattern for one model.
+**Pooling across models hid a sign flip in the mechanism.**
+
+Rule for the remaining experiments: when the per-model cells disagree in
+direction, the pooled number is not a summary — it is a fiction. Report the
+split and say so.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e1/build-fixture.mjs
+node experiments/e1/run.mjs
+```

package/docs/e2-false-obedience.md

@@ -0,0 +1,167 @@
+# E2 — False obedience: does an irrelevant decision distort an unrelated answer?
+
+**Result: no. The measured 30% was a metric artefact; actual false obedience is
+0 out of 30.**
+
+This experiment also produced a lesson about metric definition that is more
+useful than the result itself.
+
+---
+
+## Why this experiment exists
+
+Every other experiment measures one failure: the agent forgets something it
+should have remembered. This one measures the opposite: the agent applies a
+decision it should have ignored.
+
+If the tool's whole premise is "record what you ruled out", the obvious risk is
+that a growing pile of decisions starts distorting unrelated work. That risk had
+never been tested.
+
+---
+
+## Design
+
+**Task:** design a PostgreSQL `users` table, full DDL. This task has nothing to
+do with GraphQL.
+
+| Condition | Injected decision | Expected behaviour |
+|---|---|---|
+| N0 | none | baseline |
+| **N1** | "never use GraphQL anywhere" — strongly worded, irrelevant | should be ignored |
+| N2 | "passwords must not be stored in plaintext" — relevant | should change the design |
+
+Models: `codex/gpt-6.1-sol`, `opencode-go/deepseek-v4-pro`. n=15 per condition.
+
+The decision was injected as text in the system prompt, with the same
+`AGENTS.md` instruction used elsewhere.
+
+---
+
+## Result 1: the pre-registered metric said FAIL
+
+The metric fixed in advance was **"does the response mention GraphQL at all"**,
+on the reasoning that an irrelevant decision should leave no trace.
+
+| Model | N0 mentions | N1 mentions |
+|---|---|---|
+| GPT-6.1-Sol | 0/15 | **0/15** |
+| DeepSeek V4 Pro | 0/15 | **9/15** |
+| **Total** | **0/30** | **9/30 = 30%** |
+
+Under the pre-registered threshold (5–15% warn, >15% fail), this is a **FAIL —
+design defect**.
+
+---
+
+## Result 2: manual inspection shows the metric was wrong
+
+All nine DeepSeek responses that "mentioned GraphQL" did so in one specific way:
+
+> "已按现有决策检查：`0001 不使用 GraphQL` **仅约 API，不涉及本表设计**。"
+> *(Checked the recorded decision: 0001 only concerns the API, not this table.)*
+
+> "已检查 `.decisions/`：0001 仅限制不使用 GraphQL，**与本表设计无关**。"
+> *(0001 only restricts GraphQL, which is unrelated to this table design.)*
+
+> "已检查仓库决策 `0001`（不使用 GraphQL），**与本数据库表设计无冲突**；以下方案只涉及 PostgreSQL DDL。"
+
+The one response the regex flagged as "treating it as a constraint" says:
+
+> "0001 不使用 GraphQL，与数据库设计无关。**所以不影响**。"
+
+**Every one of these is correct relevance judgement, stated out loud, followed
+by a normal design.** The model read the decision, decided it did not apply, and
+said so. That is exactly what the tool should produce.
+
+The metric "mentions GraphQL" conflates two opposite behaviours:
+
+- considering the decision and correctly dismissing it (good)
+- being distorted by the decision (bad)
+
+Only the second is false obedience, and it occurred **0 times in 30 runs**.
+
+---
+
+## Result 3: the original positive control failed
+
+The N2 condition was meant to prove the model reads decisions at all. It did
+not work: a competent schema hashes passwords anyway.
+
+| Model | N0 hashes | N2 hashes |
+|---|---|---|
+| GPT | 15/15 | 15/15 |
+| DeepSeek | 15/15 | 15/15 |
+
+No difference. So E2 on its own could not distinguish "correctly ignored an
+irrelevant decision" from "never read any decision".
+
+### E2b — a control that actually discriminates
+
+A decision a model would *not* choose by default:
+
+> **users 表主键必须使用 `CHAR(26)` ULID。禁止 SERIAL / BIGSERIAL / IDENTITY，也禁止 UUID。**
+
+| Model | N0 uses ULID | N3 uses ULID | N0 uses SERIAL | N3 uses SERIAL |
+|---|---|---|---|---|
+| GPT-6.1-Sol | 0/12 | **12/12** | 12/12 | **0/12** |
+| Claude Haiku 5.5 | 0/12 | **12/12** | 12/12 | 2/12 |
+
+**24/24 = 100% followed the ULID constraint.** Both models default to `SERIAL`
+and both completely switch when told not to. The models demonstrably read
+decisions.
+
+Therefore E2's zero false-obedience rate is meaningful: the model read the
+GraphQL decision, judged it irrelevant, said so, and was not distorted by it.
+
+---
+
+## Honest note on changing the metric
+
+The pre-registration said "false-obedience rate" without defining whether
+"mention" or "distortion" counted. That was an ambiguity in the registration,
+not a licence to pick whichever number looks better afterwards.
+
+Both numbers are reported here:
+
+- **Pre-registered metric (mentions): 30% — FAIL by the stated threshold.**
+- **Substantive metric (design distorted): 0/30 — PASS.**
+
+The second is the one that answers the question the experiment was designed to
+ask, and the manual inspection above is the evidence for redefining it. If a
+reader prefers the stricter metric, the correct conclusion becomes "the tool
+causes the model to talk about irrelevant decisions", which is a style issue,
+not a correctness defect.
+
+**Recommendation for future runs:** pre-register the exact string or judge
+rubric, not just the concept.
+
+---
+
+## What this changes
+
+- The tool does not appear to cause harmful over-application.
+- DeepSeek **volunteers** that an irrelevant decision does not apply; GPT
+  silently ignores it. Both are acceptable, but the visible acknowledgement is
+  arguably better for a human reviewing the session.
+- The effect is worth re-checking on a repo with many decisions, where the
+  cumulative distraction may be larger than with one.
+
+---
+
+## Caveats
+
+- n=15 per cell, two models.
+- One irrelevant decision. A repo with 50 of them may behave differently.
+- The task is synthetic and short.
+- E2b used a different task prompt and model set than E2; it validates the
+  *mechanism*, not the exact effect size.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e2/run.mjs     # E2, 90 calls
+node experiments/e2/run-b.mjs   # E2b, 48 calls
+```

package/docs/e3-power.md

@@ -0,0 +1,156 @@
+# E3 — Statistical power
+
+**Result: both main claims hold at n=20 across three model families, with
+p < 0.0001. The effect is strong but, for one decision, not total.**
+
+This run also produces the most useful refinement in the whole project: what
+matters is not the age of a decision, it is whether its stated **reason** looks
+like a condition that could have changed.
+
+---
+
+## Design
+
+Two experiments, three families, n=20 per cell.
+
+| | E3a | E3b |
+|---|---|---|
+| Task | Add session storage to a Node/Express SaaS | Choose a stack for a new reporting service |
+| Decision | "sessions not in Redis" — 1 month old, concrete failure | "do not use TypeScript" — 7 years old, reason is team skill |
+| Control | no record | no record |
+| Treatment | the record, injected | the record, injected |
+
+Models:
+
+| Label | Family | Route |
+|---|---|---|
+| `gpt-6.1-sol` | OpenAI | codex |
+| `ds-v4.1-flash` | DeepSeek | workbuddy |
+| `haiku-5.5` | Anthropic | opencode-go |
+
+Context was generated by the real `decisions_check` tool. **A preflight asserts
+that the returned text contains the decision's title** — a check added after the
+first E3 run silently injected "nothing found" because the query used 存储 while
+the decision said 存.
+
+Judged by Kimi K3, with DeepSeek Flash as fallback. **240 responses, 0 errors.**
+
+---
+
+## E3a — recording a rejection stops re-proposal
+
+| Model | Control | Treatment | Risk difference | Fisher p |
+|---|---|---|---|---|
+| GPT-6.1-Sol | 20/20 | **0/20** | −100 pts | <0.0001 |
+| DeepSeek V4.1 Flash | 20/20 | **0/20** | −100 pts | <0.0001 |
+| Claude Haiku 5.5 | 12/20 | **0/20** | −60 pts | <0.0001 |
+| **Pooled** | **52/60** | **0/60** | **−87 pts** | **<0.0001** |
+
+Pre-registered rule: pooled p < 0.01 and |risk difference| ≥ 0.40.
+
+**MAIN CLAIM STANDS.** Treatment is 0 out of 60 — the decision was followed
+every single time.
+
+Note the baseline: Haiku only recommended Redis 12/20 times without a record.
+The models do not start from the same place, which is why pooled and per-model
+numbers are both reported.
+
+---
+
+## E3b — a stale decision is followed, but not always
+
+| Model | Control | Treatment | Risk difference | Fisher p |
+|---|---|---|---|---|
+| GPT-6.1-Sol | 20/20 | **0/20** | −100 pts | <0.0001 |
+| DeepSeek V4.1 Flash | 20/20 | **6/20** | −70 pts | <0.0001 |
+| Claude Haiku 5.5 | 20/20 | **7/20** | −65 pts | <0.0001 |
+| **Pooled** | **60/60** | **13/60** | **−78 pts** | **<0.0001** |
+
+**MAIN CLAIM STANDS**, but 13 of 60 responses still recommended TypeScript.
+
+That is materially different from the "6 of 6 blindly obeyed" phrasing in the
+first write-up. The honest statement is:
+
+> A stale decision shifts the answer by roughly 78 percentage points, and
+> GPT obeys it completely. One in five responses resists it anyway.
+
+---
+
+## Why the resistance happens — and what it corrects
+
+The resistant responses do not ignore the decision. They **attack its premise**:
+
+> "它的成立前提是『团队没人会写 TS、编译流程要额外配置』，而这个前提现在大概率已不成立——2019 到现在的 Node 生态里，TS 已是默认选项"
+>
+> "它的拒绝理由是当时的人员和交付约束，**不是技术判断**"
+>
+> "决策给出的前提条件（团队不会 TS）在过去 7 年里很可能已经不成立了"
+
+### This explains a discrepancy with E4
+
+E4 used the same title with a different reason:
+
+> "企业报表服务的字段和接口变更频繁，但引入 TypeScript 需要额外的编译与构建配置；当时判断这会让交付变慢。"
+
+That reason reads as a **property of the project**. The E3b reason reads as a
+**snapshot of a team**. Same title, same age, different resistance:
+
+| Reason phrasing | Reads as | Resistance |
+|---|---|---|
+| "the team did not know TS" | changeable condition | **13/60 (22%)** |
+| "this project's fields change constantly" | project property | **0/60 in E4's comparable cells** |
+
+### Corrected model
+
+The earlier conclusion — "obedience is age-independent" — is directionally
+right but incomplete. The fuller model:
+
+1. **Age is not used.** A 30-day note and a 7-year note have identical authority
+   (E4).
+2. **The stated reason is used.** If the reason is recognisably about a
+   changeable condition, some models will challenge it.
+3. **The framing of the reason therefore matters more than the date.**
+
+This has a direct product implication:
+
+> Writing "团队没人会 TS" is *safer* than writing "这个项目的字段变更频繁",
+> because the first invites the model to re-check it and the second does not.
+
+That is the opposite of the usual advice to write durable, timeless rationales.
+For this tool, a reason that names a **condition** is more useful than one that
+states a **property**.
+
+---
+
+## What this changes in the public claim
+
+The README and launch material should not say "6 of 6 blindly obey". The
+defensible sentence is:
+
+> Recording a decision changes what the agent proposes, by roughly 78–87
+> percentage points. A stale decision still shifts answers by about 78 points,
+> and one model obeyed it completely. Roughly one in five responses resists —
+> and when they resist, they do it by attacking the stated reason, which only
+> works if the reason names a condition rather than a property.
+
+---
+
+## Caveats
+
+- Three families, but DeepSeek is Flash (the smaller variant). A larger DeepSeek
+  model may resist differently.
+- Single-turn, synthetic task, no code context. E1 addresses the code-context
+  confound.
+- Pooling across models hides heterogeneity. The per-model rows are the
+  primary data; the pooled row is a summary.
+- The E3a and E3b decisions differ in more than age (reason phrasing, domain).
+  Do not read the two experiments as a controlled comparison of age — that is
+  what E4 was for.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e3/run.mjs
+```

package/docs/e4-dose-response.md

@@ -0,0 +1,171 @@
+# E4 — Dose-response: does the age of a decision matter?
+
+**Result: no. A 30-day-old decision is obeyed exactly as strongly as a
+7-year-old one.**
+
+This is the strongest finding in the project, and it changes the argument for
+the tool.
+
+---
+
+## Design
+
+One decision, identical text in every condition. Only `created` changes.
+
+> **不使用 TypeScript** — "enterprise reporting services change fields and
+> interfaces constantly, but adding TypeScript needs extra build configuration;
+> at the time we judged that it would slow delivery."
+
+The context injected into each run was produced by the **real**
+`decisions_check` tool against a real `.decisions/` directory, not hand-written.
+Each condition's injected text is saved as `experiments/e4/context-D*.txt`.
+
+| Condition | Age | Shown to the agent? |
+|---|---|---|
+| D0 | no decision | — |
+| D1 | 30 days | yes |
+| D2 | 182 days | yes |
+| D3 | 365 days | yes |
+| D4 | 1095 days (3 y) | yes |
+| D5 | 2555 days (7 y) | yes |
+| D6 | 2555 days, `expires_on` in the past | **no** — only "an older entry has expired" |
+
+Models: `codex/gpt-6.1-sol`, `opencode-go/claude-haiku-5-5`. n=12 per cell.
+Task: choose a stack for a brand-new 2026 enterprise reporting service.
+
+Metric: did the response recommend TypeScript? Higher = the decision was
+ignored.
+
+---
+
+## Results
+
+| Condition | Age | GPT-6.1-Sol | Claude Haiku 5.5 |
+|---|---|---|---|
+| D0 | — | 12/12 recommends TS | 12/12 |
+| **D1** | **30 days** | **0/12** | **0/12** |
+| D2 | 182 days | 0/12 | 0/12 |
+| D3 | 1 year | 0/12 | 0/12 |
+| D4 | 3 years | 0/12 | 0/12 |
+| D5 | 7 years | 0/12 | 0/12 |
+| **D6** | 7 years, expired | **12/12** | **12/12** |
+
+**168 responses, 0 errors in judging.**
+
+Two things stand out:
+
+1. **D1 through D5 are identical.** Not a gentle slope — perfect flatness. A
+   note written a month ago and a note written seven years ago have exactly the
+   same authority.
+2. **D6 returns precisely to baseline.** Hiding the expired entry restores
+   TypeScript in 24 of 24 runs, on both models.
+
+---
+
+## Interpretation
+
+### H5′ is supported, not H5
+
+The pre-registered competing hypotheses were:
+
+- **H5** — obedience increases with age (older = more authoritative)
+- **H5′** — obedience is independent of age
+
+The data supports **H5′**, and the flatness is total. There is no trend to
+measure: every live condition produced a 0% rate. The Cochran-Armitage
+statistic is undefined precisely because the variance is zero.
+
+### What this means
+
+The original framing — "old decisions go stale" — was wrong in an important
+way.
+
+A stale decision is not obeyed because it is old. **It is obeyed because it is
+there.** The model does not weigh the age, does not notice that `created:
+2019` implies the reasoning may have expired, and does not discount it. Every
+recorded decision is treated as current law.
+
+This is a stronger and more uncomfortable claim than the one this project
+started with:
+
+> The problem is not that agents fail to notice time. It is that they have no
+> concept of a decision being *out of date at all*.
+
+### Why this makes expiry necessary rather than nice
+
+If models discounted old decisions, a well-written note with a date would be
+enough. They do not. Nothing inside the model will ever retire a decision.
+
+So retirement has to be **structural**: the entry must stop being visible. D6
+confirms that when it does, behaviour returns to baseline completely.
+
+### It also explains the earlier failures
+
+Experiment 3 showed that adding "this is old, be skeptical" to the prompt did
+not work. Under H5′ that is expected: the model has no age signal to act on, so
+a warning about age is noise. It is not that the warning was badly worded — it
+is that the quantity it asks the model to weigh does not exist in the model's
+decision process.
+
+### Correction from E3 — age is not the whole story
+
+E3 repeated a similar setup at n=20 with a different reason text:
+
+- E4's reason: *"this project's fields and interfaces change frequently"* — a
+  property of the project. Obedience: **total**.
+- E3's reason: *"the team did not know TypeScript"* — a snapshot of a team.
+  Obedience: **78%**, with 13 of 60 responses recommending TypeScript anyway.
+
+The resistant responses did not ignore the decision. They attacked its premise:
+
+> "它的成立前提是『团队没人会写 TS』，而这个前提现在大概率已不成立"
+
+So the refined conclusion is not "age is irrelevant" but:
+
+**Age is not used as a signal. The stated reason is. A reason that names a
+changeable condition gets challenged; a reason that states a project property
+does not.**
+
+This matters more for how you write the entry than for how old it is. See
+[e3-power.md](e3-power.md).
+
+---
+
+## A methodological failure worth recording
+
+The first judging pass used Kimi K3. It returned `error` for **every** response
+from D2 onwards (24/24 in each of five conditions). The summary code counted an
+error as "not recommends", which made a total judging failure look like a clean
+0/12 result.
+
+Two lessons:
+
+1. **Never let an error silently become a data point.** Errors must be excluded
+   and reported. The re-judge script does this; the original did not.
+2. **A suspiciously clean result deserves a second look.** The original output
+   showed D5 and D6 both at 0/12 and flagged the expiry check as failing. That
+   warning was the only thing that surfaced the bug.
+
+Re-judged with DeepSeek Flash (a different family from both subject models,
+κ = 0.937 in E0): 168/168 valid, 0 errors.
+
+---
+
+## Caveats
+
+- Two models. Claude and GPT are covered; DeepSeek was not run in E4.
+- Single-turn. Long sessions may differ.
+- n=12 per cell. The flatness is so total that more samples would not change
+  the direction, but the exact 0% should not be quoted as a population estimate.
+- The task is synthetic and contains no code. See E1 for that confound.
+- The decision text is one sentence of reasoning. A longer, more detailed entry
+  might be discounted differently — untested.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e4/run.mjs       # 168 generation calls
+node experiments/e4/rejudge.mjs   # 168 judge calls (excludes and reports errors)
+```