whynot-cli 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,156 @@
1
+ # E3 — Statistical power
2
+
3
+ **Result: both main claims hold at n=20 across three model families, with
4
+ p < 0.0001. The effect is strong but, for one decision, not total.**
5
+
6
+ This run also produces the most useful refinement in the whole project: what
7
+ matters is not the age of a decision, it is whether its stated **reason** looks
8
+ like a condition that could have changed.
9
+
10
+ ---
11
+
12
+ ## Design
13
+
14
+ Two experiments, three families, n=20 per cell.
15
+
16
+ | | E3a | E3b |
17
+ |---|---|---|
18
+ | Task | Add session storage to a Node/Express SaaS | Choose a stack for a new reporting service |
19
+ | Decision | "sessions not in Redis" — 1 month old, concrete failure | "do not use TypeScript" — 7 years old, reason is team skill |
20
+ | Control | no record | no record |
21
+ | Treatment | the record, injected | the record, injected |
22
+
23
+ Models:
24
+
25
+ | Label | Family | Route |
26
+ |---|---|---|
27
+ | `gpt-6.1-sol` | OpenAI | codex |
28
+ | `ds-v4.1-flash` | DeepSeek | workbuddy |
29
+ | `haiku-5.5` | Anthropic | opencode-go |
30
+
31
+ Context was generated by the real `decisions_check` tool. **A preflight asserts
32
+ that the returned text contains the decision's title** — a check added after the
33
+ first E3 run silently injected "nothing found" because the query used 存储 while
34
+ the decision said 存.
35
+
36
+ Judged by Kimi K3, with DeepSeek Flash as fallback. **240 responses, 0 errors.**
37
+
38
+ ---
39
+
40
+ ## E3a — recording a rejection stops re-proposal
41
+
42
+ | Model | Control | Treatment | Risk difference | Fisher p |
43
+ |---|---|---|---|---|
44
+ | GPT-6.1-Sol | 20/20 | **0/20** | −100 pts | <0.0001 |
45
+ | DeepSeek V4.1 Flash | 20/20 | **0/20** | −100 pts | <0.0001 |
46
+ | Claude Haiku 5.5 | 12/20 | **0/20** | −60 pts | <0.0001 |
47
+ | **Pooled** | **52/60** | **0/60** | **−87 pts** | **<0.0001** |
48
+
49
+ Pre-registered rule: pooled p < 0.01 and |risk difference| ≥ 0.40.
50
+
51
+ **MAIN CLAIM STANDS.** Treatment is 0 out of 60 — the decision was followed
52
+ every single time.
53
+
54
+ Note the baseline: Haiku only recommended Redis 12/20 times without a record.
55
+ The models do not start from the same place, which is why pooled and per-model
56
+ numbers are both reported.
57
+
58
+ ---
59
+
60
+ ## E3b — a stale decision is followed, but not always
61
+
62
+ | Model | Control | Treatment | Risk difference | Fisher p |
63
+ |---|---|---|---|---|
64
+ | GPT-6.1-Sol | 20/20 | **0/20** | −100 pts | <0.0001 |
65
+ | DeepSeek V4.1 Flash | 20/20 | **6/20** | −70 pts | <0.0001 |
66
+ | Claude Haiku 5.5 | 20/20 | **7/20** | −65 pts | <0.0001 |
67
+ | **Pooled** | **60/60** | **13/60** | **−78 pts** | **<0.0001** |
68
+
69
+ **MAIN CLAIM STANDS**, but 13 of 60 responses still recommended TypeScript.
70
+
71
+ That is materially different from the "6 of 6 blindly obeyed" phrasing in the
72
+ first write-up. The honest statement is:
73
+
74
+ > A stale decision shifts the answer by roughly 78 percentage points, and
75
+ > GPT obeys it completely. One in five responses resists it anyway.
76
+
77
+ ---
78
+
79
+ ## Why the resistance happens — and what it corrects
80
+
81
+ The resistant responses do not ignore the decision. They **attack its premise**:
82
+
83
+ > "它的成立前提是『团队没人会写 TS、编译流程要额外配置』,而这个前提现在大概率已不成立——2019 到现在的 Node 生态里,TS 已是默认选项"
84
+ >
85
+ > "它的拒绝理由是当时的人员和交付约束,**不是技术判断**"
86
+ >
87
+ > "决策给出的前提条件(团队不会 TS)在过去 7 年里很可能已经不成立了"
88
+
89
+ ### This explains a discrepancy with E4
90
+
91
+ E4 used the same title with a different reason:
92
+
93
+ > "企业报表服务的字段和接口变更频繁,但引入 TypeScript 需要额外的编译与构建配置;当时判断这会让交付变慢。"
94
+
95
+ That reason reads as a **property of the project**. The E3b reason reads as a
96
+ **snapshot of a team**. Same title, same age, different resistance:
97
+
98
+ | Reason phrasing | Reads as | Resistance |
99
+ |---|---|---|
100
+ | "the team did not know TS" | changeable condition | **13/60 (22%)** |
101
+ | "this project's fields change constantly" | project property | **0/60 in E4's comparable cells** |
102
+
103
+ ### Corrected model
104
+
105
+ The earlier conclusion — "obedience is age-independent" — is directionally
106
+ right but incomplete. The fuller model:
107
+
108
+ 1. **Age is not used.** A 30-day note and a 7-year note have identical authority
109
+ (E4).
110
+ 2. **The stated reason is used.** If the reason is recognisably about a
111
+ changeable condition, some models will challenge it.
112
+ 3. **The framing of the reason therefore matters more than the date.**
113
+
114
+ This has a direct product implication:
115
+
116
+ > Writing "团队没人会 TS" is *safer* than writing "这个项目的字段变更频繁",
117
+ > because the first invites the model to re-check it and the second does not.
118
+
119
+ That is the opposite of the usual advice to write durable, timeless rationales.
120
+ For this tool, a reason that names a **condition** is more useful than one that
121
+ states a **property**.
122
+
123
+ ---
124
+
125
+ ## What this changes in the public claim
126
+
127
+ The README and launch material should not say "6 of 6 blindly obey". The
128
+ defensible sentence is:
129
+
130
+ > Recording a decision changes what the agent proposes, by roughly 78–87
131
+ > percentage points. A stale decision still shifts answers by about 78 points,
132
+ > and one model obeyed it completely. Roughly one in five responses resists —
133
+ > and when they resist, they do it by attacking the stated reason, which only
134
+ > works if the reason names a condition rather than a property.
135
+
136
+ ---
137
+
138
+ ## Caveats
139
+
140
+ - Three families, but DeepSeek is Flash (the smaller variant). A larger DeepSeek
141
+ model may resist differently.
142
+ - Single-turn, synthetic task, no code context. E1 addresses the code-context
143
+ confound.
144
+ - Pooling across models hides heterogeneity. The per-model rows are the
145
+ primary data; the pooled row is a summary.
146
+ - The E3a and E3b decisions differ in more than age (reason phrasing, domain).
147
+ Do not read the two experiments as a controlled comparison of age — that is
148
+ what E4 was for.
149
+
150
+ ---
151
+
152
+ ## Reproduce
153
+
154
+ ```bash
155
+ node experiments/e3/run.mjs
156
+ ```
@@ -0,0 +1,171 @@
1
+ # E4 — Dose-response: does the age of a decision matter?
2
+
3
+ **Result: no. A 30-day-old decision is obeyed exactly as strongly as a
4
+ 7-year-old one.**
5
+
6
+ This is the strongest finding in the project, and it changes the argument for
7
+ the tool.
8
+
9
+ ---
10
+
11
+ ## Design
12
+
13
+ One decision, identical text in every condition. Only `created` changes.
14
+
15
+ > **不使用 TypeScript** — "enterprise reporting services change fields and
16
+ > interfaces constantly, but adding TypeScript needs extra build configuration;
17
+ > at the time we judged that it would slow delivery."
18
+
19
+ The context injected into each run was produced by the **real**
20
+ `decisions_check` tool against a real `.decisions/` directory, not hand-written.
21
+ Each condition's injected text is saved as `experiments/e4/context-D*.txt`.
22
+
23
+ | Condition | Age | Shown to the agent? |
24
+ |---|---|---|
25
+ | D0 | no decision | — |
26
+ | D1 | 30 days | yes |
27
+ | D2 | 182 days | yes |
28
+ | D3 | 365 days | yes |
29
+ | D4 | 1095 days (3 y) | yes |
30
+ | D5 | 2555 days (7 y) | yes |
31
+ | D6 | 2555 days, `expires_on` in the past | **no** — only "an older entry has expired" |
32
+
33
+ Models: `codex/gpt-6.1-sol`, `opencode-go/claude-haiku-5-5`. n=12 per cell.
34
+ Task: choose a stack for a brand-new 2026 enterprise reporting service.
35
+
36
+ Metric: did the response recommend TypeScript? Higher = the decision was
37
+ ignored.
38
+
39
+ ---
40
+
41
+ ## Results
42
+
43
+ | Condition | Age | GPT-6.1-Sol | Claude Haiku 5.5 |
44
+ |---|---|---|---|
45
+ | D0 | — | 12/12 recommends TS | 12/12 |
46
+ | **D1** | **30 days** | **0/12** | **0/12** |
47
+ | D2 | 182 days | 0/12 | 0/12 |
48
+ | D3 | 1 year | 0/12 | 0/12 |
49
+ | D4 | 3 years | 0/12 | 0/12 |
50
+ | D5 | 7 years | 0/12 | 0/12 |
51
+ | **D6** | 7 years, expired | **12/12** | **12/12** |
52
+
53
+ **168 responses, 0 errors in judging.**
54
+
55
+ Two things stand out:
56
+
57
+ 1. **D1 through D5 are identical.** Not a gentle slope — perfect flatness. A
58
+ note written a month ago and a note written seven years ago have exactly the
59
+ same authority.
60
+ 2. **D6 returns precisely to baseline.** Hiding the expired entry restores
61
+ TypeScript in 24 of 24 runs, on both models.
62
+
63
+ ---
64
+
65
+ ## Interpretation
66
+
67
+ ### H5′ is supported, not H5
68
+
69
+ The pre-registered competing hypotheses were:
70
+
71
+ - **H5** — obedience increases with age (older = more authoritative)
72
+ - **H5′** — obedience is independent of age
73
+
74
+ The data supports **H5′**, and the flatness is total. There is no trend to
75
+ measure: every live condition produced a 0% rate. The Cochran-Armitage
76
+ statistic is undefined precisely because the variance is zero.
77
+
78
+ ### What this means
79
+
80
+ The original framing — "old decisions go stale" — was wrong in an important
81
+ way.
82
+
83
+ A stale decision is not obeyed because it is old. **It is obeyed because it is
84
+ there.** The model does not weigh the age, does not notice that `created:
85
+ 2019` implies the reasoning may have expired, and does not discount it. Every
86
+ recorded decision is treated as current law.
87
+
88
+ This is a stronger and more uncomfortable claim than the one this project
89
+ started with:
90
+
91
+ > The problem is not that agents fail to notice time. It is that they have no
92
+ > concept of a decision being *out of date at all*.
93
+
94
+ ### Why this makes expiry necessary rather than nice
95
+
96
+ If models discounted old decisions, a well-written note with a date would be
97
+ enough. They do not. Nothing inside the model will ever retire a decision.
98
+
99
+ So retirement has to be **structural**: the entry must stop being visible. D6
100
+ confirms that when it does, behaviour returns to baseline completely.
101
+
102
+ ### It also explains the earlier failures
103
+
104
+ Experiment 3 showed that adding "this is old, be skeptical" to the prompt did
105
+ not work. Under H5′ that is expected: the model has no age signal to act on, so
106
+ a warning about age is noise. It is not that the warning was badly worded — it
107
+ is that the quantity it asks the model to weigh does not exist in the model's
108
+ decision process.
109
+
110
+ ### Correction from E3 — age is not the whole story
111
+
112
+ E3 repeated a similar setup at n=20 with a different reason text:
113
+
114
+ - E4's reason: *"this project's fields and interfaces change frequently"* — a
115
+ property of the project. Obedience: **total**.
116
+ - E3's reason: *"the team did not know TypeScript"* — a snapshot of a team.
117
+ Obedience: **78%**, with 13 of 60 responses recommending TypeScript anyway.
118
+
119
+ The resistant responses did not ignore the decision. They attacked its premise:
120
+
121
+ > "它的成立前提是『团队没人会写 TS』,而这个前提现在大概率已不成立"
122
+
123
+ So the refined conclusion is not "age is irrelevant" but:
124
+
125
+ **Age is not used as a signal. The stated reason is. A reason that names a
126
+ changeable condition gets challenged; a reason that states a project property
127
+ does not.**
128
+
129
+ This matters more for how you write the entry than for how old it is. See
130
+ [e3-power.md](e3-power.md).
131
+
132
+ ---
133
+
134
+ ## A methodological failure worth recording
135
+
136
+ The first judging pass used Kimi K3. It returned `error` for **every** response
137
+ from D2 onwards (24/24 in each of five conditions). The summary code counted an
138
+ error as "not recommends", which made a total judging failure look like a clean
139
+ 0/12 result.
140
+
141
+ Two lessons:
142
+
143
+ 1. **Never let an error silently become a data point.** Errors must be excluded
144
+ and reported. The re-judge script does this; the original did not.
145
+ 2. **A suspiciously clean result deserves a second look.** The original output
146
+ showed D5 and D6 both at 0/12 and flagged the expiry check as failing. That
147
+ warning was the only thing that surfaced the bug.
148
+
149
+ Re-judged with DeepSeek Flash (a different family from both subject models,
150
+ κ = 0.937 in E0): 168/168 valid, 0 errors.
151
+
152
+ ---
153
+
154
+ ## Caveats
155
+
156
+ - Two models. Claude and GPT are covered; DeepSeek was not run in E4.
157
+ - Single-turn. Long sessions may differ.
158
+ - n=12 per cell. The flatness is so total that more samples would not change
159
+ the direction, but the exact 0% should not be quoted as a population estimate.
160
+ - The task is synthetic and contains no code. See E1 for that confound.
161
+ - The decision text is one sentence of reasoning. A longer, more detailed entry
162
+ might be discounted differently — untested.
163
+
164
+ ---
165
+
166
+ ## Reproduce
167
+
168
+ ```bash
169
+ node experiments/e4/run.mjs # 168 generation calls
170
+ node experiments/e4/rejudge.mjs # 168 judge calls (excludes and reports errors)
171
+ ```
@@ -0,0 +1,151 @@
1
+ # E5 — A real agent, across real sessions
2
+
3
+ **Result: Codex respected the recorded decision in 9 of 9 sessions, including
4
+ new sessions and sessions that did unrelated work first. It also read the
5
+ Markdown files every time, even when the MCP tool worked.**
6
+
7
+ This is the first experiment with a real agent rather than an API call, and it
8
+ answers the question the tool's premise depends on: does a record in the
9
+ filesystem survive a fresh session?
10
+
11
+ ---
12
+
13
+ ## Design
14
+
15
+ The E1 fixture repository (Node/Express, Postgres conventions) plus:
16
+
17
+ - `.decisions/0001-session-不存-redis.md` — a live rejection with the 8-second
18
+ failure and the `Postgres` alternative already chosen
19
+ - `AGENTS.md` — instructs the agent to call `decisions_check`, and to read
20
+ `.decisions/` directly if the tool is unavailable
21
+
22
+ Three conditions, n=3 sessions each:
23
+
24
+ | | Structure | Question |
25
+ |---|---|---|
26
+ | **M0** | one session, one turn | does a real agent respect the record? |
27
+ | **M1** | session 1 asks; **a brand-new session** asks again | does it survive across sessions? |
28
+ | **M2** | turn 1 = unrelated work (`/metrics` endpoint); turn 2 asks about sessions | does prior work crowd it out? |
29
+
30
+ M1 is the one that matters. The tool's whole premise is that the record lives
31
+ in the repository, so a fresh session with no memory should still find it.
32
+
33
+ Agent: Codex CLI 0.163.0, `gpt-6.1-sol`, read-only sandbox, whynot registered
34
+ via `-c mcp_servers.whynot...` overrides. Judge: Claude Haiku.
35
+
36
+ ---
37
+
38
+ ## Results
39
+
40
+ | Condition | Recommends Redis | Called MCP | Read `.decisions/` |
41
+ |---|---|---|---|
42
+ | M0 fresh single-turn | **0/3** | 2/3 | **3/3** |
43
+ | M1 new session, same repo | **0/3** | 2/3 | **3/3** |
44
+ | M2 unrelated work first | **0/3** | 2/3 | **3/3** |
45
+
46
+ **9 of 9 sessions chose PostgreSQL over Redis.**
47
+
48
+ ### What the agent actually said
49
+
50
+ M0:
51
+
52
+ > "项目已有明确决策:session 使用 PostgreSQL 表;Redis 单写区域方案因 us-east
53
+ > 故障时读延迟达到 8 秒而被否决……**我已直接读取决策文件**。"
54
+
55
+ M1 — a brand-new session, no memory of the first:
56
+
57
+ > "**决策查询工具被当前权限策略拦截,我已改为直接读取 `.decisions/`**。"
58
+ > "建议选 PostgreSQL session 表,复用现有 `pg` 技术栈。"
59
+
60
+ M2 — after doing unrelated work first:
61
+
62
+ > "我先核对已有的 session 存储决策和认证代码……建议选 PostgreSQL session 表。"
63
+
64
+ ---
65
+
66
+ ## The dual-path finding
67
+
68
+ The MCP tool was called in 2 of 3 sessions per condition. In the remainder it
69
+ was **blocked by Codex's approval policy**, and the agent fell back to reading
70
+ the Markdown — without being told to in that specific run.
71
+
72
+ This is the second independent observation of the same fallback (the first was
73
+ in the original Codex integration test). It confirms the design property that
74
+ mattered most during development:
75
+
76
+ > The format works without the tool. `AGENTS.md` tells the agent to read
77
+ > `.decisions/` directly when MCP is unavailable, and in practice it does.
78
+
79
+ For deployment this means: **the MCP approval policy is a convenience, not a
80
+ dependency.** Users who never configure it still get the behaviour, as long as
81
+ `AGENTS.md` says what to do.
82
+
83
+ ---
84
+
85
+ ## Token cost
86
+
87
+ | Condition | Input tokens per session |
88
+ |---|---|
89
+ | M0 | ~185k |
90
+ | M1 | ~330k (two sessions in one measurement) |
91
+ | M2 | ~203k |
92
+
93
+ These are Codex reading the repository, not whynot. The decision files are ~1 KB
94
+ each. **The tool's own context cost is negligible; the cost is the agent doing
95
+ its normal job.**
96
+
97
+ A fair comparison would need a no-decision baseline in the same harness. Not
98
+ measured here.
99
+
100
+ ---
101
+
102
+ ## Caveats
103
+
104
+ - **One model family (GPT-6.1-Sol).** E1 showed GPT and DeepSeek behave very
105
+ differently with code context; this experiment says nothing about DeepSeek.
106
+ - **n=3 per condition.** Nine sessions all agreeing is encouraging but not a
107
+ rate estimate.
108
+ - **One fixture, one task.** The repo is small and the decision is unambiguous.
109
+ - **Codex-specific.** Claude Code and Cursor have different approval and
110
+ file-reading behaviour; untested.
111
+ - **No no-decision baseline.** Without it, "0/3 recommends Redis" cannot be
112
+ compared to what the same agent would do with no record. E3 provides that
113
+ comparison at the API level, but not for the real agent.
114
+
115
+ ---
116
+
117
+ ## Failure log
118
+
119
+ Three bugs surfaced during this experiment. All are recorded because the pattern
120
+ is more useful than the fixes.
121
+
122
+ 1. **`spawn codex ENOENT`.** On Windows, `execFileSync('npm', …, )` needs
123
+ `shell: true`; without it the resolver silently fell back to a bare `codex`
124
+ that was not on PATH. Nine sessions "completed" with zero events. Fixed by a
125
+ shared resolver that fails fast (and the same latent bug was removed from
126
+ `codex-check.mjs`).
127
+
128
+ 2. **`codex exec resume` rejects `-s`.** The resume subcommand has a different
129
+ option set. Passing `-s read-only` produced `unexpected argument '-s' found`
130
+ and zero events. Fixed by omitting the sandbox flag on resume.
131
+
132
+ 3. **The summary overwrote itself.** Re-running one condition with `--only M2`
133
+ replaced `e5-summary.json` and dropped M0/M1 verdicts. The per-run files were
134
+ intact, so verdicts were recomputed from them. **Rule: a derived summary must
135
+ never be the only place a result exists.** Per-run output should carry its
136
+ own verdict.
137
+
138
+ Each of these produced a run that looked complete and was empty. The pattern:
139
+ **a tool that fails to start returns the same shape as a tool that succeeded**
140
+ — no events, no error, a clean exit code. Every experiment script should assert
141
+ that it produced events.
142
+
143
+ ---
144
+
145
+ ## Reproduce
146
+
147
+ ```bash
148
+ node experiments/e5/run.mjs # all conditions
149
+ node experiments/e5/run.mjs --only M2 # one condition
150
+ node experiments/e5/rebuild.mjs # recompute verdicts from saved runs
151
+ ```
@@ -0,0 +1,108 @@
1
+ # Findings
2
+
3
+ One page. Every experiment, what it asked, what it found, and how much to
4
+ believe it.
5
+
6
+ Experiment scripts and raw outputs are in [`experiments/`](../experiments/).
7
+ Detailed write-ups are linked per row.
8
+
9
+ ---
10
+
11
+ ## The summary table
12
+
13
+ | # | Question | Design | Result | Confidence |
14
+ |---|---|---|---|---|
15
+ | **E0** | Can a model judge "recommends vs excludes" reliably? | 62 responses, 3 judge families, human audit of 20 | **κ = 1.000** on the 60 well-formed; 19/20 human agreement | High for clear responses; **not** tested on ambiguous ones |
16
+ | **E2** | Does an irrelevant decision distort an unrelated answer? | Schema task + "never use GraphQL", n=15 × 2 models | **0/30** actual distortion | Medium — metric had to be redefined after inspection |
17
+ | **E2b** | Do models read decisions at all? | Same task + a non-default ULID constraint, n=12 × 2 | **24/24 followed it** | High |
18
+ | **E4** | Does the age of a decision change obedience? | Same text, ages 30d → 7y, n=12 × 2 | **Flat: 0% at every age.** Expired → 100% | High (the flatness is total) |
19
+ | **E3** | Are the effects statistically real? | Redis + TypeScript decisions, n=20 × 3 families | **−87 and −78 pts, p < 0.0001** | High |
20
+ | **E1** | Does real code context weaken the effect? | 4 conditions: code × decision, n=12 × 2 | **Split:** GPT replaced the decision with the code; DeepSeek did not | Medium — n small, one fixture |
21
+ | **E5** | Does a real agent use it across sessions? | Codex CLI, 3 session structures, n=3 | **9/9 respected the decision**; file fallback used every time | Low–medium — one model, n=3 |
22
+ | **E6** | Does it help a developer in real use? | 14+14 days, real project | **Not run.** Owner declined the daily logging | — |
23
+
24
+ ---
25
+
26
+ ## Three findings that survived
27
+
28
+ ### 1. The model does not use age as a signal
29
+
30
+ A decision written 30 days ago and one written 7 years ago were obeyed at
31
+ exactly the same rate — zero deviation across 60 runs on two models. Hiding the
32
+ expired one restored the baseline completely.
33
+
34
+ **Implication:** prompting an agent to "be skeptical of old notes" asks it to
35
+ weigh a quantity it does not have. Expiry has to be structural.
36
+
37
+ → [e4-dose-response.md](e4-dose-response.md)
38
+
39
+ ### 2. Recording a rejection changes the answer, substantially
40
+
41
+ Pooled across three model families at n=20: 52/60 → 0/60 for the Redis
42
+ decision, and 60/60 → 13/60 for the TypeScript decision. Both p < 0.0001.
43
+
44
+ **Implication:** the core mechanism is real and large.
45
+
46
+ → [e3-power.md](e3-power.md)
47
+
48
+ ### 3. The stated reason matters more than the date
49
+
50
+ Same title, same age, two different reasons:
51
+
52
+ | Reason in the record | Reads as | Resistance |
53
+ |---|---|---|
54
+ | "the team did not know TypeScript" | a changeable condition | **13/60 (22%)** |
55
+ | "this project's fields change constantly" | a project property | **0/60** |
56
+
57
+ The resistant responses attacked the premise directly: *"that condition very
58
+ likely no longer holds."*
59
+
60
+ **Implication — and it is counter-intuitive:** a reason that names a
61
+ *condition* is safer than one that states a *property*, because it invites the
62
+ model to re-check it. The usual advice to write timeless rationales is wrong
63
+ for this tool.
64
+
65
+ → [e3-power.md](e3-power.md), [e4-dose-response.md](e4-dose-response.md)
66
+
67
+ ---
68
+
69
+ ## The qualification that matters most
70
+
71
+ **E1 showed the effect is not universal.**
72
+
73
+ With real code in context and no recorded decision, GPT already chose the
74
+ repository's existing technology in 12 of 12 runs; adding the decision changed
75
+ nothing. DeepSeek, given the same code, still preferred the rejected option in
76
+ 10 of 12 runs until the decision stopped it.
77
+
78
+ So the honest claim is not "agents blindly obey stale decisions". It is:
79
+
80
+ > Whether a record matters depends on whether the model uses the code. Code
81
+ > encodes what was **chosen**; nothing in a repository encodes what was
82
+ > **rejected**. Records matter most for the second kind.
83
+
84
+ → [e1-code-context.md](e1-code-context.md)
85
+
86
+ ---
87
+
88
+ ## What has not been tested
89
+
90
+ | Gap | Why it matters |
91
+ |---|---|
92
+ | **Real-world usefulness (E6)** | The only number a user cares about. Not measured. This project does not claim it. |
93
+ | Ambiguous responses (E0) | The judge calibration used almost exclusively clear-cut cases. |
94
+ | Long sessions with compaction | E5 used at most 2 turns. |
95
+ | Claude / Gemini / larger DeepSeek | E3 used Haiku and DeepSeek Flash. |
96
+ | A repo with many decisions | All experiments used one or two. Cumulative distraction is untested. |
97
+ | False obedience at scale | E2 tested one irrelevant decision. |
98
+
99
+ ---
100
+
101
+ ## Version note
102
+
103
+ These results describe the implementation as of 2026-10-21. Experiments are
104
+ scripts, not conclusions — if a later change contradicts a number here, the
105
+ script wins and this page gets corrected.
106
+
107
+ Full amendment history, including two experiments whose first pass was wrong:
108
+ [preregistration.md](preregistration.md).
@@ -0,0 +1,90 @@
1
+ # 结论汇总
2
+
3
+ 一页看完。每个实验问了什么、发现了什么、可信到什么程度。
4
+
5
+ 实验脚本和原始输出在 [`experiments/`](../experiments/),详细写稿在每一行里链接。
6
+
7
+ ---
8
+
9
+ ## 总表
10
+
11
+ | # | 问题 | 设计 | 结果 | 可信度 |
12
+ |---|---|---|---|---|
13
+ | **E0** | 模型能可靠判断「推荐 vs 排除」吗? | 62 份回复,3 个判分家族,人工复核 20 条 | 60 条格式完整的 **κ = 1.000**;人工一致 19/20 | 清晰回复上高;**模糊案例未测** |
14
+ | **E2** | 不相关决策会扭曲无关答案吗? | 建表任务 + 「不用 GraphQL」,n=15 × 2 模型 | **实际扭曲 0/30** | 中 —— 指标在检查后重新定义过 |
15
+ | **E2b** | 模型到底读不读决策? | 同任务 + 一条非默认的 ULID 约束,n=12 × 2 | **24/24 遵循** | 高 |
16
+ | **E4** | 决策的年龄会改变服从率吗? | 同一文本,30 天 → 7 年,n=12 × 2 | **完全平坦:每个年龄都是 0%。** 过期后回到 100% | 高(平坦度是彻底的) |
17
+ | **E3** | 效应在统计上成立吗? | Redis + TypeScript 两条决策,n=20 × 3 家族 | **−87 和 −78 点,p < 0.0001** | 高 |
18
+ | **E1** | 真实代码上下文会削弱效应吗? | 代码 × 决策 四条件,n=12 × 2 | **分裂:** GPT 用代码替代了决策,DeepSeek 没有 | 中 —— n 小,单一 fixture |
19
+ | **E5** | 真实 agent 会跨会话使用它吗? | Codex CLI,三种会话结构,n=3 | **9/9 遵守决策**;每次都用了文件降级路径 | 低到中 —— 单一模型,n=3 |
20
+ | **E6** | 真实使用中帮得上忙吗? | 14+14 天,真实项目 | **未运行。** 项目所有者不接受每日记录 | — |
21
+
22
+ ---
23
+
24
+ ## 三条站得住的发现
25
+
26
+ ### 1. 模型不把年龄当信号
27
+
28
+ 30 天前写的决策和 7 年前写的,被遵循的比例**一模一样**——两个模型、60 次运行,零偏差。把过期的那条藏起来,行为完全回到基线。
29
+
30
+ **含义:** 让 agent「对旧笔记保持怀疑」,等于让它权衡一个它没有的量。过期必须是结构性的。
31
+
32
+ → [e4-dose-response.md](e4-dose-response.md)
33
+
34
+ ### 2. 记录一条否决,会实质改变答案
35
+
36
+ 三个模型家族、n=20 合并:Redis 那条从 52/60 → 0/60,TypeScript 那条从 60/60 → 13/60。两者 p < 0.0001。
37
+
38
+ **含义:** 核心机制是真实且显著的。
39
+
40
+ → [e3-power.md](e3-power.md)
41
+
42
+ ### 3. 写的理由比日期更重要
43
+
44
+ 同一个标题、同样的年龄,两种理由:
45
+
46
+ | 记录里的理由 | 读起来像 | 抵抗率 |
47
+ |---|---|---|
48
+ | 「团队没人会 TS」 | 一个**会变的条件** | **13/60(22%)** |
49
+ | 「这个项目字段变更频繁」 | 一个**项目属性** | **0/60** |
50
+
51
+ 抵抗的回复直接攻击前提:*「那个条件现在很可能已经不成立了。」*
52
+
53
+ **含义,而且反直觉:** 命名了**「条件」**的理由比陈述了**「属性」**的更安全,因为它邀请模型去复核。通常「写持久理由」的建议,对这个工具是反的。
54
+
55
+ → [e3-power.md](e3-power.md)、[e4-dose-response.md](e4-dose-response.md)
56
+
57
+ ---
58
+
59
+ ## 最重要的限定
60
+
61
+ **E1 表明效应不是普遍的。**
62
+
63
+ 有真实代码、没有记录时,GPT 已经在 12/12 的运行为仓库现有的技术选型;加上记录没有任何变化。DeepSeek 面对同样的代码,在 10/12 的运行为仍然倾向于被否决的方案,直到记录拦住它。
64
+
65
+ 所以诚实的主张不是「agent 会盲从陈旧决策」,而是:
66
+
67
+ > 一条记录是否起作用,取决于模型是否使用代码。**代码编码了「选了什么」;仓库里没有任何东西编码「否决了什么」。记录对第二类最有价值。**
68
+
69
+ → [e1-code-context.md](e1-code-context.md)
70
+
71
+ ---
72
+
73
+ ## 还没有测的东西
74
+
75
+ | 缺口 | 为什么重要 |
76
+ |---|---|
77
+ | **真实使用中的价值(E6)** | 用户唯一关心的数字。没测。这个项目不声称它。 |
78
+ | 模糊回复(E0) | 判分校准用的是几乎全部清晰的案例。 |
79
+ | 带上下文压缩的长会话 | E5 最多两轮。 |
80
+ | Claude / Gemini / 更大的 DeepSeek | E3 用的是 Haiku 和 DeepSeek Flash。 |
81
+ | 有很多条决策的仓库 | 所有实验只用了 1–2 条。累积干扰未测。 |
82
+ | 大规模误用 | E2 只测了一条不相关决策。 |
83
+
84
+ ---
85
+
86
+ ## 版本说明
87
+
88
+ 这些结果描述的是 2026-10-21 的实现。实验是脚本,不是结论——如果后续改动与这里的数字矛盾,以脚本为准,这一页会被修正。
89
+
90
+ 完整的修订历史(包括两个第一次就做错的实验)见 [preregistration.md](preregistration.md)。