whynot-cli 0.0.0-stage → 0.1.0

package/docs/design.md

@@ -0,0 +1,139 @@
+# Design
+
+Why the tool looks like this.
+
+## The problem
+
+A coding agent has no memory of last week. Every session starts from nothing.
+That is mostly fine: the code is the state, and the agent can read it. What it
+cannot read is **why you rejected an approach**.
+
+That knowledge has three properties that make it hard:
+
+- It does not exist in the repository. It exists in someone's head.
+- It decays. Six months later, the person remembers "we tried something" but
+  not what broke.
+- It is expensive to rediscover. The agent will happily spend tokens walking
+  the same dead end again.
+
+## The finding that shaped the design
+
+We recorded a 2019 decision — "do not use TypeScript, the team does not know
+it" — and asked two model families to choose a stack for a brand-new 2026
+service. Every response followed the seven-year-old note. Adding a warning to
+the prompt ("this is old, be skeptical") did not fix it. Making the entry
+expire did.
+
+The full numbers are in [validation.md](validation.md).
+
+The conclusion: **an agent cannot be relied on to weigh a stale instruction
+correctly.** So the design does not ask it to. An expired entry is not shown to
+the agent at all.
+
+This one decision explains most of the rest of the design.
+
+## Four decisions
+
+### 1. Data is Markdown in git
+
+Everything else follows from this.
+
+- A human can read and edit it without the tool.
+- `git diff` gives review for free.
+- `git revert` gives an undo for free.
+- If the project is abandoned, the data is still usable with `grep`.
+
+Most tools in this space store decisions in a database or a vector index. When
+those projects die, users lose the data along with the tool. This project is
+designed so that cannot happen.
+
+### 2. Expiry is the default
+
+Every entry carries an expiry date. `rejected` entries last 90 days, `decision`
+180, `constraint` 365. When it lapses, the entry stops reaching the agent.
+
+An expired entry is not deleted. `whynot stale` surfaces it, and a human either
+renews it or retires it. Both are explicit actions.
+
+`expires_on: never` exists for genuinely permanent constraints, but it must be
+written deliberately. Permanence should be a decision, not a default.
+
+### 3. No embeddings, no model calls on write
+
+Retrieval is deliberately simple: case-insensitive keyword matching over titles,
+tags, and bodies, scored by how many query terms hit.
+
+Two reasons:
+
+- **Auditability.** When an entry appears in the agent's context, it is obvious
+  why: the words matched. With embeddings, nobody can explain it.
+- **Cost and failure modes.** Vector indexing goes stale. LLM extraction is
+  expensive and is the main source of corrupted memories in the tools that use
+  it. Both are avoidable here, so they are avoided.
+
+The trade-off is real: keyword matching misses paraphrases. That is why
+`decisions_check` with no query returns a compact index of everything live, so
+the agent can see the full list cheaply before narrowing down.
+
+### 4. The format works without the tool
+
+The agent gets `decisions_check` through MCP. If MCP is unavailable, it can
+still read `.decisions/` directly — that is just Markdown.
+
+This was not a theoretical property. In the Codex integration test, the MCP
+call was blocked by an approval policy, and the agent fell back to reading the
+files and still respected the decision.
+
+## Form factor: four layers, not one
+
+A common mistake in this category is picking one form and forcing everything
+through it. Each concern here belongs to a different layer:
+
+| Layer | Form | Why it has to be this |
+|---|---|---|
+| Data | `.decisions/*.md` in git | Works with zero tooling. The durable asset. |
+| Behaviour | A block in `AGENTS.md` | Making an agent *proactively record* a rejection is an instruction, not an API. |
+| Capability | MCP server | Pull-based, structured, works across agents. |
+| Human loop | CLI | Renewing and retiring are human decisions. |
+| Distribution | npm and agent plugins | Zero-install; meets people where they already are. |
+| **Not** | Hosted API | Decisions about a codebase are the most sensitive artifact in the repo. Hosting them defeats the purpose. |
+
+## Cross-platform
+
+The project targets Windows, macOS, and Linux, because the failure modes are
+different on each.
+
+- **Line endings.** `init` writes `*.md text eol=lf` to `.gitattributes`.
+  Without it, Windows checkouts write CRLF and every diff shows the whole file.
+- **Filenames.** Windows forbids `\ / : * ? " < > |`, and Windows and macOS are
+  case-insensitive. Filenames are slugged to `[a-z0-9]` plus CJK, which
+  eliminates a class of "works on Linux" bugs.
+- **No background process.** Windows file watching is unreliable and needs
+  native dependencies. The tool is invoked, never resident.
+- **No git hooks.** Cross-platform shell in hooks is a minefield. Not used.
+
+## Deliberately not built
+
+- Vector search or embeddings.
+- Any model call in the write path.
+- A daemon or file watcher.
+- A web UI.
+- Automatic detection of rejections (a hook is possible later, but it is not
+  required for the core loop).
+- Team sync. `git` already does this.
+- Multi-repo aggregation.
+
+Each one is a real feature someone has asked for. Each one also makes the tool
+heavier, and this category has a graveyard full of heavier tools.
+
+## Open questions
+
+- **Does the repeated-mistake rate actually drop in real use?** The experiments
+  measure single-turn behaviour. The claim that matters is longitudinal and has
+  not been measured yet.
+- **How often does expiry lose knowledge people still need?** The trade is
+  deliberate, but the ratio is unknown. `whynot stale` is where that shows up.
+- **Does this survive being used by a team?** The design assumes one person
+  curating. With five people, who renews what?
+- **Is 90 days right for a rejection?** It is a guess. Different codebases move
+  at different speeds.

package/docs/design.zh-CN.md

@@ -0,0 +1,102 @@
+# 设计说明
+
+这个工具为什么长这样。
+
+## 问题
+
+编码 agent 对上周发生的事没有记忆。每次会话都从零开始。这本身问题不大——代码就是状态，agent 可以自己读。但它读不到的是：**你为什么否决了某个方案。**
+
+这类知识有三个麻烦的性质：
+
+- 它不在仓库里，在某个人脑子里。
+- 它会腐烂。半年后，那个人只记得「当时好像试过」，不记得具体哪里崩了。
+- 重新发现它的代价很高。agent 会毫不心疼地再走一遍那条死路。
+
+## 决定了整个设计的那个发现
+
+我们记录了一条 2019 年的决策——「不要用 TypeScript，团队没人会」——然后让两个模型家族给一个 **2026 年的全新服务**做技术选型。**每一次回复都遵循了那条七年前的笔记。** 在提示词里加一句「这条很旧，请保持怀疑」没有修好。让条目过期，才修好。
+
+完整数字见 [validation.zh-CN.md](validation.zh-CN.md)。
+
+结论是：**不能指望 agent 正确地权衡一条陈旧指令。** 所以设计上干脆不要求它权衡——**过期条目根本不出现在 agent 的视野里。**
+
+这一条决定了后面大部分设计。
+
+## 四个决定
+
+### 1. 数据是 git 里的 Markdown
+
+其他一切都由此而来。
+
+- 人不用工具也能读、能改。
+- `git diff` 免费提供 review。
+- `git revert` 免费提供撤销。
+- 就算项目被放弃，数据依然能用 `grep` 检索。
+
+这个领域里大多数工具把决策存进数据库或向量索引。那些项目死掉时，用户会连数据一起丢。这个项目从设计上就不允许这种事发生。
+
+### 2. 默认过期
+
+每条记录都带一个到期日。否决类 90 天，决策类 180 天，约束类 365 天。到期后，条目不再到达 agent。
+
+过期条目**不会被删除**。`whynot stale` 会把它列出来，由人决定续期还是归档。两者都是显式动作。
+
+`expires_on: never` 是为真正永久的约束准备的，但必须刻意写出来。**永久应该是一个决定，不是一个默认值。**
+
+### 3. 不用 embedding，写入时不调模型
+
+检索刻意做得很简单：对标题、标签、正文做大小写不敏感的关键词匹配，按命中的查询词数量排序。
+
+两个理由：
+
+- **可审计。** 当一条记录出现在 agent 的上下文里，原因一目了然：词对上了。用 embedding 的话，没人解释得清它为什么在那儿。
+- **成本与失败模式。** 向量索引会过期。用 LLM 抽取记忆很贵，而且正是这类工具里记忆被污染的主要来源。这两样在这里都可以避免，所以就不做。
+
+代价是真实的：关键词匹配会漏掉改写的说法。所以 `decisions_check` 在不带查询条件时会返回一个紧凑的索引，让 agent 先廉价地看到全貌，再往下缩小范围。
+
+### 4. 没有工具时，格式依然可用
+
+agent 通过 MCP 调用 `decisions_check`。如果 MCP 不可用，它仍然可以直接读 `.decisions/`——那不过是 Markdown。
+
+这不是理论上的性质。在 Codex 集成测试里，MCP 调用被审批策略拦住了，agent 自动降级去读文件，并且**依然遵守了那条决策**。
+
+## 形态：四层，不是一个
+
+这个品类里一个常见的错误是选一种形态，然后把所有事都塞进去。这里的每件事属于不同的层：
+
+| 层 | 形态 | 为什么必须是这个 |
+|---|---|---|
+| 数据 | git 里的 `.decisions/*.md` | 零工具依赖也能用。这是唯一不会死的资产。 |
+| 行为 | `AGENTS.md` 里的一段指令 | 让 agent **主动记录**否决，是指令问题，不是 API 能力问题。 |
+| 能力 | MCP server | 拉取式、结构化、跨 agent 通用。 |
+| 人的回路 | CLI | 续期和归档是人的决定。 |
+| 分发 | npm + agent 插件 | 零安装，在用户已经在的地方出现。 |
+| **不是** | 托管 API | 关于代码库的决策是仓库里最敏感的东西。托管它就等于毁掉整个前提。 |
+
+## 跨平台
+
+目标平台是 Windows、macOS、Linux，因为三者的坑不一样。
+
+- **换行符。** `init` 会往 `.gitattributes` 写 `*.md text eol=lf`。不写的话，Windows 上会写入 CRLF，每次 diff 都显示整个文件被改。
+- **文件名。** Windows 禁止 `\ / : * ? " < > |`，而且 Windows 和 macOS 大小写不敏感。文件名会被 slug 成 `[a-z0-9]` 加中日韩字符，从根上消除一整类「Linux 上好的、换个系统就崩」的问题。
+- **没有常驻进程。** Windows 的文件监听不可靠且需要原生依赖。工具只被调用，不常驻。
+- **不用 git hooks。** 跨平台的 shell 在 hook 里是雷区。所以不用。
+
+## 刻意不做的东西
+
+- 向量检索 / embedding
+- 写入路径上的任何模型调用
+- 守护进程或文件监听
+- Web UI
+- 自动检测否决（以后可以用 hook，但核心闭环不依赖它）
+- 团队同步。`git` 本身就是。
+- 多仓库聚合
+
+每一条都是真的有人要过的功能。每一条也都会让工具更重，而这个品类的坟场里全是更重的工具。
+
+## 还没有答案的问题
+
+- **真实使用中，重复犯错率真的会降吗？** 实验测的是单轮行为。真正重要的主张是长期的，而那个还没测。
+- **过期多久会丢掉人还需要的信息？** 这是一个刻意的取舍，但比例未知。`whynot stale` 是它显现的地方。
+- **团队用它撑得住吗？** 设计假设是一个人维护。五个人时，谁负责续期？
+- **否决类 90 天合适吗？** 这是猜的。不同代码库的节奏不一样。

package/docs/e1-code-context.md

@@ -0,0 +1,169 @@
+# E1 — Does real code context change the effect?
+
+**Result: it depends on the model, and the pooled number hides that. GPT
+replaces the decision with the code; DeepSeek does not.**
+
+This is the experiment that most threatens the project's claim, and the result
+is genuinely mixed. It also produces the sharpest statement of what the tool is
+actually for.
+
+---
+
+## Why this experiment exists
+
+Every earlier experiment gave the model a task with no code. That leaves an
+obvious alternative explanation:
+
+> The model obeyed the decision only because it had nothing else to reason from.
+
+If true, the public claim would have to change from "agents blindly obey stale
+decisions" to "agents obey any given constraint when starved of context".
+
+---
+
+## Design
+
+A real Node/Express project (`experiments/e1/fixture/`, 19 files, ~410 lines)
+with conventions that matter: Postgres for all app data, service layer takes
+`tenantId` first, `REDIS_URL` declared in config but unused, README notes that
+Redis was being evaluated.
+
+Four conditions, n=12 per model per cell:
+
+| | Code context | Decision |
+|---|---|---|
+| C0 | no | no |
+| C1 | no | yes |
+| C2 | yes | no |
+| C3 | yes | yes |
+
+Task: add session storage to this project, 100k DAU, two regions.
+
+Models: GPT-6.1-Sol, DeepSeek V4.1 Flash. Judge: Claude Haiku (third family).
+**96 responses, 0 judging errors.**
+
+The decision was injected through the real `decisions_check` tool; the script
+aborts if the returned context does not contain the entry.
+
+---
+
+## Results
+
+| Condition | GPT | DeepSeek | Pooled |
+|---|---|---|---|
+| C0 no code, no decision | 10/12 | 12/12 | **22/24 (92%)** |
+| C1 no code + decision | 0/12 | 0/12 | **0/24 (0%)** |
+| C2 code, no decision | **0/12** | **10/12** | **10/24 (42%)** |
+| C3 code + decision | 0/12 | 0/12 | **0/24 (0%)** |
+
+Derived effects:
+
+| Effect | Size |
+|---|---|
+| Decision, without code (C1 − C0) | −92 pts |
+| Decision, with code (C3 − C2) | −42 pts |
+| Code alone, no decision (C2 − C0) | −50 pts |
+
+The pooled H3 test — "does the decision still work with code?" — reads as
+**0 pts change** and the automated verdict said "H3 not supported".
+
+**That verdict is wrong, because the two models do opposite things.**
+
+---
+
+## The split
+
+### GPT: the code replaces the decision
+
+With code and no decision, GPT already refuses Redis in 12 of 12 runs. It says
+so explicitly:
+
+> "现有 `REDIS_URL` 只是历史配置，不应成为选用 Redis 的理由。"
+> *(The existing REDIS_URL is just historical config; it is not a reason to pick Redis.)*
+
+> "现有项目的数据已经全部放在 PostgreSQL，增加 session 表能直接复用连接池、备份和运维体系。暂时不引入 Redis。"
+
+Adding the recorded decision changes nothing (0/12 → 0/12). For GPT, **the code
+is sufficient**; the decision is redundant in this case.
+
+### DeepSeek: the code is not enough
+
+DeepSeek reads the same code and still recommends Redis in 10 of 12 runs. Only
+when the decision is present does it stop (0/12).
+
+For DeepSeek, the decision remains **decisive even with the code in front of it**.
+
+---
+
+## What this actually says
+
+The pooled framing ("the effect survives code context") is technically what the
+numbers say and substantively wrong. The honest statement:
+
+> Whether a recorded decision matters depends on whether the model uses the code.
+> For a model that reads the repository and infers its conventions, a decision
+> that the code already implies is redundant. For a model that does not, the
+> decision is the only thing that stops it.
+
+### And it sharpens the product thesis
+
+The fixture's code makes "use Postgres" obvious — that is what code does, it
+encodes the choices that were made. **What code cannot encode is the
+alternative that was rejected.** Nothing in the repository says "we tried Redis
+and it failed at 8 seconds", because a rejected option leaves no artifact.
+
+That is exactly the gap the tool fills, and E1 shows it concretely:
+
+- GPT inferred the chosen option from the code, so the decision added nothing.
+- Neither model could infer the **rejection** from the code, and for DeepSeek
+  that mattered at 10/12.
+
+So the correct positioning is narrower and stronger than the original:
+
+> Don't record decisions the code already implies. Record the alternatives that
+> were rejected, because nothing else in the repository remembers them.
+
+---
+
+## What to change in the public claim
+
+The README should not say "agents blindly obey stale decisions" without
+qualification. The defensible version:
+
+> A recorded rejection changes what an agent proposes by 42–92 percentage
+> points, depending on the model and on whether the repository already implies
+> the answer. Models that read the code can infer what was chosen; none of them
+> can infer what was rejected.
+
+---
+
+## Caveats
+
+- Two models. The split may look different on Claude or on a larger DeepSeek.
+- One fixture, written by us, with deliberately strong conventions. A messier
+  repo might not substitute for the decision as cleanly.
+- n=12 per cell. The GPT/DeepSeek split is stark (0/12 vs 10/12), but the exact
+  rates should not be treated as population estimates.
+- The task is still synthetic. E5 (multi-turn with a real agent) is the next
+  step for external validity.
+
+---
+
+## A reporting lesson
+
+The script printed "H3 NOT SUPPORTED — the effect survives code context" based
+on pooled rates. The per-model table shows the opposite pattern for one model.
+**Pooling across models hid a sign flip in the mechanism.**
+
+Rule for the remaining experiments: when the per-model cells disagree in
+direction, the pooled number is not a summary — it is a fiction. Report the
+split and say so.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e1/build-fixture.mjs
+node experiments/e1/run.mjs
+```

package/docs/e2-false-obedience.md

@@ -0,0 +1,167 @@
+# E2 — False obedience: does an irrelevant decision distort an unrelated answer?
+
+**Result: no. The measured 30% was a metric artefact; actual false obedience is
+0 out of 30.**
+
+This experiment also produced a lesson about metric definition that is more
+useful than the result itself.
+
+---
+
+## Why this experiment exists
+
+Every other experiment measures one failure: the agent forgets something it
+should have remembered. This one measures the opposite: the agent applies a
+decision it should have ignored.
+
+If the tool's whole premise is "record what you ruled out", the obvious risk is
+that a growing pile of decisions starts distorting unrelated work. That risk had
+never been tested.
+
+---
+
+## Design
+
+**Task:** design a PostgreSQL `users` table, full DDL. This task has nothing to
+do with GraphQL.
+
+| Condition | Injected decision | Expected behaviour |
+|---|---|---|
+| N0 | none | baseline |
+| **N1** | "never use GraphQL anywhere" — strongly worded, irrelevant | should be ignored |
+| N2 | "passwords must not be stored in plaintext" — relevant | should change the design |
+
+Models: `codex/gpt-6.1-sol`, `opencode-go/deepseek-v4-pro`. n=15 per condition.
+
+The decision was injected as text in the system prompt, with the same
+`AGENTS.md` instruction used elsewhere.
+
+---
+
+## Result 1: the pre-registered metric said FAIL
+
+The metric fixed in advance was **"does the response mention GraphQL at all"**,
+on the reasoning that an irrelevant decision should leave no trace.
+
+| Model | N0 mentions | N1 mentions |
+|---|---|---|
+| GPT-6.1-Sol | 0/15 | **0/15** |
+| DeepSeek V4 Pro | 0/15 | **9/15** |
+| **Total** | **0/30** | **9/30 = 30%** |
+
+Under the pre-registered threshold (5–15% warn, >15% fail), this is a **FAIL —
+design defect**.
+
+---
+
+## Result 2: manual inspection shows the metric was wrong
+
+All nine DeepSeek responses that "mentioned GraphQL" did so in one specific way:
+
+> "已按现有决策检查：`0001 不使用 GraphQL` **仅约 API，不涉及本表设计**。"
+> *(Checked the recorded decision: 0001 only concerns the API, not this table.)*
+
+> "已检查 `.decisions/`：0001 仅限制不使用 GraphQL，**与本表设计无关**。"
+> *(0001 only restricts GraphQL, which is unrelated to this table design.)*
+
+> "已检查仓库决策 `0001`（不使用 GraphQL），**与本数据库表设计无冲突**；以下方案只涉及 PostgreSQL DDL。"
+
+The one response the regex flagged as "treating it as a constraint" says:
+
+> "0001 不使用 GraphQL，与数据库设计无关。**所以不影响**。"
+
+**Every one of these is correct relevance judgement, stated out loud, followed
+by a normal design.** The model read the decision, decided it did not apply, and
+said so. That is exactly what the tool should produce.
+
+The metric "mentions GraphQL" conflates two opposite behaviours:
+
+- considering the decision and correctly dismissing it (good)
+- being distorted by the decision (bad)
+
+Only the second is false obedience, and it occurred **0 times in 30 runs**.
+
+---
+
+## Result 3: the original positive control failed
+
+The N2 condition was meant to prove the model reads decisions at all. It did
+not work: a competent schema hashes passwords anyway.
+
+| Model | N0 hashes | N2 hashes |
+|---|---|---|
+| GPT | 15/15 | 15/15 |
+| DeepSeek | 15/15 | 15/15 |
+
+No difference. So E2 on its own could not distinguish "correctly ignored an
+irrelevant decision" from "never read any decision".
+
+### E2b — a control that actually discriminates
+
+A decision a model would *not* choose by default:
+
+> **users 表主键必须使用 `CHAR(26)` ULID。禁止 SERIAL / BIGSERIAL / IDENTITY，也禁止 UUID。**
+
+| Model | N0 uses ULID | N3 uses ULID | N0 uses SERIAL | N3 uses SERIAL |
+|---|---|---|---|---|
+| GPT-6.1-Sol | 0/12 | **12/12** | 12/12 | **0/12** |
+| Claude Haiku 5.5 | 0/12 | **12/12** | 12/12 | 2/12 |
+
+**24/24 = 100% followed the ULID constraint.** Both models default to `SERIAL`
+and both completely switch when told not to. The models demonstrably read
+decisions.
+
+Therefore E2's zero false-obedience rate is meaningful: the model read the
+GraphQL decision, judged it irrelevant, said so, and was not distorted by it.
+
+---
+
+## Honest note on changing the metric
+
+The pre-registration said "false-obedience rate" without defining whether
+"mention" or "distortion" counted. That was an ambiguity in the registration,
+not a licence to pick whichever number looks better afterwards.
+
+Both numbers are reported here:
+
+- **Pre-registered metric (mentions): 30% — FAIL by the stated threshold.**
+- **Substantive metric (design distorted): 0/30 — PASS.**
+
+The second is the one that answers the question the experiment was designed to
+ask, and the manual inspection above is the evidence for redefining it. If a
+reader prefers the stricter metric, the correct conclusion becomes "the tool
+causes the model to talk about irrelevant decisions", which is a style issue,
+not a correctness defect.
+
+**Recommendation for future runs:** pre-register the exact string or judge
+rubric, not just the concept.
+
+---
+
+## What this changes
+
+- The tool does not appear to cause harmful over-application.
+- DeepSeek **volunteers** that an irrelevant decision does not apply; GPT
+  silently ignores it. Both are acceptable, but the visible acknowledgement is
+  arguably better for a human reviewing the session.
+- The effect is worth re-checking on a repo with many decisions, where the
+  cumulative distraction may be larger than with one.
+
+---
+
+## Caveats
+
+- n=15 per cell, two models.
+- One irrelevant decision. A repo with 50 of them may behave differently.
+- The task is synthetic and short.
+- E2b used a different task prompt and model set than E2; it validates the
+  *mechanism*, not the exact effect size.
+
+---
+
+## Reproduce
+
+```bash
+node experiments/e2/run.mjs     # E2, 90 calls
+node experiments/e2/run-b.mjs   # E2b, 48 calls
+```