whynot-cli 0.0.0-stage → 0.1.1

package/docs/preregistration.md

@@ -0,0 +1,257 @@
+# Pre-registration
+
+Written before the supplementary experiments were run, so that the decision
+rules cannot be adjusted after seeing the results.
+
+Created: 2026-10-20
+Status: active. Do not edit after data collection begins.
+
+---
+
+## Why this file exists
+
+The original experiments reported numbers after the fact, with thresholds
+chosen implicitly. That makes it impossible for a reader to tell whether a
+result was predicted or fitted. This document fixes the hypotheses, sample
+sizes, and decision rules in advance.
+
+If a result contradicts what is written here, the honest move is to publish the
+contradiction, not to rewrite the rule.
+
+---
+
+## Hypotheses
+
+### H1 — stale decisions are obeyed
+An agent given a decision that no longer holds will follow it rather than
+question it.
+
+**Prediction:** in the stale condition, ≥70% of responses avoid the rejected
+option.
+
+### H2 — expiry removes the effect
+If an expired entry is not shown to the agent at all, H1's effect disappears.
+
+**Prediction:** in the expired condition, the rate of avoiding the option falls
+to within 15 percentage points of the no-record baseline.
+
+### H3 — code context reduces it (the "information vacuum" hypothesis)
+The original tasks contained no code, so the model had nothing to reason from
+except the note. With a real codebase in context, the note's influence should
+be weaker.
+
+**Prediction:** with code context, the avoidance rate in the stale condition is
+at least 20 points lower than without.
+
+**This is the hypothesis most likely to be true and least flattering to the
+project.** If it holds, the public claim must be rewritten from "agents blindly
+obey stale decisions" to "agents obey given constraints when they have no other
+information".
+
+### H4 — decisions are over-applied (false obedience)
+An agent given a strong but irrelevant decision will let it distort an unrelated
+answer.
+
+**Prediction:** false-obedience rate < 10%.
+
+### H5 — obedience increases with age
+Older decisions are obeyed more often.
+
+**Prediction:** monotonic increase across 1 month → 7 years.
+
+### H5' — obedience is age-independent
+The competing hypothesis. Models have no sense of how old a note is.
+
+**Prediction:** flat across 1 month → 7 years.
+
+H5 and H5' are mutually exclusive. **Both are publishable.** H5' is the stronger
+finding because it means the problem is structural, not a matter of degree.
+
+### H6 — prompt warnings do not work
+Adding "this is old, be skeptical" to the supplied context does not change
+behaviour; removing the entry does.
+
+**Status:** already tested once (experiment 3). Result: no reliable improvement.
+
+---
+
+## Experiments and decision rules
+
+### E0 — judge calibration
+
+**Design:** all 62 existing responses, blinded, scored by three judges from
+different model families (Claude Haiku, Kimi K3, DeepSeek Flash). Compared
+against human labels using Cohen's kappa.
+
+**Decision rule (fixed before running):**
+
+| worst judge κ | action |
+|---|---|
+| ≥ 0.80 | PASS — use majority vote downstream |
+| 0.60 – 0.80 | WARN — proceed, but disclose and add human spot-checks |
+| < 0.60 | **FAIL — stop all further experiments until scoring is redesigned** |
+
+**Known limitation:** the "human" labels were produced by the same assistant
+that designed the experiments, blind to condition but not independent. The user
+must audit a random subset. If the audit disagrees on more than 10% of items,
+the labels are not a valid reference and E0 is re-run with the user's labels.
+
+---
+
+### E1 — code context
+
+**Design:** three fixture repositories (small/medium/large), each with a planted
+decision. Conditions: no decision, decision injected, decision discovered by
+reading the repo.
+
+**Sample:** n=10 per condition per model, 2 models.
+
+**Decision rule:** H3 is supported if the code-context condition shows an
+avoidance rate at least 20 points lower than the injected condition, with a
+confidence interval excluding zero.
+
+---
+
+### E2 — false obedience
+
+**Design:** a strongly worded but irrelevant decision, plus a relevant-decision
+control.
+
+**Sample:** n=15 per condition, 2 models.
+
+**Decision rule:**
+
+| false-obedience rate | action |
+|---|---|
+| < 5% | acceptable |
+| 5 – 15% | add a constraint to the AGENTS.md instruction and retest |
+| > 15% | **design defect — do not ship until fixed** |
+
+---
+
+### E3 — statistical power
+
+**Design:** the Redis and TypeScript experiments at n=20 per condition, three
+model families.
+
+**Primary analysis:** pooled effect across models, Wilson 95% intervals, Fisher
+exact test, risk difference.
+
+**Decision rule:**
+- pooled p < 0.01 and risk difference ≥ 0.40 → main claim stands
+- p < 0.05 with a smaller effect → downgrade the wording
+- not significant → **publish the correction**
+
+---
+
+### E4 — dose-response
+
+**Design:** one decision, six ages (1 month, 6 months, 1 year, 3 years, 7 years,
+plus an expired 7-year control), n=12 per cell, 2 models.
+
+**Analysis:** Cochran-Armitage trend test.
+
+**Decision rule:**
+- D1 < 30% and D5 > 70% → H5 (dose effect)
+- D1..D5 all > 70% → H5' (age-independent). Rewrite the narrative accordingly.
+- D6 not close to D0 → the expiry mechanism is faulty; stop and fix.
+
+---
+
+### E5 — multi-turn with a real agent
+
+**Design:** Codex CLI against a fixture repo, 8–12 turns, three conditions.
+
+**Decision rule:** report the turn at which the decision stops influencing
+behaviour, and the token cost. No pass/fail threshold — this is descriptive.
+
+---
+
+### E6 — two-week field test
+
+**Status: not run.**
+
+**Consequence:** the project may claim that the mechanism works under test. It
+may **not** claim that using the tool reduces repeated mistakes.
+
+**Design (for anyone who wants to run it):** 14 days without the tool, 14 days
+with it, on a real project.
+
+**Primary metric:** repeated-mistake rate — how often the agent re-proposes
+something already rejected.
+
+**Decision rule:**
+- ≥ 30% reduction → the tool helps
+- < 10% reduction → it does not
+- increase → it hurts
+
+---
+
+## Amendments
+
+Amendments are recorded with the date and the reason. Changing a decision rule
+after seeing data is exactly what pre-registration exists to prevent, so any
+change is logged here and both the original and amended rules are reported.
+
+### 2026-10-21 — E2 metric definition
+
+**Original:** "false-obedience rate < 10%", without specifying what counted.
+
+**Problem found during analysis:** the implementation used "response mentions
+the irrelevant option at all". Manual inspection of the 9 flagged responses
+showed they all said the decision *did not apply* — correct behaviour, not
+obedience. The metric conflated "considered and dismissed" with "distorted by".
+
+**Amendment:** report both metrics.
+
+- mentions: 9/30 (30%) — the pre-registered reading, which is a FAIL
+- design distorted: 0/30 (0%) — the substantive reading
+
+Both are published in `e2-false-obedience.md`. The stricter number is not
+hidden and the redefinition is argued from the raw text, not from preference.
+
+**Future experiments:** pre-register the exact rubric or the exact string
+matched, not just the concept.
+
+### 2026-10-21 — E4 judge failure
+
+Kimi K3 returned `error` for every response from D2 onward. The original
+summary counted errors as non-events, turning a judging failure into a clean
+0/12. All 168 responses were re-judged with DeepSeek Flash.
+
+**Rule added:** an error is never a data point. Errors are excluded from
+denominators and reported. This rule is now enforced in the E3 script, which
+re-judges failures with a fallback judge.
+
+### 2026-10-21 — E3 query bug (caught before analysis)
+
+The first E3 run used the query `技术 选型 存储 语言` for a decision titled
+`session 不存 Redis`. No query term matched, so `decisions_check` returned
+"nothing found" and the treatment condition was identical to the control. The
+run was stopped before any analysis.
+
+**Rule added:** every experiment that injects a decision now asserts that the
+returned context contains the decision's title, and aborts otherwise.
+
+---
+
+## Analysis conventions
+
+- Report confidence intervals, never bare proportions.
+- Report the number of excluded or errored samples.
+- Report inter-rater reliability for any model-judged outcome.
+- Judge calls use `temperature: 0`.
+- Any response that is truncated or missing a conclusion is labelled `unclear`
+  and included in the denominator. Dropping it would inflate agreement.
+
+## What would make me abandon the project
+
+Stated in advance so it cannot be rationalised later:
+
+1. E0 fails (κ < 0.60) and the scoring cannot be redesigned.
+2. E2 shows > 15% false obedience and the instruction cannot fix it.
+3. E3 shows no significant effect at n=20.
+4. E6 shows no reduction in repeated mistakes over two weeks.
+
+Any one of these is sufficient. The project's value depends on the claims being
+true, not on the claims being published.

package/docs/validation.md

@@ -0,0 +1,30 @@
+# Validation
+
+This page is now an index. The consolidated results are in
+[findings.md](findings.md); the detailed write-ups are below.
+
+| Experiment | Question | Write-up |
+|---|---|---|
+| E0 | Can a model reliably judge "recommends vs excludes"? | [judge-calibration.md](judge-calibration.md) |
+| E1 | Does real code context weaken the effect? | [e1-code-context.md](e1-code-context.md) |
+| E2 / E2b | Does an irrelevant decision distort unrelated work? | [e2-false-obedience.md](e2-false-obedience.md) |
+| E3 | Are the effects statistically real? | [e3-power.md](e3-power.md) |
+| E4 | Does the age of a decision matter? | [e4-dose-response.md](e4-dose-response.md) |
+| E5 | Does a real agent use it across sessions? | [e5-real-agent.md](e5-real-agent.md) |
+| E6 | Does it help in real use? | **Not run** — see [preregistration.md](preregistration.md) |
+
+## A note on the earlier version of this page
+
+The first version of this document reported results from a single small
+experiment (n=3 per condition, two models). Those numbers appeared in the
+README and in the launch material.
+
+Several of them were misleading, and one was simply wrong:
+
+- The "6 of 6 obeyed" framing understated the variance; at n=20 the same
+  decision was resisted in 13 of 60 responses.
+- The first judging pass for the dose-response experiment silently counted
+  judge errors as data, producing a clean-looking result that did not exist.
+
+Both are documented in [preregistration.md](preregistration.md). The numbers
+that appear throughout the project now come from the experiments linked above.

package/docs/validation.zh-CN.md

@@ -0,0 +1,24 @@
+# 验证
+
+这一页现在是一个索引。汇总结果在 [findings.zh-CN.md](findings.zh-CN.md)（英文版 [findings.md](findings.md)），详细写稿在下面。
+
+| 实验 | 问题 | 写稿 |
+|---|---|---|
+| E0 | 模型能可靠地判断「推荐 vs 排除」吗？ | [judge-calibration.md](judge-calibration.md) |
+| E1 | 真实代码上下文会削弱效应吗？ | [e1-code-context.md](e1-code-context.md) |
+| E2 / E2b | 不相关的决策会扭曲无关工作吗？ | [e2-false-obedience.md](e2-false-obedience.md) |
+| E3 | 效应在统计上成立吗？ | [e3-power.md](e3-power.md) |
+| E4 | 决策的年龄重要吗？ | [e4-dose-response.md](e4-dose-response.md) |
+| E5 | 真实 agent 会跨会话使用它吗？ | [e5-real-agent.md](e5-real-agent.md) |
+| E6 | 真实使用中帮得上忙吗？ | **未运行** —— 见 [preregistration.md](preregistration.md) |
+
+## 关于这一页的旧版本
+
+这一页最早报告的是单个小实验的结果（每条件 n=3，两个模型）。那些数字曾经出现在 README 和发布材料里。
+
+其中有几个是误导性的，有一个是**错的**：
+
+- 「6/6 全部服从」这个说法低估了方差；n=20 时同一条决策在 60 次回复里被抵抗了 13 次。
+- 剂量-反应实验的第一次判分把判分错误静默地当成了数据，产出了一个看起来很干净、实际不存在的结果。
+
+两件事都记录在 [preregistration.md](preregistration.md) 里。现在项目中使用的数字都来自上面链接的实验。

package/package.json

@@ -1,6 +1,67 @@
 {
   "name": "whynot-cli",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.1",
+  "description": "Record what you ruled out, and why, so your coding agent stops re-suggesting it.",
+  "type": "module",
+  "author": "Stakenshake",
+  "homepage": "https://github.com/Stakenshake/whynot#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Stakenshake/whynot.git"
+  },
+  "bugs": {
+    "url": "https://github.com/Stakenshake/whynot/issues"
+  },
+  "bin": {
+    "whynot": "bin/whynot.js"
+  },
+  "exports": {
+    ".": "./src/index.js",
+    "./mcp": "./src/mcp.js",
+    "./audit": "./src/audit.js"
+  },
+  "files": [
+    "bin",
+    "src",
+    "assets",
+    "docs/design.md",
+    "docs/design.zh-CN.md",
+    "docs/findings.md",
+    "docs/findings.zh-CN.md",
+    "docs/validation.md",
+    "docs/validation.zh-CN.md",
+    "docs/judge-calibration.md",
+    "docs/e1-code-context.md",
+    "docs/e2-false-obedience.md",
+    "docs/e3-power.md",
+    "docs/e4-dose-response.md",
+    "docs/e5-real-agent.md",
+    "docs/preregistration.md",
+    "AGENTS.snippet.md",
+    "AGENTS.snippet.zh-CN.md",
+    "README.md",
+    "README.zh-CN.md",
+    "CONTRIBUTING.md",
+    "LICENSE"
+  ],
+  "engines": {
+    "node": ">=20"
+  },
+  "scripts": {
+    "test": "node --test \"test/*.test.js\"",
+    "mcp": "node bin/whynot.js mcp"
+  },
+  "keywords": [
+    "mcp",
+    "model-context-protocol",
+    "coding-agents",
+    "claude-code",
+    "codex",
+    "context-engineering",
+    "decision-log",
+    "architecture-decision-record",
+    "adr"
+  ],
+  "license": "MIT",
+  "dependencies": {}
+}

package/src/audit.js

@@ -0,0 +1,222 @@
+// Audit an existing repository for agent rules that are likely to be obeyed
+// long after they stopped being true.
+//
+// This is a heuristic, not a verdict. It looks for the two properties that
+// caused the failure in our experiments: absolute language with no stated
+// condition, and age. It does not call a model, and it never changes files.
+
+import fs from 'node:fs';
+import path from 'node:path';
+import { execFileSync } from 'node:child_process';
+
+export const CONFIG_CANDIDATES = [
+  'AGENTS.md',
+  'AGENTS.override.md',
+  'CLAUDE.md',
+  '.claude/CLAUDE.md',
+  'GEMINI.md',
+  '.cursorrules',
+  '.windsurfrules',
+  '.github/copilot-instructions.md',
+];
+
+// Absolute phrasing — the kind of rule that becomes a permanent constraint.
+const ABSOLUTE = [
+  /\bnever\b/i,
+  /\balways\b/i,
+  /\bmust not\b/i,
+  /\bdo not\b/i,
+  /\bdon'?t\b/i,
+  /\bavoid\b/i,
+  /不要/,
+  /不能/,
+  /必须/,
+  /禁止/,
+  /绝不/,
+  /永远/,
+  /一律/,
+  /不允许/,
+];
+
+// Conditional phrasing — the escape hatch that lets a rule expire on its own.
+const CONDITION = [
+  /\bunless\b/i,
+  /\bexcept\b/i,
+  /\bif\b/i,
+  /\bwhen\b/i,
+  /\buntil\b/i,
+  /\bonly (?:if|when)\b/i,
+  /\brevisit\b/i,
+  /除非/,
+  /如果/,
+  /若/,
+  /当/,
+  /直到/,
+  /仅在/,
+  /只有/,
+];
+
+// Instruction words carry no topic. Without this, "never" looks like a theme
+// shared by every rule in the repo.
+const STOPWORDS = new Set([
+  'never', 'always', 'must', 'dont', 'avoid', 'only', 'also', 'just', 'even',
+  'this', 'that', 'these', 'those', 'with', 'from', 'into', 'your', 'their',
+  'there', 'here', 'have', 'has', 'been', 'being', 'when', 'unless', 'except',
+  'until', 'some', 'more', 'than', 'then', 'they', 'them', 'what', 'which',
+  'code', 'file', 'files', 'project', 'repo', 'repository', 'please', 'should',
+  'would', 'could', 'make', 'sure', 'used', 'using', 'need', 'want', 'keep',
+  'every', 'each', 'other', 'into', 'over', 'under', 'about', 'after', 'before',
+]);
+
+/** Topic-ish tokens. Trailing punctuation is stripped so "redis." == "redis". */
+function topicTokens(text) {
+  return [
+    ...new Set(
+      (text.toLowerCase().match(/[a-z][a-z0-9.+#-]{2,}/g) || [])
+        .map((t) => t.replace(/[.+#-]+$/, ''))
+        .filter((t) => t.length >= 4 && !STOPWORDS.has(t))
+    ),
+  ];
+}
+
+function findConfigFiles(root) {
+  const found = [];
+
+  for (const rel of CONFIG_CANDIDATES) {
+    const p = path.join(root, rel);
+    if (fs.existsSync(p) && fs.statSync(p).isFile()) found.push(p);
+  }
+
+  // Directory-style configs.
+  for (const dir of ['.cursor/rules', '.github/instructions']) {
+    const d = path.join(root, dir);
+    if (!fs.existsSync(d) || !fs.statSync(d).isDirectory()) continue;
+    for (const f of fs.readdirSync(d)) {
+      const p = path.join(d, f);
+      if (fs.statSync(p).isFile() && /\.(md|mdc|txt)$/i.test(f)) found.push(p);
+    }
+  }
+
+  return [...new Set(found)];
+}
+
+function lastCommitDaysAgo(root, file, now) {
+  try {
+    const out = execFileSync('git', ['log', '-1', '--format=%ct', '--', path.relative(root, file)], {
+      cwd: root,
+      encoding: 'utf8',
+      stdio: ['ignore', 'pipe', 'ignore'],
+    }).trim();
+    if (!out) return null;
+    return Math.floor((now - new Date(Number(out) * 1000)) / 86_400_000);
+  } catch {
+    return null; // not a git repo, or git missing — age unknown
+  }
+}
+
+/** Lines that read like an instruction rather than prose or a heading. */
+function candidateRules(text) {
+  return text
+    .split(/\r?\n/)
+    .map((line, i) => ({ line: line.trim(), number: i + 1 }))
+    .filter(({ line }) => line.length >= 6 && !line.startsWith('#') && !line.startsWith('<!--'))
+    .filter(({ line }) => ABSOLUTE.some((re) => re.test(line)));
+}
+
+export function audit(root, { now = new Date(), staleDays = 365 } = {}) {
+  const files = findConfigFiles(root);
+  const findings = [];
+  const ages = new Map();
+
+  for (const file of files) {
+    const age = lastCommitDaysAgo(root, file, now);
+    ages.set(file, age);
+
+    let text;
+    try {
+      text = fs.readFileSync(file, 'utf8');
+    } catch {
+      continue;
+    }
+
+    for (const rule of candidateRules(text)) {
+      const hasCondition = CONDITION.some((re) => re.test(rule.line));
+      const isOld = age !== null && age >= staleDays;
+      if (hasCondition) continue;
+
+      findings.push({
+        file: path.relative(root, file),
+        line: rule.number,
+        text: rule.line.replace(/\s+/g, ' ').slice(0, 140),
+        ageDays: age,
+        reasons: [
+          'absolute rule with no stated condition',
+          ...(isOld ? [`unchanged for ${age} days`] : []),
+        ],
+      });
+    }
+  }
+
+  // Same topic mentioned in several places drifts apart. Cheap proxy: shared
+  // distinctive tokens between rules in different files.
+  const byToken = new Map();
+  for (const f of findings) {
+    for (const token of topicTokens(f.text)) {
+      if (!byToken.has(token)) byToken.set(token, []);
+      byToken.get(token).push(f);
+    }
+  }
+  const duplicated = [...byToken.entries()]
+    .filter(([, hits]) => new Set(hits.map((h) => h.file)).size > 1)
+    .map(([token]) => token);
+
+  return {
+    files: files.map((f) => ({ path: path.relative(root, f), ageDays: ages.get(f) })),
+    findings,
+    duplicatedTokens: [...new Set(duplicated)].slice(0, 10),
+    scanned: files.length,
+  };
+}
+
+/** Human-readable report. Plain text so it can be pasted anywhere. */
+export function renderAudit(report) {
+  const lines = [];
+
+  if (!report.scanned) {
+    lines.push('No agent config files found.');
+    lines.push('Looked for: AGENTS.md, CLAUDE.md, GEMINI.md, .cursor/rules/*, .github/copilot-instructions.md');
+    return lines.join('\n');
+  }
+
+  lines.push(`Scanned ${report.scanned} agent config file${report.scanned === 1 ? '' : 's'}:`);
+  for (const f of report.files) {
+    lines.push(`  ${f.path}${f.ageDays === null ? '' : `  (last changed ${f.ageDays}d ago)`}`);
+  }
+  lines.push('');
+
+  if (!report.findings.length) {
+    lines.push('No unqualified absolute rules found. Nothing to review.');
+    return lines.join('\n');
+  }
+
+  lines.push(`${report.findings.length} rule${report.findings.length === 1 ? '' : 's'} look like permanent constraints:`);
+  lines.push('');
+
+  for (const f of report.findings.slice(0, 20)) {
+    lines.push(`  ${f.file}:${f.line}`);
+    lines.push(`    "${f.text}"`);
+    lines.push(`    → ${f.reasons.join('; ')}`);
+  }
+  if (report.findings.length > 20) lines.push(`  ... and ${report.findings.length - 20} more`);
+
+  if (report.duplicatedTokens.length) {
+    lines.push('');
+    lines.push(`Topics mentioned across more than one file (may have drifted): ${report.duplicatedTokens.join(', ')}`);
+  }
+
+  lines.push('');
+  lines.push('These are heuristics, not errors. A rule with no condition is fine —');
+  lines.push('until nobody remembers why it exists. whynot turns those into entries');
+  lines.push('that expire unless someone renews them.');
+  return lines.join('\n');
+}