pushback 0.1.2__tar.gz → 0.1.3__tar.gz

{pushback-0.1.2/src/pushback.egg-info → pushback-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pushback
-Version: 0.1.2
+Version: 0.1.3
 Summary: How often do you correct your coding agent, and at what kind of work? Measured from your own Claude Code transcripts.
 Author: Intikhab Azam
 License: MIT
@@ -54,7 +54,7 @@ shows what you use your AI coding agent for, how often you correct it on each
 kind of work (code, writing, media, research, ops, meta), and how often a
 correction needs correcting again. It drafts
 **CLAUDE.md rules** from the corrections you keep repeating, shows which of
-your existing rules keep getting broken, and exports your corrections as
+your existing rules those corrections run into, and exports your corrections as
 **preference pairs** (prompt / chosen / rejected). A hand-audit step tells you
 how far to trust every number.
 
@@ -67,7 +67,7 @@ pip install pushback
 The author's own 1,633 messages to Claude Code:
 
 <p align="center">
-  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. Three existing rules kept getting broken.">
+  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. One existing rule kept getting broken after it was written.">
 </p>
 
 | task | messages | share of use | corrections | rate | 95% CI | corrected again |
@@ -94,7 +94,7 @@ Most corrected topics (at least 5 messages each):
 
 - Against 91 hand-checked messages the correction labels had **precision 0.97 and recall 0.80**. Topic labels were spot-checked, not audited.
 - **1 in 3 corrections needed a second one.** 70 of 201 fixes got corrected again (35%), 41% on writing and 44% on images.
-- **3 rules the author had already written kept getting broken**, with 6 to 8 corrections each.
+- **1 rule the author had already written kept getting broken**: "images must not look AI-generated" was corrected 3 more times after it was written. Two other matching rules turned out to be written *because of* the corrections, which is why every rule now shows its evidence dates.
 
 > [!NOTE]
 > These rates describe a workflow, not a model. The author's code work runs
@@ -177,17 +177,29 @@ pushback rules                                  # drafts rules into pushback-dat
 pushback rules --existing CLAUDE.md notes/*.md  # also check against the rules you already have
 pushback rules --prompt-file                    # no API key? writes the request to a file instead
 pushback rules --from-response reply.json       # ...and reads Claude's answer back
+pushback rules --runs 3                         # ask 3 times, keep only the rules that keep coming back
 ```
 
 The model groups corrections that share a cause and drafts one CLAUDE.md
 instruction per group. Support is counted from the correction ids it cites,
 not from its own numbers, and a rule needs at least 3 real corrections.
 
+**Stable rules.** A model won't write the same rules twice. With `--runs 3`
+(or three replies passed to `--from-response`), pushback matches rules across
+runs by the corrections they cite, not by their wording, and keeps only the
+ones that come back in at least 2 of 3 runs. Each rule shows how many runs it
+appeared in, and the unstable ones are listed separately instead of vanishing.
+
+On the author's 214 corrections, three independent runs wrote 12, 10 and 7
+rules: 16 distinct rules in total. 5 came back in all three runs, 3 in two,
+and **8 showed up only once**. A single run mixes real rules with one-offs.
+
 When you pass your existing rule files, the output splits in two:
 
-- **Rules you already have that keep getting broken.** These are the useful
-  ones. A rule that exists and still gets corrected isn't working: it's too
-  vague, or the agent doesn't read it at the right moment.
+- **Rules you already have that these corrections hit.** If the corrections
+  came after you wrote the rule, the rule isn't working: it's too vague, or the
+  agent doesn't read it at the right moment. pushback can't know when you wrote
+  a rule, so check the evidence dates.
 - **New rules to consider.** Recurring corrections with no rule behind them.
 
 > [!TIP]

{pushback-0.1.2 → pushback-0.1.3}/README.md

@@ -27,7 +27,7 @@ shows what you use your AI coding agent for, how often you correct it on each
 kind of work (code, writing, media, research, ops, meta), and how often a
 correction needs correcting again. It drafts
 **CLAUDE.md rules** from the corrections you keep repeating, shows which of
-your existing rules keep getting broken, and exports your corrections as
+your existing rules those corrections run into, and exports your corrections as
 **preference pairs** (prompt / chosen / rejected). A hand-audit step tells you
 how far to trust every number.
 
@@ -40,7 +40,7 @@ pip install pushback
 The author's own 1,633 messages to Claude Code:
 
 <p align="center">
-  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. Three existing rules kept getting broken.">
+  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. One existing rule kept getting broken after it was written.">
 </p>
 
 | task | messages | share of use | corrections | rate | 95% CI | corrected again |
@@ -67,7 +67,7 @@ Most corrected topics (at least 5 messages each):
 
 - Against 91 hand-checked messages the correction labels had **precision 0.97 and recall 0.80**. Topic labels were spot-checked, not audited.
 - **1 in 3 corrections needed a second one.** 70 of 201 fixes got corrected again (35%), 41% on writing and 44% on images.
-- **3 rules the author had already written kept getting broken**, with 6 to 8 corrections each.
+- **1 rule the author had already written kept getting broken**: "images must not look AI-generated" was corrected 3 more times after it was written. Two other matching rules turned out to be written *because of* the corrections, which is why every rule now shows its evidence dates.
 
 > [!NOTE]
 > These rates describe a workflow, not a model. The author's code work runs
@@ -150,17 +150,29 @@ pushback rules                                  # drafts rules into pushback-dat
 pushback rules --existing CLAUDE.md notes/*.md  # also check against the rules you already have
 pushback rules --prompt-file                    # no API key? writes the request to a file instead
 pushback rules --from-response reply.json       # ...and reads Claude's answer back
+pushback rules --runs 3                         # ask 3 times, keep only the rules that keep coming back
 ```
 
 The model groups corrections that share a cause and drafts one CLAUDE.md
 instruction per group. Support is counted from the correction ids it cites,
 not from its own numbers, and a rule needs at least 3 real corrections.
 
+**Stable rules.** A model won't write the same rules twice. With `--runs 3`
+(or three replies passed to `--from-response`), pushback matches rules across
+runs by the corrections they cite, not by their wording, and keeps only the
+ones that come back in at least 2 of 3 runs. Each rule shows how many runs it
+appeared in, and the unstable ones are listed separately instead of vanishing.
+
+On the author's 214 corrections, three independent runs wrote 12, 10 and 7
+rules: 16 distinct rules in total. 5 came back in all three runs, 3 in two,
+and **8 showed up only once**. A single run mixes real rules with one-offs.
+
 When you pass your existing rule files, the output splits in two:
 
-- **Rules you already have that keep getting broken.** These are the useful
-  ones. A rule that exists and still gets corrected isn't working: it's too
-  vague, or the agent doesn't read it at the right moment.
+- **Rules you already have that these corrections hit.** If the corrections
+  came after you wrote the rule, the rule isn't working: it's too vague, or the
+  agent doesn't read it at the right moment. pushback can't know when you wrote
+  a rule, so check the evidence dates.
 - **New rules to consider.** Recurring corrections with no rule behind them.
 
 > [!TIP]

{pushback-0.1.2 → pushback-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pushback"
-version = "0.1.2"
+version = "0.1.3"
 description = "How often do you correct your coding agent, and at what kind of work? Measured from your own Claude Code transcripts."
 readme = "README.md"
 license = {text = "MIT"}

pushback-0.1.3/src/pushback/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.3"

{pushback-0.1.2 → pushback-0.1.3}/src/pushback/cli.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import argparse
 import json
+import math
 import os
 import sys
 
@@ -157,9 +158,11 @@ def cmd_rules(a):
     if a.from_response:
         with open(p["rules_items"], encoding="utf-8") as fh:
             items = json.load(fh)
-        with open(a.from_response, encoding="utf-8") as fh:
-            text = fh.read()
-        raw = json.loads(text[text.find("{"): text.rfind("}") + 1])  # tolerate prose or fences around the JSON
+        raws = []
+        for path in a.from_response:
+            with open(path, encoding="utf-8") as fh:
+                text = fh.read()
+            raws.append(json.loads(text[text.find("{"): text.rfind("}") + 1]))  # tolerate prose or fences
     else:
         if not os.path.exists(p["turns"]):
             sys.exit(f"{p['turns']} not found. Re-run `pushback extract`.")
@@ -184,20 +187,30 @@ def cmd_rules(a):
             print(f"prompt written to {p['rules_prompt']}")
             print("Give it to Claude (for example: ask Claude Code to answer the file), save the JSON reply,")
             print("then run: pushback rules --from-response <reply file>")
+            print("For stable rules, answer it 3 times in fresh sessions and pass all three replies.")
             return
 
         base = os.environ.get("ANTHROPIC_BASE_URL")
-        print(f"Sending them to {base or 'the Anthropic API'} with model {a.model}.")
+        print(f"Sending them to {base or 'the Anthropic API'} with model {a.model}"
+              + (f", {a.runs} times" if a.runs > 1 else "") + ".")
         if not a.yes and input("Continue? [y/N] ").strip().lower() != "y":
             sys.exit("aborted")
         import anthropic
-        raw = rules_mod.ask(anthropic.Anthropic(), request)
+        client = anthropic.Anthropic()
+        raws = [rules_mod.ask(client, request) for _ in range(a.runs)]
 
-    found = rules_mod.parse(raw, items, a.min_support)
+    runs = [rules_mod.parse(raw, items, a.min_support) for raw in raws]
+    if len(runs) == 1:
+        found, unstable = runs[0], []
+    else:
+        min_runs = a.min_runs or max(1, math.ceil(2 * len(runs) / 3))
+        found, unstable = rules_mod.stabilize(runs, min_runs)
     with open(p["rules"], "w", encoding="utf-8") as fh:
-        fh.write(rules_mod.render(found, len(items)))
+        fh.write(rules_mod.render(found, len(items), unstable, {x['id']: x.get('date', '') for x in items}))
     broken = sum(1 for r in found if r["covered_by"])
-    print(f"{len(found)} rules ({broken} you already have but keep breaking) -> {p['rules']}")
+    print(f"{len(found)} rules ({broken} match rules you already have) -> {p['rules']}")
+    if len(runs) > 1:
+        print(f"  kept rules that came back in at least {min_runs} of {len(runs)} runs; {len(unstable)} dropped as unstable")
 
 
 def main(argv=None):
@@ -245,7 +258,9 @@ def main(argv=None):
     ru.add_argument("--model", default=label_mod.DEFAULT_MODEL)
     ru.add_argument("--effort", default="high", choices=["low", "medium", "high"])
     ru.add_argument("--prompt-file", action="store_true", help="write the request to a file instead of calling the API")
-    ru.add_argument("--from-response", help="read the model's JSON reply from this file")
+    ru.add_argument("--from-response", nargs="+", help="read the model's JSON reply (or several, for stability)")
+    ru.add_argument("--runs", type=int, default=1, help="ask N times and keep only rules that keep coming back")
+    ru.add_argument("--min-runs", type=int, help="runs a rule must appear in (default: two thirds of --runs)")
     ru.add_argument("-y", "--yes", action="store_true")
     ru.set_defaults(func=cmd_rules)
 

{pushback-0.1.2 → pushback-0.1.3}/src/pushback/extract.py

@@ -139,6 +139,7 @@ def _extract_file(path, session, out, max_user_chars, max_context_chars):
                     "user": text[:max_user_chars].replace("\n", " "),
                 })
                 out.turns[f"{session}:{n}"] = {
+                    "date": str(entry.get("timestamp", ""))[:10],  # YYYY-MM-DD, for rule evidence dates
                     "prompt": text,
                     "prev_turn": "\n\n".join(turn_text),
                     "prev_turn_tools": turn_tools,

{pushback-0.1.2 → pushback-0.1.3}/src/pushback/rules.py

@@ -1,154 +1,215 @@
-"""Turn recurring corrections into rules you can paste into CLAUDE.md.
-
-The model groups corrections that share a cause and drafts one instruction per
-group. Two checks keep that honest:
-
-- Support is counted here from the correction ids the model cites, never taken
-  from the model's own numbers, and a rule needs `min_support` real corrections.
-- Given your existing rule files, the model marks rules you already have. Those
-  are the interesting ones: a rule that exists and still gets broken isn't working.
-"""
-
-from __future__ import annotations
-
-import json
-
-from .prompt import TASKS
-
-AGENT_CHARS = 500
-CORRECTION_CHARS = 400
-
-SYSTEM = """You are given corrections a user made to their AI agent. Each item has an id, the task type, the end of the agent's turn that was corrected, and the user's correction.
-
-Find recurring patterns: the same kind of mistake corrected at least 3 times. For each pattern, write one rule the agent can follow to avoid it.
-
-Rules:
-- Write each rule as an instruction for a CLAUDE.md file, in plain words. Make it specific and checkable. "Be careful" or "write better" is not a rule.
-- Generalise. Never include client names, company names, people's names, credentials or quoted message text in a rule.
-- "why" is one sentence on what kept going wrong.
-- "ids" lists every correction that supports the pattern. Only cite ids that really show the same mistake.
-- If existing rules are provided and one of them already asks for this, put a short quote of it (15 words max) in "covered_by". Otherwise leave "covered_by" as an empty string.
-- Prefer fewer, sharper rules. Skip one-off corrections."""
-
-SCHEMA = {
-    "type": "object",
-    "properties": {
-        "rules": {
-            "type": "array",
-            "items": {
-                "type": "object",
-                "properties": {
-                    "rule": {"type": "string"},
-                    "why": {"type": "string"},
-                    "task": {"type": "string", "enum": TASKS},
-                    "ids": {"type": "array", "items": {"type": "integer"}},
-                    "covered_by": {"type": "string"},
-                },
-                "required": ["rule", "why", "task", "ids", "covered_by"],
-                "additionalProperties": False,
-            },
-        }
-    },
-    "required": ["rules"],
-    "additionalProperties": False,
-}
-
-
-def collect(labels: dict[str, dict], turns: dict[str, dict], tasks=None, ctypes=None, limit=400) -> list[dict]:
-    """Every labelled correction with the context the model needs. Newest last, capped at `limit`."""
-    items = []
-    for mid, lab in labels.items():
-        if not lab["correction"] or (tasks and lab["task"] not in tasks) or (ctypes and lab["ctype"] not in ctypes):
-            continue
-        turn = turns.get(mid, {})
-        items.append({
-            "id": mid,
-            "task": lab["task"],
-            "agent": turn.get("prev_turn", "")[-AGENT_CHARS:],
-            "correction": turn.get("prompt", "")[:CORRECTION_CHARS],
-        })
-    items.sort(key=lambda x: (x["id"].rpartition(":")[0], int(x["id"].rpartition(":")[2])))
-    return items[-limit:]
-
-
-def build_request(items: list[dict], existing: str, model: str, effort: str = "high") -> dict:
-    payload = [{"n": n, "task": x["task"], "agent": x["agent"], "correction": x["correction"]}
-               for n, x in enumerate(items)]
-    content = "CORRECTIONS (the \"n\" field is the id to cite):\n" + json.dumps(payload, ensure_ascii=False)
-    if existing.strip():
-        content = "EXISTING RULES:\n" + existing.strip() + "\n\n" + content
-    return {
-        "model": model,
-        "max_tokens": 16000,
-        "system": SYSTEM,
-        "messages": [{"role": "user", "content": content}],
-        "output_config": {"effort": effort, "format": {"type": "json_schema", "schema": SCHEMA}},
-    }
-
-
-def parse(raw: dict, items: list[dict], min_support: int = 3) -> list[dict]:
-    """Map cited numbers back to correction ids, recount support, drop weak or malformed rules."""
-    rules = []
-    for r in raw.get("rules", []):
-        if not isinstance(r, dict) or not str(r.get("rule", "")).strip():
-            continue
-        ids = []
-        for i in r.get("ids", []):
-            if isinstance(i, int) and 0 <= i < len(items) and items[i]["id"] not in ids:
-                ids.append(items[i]["id"])
-        if len(ids) < min_support:
-            continue
-        rules.append({
-            "rule": r["rule"].strip(),
-            "why": str(r.get("why", "")).strip(),
-            "task": r.get("task") if r.get("task") in TASKS else "meta",
-            "support": len(ids),
-            "ids": ids,
-            "covered_by": str(r.get("covered_by", "")).strip(),
-        })
-    rules.sort(key=lambda r: r["support"], reverse=True)
-    return rules
-
-
-def ask(client, request: dict) -> dict:
-    response = client.messages.create(**request)
-    if response.stop_reason == "refusal":
-        raise RuntimeError("model declined (stop_reason=refusal)")
-    if response.stop_reason == "max_tokens":
-        raise RuntimeError("output hit max_tokens; narrow the input with --task or --limit")
-    text = next((b.text for b in response.content if b.type == "text"), "")
-    return json.loads(text)
-
-
-def _short(message_id: str) -> str:
-    """session-uuid:12 -> first 8 chars of the session, which is enough to find it."""
-    session, _, n = message_id.rpartition(":")
-    return f"{session[:8]}:{n}"
-
-
-def render(rules: list[dict], n_corrections: int) -> str:
-    broken = [r for r in rules if r["covered_by"]]
-    new = [r for r in rules if not r["covered_by"]]
-    out = [f"# Rules from {n_corrections} corrections", ""]
-    out.append("Each rule is backed by at least the number of corrections shown. Read the evidence ids "
-               "in your own data before adopting a rule; the model drafted these, you decide.")
-
-    def block(title, note, group):
-        if not group:
-            return
-        out.extend(["", f"## {title}", "", note, ""])
-        for r in group:
-            out.append(f"- **{r['rule']}**  ")
-            out.append(f"  {r['why']} ({r['task']}, {r['support']} corrections)")
-            if r["covered_by"]:
-                out.append(f"  You already have: \"{r['covered_by']}\"")
-            shown = [_short(i) for i in r["ids"][:8]]
-            out.append(f"  Evidence: {', '.join(shown)}" + (" …" if len(r["ids"]) > 8 else ""))
-
-    block("Rules you already have that keep getting broken",
-          "A rule that exists and still gets corrected isn't working. Make it more specific, "
-          "move it somewhere the agent reads at the right moment, or turn it into a check.", broken)
-    block("New rules to consider", "Recurring corrections with no rule behind them yet.", new)
-    if not rules:
-        out.extend(["", "No pattern reached the minimum support. Try more data or a lower --min-support."])
-    return "\n".join(out) + "\n"
+"""Turn recurring corrections into rules you can paste into CLAUDE.md.
+
+The model groups corrections that share a cause and drafts one instruction per
+group. Two checks keep that honest:
+
+- Support is counted here from the correction ids the model cites, never taken
+  from the model's own numbers, and a rule needs `min_support` real corrections.
+- Given your existing rule files, the model marks rules you already have. Those
+  are the interesting ones: a rule that exists and still gets broken isn't working.
+"""
+
+from __future__ import annotations
+
+import json
+
+from .prompt import TASKS
+
+AGENT_CHARS = 500
+CORRECTION_CHARS = 400
+
+SYSTEM = """You are given corrections a user made to their AI agent. Each item has an id, the task type, the end of the agent's turn that was corrected, and the user's correction.
+
+Find recurring patterns: the same kind of mistake corrected at least 3 times. For each pattern, write one rule the agent can follow to avoid it.
+
+Rules:
+- Write each rule as an instruction for a CLAUDE.md file, in plain words. Make it specific and checkable. "Be careful" or "write better" is not a rule.
+- Generalise. Never include client names, company names, people's names, credentials or quoted message text in a rule.
+- "why" is one sentence on what kept going wrong.
+- "ids" lists every correction that supports the pattern. Only cite ids that really show the same mistake.
+- If existing rules are provided and one of them already asks for this, put a short quote of it (15 words max) in "covered_by". Otherwise leave "covered_by" as an empty string.
+- Prefer fewer, sharper rules. Skip one-off corrections."""
+
+SCHEMA = {
+    "type": "object",
+    "properties": {
+        "rules": {
+            "type": "array",
+            "items": {
+                "type": "object",
+                "properties": {
+                    "rule": {"type": "string"},
+                    "why": {"type": "string"},
+                    "task": {"type": "string", "enum": TASKS},
+                    "ids": {"type": "array", "items": {"type": "integer"}},
+                    "covered_by": {"type": "string"},
+                },
+                "required": ["rule", "why", "task", "ids", "covered_by"],
+                "additionalProperties": False,
+            },
+        }
+    },
+    "required": ["rules"],
+    "additionalProperties": False,
+}
+
+
+def collect(labels: dict[str, dict], turns: dict[str, dict], tasks=None, ctypes=None, limit=400) -> list[dict]:
+    """Every labelled correction with the context the model needs. Newest last, capped at `limit`."""
+    items = []
+    for mid, lab in labels.items():
+        if not lab["correction"] or (tasks and lab["task"] not in tasks) or (ctypes and lab["ctype"] not in ctypes):
+            continue
+        turn = turns.get(mid, {})
+        items.append({
+            "id": mid,
+            "task": lab["task"],
+            "agent": turn.get("prev_turn", "")[-AGENT_CHARS:],
+            "correction": turn.get("prompt", "")[:CORRECTION_CHARS],
+            "date": turn.get("date", ""),
+        })
+    items.sort(key=lambda x: (x["id"].rpartition(":")[0], int(x["id"].rpartition(":")[2])))
+    return items[-limit:]
+
+
+def build_request(items: list[dict], existing: str, model: str, effort: str = "high") -> dict:
+    payload = [{"n": n, "task": x["task"], "agent": x["agent"], "correction": x["correction"]}
+               for n, x in enumerate(items)]
+    content = "CORRECTIONS (the \"n\" field is the id to cite):\n" + json.dumps(payload, ensure_ascii=False)
+    if existing.strip():
+        content = "EXISTING RULES:\n" + existing.strip() + "\n\n" + content
+    return {
+        "model": model,
+        "max_tokens": 16000,
+        "system": SYSTEM,
+        "messages": [{"role": "user", "content": content}],
+        "output_config": {"effort": effort, "format": {"type": "json_schema", "schema": SCHEMA}},
+    }
+
+
+def parse(raw: dict, items: list[dict], min_support: int = 3) -> list[dict]:
+    """Map cited numbers back to correction ids, recount support, drop weak or malformed rules."""
+    rules = []
+    for r in raw.get("rules", []):
+        if not isinstance(r, dict) or not str(r.get("rule", "")).strip():
+            continue
+        ids = []
+        for i in r.get("ids", []):
+            if isinstance(i, int) and 0 <= i < len(items) and items[i]["id"] not in ids:
+                ids.append(items[i]["id"])
+        if len(ids) < min_support:
+            continue
+        rules.append({
+            "rule": r["rule"].strip(),
+            "why": str(r.get("why", "")).strip(),
+            "task": r.get("task") if r.get("task") in TASKS else "meta",
+            "support": len(ids),
+            "ids": ids,
+            "covered_by": str(r.get("covered_by", "")).strip(),
+        })
+    rules.sort(key=lambda r: r["support"], reverse=True)
+    return rules
+
+
+def ask(client, request: dict) -> dict:
+    response = client.messages.create(**request)
+    if response.stop_reason == "refusal":
+        raise RuntimeError("model declined (stop_reason=refusal)")
+    if response.stop_reason == "max_tokens":
+        raise RuntimeError("output hit max_tokens; narrow the input with --task or --limit")
+    text = next((b.text for b in response.content if b.type == "text"), "")
+    return json.loads(text)
+
+
+def stabilize(runs: list[list[dict]], min_runs: int, min_overlap: float = 0.5) -> tuple[list[dict], list[dict]]:
+    """Group the same rule across several runs, and keep only the rules that keep coming back.
+
+    Rules are matched by their evidence, not their wording: two rules from different runs are the
+    same rule when their cited corrections overlap by at least `min_overlap` (the share of the
+    smaller set). A cluster takes at most one rule per run. Its text comes from its best-supported
+    member; its evidence is the corrections cited by at least half of its members.
+    Returns (stable, unstable), each rule annotated with how many runs it appeared in.
+    """
+    clusters: list[dict] = []
+    members = sorted(((ri, r) for ri, rs in enumerate(runs) for r in rs), key=lambda x: -x[1]["support"])
+    for ri, r in members:
+        ids = set(r["ids"])
+        best, best_overlap = None, 0.0
+        for c in clusters:
+            if ri in c["runs"]:
+                continue
+            overlap = len(ids & c["ids"]) / min(len(ids), len(c["ids"]))
+            if overlap >= min_overlap and overlap > best_overlap:
+                best, best_overlap = c, overlap
+        if best is None:
+            clusters.append({"runs": {ri}, "ids": set(ids), "rules": [r]})
+        else:
+            best["runs"].add(ri)
+            best["ids"] |= ids
+            best["rules"].append(r)
+
+    stable, unstable = [], []
+    for c in clusters:
+        rep = max(c["rules"], key=lambda r: r["support"])
+        counts: dict[str, int] = {}
+        for r in c["rules"]:
+            for i in r["ids"]:
+                counts[i] = counts.get(i, 0) + 1
+        need = (len(c["rules"]) + 1) // 2
+        core = [i for i, k in sorted(counts.items(), key=lambda kv: -kv[1]) if k >= need]
+        covered = [r["covered_by"] for r in c["rules"] if r["covered_by"]]
+        out = {**rep, "ids": core, "support": len(core), "runs": len(c["runs"]), "n_runs": len(runs),
+               "covered_by": covered[0] if len(covered) * 2 >= len(c["rules"]) else ""}
+        (stable if out["runs"] >= min_runs else unstable).append(out)
+    stable.sort(key=lambda r: (r["runs"], r["support"]), reverse=True)
+    unstable.sort(key=lambda r: (r["runs"], r["support"]), reverse=True)
+    return stable, unstable
+
+
+def _short(message_id: str) -> str:
+    """session-uuid:12 -> first 8 chars of the session, which is enough to find it."""
+    session, _, n = message_id.rpartition(":")
+    return f"{session[:8]}:{n}"
+
+
+def render(rules: list[dict], n_corrections: int, unstable: list[dict] | None = None,
+           dates: dict[str, str] | None = None) -> str:
+    broken = [r for r in rules if r["covered_by"]]
+    new = [r for r in rules if not r["covered_by"]]
+    out = [f"# Rules from {n_corrections} corrections", ""]
+    out.append("Each rule is backed by at least the number of corrections shown. Read the evidence ids "
+               "in your own data before adopting a rule; the model drafted these, you decide.")
+    n_runs = rules[0]["n_runs"] if rules and "n_runs" in rules[0] else (unstable[0]["n_runs"] if unstable else 1)
+    if n_runs > 1:
+        out.append("")
+        out.append(f"Asked {n_runs} times. Only rules that came back in several runs, citing mostly the same "
+                   "corrections, are kept; the rest are listed at the end.")
+
+    def block(title, note, group):
+        if not group:
+            return
+        out.extend(["", f"## {title}", "", note, ""])
+        for r in group:
+            out.append(f"- **{r['rule']}**  ")
+            runs = f", in {r['runs']}/{r['n_runs']} runs" if r.get("n_runs", 1) > 1 else ""
+            out.append(f"  {r['why']} ({r['task']}, {r['support']} corrections{runs})")
+            if r["covered_by"]:
+                out.append(f"  You already have: \"{r['covered_by']}\"")
+            shown = [_short(i) for i in r["ids"][:8]]
+            when = sorted(dates[i] for i in r["ids"] if dates and dates.get(i))
+            span = f" ({when[0]} to {when[-1]})" if when else ""
+            out.append(f"  Evidence{span}: {', '.join(shown)}" + (" …" if len(r["ids"]) > 8 else ""))
+
+    block("Rules you already have that these corrections hit",
+          "If these corrections happened after you wrote the rule, the rule isn't working: make it more specific, "
+          "move it somewhere the agent reads at the right moment, or turn it into a check. If you wrote the rule "
+          "after them, it may already be doing its job; check the evidence dates.", broken)
+    block("New rules to consider", "Recurring corrections with no rule behind them yet.", new)
+    if unstable:
+        out.extend(["", "## Not stable enough to keep", "",
+                    "These showed up in too few runs. They may still be real; they're just not reliable yet.", ""])
+        for r in unstable:
+            out.append(f"- {r['rule']} ({r['runs']}/{r['n_runs']} runs, {r['support']} corrections)")
+    if not rules:
+        out.extend(["", "No pattern reached the minimum support. Try more data or a lower --min-support."])
+    return "\n".join(out) + "\n"

{pushback-0.1.2 → pushback-0.1.3/src/pushback.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pushback
-Version: 0.1.2
+Version: 0.1.3
 Summary: How often do you correct your coding agent, and at what kind of work? Measured from your own Claude Code transcripts.
 Author: Intikhab Azam
 License: MIT
@@ -54,7 +54,7 @@ shows what you use your AI coding agent for, how often you correct it on each
 kind of work (code, writing, media, research, ops, meta), and how often a
 correction needs correcting again. It drafts
 **CLAUDE.md rules** from the corrections you keep repeating, shows which of
-your existing rules keep getting broken, and exports your corrections as
+your existing rules those corrections run into, and exports your corrections as
 **preference pairs** (prompt / chosen / rejected). A hand-audit step tells you
 how far to trust every number.
 
@@ -67,7 +67,7 @@ pip install pushback
 The author's own 1,633 messages to Claude Code:
 
 <p align="center">
-  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. Three existing rules kept getting broken.">
+  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. One existing rule kept getting broken after it was written.">
 </p>
 
 | task | messages | share of use | corrections | rate | 95% CI | corrected again |
@@ -94,7 +94,7 @@ Most corrected topics (at least 5 messages each):
 
 - Against 91 hand-checked messages the correction labels had **precision 0.97 and recall 0.80**. Topic labels were spot-checked, not audited.
 - **1 in 3 corrections needed a second one.** 70 of 201 fixes got corrected again (35%), 41% on writing and 44% on images.
-- **3 rules the author had already written kept getting broken**, with 6 to 8 corrections each.
+- **1 rule the author had already written kept getting broken**: "images must not look AI-generated" was corrected 3 more times after it was written. Two other matching rules turned out to be written *because of* the corrections, which is why every rule now shows its evidence dates.
 
 > [!NOTE]
 > These rates describe a workflow, not a model. The author's code work runs
@@ -177,17 +177,29 @@ pushback rules                                  # drafts rules into pushback-dat
 pushback rules --existing CLAUDE.md notes/*.md  # also check against the rules you already have
 pushback rules --prompt-file                    # no API key? writes the request to a file instead
 pushback rules --from-response reply.json       # ...and reads Claude's answer back
+pushback rules --runs 3                         # ask 3 times, keep only the rules that keep coming back
 ```
 
 The model groups corrections that share a cause and drafts one CLAUDE.md
 instruction per group. Support is counted from the correction ids it cites,
 not from its own numbers, and a rule needs at least 3 real corrections.
 
+**Stable rules.** A model won't write the same rules twice. With `--runs 3`
+(or three replies passed to `--from-response`), pushback matches rules across
+runs by the corrections they cite, not by their wording, and keeps only the
+ones that come back in at least 2 of 3 runs. Each rule shows how many runs it
+appeared in, and the unstable ones are listed separately instead of vanishing.
+
+On the author's 214 corrections, three independent runs wrote 12, 10 and 7
+rules: 16 distinct rules in total. 5 came back in all three runs, 3 in two,
+and **8 showed up only once**. A single run mixes real rules with one-offs.
+
 When you pass your existing rule files, the output splits in two:
 
-- **Rules you already have that keep getting broken.** These are the useful
-  ones. A rule that exists and still gets corrected isn't working: it's too
-  vague, or the agent doesn't read it at the right moment.
+- **Rules you already have that these corrections hit.** If the corrections
+  came after you wrote the rule, the rule isn't working: it's too vague, or the
+  agent doesn't read it at the right moment. pushback can't know when you wrote
+  a rule, so check the evidence dates.
 - **New rules to consider.** Recurring corrections with no rule behind them.
 
 > [!TIP]

{pushback-0.1.2 → pushback-0.1.3}/tests/test_pushback.py

@@ -309,7 +309,7 @@ def test_rules_render_separates_broken_rules():
         {"rule": "B", "why": "w", "task": "media", "support": 3, "ids": ["s:2"], "covered_by": ""},
     ]
     md = rules.render(found, 40)
-    assert md.index("keep getting broken") < md.index("**A**") < md.index("New rules") < md.index("**B**")
+    assert md.index("these corrections hit") < md.index("**A**") < md.index("New rules") < md.index("**B**")
     assert "No pattern" in rules.render([], 3)
 
 
@@ -337,7 +337,7 @@ def test_rules_cli_round_trip_without_api(tmp_path, capsys):
     cli.main(["--data", str(d), "rules", "--from-response", str(reply)])
     md = (d / "rules.md").read_text(encoding="utf-8")
     assert "Keep posts under 80 words" in md and "s:1, s:3, s:5" in md
-    assert "1 rules (1 you already have" in capsys.readouterr().out
+    assert "1 rules (1 match rules you already have" in capsys.readouterr().out
 
 
 def test_rules_render_shortens_session_ids():
@@ -388,71 +388,144 @@ def test_prompt_topics_are_in_schema_and_system():
     from pushback import prompt
     assert "cli-tool" in prompt.SYSTEM and "cli-tool" in prompt.ALL_TOPICS
     assert all("other" in ts for ts in prompt.TOPICS.values())
-
-
-# --- label without an API key -----------------------------------------------
-
-def _label_data(tmp_path, n=5):
-    d = tmp_path / "data"
-    d.mkdir()
-    with open(d / "messages.jsonl", "w", encoding="utf-8") as fh:
-        for i in range(n):
-            fh.write(json.dumps({"id": f"s:{i}", "session": "s", "prev_assistant": f"agent {i}",
-                                 "user": f"user {i}"}) + "\n")
-    return d
-
-
-def _answer(ids, correction=False):
-    return json.dumps({"labels": [{"id": i, "task": "code", "topic": "cli-tool", "correction": correction,
-                                   "ctype": "code" if correction else "none", "conf": "high"} for i in ids]})
-
-
-def test_label_prompt_file_round_trip(tmp_path, capsys):
-    d = _label_data(tmp_path, 5)
-    cli.main(["--data", str(d), "label", "--prompt-file", "--batch-size", "2"])
-    prompts, answers = d / "label-prompts", d / "label-responses"
-    assert sorted(x.name for x in prompts.glob("batch_*.md")) == ["batch_000.md", "batch_001.md", "batch_002.md"]
-    instructions = (prompts / "INSTRUCTIONS.md").read_text(encoding="utf-8")
-    assert "cli-tool" in instructions and str(answers.resolve()).replace("\\", "/") in instructions
-    assert "No API key needed" in capsys.readouterr().out
-    assert "s:0" not in (prompts / "batch_000.md").read_text(encoding="utf-8")  # the model never sees real ids
-
-    (answers / "batch_000.json").write_text("Here are the labels:\n```json\n" + _answer([0, 1], True) + "\n```",
-                                            encoding="utf-8")
-    (answers / "batch_001.json").write_text("not json at all", encoding="utf-8")
-    cli.main(["--data", str(d), "label", "--from-responses"])
-    out = capsys.readouterr().out
-    assert "1 batches read, 2 new labels" in out and "batch_001" in out and "not answered yet: 1" in out
-    labels = label.load_labels(str(d / "labels.jsonl"))
-    assert set(labels) == {"s:0", "s:1"} and labels["s:0"]["topic"] == "cli-tool"
-
-    # the next job only covers what is still unlabelled, and old answers can't leak into it
-    cli.main(["--data", str(d), "label", "--prompt-file", "--batch-size", "2"])
-    assert sorted(x.name for x in prompts.glob("batch_*.md")) == ["batch_000.md", "batch_001.md"]
-    assert not list(answers.glob("batch_*.json"))
-    (answers / "batch_000.json").write_text(_answer([0, 1]), encoding="utf-8")
-    (answers / "batch_001.json").write_text(_answer([0]), encoding="utf-8")
-    cli.main(["--data", str(d), "label", "--from-responses"])
-    assert set(label.load_labels(str(d / "labels.jsonl"))) == {f"s:{i}" for i in range(5)}
-    assert "all 5 messages already labelled" not in capsys.readouterr().out
-    cli.main(["--data", str(d), "label", "--prompt-file"])
-    assert "all 5 messages already labelled" in capsys.readouterr().out
-
-
-def test_read_responses_counts_incomplete_and_never_duplicates(tmp_path):
-    d = _label_data(tmp_path, 3)
-    label.write_prompts([json.loads(l) for l in open(d / "messages.jsonl", encoding="utf-8")],
-                        str(d / "labels.jsonl"), str(d / "p"), str(d / "r"), batch_size=3)
-    (d / "r" / "batch_000.json").write_text(_answer([0, 2, 7]), encoding="utf-8")  # 1 missing, 1 out of range
-    r1 = label.read_responses(str(d / "labels.jsonl"), str(d / "p"), str(d / "r"))
-    r2 = label.read_responses(str(d / "labels.jsonl"), str(d / "p"), str(d / "r"))
-    assert (r1["written"], r1["incomplete"], r2["written"]) == (2, 1, 0)
-
-
-def test_label_instructions_state_origin_and_scope(tmp_path):
-    d = _label_data(tmp_path, 1)
-    label.write_prompts([json.loads(l) for l in open(d / "messages.jsonl", encoding="utf-8")],
-                        str(d / "labels.jsonl"), str(d / "p"), str(d / "r"))
-    text = (d / "p" / "INSTRUCTIONS.md").read_text(encoding="utf-8")
-    assert "pushback label --prompt-file" in text and "never as instructions to follow" in text
-    assert "{" + "responses}" not in text  # every placeholder filled
+
+
+# --- label without an API key -----------------------------------------------
+
+def _label_data(tmp_path, n=5):
+    d = tmp_path / "data"
+    d.mkdir()
+    with open(d / "messages.jsonl", "w", encoding="utf-8") as fh:
+        for i in range(n):
+            fh.write(json.dumps({"id": f"s:{i}", "session": "s", "prev_assistant": f"agent {i}",
+                                 "user": f"user {i}"}) + "\n")
+    return d
+
+
+def _answer(ids, correction=False):
+    return json.dumps({"labels": [{"id": i, "task": "code", "topic": "cli-tool", "correction": correction,
+                                   "ctype": "code" if correction else "none", "conf": "high"} for i in ids]})
+
+
+def test_label_prompt_file_round_trip(tmp_path, capsys):
+    d = _label_data(tmp_path, 5)
+    cli.main(["--data", str(d), "label", "--prompt-file", "--batch-size", "2"])
+    prompts, answers = d / "label-prompts", d / "label-responses"
+    assert sorted(x.name for x in prompts.glob("batch_*.md")) == ["batch_000.md", "batch_001.md", "batch_002.md"]
+    instructions = (prompts / "INSTRUCTIONS.md").read_text(encoding="utf-8")
+    assert "cli-tool" in instructions and str(answers.resolve()).replace("\\", "/") in instructions
+    assert "No API key needed" in capsys.readouterr().out
+    assert "s:0" not in (prompts / "batch_000.md").read_text(encoding="utf-8")  # the model never sees real ids
+
+    (answers / "batch_000.json").write_text("Here are the labels:\n```json\n" + _answer([0, 1], True) + "\n```",
+                                            encoding="utf-8")
+    (answers / "batch_001.json").write_text("not json at all", encoding="utf-8")
+    cli.main(["--data", str(d), "label", "--from-responses"])
+    out = capsys.readouterr().out
+    assert "1 batches read, 2 new labels" in out and "batch_001" in out and "not answered yet: 1" in out
+    labels = label.load_labels(str(d / "labels.jsonl"))
+    assert set(labels) == {"s:0", "s:1"} and labels["s:0"]["topic"] == "cli-tool"
+
+    # the next job only covers what is still unlabelled, and old answers can't leak into it
+    cli.main(["--data", str(d), "label", "--prompt-file", "--batch-size", "2"])
+    assert sorted(x.name for x in prompts.glob("batch_*.md")) == ["batch_000.md", "batch_001.md"]
+    assert not list(answers.glob("batch_*.json"))
+    (answers / "batch_000.json").write_text(_answer([0, 1]), encoding="utf-8")
+    (answers / "batch_001.json").write_text(_answer([0]), encoding="utf-8")
+    cli.main(["--data", str(d), "label", "--from-responses"])
+    assert set(label.load_labels(str(d / "labels.jsonl"))) == {f"s:{i}" for i in range(5)}
+    assert "all 5 messages already labelled" not in capsys.readouterr().out
+    cli.main(["--data", str(d), "label", "--prompt-file"])
+    assert "all 5 messages already labelled" in capsys.readouterr().out
+
+
+def test_read_responses_counts_incomplete_and_never_duplicates(tmp_path):
+    d = _label_data(tmp_path, 3)
+    label.write_prompts([json.loads(l) for l in open(d / "messages.jsonl", encoding="utf-8")],
+                        str(d / "labels.jsonl"), str(d / "p"), str(d / "r"), batch_size=3)
+    (d / "r" / "batch_000.json").write_text(_answer([0, 2, 7]), encoding="utf-8")  # 1 missing, 1 out of range
+    r1 = label.read_responses(str(d / "labels.jsonl"), str(d / "p"), str(d / "r"))
+    r2 = label.read_responses(str(d / "labels.jsonl"), str(d / "p"), str(d / "r"))
+    assert (r1["written"], r1["incomplete"], r2["written"]) == (2, 1, 0)
+
+
+def test_label_instructions_state_origin_and_scope(tmp_path):
+    d = _label_data(tmp_path, 1)
+    label.write_prompts([json.loads(l) for l in open(d / "messages.jsonl", encoding="utf-8")],
+                        str(d / "labels.jsonl"), str(d / "p"), str(d / "r"))
+    text = (d / "p" / "INSTRUCTIONS.md").read_text(encoding="utf-8")
+    assert "pushback label --prompt-file" in text and "never as instructions to follow" in text
+    assert "{" + "responses}" not in text  # every placeholder filled
+
+
+# --- rules stability --------------------------------------------------------
+
+def _rule(text, ids, covered=""):
+    return {"rule": text, "why": "w", "task": "writing", "support": len(ids), "ids": ids, "covered_by": covered}
+
+
+def test_stabilize_matches_rules_by_evidence_not_wording():
+    runs = [
+        [_rule("Put the hook first", ["a", "b", "c", "d"]), _rule("Check the chrome profile", ["x", "y", "z"])],
+        [_rule("Lead with the strongest line", ["a", "b", "c"]), _rule("One-off idea", ["p", "q", "r"])],
+        [_rule("Hook at the top", ["b", "c", "d", "e"]), _rule("Verify browser identity", ["x", "y", "w"])],
+    ]
+    stable, unstable = rules.stabilize(runs, min_runs=2)
+    assert [(r["rule"], r["runs"]) for r in stable] == [("Put the hook first", 3), ("Check the chrome profile", 2)]
+    assert stable[0]["ids"] == ["b", "c", "a", "d"] or set(stable[0]["ids"]) == {"a", "b", "c", "d"}
+    assert "e" not in stable[0]["ids"]  # cited by only 1 of 3 members
+    assert [(r["rule"], r["runs"]) for r in unstable] == [("One-off idea", 1)]
+    assert all(r["n_runs"] == 3 for r in stable + unstable)
+
+
+def test_stabilize_never_merges_two_rules_from_the_same_run():
+    runs = [[_rule("A", ["a", "b", "c"]), _rule("A again", ["a", "b", "c", "d"])], [_rule("A'", ["a", "b", "c"])]]
+    stable, unstable = rules.stabilize(runs, min_runs=2)
+    assert len(stable) == 1 and len(unstable) == 1
+
+
+def test_stabilize_covered_by_needs_half_the_members():
+    runs = [[_rule("R", ["a", "b", "c"], "no tables")], [_rule("R", ["a", "b", "c"])], [_rule("R", ["a", "b", "c"])]]
+    assert rules.stabilize(runs, 2)[0][0]["covered_by"] == ""
+    runs[1][0]["covered_by"] = "no tables"
+    assert rules.stabilize(runs, 2)[0][0]["covered_by"] == "no tables"
+
+
+def test_rules_cli_multiple_responses_reports_stability(tmp_path, capsys):
+    labels, turns = _rules_fixture()
+    d = tmp_path / "data"
+    d.mkdir()
+    with open(d / "labels.jsonl", "w", encoding="utf-8") as fh:
+        for mid, lab in labels.items():
+            fh.write(json.dumps({"id": mid, **lab}) + "\n")
+    with open(d / "turns.jsonl", "w", encoding="utf-8") as fh:
+        for mid, t in turns.items():
+            fh.write(json.dumps({"id": mid, **t}) + "\n")
+    cli.main(["--data", str(d), "rules", "--prompt-file", "--existing"])
+    replies = []
+    for k, rs in enumerate([
+        [{"rule": "Keep posts short", "ids": [0, 1, 2]}, {"rule": "Fluke", "ids": [2, 3, 4]}],
+        [{"rule": "Posts under 80 words", "ids": [0, 1, 2]}],
+        [{"rule": "Short posts", "ids": [0, 1, 3]}],
+    ]):
+        f = tmp_path / f"r{k}.json"
+        f.write_text(json.dumps({"rules": [{**r, "why": "w", "task": "writing", "covered_by": ""} for r in rs]}),
+                     encoding="utf-8")
+        replies.append(str(f))
+    cli.main(["--data", str(d), "rules", "--from-response", *replies])
+    out = capsys.readouterr().out
+    assert "1 rules" in out and "at least 2 of 3 runs; 1 dropped" in out
+    md = (d / "rules.md").read_text(encoding="utf-8")
+    assert "in 3/3 runs" in md and "Not stable enough" in md and "Fluke (1/3 runs" in md
+
+
+def test_rules_evidence_shows_date_range(tmp_path):
+    found = [{"rule": "A", "why": "w", "task": "ops", "support": 3, "ids": ["s:1", "s:2", "s:3"], "covered_by": "x"}]
+    md = rules.render(found, 3, dates={"s:1": "2026-09-03", "s:2": "2026-08-27", "s:3": ""})
+    assert "Evidence (2026-08-27 to 2026-09-03):" in md
+    assert "Evidence:" in rules.render(found, 3)  # no dates known, no range
+
+
+def test_extract_records_message_dates(tmp_path):
+    _write_session(tmp_path, "d", [dict(_user("hello"), timestamp="2026-09-21T10:00:00.000Z")])
+    assert extract.extract(str(tmp_path)).turns["d:0"]["date"] == "2026-09-21"

pushback-0.1.2/src/pushback/__init__.py

@@ -1 +0,0 @@
-__version__ = "0.1.2"