pushback 0.1.1__tar.gz → 0.1.3__tar.gz

{pushback-0.1.1/src/pushback.egg-info → pushback-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pushback
-Version: 0.1.1
+Version: 0.1.3
 Summary: How often do you correct your coding agent, and at what kind of work? Measured from your own Claude Code transcripts.
 Author: Intikhab Azam
 License: MIT
@@ -54,7 +54,7 @@ shows what you use your AI coding agent for, how often you correct it on each
 kind of work (code, writing, media, research, ops, meta), and how often a
 correction needs correcting again. It drafts
 **CLAUDE.md rules** from the corrections you keep repeating, shows which of
-your existing rules keep getting broken, and exports your corrections as
+your existing rules those corrections run into, and exports your corrections as
 **preference pairs** (prompt / chosen / rejected). A hand-audit step tells you
 how far to trust every number.
 
@@ -67,7 +67,7 @@ pip install pushback
 The author's own 1,633 messages to Claude Code:
 
 <p align="center">
-  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. Three existing rules kept getting broken.">
+  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. One existing rule kept getting broken after it was written.">
 </p>
 
 | task | messages | share of use | corrections | rate | 95% CI | corrected again |
@@ -94,7 +94,7 @@ Most corrected topics (at least 5 messages each):
 
 - Against 91 hand-checked messages the correction labels had **precision 0.97 and recall 0.80**. Topic labels were spot-checked, not audited.
 - **1 in 3 corrections needed a second one.** 70 of 201 fixes got corrected again (35%), 41% on writing and 44% on images.
-- **3 rules the author had already written kept getting broken**, with 6 to 8 corrections each.
+- **1 rule the author had already written kept getting broken**: "images must not look AI-generated" was corrected 3 more times after it was written. Two other matching rules turned out to be written *because of* the corrections, which is why every rule now shows its evidence dates.
 
 > [!NOTE]
 > These rates describe a workflow, not a model. The author's code work runs
@@ -121,10 +121,20 @@ pushback export             # your corrections as prompt/chosen/rejected pairs
 `ant auth login` profile. It defaults to `claude-opus-5` at low effort; change
 it with `--model`. To send through a gateway, set `ANTHROPIC_BASE_URL`.
 
+> [!TIP]
+> **No API key?** Every step works without one:
+> ```bash
+> pushback label --prompt-file       # writes the labelling job as batch files
+> # in Claude Code: "read <path it prints>/INSTRUCTIONS.md and follow it"
+> pushback label --from-responses    # reads the answers back, with the same checks as the API path
+> ```
+> Your messages are still read by Claude, through Claude Code on your
+> subscription. Unanswered or broken batches are listed and stay unlabelled.
+
 | command | what it does | leaves your machine? |
 |---|---|---|
 | `extract` | pulls the messages you typed, plus full agent turns, out of transcripts | no |
-| `label` | tags each message with a task type and whether it's a correction | yes, to the model provider, after asking |
+| `label` | tags each message with a task, a topic and whether it's a correction | yes: to the API after asking, or through Claude Code with `--prompt-file` |
 | `audit` | shows you a stratified sample to judge by hand | no |
 | `report` | what you use the agent for, how often you correct it per task and per topic (frontend, cli-tool, social-post, ...), how often a fix gets corrected again, with 95% intervals | no |
 | `rules` | groups recurring corrections into CLAUDE.md rules | yes, or no with `--prompt-file` |
@@ -167,22 +177,35 @@ pushback rules                                  # drafts rules into pushback-dat
 pushback rules --existing CLAUDE.md notes/*.md  # also check against the rules you already have
 pushback rules --prompt-file                    # no API key? writes the request to a file instead
 pushback rules --from-response reply.json       # ...and reads Claude's answer back
+pushback rules --runs 3                         # ask 3 times, keep only the rules that keep coming back
 ```
 
 The model groups corrections that share a cause and drafts one CLAUDE.md
 instruction per group. Support is counted from the correction ids it cites,
 not from its own numbers, and a rule needs at least 3 real corrections.
 
+**Stable rules.** A model won't write the same rules twice. With `--runs 3`
+(or three replies passed to `--from-response`), pushback matches rules across
+runs by the corrections they cite, not by their wording, and keeps only the
+ones that come back in at least 2 of 3 runs. Each rule shows how many runs it
+appeared in, and the unstable ones are listed separately instead of vanishing.
+
+On the author's 214 corrections, three independent runs wrote 12, 10 and 7
+rules: 16 distinct rules in total. 5 came back in all three runs, 3 in two,
+and **8 showed up only once**. A single run mixes real rules with one-offs.
+
 When you pass your existing rule files, the output splits in two:
 
-- **Rules you already have that keep getting broken.** These are the useful
-  ones. A rule that exists and still gets corrected isn't working: it's too
-  vague, or the agent doesn't read it at the right moment.
+- **Rules you already have that these corrections hit.** If the corrections
+  came after you wrote the rule, the rule isn't working: it's too vague, or the
+  agent doesn't read it at the right moment. pushback can't know when you wrote
+  a rule, so check the evidence dates.
 - **New rules to consider.** Recurring corrections with no rule behind them.
 
 > [!TIP]
 > No API key? `--prompt-file` writes the whole request to `rules-prompt.md`.
 > Ask Claude Code to answer it, save the JSON reply, then run `--from-response`.
+> `label` has the same option, so the whole pipeline runs without a key.
 
 ## Export your corrections as preference pairs
 
@@ -224,7 +247,8 @@ answer to pair with.
 - `extract`, `audit`, `report` and `export` never leave your machine. `report` prints counts only.
 - `label` and `rules` send each message, plus the end of the agent reply before
   it, to the model provider. If your logs contain client work, check that
-  provider's data policy first. Both ask before they send anything.
+  provider's data policy first. Both ask before they send anything. With
+  `--prompt-file` nothing is sent by pushback; Claude Code reads the files instead.
 - `pushback-data/` holds your raw messages. It's in `.gitignore`. Keep it there.
 - Your transcripts probably contain other people's information. Keep exports
   local unless every conversation in them is yours to share.
@@ -251,6 +275,11 @@ answer to pair with.
 - Claude Code transcripts only, for now. Codex and Cursor logs are not read yet.
 - Task type is judged from the last agent reply and your message, not the whole session.
 - The rubric was tuned on one person's logs.
+- Labels aren't perfectly stable between runs. Two independent runs on the same
+  60 messages agreed on **correction 93%** of the time, on **task 70%**, and on
+  **topic 76%** when the task matched. Disagreements sit at fuzzy edges
+  (writing vs meta, code vs ops), mostly short "ok do that" messages. Treat
+  small differences between topics as noise.
 
 ## Contributing
 

{pushback-0.1.1 → pushback-0.1.3}/README.md

@@ -27,7 +27,7 @@ shows what you use your AI coding agent for, how often you correct it on each
 kind of work (code, writing, media, research, ops, meta), and how often a
 correction needs correcting again. It drafts
 **CLAUDE.md rules** from the corrections you keep repeating, shows which of
-your existing rules keep getting broken, and exports your corrections as
+your existing rules those corrections run into, and exports your corrections as
 **preference pairs** (prompt / chosen / rejected). A hand-audit step tells you
 how far to trust every number.
 
@@ -40,7 +40,7 @@ pip install pushback
 The author's own 1,633 messages to Claude Code:
 
 <p align="center">
-  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. Three existing rules kept getting broken.">
+  <img src="assets/rates.svg" width="100%" alt="Correction rate by task: media 43.0%, writing 24.7%, research 6.9%, code 5.9%, meta 4.7%, ops 4.5%. One in three corrections needed a second correction. One existing rule kept getting broken after it was written.">
 </p>
 
 | task | messages | share of use | corrections | rate | 95% CI | corrected again |
@@ -67,7 +67,7 @@ Most corrected topics (at least 5 messages each):
 
 - Against 91 hand-checked messages the correction labels had **precision 0.97 and recall 0.80**. Topic labels were spot-checked, not audited.
 - **1 in 3 corrections needed a second one.** 70 of 201 fixes got corrected again (35%), 41% on writing and 44% on images.
-- **3 rules the author had already written kept getting broken**, with 6 to 8 corrections each.
+- **1 rule the author had already written kept getting broken**: "images must not look AI-generated" was corrected 3 more times after it was written. Two other matching rules turned out to be written *because of* the corrections, which is why every rule now shows its evidence dates.
 
 > [!NOTE]
 > These rates describe a workflow, not a model. The author's code work runs
@@ -94,10 +94,20 @@ pushback export             # your corrections as prompt/chosen/rejected pairs
 `ant auth login` profile. It defaults to `claude-opus-5` at low effort; change
 it with `--model`. To send through a gateway, set `ANTHROPIC_BASE_URL`.
 
+> [!TIP]
+> **No API key?** Every step works without one:
+> ```bash
+> pushback label --prompt-file       # writes the labelling job as batch files
+> # in Claude Code: "read <path it prints>/INSTRUCTIONS.md and follow it"
+> pushback label --from-responses    # reads the answers back, with the same checks as the API path
+> ```
+> Your messages are still read by Claude, through Claude Code on your
+> subscription. Unanswered or broken batches are listed and stay unlabelled.
+
 | command | what it does | leaves your machine? |
 |---|---|---|
 | `extract` | pulls the messages you typed, plus full agent turns, out of transcripts | no |
-| `label` | tags each message with a task type and whether it's a correction | yes, to the model provider, after asking |
+| `label` | tags each message with a task, a topic and whether it's a correction | yes: to the API after asking, or through Claude Code with `--prompt-file` |
 | `audit` | shows you a stratified sample to judge by hand | no |
 | `report` | what you use the agent for, how often you correct it per task and per topic (frontend, cli-tool, social-post, ...), how often a fix gets corrected again, with 95% intervals | no |
 | `rules` | groups recurring corrections into CLAUDE.md rules | yes, or no with `--prompt-file` |
@@ -140,22 +150,35 @@ pushback rules                                  # drafts rules into pushback-dat
 pushback rules --existing CLAUDE.md notes/*.md  # also check against the rules you already have
 pushback rules --prompt-file                    # no API key? writes the request to a file instead
 pushback rules --from-response reply.json       # ...and reads Claude's answer back
+pushback rules --runs 3                         # ask 3 times, keep only the rules that keep coming back
 ```
 
 The model groups corrections that share a cause and drafts one CLAUDE.md
 instruction per group. Support is counted from the correction ids it cites,
 not from its own numbers, and a rule needs at least 3 real corrections.
 
+**Stable rules.** A model won't write the same rules twice. With `--runs 3`
+(or three replies passed to `--from-response`), pushback matches rules across
+runs by the corrections they cite, not by their wording, and keeps only the
+ones that come back in at least 2 of 3 runs. Each rule shows how many runs it
+appeared in, and the unstable ones are listed separately instead of vanishing.
+
+On the author's 214 corrections, three independent runs wrote 12, 10 and 7
+rules: 16 distinct rules in total. 5 came back in all three runs, 3 in two,
+and **8 showed up only once**. A single run mixes real rules with one-offs.
+
 When you pass your existing rule files, the output splits in two:
 
-- **Rules you already have that keep getting broken.** These are the useful
-  ones. A rule that exists and still gets corrected isn't working: it's too
-  vague, or the agent doesn't read it at the right moment.
+- **Rules you already have that these corrections hit.** If the corrections
+  came after you wrote the rule, the rule isn't working: it's too vague, or the
+  agent doesn't read it at the right moment. pushback can't know when you wrote
+  a rule, so check the evidence dates.
 - **New rules to consider.** Recurring corrections with no rule behind them.
 
 > [!TIP]
 > No API key? `--prompt-file` writes the whole request to `rules-prompt.md`.
 > Ask Claude Code to answer it, save the JSON reply, then run `--from-response`.
+> `label` has the same option, so the whole pipeline runs without a key.
 
 ## Export your corrections as preference pairs
 
@@ -197,7 +220,8 @@ answer to pair with.
 - `extract`, `audit`, `report` and `export` never leave your machine. `report` prints counts only.
 - `label` and `rules` send each message, plus the end of the agent reply before
   it, to the model provider. If your logs contain client work, check that
-  provider's data policy first. Both ask before they send anything.
+  provider's data policy first. Both ask before they send anything. With
+  `--prompt-file` nothing is sent by pushback; Claude Code reads the files instead.
 - `pushback-data/` holds your raw messages. It's in `.gitignore`. Keep it there.
 - Your transcripts probably contain other people's information. Keep exports
   local unless every conversation in them is yours to share.
@@ -224,6 +248,11 @@ answer to pair with.
 - Claude Code transcripts only, for now. Codex and Cursor logs are not read yet.
 - Task type is judged from the last agent reply and your message, not the whole session.
 - The rubric was tuned on one person's logs.
+- Labels aren't perfectly stable between runs. Two independent runs on the same
+  60 messages agreed on **correction 93%** of the time, on **task 70%**, and on
+  **topic 76%** when the task matched. Disagreements sit at fuzzy edges
+  (writing vs meta, code vs ops), mostly short "ok do that" messages. Treat
+  small differences between topics as noise.
 
 ## Contributing
 

{pushback-0.1.1 → pushback-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pushback"
-version = "0.1.1"
+version = "0.1.3"
 description = "How often do you correct your coding agent, and at what kind of work? Measured from your own Claude Code transcripts."
 readme = "README.md"
 license = {text = "MIT"}

pushback-0.1.3/src/pushback/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.3"