agent-coach-cli 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- agent_coach_cli-0.1.0/LICENSE +21 -0
- agent_coach_cli-0.1.0/PKG-INFO +242 -0
- agent_coach_cli-0.1.0/README.md +221 -0
- agent_coach_cli-0.1.0/agent_coach_cli.egg-info/PKG-INFO +242 -0
- agent_coach_cli-0.1.0/agent_coach_cli.egg-info/SOURCES.txt +23 -0
- agent_coach_cli-0.1.0/agent_coach_cli.egg-info/dependency_links.txt +1 -0
- agent_coach_cli-0.1.0/agent_coach_cli.egg-info/entry_points.txt +2 -0
- agent_coach_cli-0.1.0/agent_coach_cli.egg-info/top_level.txt +1 -0
- agent_coach_cli-0.1.0/agentcoach/__init__.py +1 -0
- agent_coach_cli-0.1.0/agentcoach/__main__.py +3 -0
- agent_coach_cli-0.1.0/agentcoach/cli.py +254 -0
- agent_coach_cli-0.1.0/agentcoach/collect.py +336 -0
- agent_coach_cli-0.1.0/agentcoach/heuristics.py +134 -0
- agent_coach_cli-0.1.0/agentcoach/i18n.py +88 -0
- agent_coach_cli-0.1.0/agentcoach/judge.py +328 -0
- agent_coach_cli-0.1.0/agentcoach/paths.py +24 -0
- agent_coach_cli-0.1.0/agentcoach/redact.py +52 -0
- agent_coach_cli-0.1.0/agentcoach/report.py +559 -0
- agent_coach_cli-0.1.0/agentcoach/rubric/default.json +561 -0
- agent_coach_cli-0.1.0/agentcoach/rubric.py +46 -0
- agent_coach_cli-0.1.0/agentcoach/store.py +102 -0
- agent_coach_cli-0.1.0/pyproject.toml +38 -0
- agent_coach_cli-0.1.0/setup.cfg +4 -0
- agent_coach_cli-0.1.0/tests/test_collect.py +176 -0
- agent_coach_cli-0.1.0/tests/test_release.py +44 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 agent-coach contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,242 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: agent-coach-cli
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Grade and coach your Claude Code and Codex prompts from your own conversation history, against Anthropic's and OpenAI's official guides.
|
|
5
|
+
License-Expression: MIT
|
|
6
|
+
Project-URL: Homepage, https://github.com/1017toa/agent-coach
|
|
7
|
+
Project-URL: Issues, https://github.com/1017toa/agent-coach/issues
|
|
8
|
+
Keywords: claude-code,codex,openai-codex,anthropic,claude,prompt-engineering,prompting,coaching,ai-agents,coding-agents,agentic-coding,developer-tools,cli,claude-md,agents-md
|
|
9
|
+
Classifier: Programming Language :: Python :: 3
|
|
10
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
11
|
+
Classifier: Operating System :: OS Independent
|
|
12
|
+
Classifier: Topic :: Software Development
|
|
13
|
+
Classifier: Environment :: Console
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Natural Language :: English
|
|
16
|
+
Classifier: Natural Language :: Korean
|
|
17
|
+
Requires-Python: >=3.10
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
License-File: LICENSE
|
|
20
|
+
Dynamic: license-file
|
|
21
|
+
|
|
22
|
+
<div align="center">
|
|
23
|
+
|
|
24
|
+
<h1>agent-coach</h1>
|
|
25
|
+
|
|
26
|
+
<p><b>Grade and coach your Claude Code and Codex prompts from your own conversation history, against Anthropic's and OpenAI's official guides.</b></p>
|
|
27
|
+
|
|
28
|
+
<p>
|
|
29
|
+
<a href="https://github.com/1017toa/agent-coach/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT license"></a>
|
|
30
|
+
<a href="https://pypi.org/project/agent-coach-cli/"><img src="https://img.shields.io/pypi/v/agent-coach-cli.svg?label=PyPI" alt="PyPI"></a>
|
|
31
|
+
<img src="https://img.shields.io/badge/python-3.10%2B-3776AB.svg?logo=python&logoColor=white" alt="Python 3.10+">
|
|
32
|
+
<img src="https://img.shields.io/badge/dependencies-0-brightgreen.svg" alt="Zero dependencies">
|
|
33
|
+
<img src="https://img.shields.io/badge/works%20with-Claude%20Code%20%7C%20Codex-8A63D2.svg" alt="Works with Claude Code and Codex">
|
|
34
|
+
<img src="https://img.shields.io/badge/runs-100%25%20local-555.svg" alt="Runs locally">
|
|
35
|
+
</p>
|
|
36
|
+
|
|
37
|
+
<p>
|
|
38
|
+
<a href="#quick-start">Quick start</a> ·
|
|
39
|
+
<a href="#what-it-grades">What it grades</a> ·
|
|
40
|
+
<a href="#missed-responses">Missed responses</a> ·
|
|
41
|
+
<a href="#faq">FAQ</a> ·
|
|
42
|
+
<a href="https://github.com/1017toa/agent-coach/blob/main/README.ko.md">한국어</a>
|
|
43
|
+
</p>
|
|
44
|
+
|
|
45
|
+
<picture>
|
|
46
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://github.com/1017toa/agent-coach/raw/main/docs/images/overview-dark.png">
|
|
47
|
+
<img alt="agent-coach report: overall score, the three habits to work on, a radar chart of 11 prompting habits and a weakest-first list" src="https://github.com/1017toa/agent-coach/raw/main/docs/images/overview-light.png" width="900">
|
|
48
|
+
</picture>
|
|
49
|
+
|
|
50
|
+
<sub>Demo report built from made-up conversations (<code>python scripts/demo_report.py</code>).</sub>
|
|
51
|
+
|
|
52
|
+
</div>
|
|
53
|
+
|
|
54
|
+
**agent-coach** is a command-line tool that reads the transcripts Claude Code and OpenAI Codex already save on your machine (`~/.claude/projects`, `~/.codex/sessions`) and scores how you direct the agent on 11 habits taken from the official Anthropic and OpenAI guides and academies. It rewrites prompts you actually typed into better versions, finds the turns where you didn't get what you wanted, and tracks your scores run after run so you can see whether you're improving.
|
|
55
|
+
|
|
56
|
+
## Quick start
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
pipx install agent-coach-cli # or: uv tool install agent-coach-cli · pip install agent-coach-cli
|
|
60
|
+
agent-coach run
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
The package on PyPI is `agent-coach-cli`; the command it installs is `agent-coach`. `run` grades your recent conversations through the `claude` or `codex` CLI you're already signed in to, saves the result and opens an HTML report. No API key, no server, no dependencies. Want to look before grading anything? `agent-coach scan` lists what would be graded, without calling a model.
|
|
64
|
+
|
|
65
|
+
### Or use it inside Claude Code or Codex
|
|
66
|
+
|
|
67
|
+
Install the plugin and let the agent run it and coach you in the chat, with nothing else to install.
|
|
68
|
+
|
|
69
|
+
```text
|
|
70
|
+
# Claude Code
|
|
71
|
+
/plugin install agent-coach --marketplace 1017toa/agent-coach
|
|
72
|
+
/agent-coach:coach
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
# Codex
|
|
77
|
+
codex plugin marketplace add 1017toa/agent-coach
|
|
78
|
+
codex plugin add agent-coach@agent-coach
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Then ask Codex to "grade my prompting with agent-coach". The skill previews what will be graded, runs the grading, and walks you through your weakest habits, rewrites and rules worth adding to `CLAUDE.md` / `AGENTS.md`.
|
|
82
|
+
|
|
83
|
+
## Highlights
|
|
84
|
+
|
|
85
|
+
- **Grounded in the official guides.** Every habit cites the guide section it comes from: Claude Code best practices, Claude Academy, Codex best practices, OpenAI Academy ([rubric](https://github.com/1017toa/agent-coach/blob/main/agentcoach/rubric/default.json)).
|
|
86
|
+
- **Rewrites of your real prompts.** "You wrote this; this would have worked better, because…", not generic tips.
|
|
87
|
+
- **Short prompts aren't penalized.** In a long conversation, "ok do it" is often all you need. Coaching targets turns that actually missed, and only when the cause was a gap in your prompt.
|
|
88
|
+
- **Finds rules worth writing down.** Gaps that recur across conversations become one-line rules, with where to put them: `CLAUDE.md` / `AGENTS.md`, memory, a skill, or a hook.
|
|
89
|
+
- **Tracks progress.** Each run is saved under `~/.agent-coach/history/`, so your trend survives after the tools delete old transcripts.
|
|
90
|
+
- **Claude Code and Codex on one scale**, in one report.
|
|
91
|
+
- **Private by default.** Secrets and contact details are masked before anything reaches the judge, and results stay on your machine.
|
|
92
|
+
- **English and Korean.** Report, CLI and coaching in either language (`--lang en|ko`).
|
|
93
|
+
|
|
94
|
+
## Supported tools
|
|
95
|
+
|
|
96
|
+
| Tool | Transcripts read from | Notes |
|
|
97
|
+
|---|---|---|
|
|
98
|
+
| Claude Code (CLI, desktop, IDE) | `~/.claude/projects/**/*.jsonl` | Scripted `claude -p` runs and app-generated turns are skipped |
|
|
99
|
+
| OpenAI Codex (CLI, IDE) | `~/.codex/sessions/**/rollout-*.jsonl` | `codex exec` and agent-spawned sessions are skipped |
|
|
100
|
+
|
|
101
|
+
Judge: `claude` CLI, `codex` CLI, or offline keyword scoring (`--judge heuristic`).
|
|
102
|
+
|
|
103
|
+
## How it works
|
|
104
|
+
|
|
105
|
+
```
|
|
106
|
+
~/.claude, ~/.codex transcripts
|
|
107
|
+
│
|
|
108
|
+
● collect your own typed turns only (automation, slash commands, app messages filtered out)
|
|
109
|
+
│
|
|
110
|
+
● redact keys, emails, phone / ID / card numbers, addresses, names you list
|
|
111
|
+
│
|
|
112
|
+
● judge one call per conversation via your claude / codex CLI, rubric v2 (11 habits)
|
|
113
|
+
│
|
|
114
|
+
● save ~/.agent-coach/history/run-*.json
|
|
115
|
+
│
|
|
116
|
+
✓ report one self-contained HTML file: overview · coaching · missed responses · conversations · progress
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
## What it grades
|
|
120
|
+
|
|
121
|
+
Eleven habits, each scored 0 / 1 / 2 only in the conversations where it mattered, averaged to 0–100.
|
|
122
|
+
|
|
123
|
+
| Habit | What the guides recommend |
|
|
124
|
+
|---|---|
|
|
125
|
+
| Specific context | State the goal and point at the file, screen and symptom from the start |
|
|
126
|
+
| Done criteria and verification | Give a pass/fail check (tests, build, screenshot) and let the agent run it |
|
|
127
|
+
| Constraints and preferences | Say what must not change and which approach you prefer |
|
|
128
|
+
| Hand over the real material | Attach the @file, screenshot or full error instead of describing it |
|
|
129
|
+
| Scope and output shape | Minimal or full version; format and length of the result |
|
|
130
|
+
| Plan or interview first | For big or fuzzy work, plan or get interviewed before building |
|
|
131
|
+
| Stop points and working style | How far to go alone, where to stop and ask, how to report |
|
|
132
|
+
| Ask for evidence, review the result | Commands, output, diff; a fresh session to review |
|
|
133
|
+
| Course correction | After two failed corrections, start fresh with a better prompt |
|
|
134
|
+
| Move repeated rules into files | Put recurring rules in CLAUDE.md, AGENTS.md or a skill |
|
|
135
|
+
| One outcome per conversation | New work in a new conversation; tidy long ones |
|
|
136
|
+
|
|
137
|
+
<img alt="Coaching tab: a habit's score, rewrites of the user's own prompts side by side with better versions, and the official guidance behind it" src="https://github.com/1017toa/agent-coach/raw/main/docs/images/coaching.png" width="900">
|
|
138
|
+
|
|
139
|
+
## Missed responses
|
|
140
|
+
|
|
141
|
+
Context builds up over a conversation, so a short instruction can be exactly right. agent-coach never marks a prompt down for being short. It looks for **turns that didn't get what you wanted**, judged by what happened next:
|
|
142
|
+
|
|
143
|
+
- **Signals**: you corrected the result, repeated an earlier instruction, asked to redo or undo, reported damage, showed frustration, or the agent had to ask back.
|
|
144
|
+
- **Causes**: *gap in your prompt* (something only you knew was missing and context didn't cover it) / *agent error* (context was enough) / *changed your mind* / *external limit*.
|
|
145
|
+
- **Coaching**: only prompt gaps get a rewrite, phrased as "the one missing piece". Gaps that recur become rule candidates.
|
|
146
|
+
|
|
147
|
+
<img alt="Missed responses tab: turns split by cause, the prompt, what happened, the one missing piece and a recurring-gap rule" src="https://github.com/1017toa/agent-coach/raw/main/docs/images/missed-responses.png" width="900">
|
|
148
|
+
|
|
149
|
+
Causes are classified only with an LLM judge; keyword scoring finds the signals and leaves the cause unclassified.
|
|
150
|
+
|
|
151
|
+
## Usage
|
|
152
|
+
|
|
153
|
+
```bash
|
|
154
|
+
agent-coach scan # preview which conversations would be graded (no model calls)
|
|
155
|
+
agent-coach run # grade → save the run → open the report
|
|
156
|
+
agent-coach summary # latest run as text: weakest habits, rewrites, rules to add
|
|
157
|
+
agent-coach history # score table per run
|
|
158
|
+
agent-coach report # rebuild the report from saved runs
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
| Option | Meaning |
|
|
162
|
+
|---|---|
|
|
163
|
+
| `--days 30` / `--all` | Time window (default: since the previous run; 14 days on the first run) |
|
|
164
|
+
| `--judge claude\|codex\|heuristic` | Which judge (default: auto) |
|
|
165
|
+
| `--model sonnet` | Judge model |
|
|
166
|
+
| `--max-sessions 20` | Max conversations to grade; each one is one model call |
|
|
167
|
+
| `--source claude` / `--source codex` | Only one tool |
|
|
168
|
+
| `--rubric my-rubric.json` | Your own rubric, for a course or a team |
|
|
169
|
+
| `--lang en\|ko` | Language of the report and coaching (default: OS locale, or `AGENT_COACH_LANG`) |
|
|
170
|
+
|
|
171
|
+
From a checkout without installing: `python -m agentcoach run`. Latest development version: `pip install git+https://github.com/1017toa/agent-coach.git`.
|
|
172
|
+
|
|
173
|
+
**Grade on a schedule.** Run `agent-coach run` every week or two; each run grades only what's new and adds to the trend. Claude Code deletes transcripts older than 30 days by default, so keep the interval shorter than that.
|
|
174
|
+
|
|
175
|
+
## How it compares
|
|
176
|
+
|
|
177
|
+
| | agent-coach | Token and cost trackers (e.g. ccusage) | Prompt eval frameworks (e.g. promptfoo) |
|
|
178
|
+
|---|---|---|---|
|
|
179
|
+
| Looks at | How **you** direct the agent | Tokens and spend | Your app's prompts and model outputs |
|
|
180
|
+
| Input | Your Claude Code / Codex history | Your Claude Code / Codex history | Test cases you write |
|
|
181
|
+
| Output | Scores per habit, rewrites, missed turns, rules to add | Usage tables | Pass/fail matrices |
|
|
182
|
+
| For | Developers, teams and courses learning to work with coding agents | Budgeting | Building LLM apps |
|
|
183
|
+
|
|
184
|
+
They answer different questions and work fine side by side.
|
|
185
|
+
|
|
186
|
+
## FAQ
|
|
187
|
+
|
|
188
|
+
### How do I improve my Claude Code or Codex prompts?
|
|
189
|
+
Run `agent-coach run` and open the **Coaching** tab. It starts with your three weakest habits and shows prompts you actually wrote next to rewritten versions, with the official guide section behind each one. Re-run in a week or two and check the **Progress** tab.
|
|
190
|
+
|
|
191
|
+
### Does it send my conversations anywhere?
|
|
192
|
+
Raw transcripts never leave your machine. With an LLM judge, the masked prompts of each graded conversation and one-line summaries of what the agent did are sent to the Claude or Codex account you're signed in to, the same place your conversations already went. `--judge heuristic` grades fully offline.
|
|
193
|
+
|
|
194
|
+
### Do I need an API key?
|
|
195
|
+
No. Grading runs through the `claude` or `codex` CLI you're already logged in to, using your existing plan. Without either, it falls back to offline keyword scoring.
|
|
196
|
+
|
|
197
|
+
### How much does a run cost?
|
|
198
|
+
One judge call per conversation, on your existing plan. Use `--max-sessions` to cap it and `agent-coach scan` to preview the count first.
|
|
199
|
+
|
|
200
|
+
### Why wasn't my short prompt marked down?
|
|
201
|
+
Because short prompts are often right. "Yes, do it" after a long discussion has all the context it needs. agent-coach only coaches a turn when what happened next shows it missed, and only rewrites it when the missing piece was something only you knew.
|
|
202
|
+
|
|
203
|
+
### Can I use my own rubric for a course or a team?
|
|
204
|
+
Yes. Copy [`agentcoach/rubric/default.json`](https://github.com/1017toa/agent-coach/blob/main/agentcoach/rubric/default.json), edit the habits and examples, and pass `--rubric your-rubric.json`. A rubric without English fields is shown as written.
|
|
205
|
+
|
|
206
|
+
### Where are results stored?
|
|
207
|
+
`~/.agent-coach/history/run-*.json` (one file per run) and `~/.agent-coach/report.html`. Set `AGENT_COACH_HOME` to move them.
|
|
208
|
+
|
|
209
|
+
### How do I hide my name or company?
|
|
210
|
+
Names can't be detected reliably, so list yours (name, romanized name, company, ...) one per line in `~/.agent-coach/redact.txt`. Keys, tokens, emails, phone, national ID and card numbers, and Korean road addresses are masked automatically.
|
|
211
|
+
|
|
212
|
+
### Does it support Cursor, Gemini CLI or Aider?
|
|
213
|
+
Not yet; Claude Code and Codex only. Support for more tools is on the roadmap, and parsers are small (see [`agentcoach/collect.py`](https://github.com/1017toa/agent-coach/blob/main/agentcoach/collect.py)).
|
|
214
|
+
|
|
215
|
+
## Sources
|
|
216
|
+
|
|
217
|
+
The default rubric (v2) is built from:
|
|
218
|
+
|
|
219
|
+
- **Anthropic**: [Best practices for Claude Code](https://code.claude.com/docs/en/best-practices), [Getting the most out of Opus 5.5](https://claude.dev/blog/getting-the-most-out-of-opus-5-5/)
|
|
220
|
+
- **Claude Academy**: [AI Fluency: Framework & Foundations](https://academy.claude.com/courses/ai-fluency-framework-foundations), [Claude Code 101](https://academy.claude.com/courses/claude-code-101), [Claude Code in Action](https://academy.claude.com/courses/claude-code-in-action), [Claude 101](https://academy.claude.com/courses/claude-101)
|
|
221
|
+
- **OpenAI**: [Codex Best practices](https://learn.chatgpt.com/guides/best-practices), [Codex Prompting](https://learn.chatgpt.com/docs/prompting)
|
|
222
|
+
- **OpenAI Academy**: [Get Started with Codex](https://academy.openai.com/public/courses/get-started-with-codex-jkhsq), [Agents and Workflows](https://academy.openai.com/public/courses/agents-and-workflows-y0qoc), [Extend Codex Workflows](https://academy.openai.com/public/courses/extend-codex-workflows-4e61p) (course bodies need sign-in; only the public overviews are used)
|
|
223
|
+
|
|
224
|
+
When the rubric changes, earlier runs have no score for new habits and the report says the versions differ.
|
|
225
|
+
|
|
226
|
+
## Roadmap
|
|
227
|
+
|
|
228
|
+
- Anonymized aggregate export for courses and teams
|
|
229
|
+
- Example custom rubrics
|
|
230
|
+
- Cursor and Gemini CLI transcripts
|
|
231
|
+
|
|
232
|
+
## Contributing
|
|
233
|
+
|
|
234
|
+
Issues and pull requests are welcome; see [CONTRIBUTING.md](https://github.com/1017toa/agent-coach/blob/main/CONTRIBUTING.md). Run the tests with:
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
python -m unittest discover -s tests
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
## License
|
|
241
|
+
|
|
242
|
+
[MIT](https://github.com/1017toa/agent-coach/blob/main/LICENSE)
|
|
@@ -0,0 +1,221 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
|
|
3
|
+
<h1>agent-coach</h1>
|
|
4
|
+
|
|
5
|
+
<p><b>Grade and coach your Claude Code and Codex prompts from your own conversation history, against Anthropic's and OpenAI's official guides.</b></p>
|
|
6
|
+
|
|
7
|
+
<p>
|
|
8
|
+
<a href="https://github.com/1017toa/agent-coach/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT license"></a>
|
|
9
|
+
<a href="https://pypi.org/project/agent-coach-cli/"><img src="https://img.shields.io/pypi/v/agent-coach-cli.svg?label=PyPI" alt="PyPI"></a>
|
|
10
|
+
<img src="https://img.shields.io/badge/python-3.10%2B-3776AB.svg?logo=python&logoColor=white" alt="Python 3.10+">
|
|
11
|
+
<img src="https://img.shields.io/badge/dependencies-0-brightgreen.svg" alt="Zero dependencies">
|
|
12
|
+
<img src="https://img.shields.io/badge/works%20with-Claude%20Code%20%7C%20Codex-8A63D2.svg" alt="Works with Claude Code and Codex">
|
|
13
|
+
<img src="https://img.shields.io/badge/runs-100%25%20local-555.svg" alt="Runs locally">
|
|
14
|
+
</p>
|
|
15
|
+
|
|
16
|
+
<p>
|
|
17
|
+
<a href="#quick-start">Quick start</a> ·
|
|
18
|
+
<a href="#what-it-grades">What it grades</a> ·
|
|
19
|
+
<a href="#missed-responses">Missed responses</a> ·
|
|
20
|
+
<a href="#faq">FAQ</a> ·
|
|
21
|
+
<a href="https://github.com/1017toa/agent-coach/blob/main/README.ko.md">한국어</a>
|
|
22
|
+
</p>
|
|
23
|
+
|
|
24
|
+
<picture>
|
|
25
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://github.com/1017toa/agent-coach/raw/main/docs/images/overview-dark.png">
|
|
26
|
+
<img alt="agent-coach report: overall score, the three habits to work on, a radar chart of 11 prompting habits and a weakest-first list" src="https://github.com/1017toa/agent-coach/raw/main/docs/images/overview-light.png" width="900">
|
|
27
|
+
</picture>
|
|
28
|
+
|
|
29
|
+
<sub>Demo report built from made-up conversations (<code>python scripts/demo_report.py</code>).</sub>
|
|
30
|
+
|
|
31
|
+
</div>
|
|
32
|
+
|
|
33
|
+
**agent-coach** is a command-line tool that reads the transcripts Claude Code and OpenAI Codex already save on your machine (`~/.claude/projects`, `~/.codex/sessions`) and scores how you direct the agent on 11 habits taken from the official Anthropic and OpenAI guides and academies. It rewrites prompts you actually typed into better versions, finds the turns where you didn't get what you wanted, and tracks your scores run after run so you can see whether you're improving.
|
|
34
|
+
|
|
35
|
+
## Quick start
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
pipx install agent-coach-cli # or: uv tool install agent-coach-cli · pip install agent-coach-cli
|
|
39
|
+
agent-coach run
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The package on PyPI is `agent-coach-cli`; the command it installs is `agent-coach`. `run` grades your recent conversations through the `claude` or `codex` CLI you're already signed in to, saves the result and opens an HTML report. No API key, no server, no dependencies. Want to look before grading anything? `agent-coach scan` lists what would be graded, without calling a model.
|
|
43
|
+
|
|
44
|
+
### Or use it inside Claude Code or Codex
|
|
45
|
+
|
|
46
|
+
Install the plugin and let the agent run it and coach you in the chat, with nothing else to install.
|
|
47
|
+
|
|
48
|
+
```text
|
|
49
|
+
# Claude Code
|
|
50
|
+
/plugin install agent-coach --marketplace 1017toa/agent-coach
|
|
51
|
+
/agent-coach:coach
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
# Codex
|
|
56
|
+
codex plugin marketplace add 1017toa/agent-coach
|
|
57
|
+
codex plugin add agent-coach@agent-coach
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Then ask Codex to "grade my prompting with agent-coach". The skill previews what will be graded, runs the grading, and walks you through your weakest habits, rewrites and rules worth adding to `CLAUDE.md` / `AGENTS.md`.
|
|
61
|
+
|
|
62
|
+
## Highlights
|
|
63
|
+
|
|
64
|
+
- **Grounded in the official guides.** Every habit cites the guide section it comes from: Claude Code best practices, Claude Academy, Codex best practices, OpenAI Academy ([rubric](https://github.com/1017toa/agent-coach/blob/main/agentcoach/rubric/default.json)).
|
|
65
|
+
- **Rewrites of your real prompts.** "You wrote this; this would have worked better, because…", not generic tips.
|
|
66
|
+
- **Short prompts aren't penalized.** In a long conversation, "ok do it" is often all you need. Coaching targets turns that actually missed, and only when the cause was a gap in your prompt.
|
|
67
|
+
- **Finds rules worth writing down.** Gaps that recur across conversations become one-line rules, with where to put them: `CLAUDE.md` / `AGENTS.md`, memory, a skill, or a hook.
|
|
68
|
+
- **Tracks progress.** Each run is saved under `~/.agent-coach/history/`, so your trend survives after the tools delete old transcripts.
|
|
69
|
+
- **Claude Code and Codex on one scale**, in one report.
|
|
70
|
+
- **Private by default.** Secrets and contact details are masked before anything reaches the judge, and results stay on your machine.
|
|
71
|
+
- **English and Korean.** Report, CLI and coaching in either language (`--lang en|ko`).
|
|
72
|
+
|
|
73
|
+
## Supported tools
|
|
74
|
+
|
|
75
|
+
| Tool | Transcripts read from | Notes |
|
|
76
|
+
|---|---|---|
|
|
77
|
+
| Claude Code (CLI, desktop, IDE) | `~/.claude/projects/**/*.jsonl` | Scripted `claude -p` runs and app-generated turns are skipped |
|
|
78
|
+
| OpenAI Codex (CLI, IDE) | `~/.codex/sessions/**/rollout-*.jsonl` | `codex exec` and agent-spawned sessions are skipped |
|
|
79
|
+
|
|
80
|
+
Judge: `claude` CLI, `codex` CLI, or offline keyword scoring (`--judge heuristic`).
|
|
81
|
+
|
|
82
|
+
## How it works
|
|
83
|
+
|
|
84
|
+
```
|
|
85
|
+
~/.claude, ~/.codex transcripts
|
|
86
|
+
│
|
|
87
|
+
● collect your own typed turns only (automation, slash commands, app messages filtered out)
|
|
88
|
+
│
|
|
89
|
+
● redact keys, emails, phone / ID / card numbers, addresses, names you list
|
|
90
|
+
│
|
|
91
|
+
● judge one call per conversation via your claude / codex CLI, rubric v2 (11 habits)
|
|
92
|
+
│
|
|
93
|
+
● save ~/.agent-coach/history/run-*.json
|
|
94
|
+
│
|
|
95
|
+
✓ report one self-contained HTML file: overview · coaching · missed responses · conversations · progress
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
## What it grades
|
|
99
|
+
|
|
100
|
+
Eleven habits, each scored 0 / 1 / 2 only in the conversations where it mattered, averaged to 0–100.
|
|
101
|
+
|
|
102
|
+
| Habit | What the guides recommend |
|
|
103
|
+
|---|---|
|
|
104
|
+
| Specific context | State the goal and point at the file, screen and symptom from the start |
|
|
105
|
+
| Done criteria and verification | Give a pass/fail check (tests, build, screenshot) and let the agent run it |
|
|
106
|
+
| Constraints and preferences | Say what must not change and which approach you prefer |
|
|
107
|
+
| Hand over the real material | Attach the @file, screenshot or full error instead of describing it |
|
|
108
|
+
| Scope and output shape | Minimal or full version; format and length of the result |
|
|
109
|
+
| Plan or interview first | For big or fuzzy work, plan or get interviewed before building |
|
|
110
|
+
| Stop points and working style | How far to go alone, where to stop and ask, how to report |
|
|
111
|
+
| Ask for evidence, review the result | Commands, output, diff; a fresh session to review |
|
|
112
|
+
| Course correction | After two failed corrections, start fresh with a better prompt |
|
|
113
|
+
| Move repeated rules into files | Put recurring rules in CLAUDE.md, AGENTS.md or a skill |
|
|
114
|
+
| One outcome per conversation | New work in a new conversation; tidy long ones |
|
|
115
|
+
|
|
116
|
+
<img alt="Coaching tab: a habit's score, rewrites of the user's own prompts side by side with better versions, and the official guidance behind it" src="https://github.com/1017toa/agent-coach/raw/main/docs/images/coaching.png" width="900">
|
|
117
|
+
|
|
118
|
+
## Missed responses
|
|
119
|
+
|
|
120
|
+
Context builds up over a conversation, so a short instruction can be exactly right. agent-coach never marks a prompt down for being short. It looks for **turns that didn't get what you wanted**, judged by what happened next:
|
|
121
|
+
|
|
122
|
+
- **Signals**: you corrected the result, repeated an earlier instruction, asked to redo or undo, reported damage, showed frustration, or the agent had to ask back.
|
|
123
|
+
- **Causes**: *gap in your prompt* (something only you knew was missing and context didn't cover it) / *agent error* (context was enough) / *changed your mind* / *external limit*.
|
|
124
|
+
- **Coaching**: only prompt gaps get a rewrite, phrased as "the one missing piece". Gaps that recur become rule candidates.
|
|
125
|
+
|
|
126
|
+
<img alt="Missed responses tab: turns split by cause, the prompt, what happened, the one missing piece and a recurring-gap rule" src="https://github.com/1017toa/agent-coach/raw/main/docs/images/missed-responses.png" width="900">
|
|
127
|
+
|
|
128
|
+
Causes are classified only with an LLM judge; keyword scoring finds the signals and leaves the cause unclassified.
|
|
129
|
+
|
|
130
|
+
## Usage
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
agent-coach scan # preview which conversations would be graded (no model calls)
|
|
134
|
+
agent-coach run # grade → save the run → open the report
|
|
135
|
+
agent-coach summary # latest run as text: weakest habits, rewrites, rules to add
|
|
136
|
+
agent-coach history # score table per run
|
|
137
|
+
agent-coach report # rebuild the report from saved runs
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
| Option | Meaning |
|
|
141
|
+
|---|---|
|
|
142
|
+
| `--days 30` / `--all` | Time window (default: since the previous run; 14 days on the first run) |
|
|
143
|
+
| `--judge claude\|codex\|heuristic` | Which judge (default: auto) |
|
|
144
|
+
| `--model sonnet` | Judge model |
|
|
145
|
+
| `--max-sessions 20` | Max conversations to grade; each one is one model call |
|
|
146
|
+
| `--source claude` / `--source codex` | Only one tool |
|
|
147
|
+
| `--rubric my-rubric.json` | Your own rubric, for a course or a team |
|
|
148
|
+
| `--lang en\|ko` | Language of the report and coaching (default: OS locale, or `AGENT_COACH_LANG`) |
|
|
149
|
+
|
|
150
|
+
From a checkout without installing: `python -m agentcoach run`. Latest development version: `pip install git+https://github.com/1017toa/agent-coach.git`.
|
|
151
|
+
|
|
152
|
+
**Grade on a schedule.** Run `agent-coach run` every week or two; each run grades only what's new and adds to the trend. Claude Code deletes transcripts older than 30 days by default, so keep the interval shorter than that.
|
|
153
|
+
|
|
154
|
+
## How it compares
|
|
155
|
+
|
|
156
|
+
| | agent-coach | Token and cost trackers (e.g. ccusage) | Prompt eval frameworks (e.g. promptfoo) |
|
|
157
|
+
|---|---|---|---|
|
|
158
|
+
| Looks at | How **you** direct the agent | Tokens and spend | Your app's prompts and model outputs |
|
|
159
|
+
| Input | Your Claude Code / Codex history | Your Claude Code / Codex history | Test cases you write |
|
|
160
|
+
| Output | Scores per habit, rewrites, missed turns, rules to add | Usage tables | Pass/fail matrices |
|
|
161
|
+
| For | Developers, teams and courses learning to work with coding agents | Budgeting | Building LLM apps |
|
|
162
|
+
|
|
163
|
+
They answer different questions and work fine side by side.
|
|
164
|
+
|
|
165
|
+
## FAQ
|
|
166
|
+
|
|
167
|
+
### How do I improve my Claude Code or Codex prompts?
|
|
168
|
+
Run `agent-coach run` and open the **Coaching** tab. It starts with your three weakest habits and shows prompts you actually wrote next to rewritten versions, with the official guide section behind each one. Re-run in a week or two and check the **Progress** tab.
|
|
169
|
+
|
|
170
|
+
### Does it send my conversations anywhere?
|
|
171
|
+
Raw transcripts never leave your machine. With an LLM judge, the masked prompts of each graded conversation and one-line summaries of what the agent did are sent to the Claude or Codex account you're signed in to, the same place your conversations already went. `--judge heuristic` grades fully offline.
|
|
172
|
+
|
|
173
|
+
### Do I need an API key?
|
|
174
|
+
No. Grading runs through the `claude` or `codex` CLI you're already logged in to, using your existing plan. Without either, it falls back to offline keyword scoring.
|
|
175
|
+
|
|
176
|
+
### How much does a run cost?
|
|
177
|
+
One judge call per conversation, on your existing plan. Use `--max-sessions` to cap it and `agent-coach scan` to preview the count first.
|
|
178
|
+
|
|
179
|
+
### Why wasn't my short prompt marked down?
|
|
180
|
+
Because short prompts are often right. "Yes, do it" after a long discussion has all the context it needs. agent-coach only coaches a turn when what happened next shows it missed, and only rewrites it when the missing piece was something only you knew.
|
|
181
|
+
|
|
182
|
+
### Can I use my own rubric for a course or a team?
|
|
183
|
+
Yes. Copy [`agentcoach/rubric/default.json`](https://github.com/1017toa/agent-coach/blob/main/agentcoach/rubric/default.json), edit the habits and examples, and pass `--rubric your-rubric.json`. A rubric without English fields is shown as written.
|
|
184
|
+
|
|
185
|
+
### Where are results stored?
|
|
186
|
+
`~/.agent-coach/history/run-*.json` (one file per run) and `~/.agent-coach/report.html`. Set `AGENT_COACH_HOME` to move them.
|
|
187
|
+
|
|
188
|
+
### How do I hide my name or company?
|
|
189
|
+
Names can't be detected reliably, so list yours (name, romanized name, company, ...) one per line in `~/.agent-coach/redact.txt`. Keys, tokens, emails, phone, national ID and card numbers, and Korean road addresses are masked automatically.
|
|
190
|
+
|
|
191
|
+
### Does it support Cursor, Gemini CLI or Aider?
|
|
192
|
+
Not yet; Claude Code and Codex only. Support for more tools is on the roadmap, and parsers are small (see [`agentcoach/collect.py`](https://github.com/1017toa/agent-coach/blob/main/agentcoach/collect.py)).
|
|
193
|
+
|
|
194
|
+
## Sources
|
|
195
|
+
|
|
196
|
+
The default rubric (v2) is built from:
|
|
197
|
+
|
|
198
|
+
- **Anthropic**: [Best practices for Claude Code](https://code.claude.com/docs/en/best-practices), [Getting the most out of Opus 5.5](https://claude.dev/blog/getting-the-most-out-of-opus-5-5/)
|
|
199
|
+
- **Claude Academy**: [AI Fluency: Framework & Foundations](https://academy.claude.com/courses/ai-fluency-framework-foundations), [Claude Code 101](https://academy.claude.com/courses/claude-code-101), [Claude Code in Action](https://academy.claude.com/courses/claude-code-in-action), [Claude 101](https://academy.claude.com/courses/claude-101)
|
|
200
|
+
- **OpenAI**: [Codex Best practices](https://learn.chatgpt.com/guides/best-practices), [Codex Prompting](https://learn.chatgpt.com/docs/prompting)
|
|
201
|
+
- **OpenAI Academy**: [Get Started with Codex](https://academy.openai.com/public/courses/get-started-with-codex-jkhsq), [Agents and Workflows](https://academy.openai.com/public/courses/agents-and-workflows-y0qoc), [Extend Codex Workflows](https://academy.openai.com/public/courses/extend-codex-workflows-4e61p) (course bodies need sign-in; only the public overviews are used)
|
|
202
|
+
|
|
203
|
+
When the rubric changes, earlier runs have no score for new habits and the report says the versions differ.
|
|
204
|
+
|
|
205
|
+
## Roadmap
|
|
206
|
+
|
|
207
|
+
- Anonymized aggregate export for courses and teams
|
|
208
|
+
- Example custom rubrics
|
|
209
|
+
- Cursor and Gemini CLI transcripts
|
|
210
|
+
|
|
211
|
+
## Contributing
|
|
212
|
+
|
|
213
|
+
Issues and pull requests are welcome; see [CONTRIBUTING.md](https://github.com/1017toa/agent-coach/blob/main/CONTRIBUTING.md). Run the tests with:
|
|
214
|
+
|
|
215
|
+
```bash
|
|
216
|
+
python -m unittest discover -s tests
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
## License
|
|
220
|
+
|
|
221
|
+
[MIT](https://github.com/1017toa/agent-coach/blob/main/LICENSE)
|