agentfluent 0.1.0__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. agentfluent-0.2.0/PKG-INFO +403 -0
  2. agentfluent-0.2.0/README.md +374 -0
  3. {agentfluent-0.1.0 → agentfluent-0.2.0}/pyproject.toml +1 -2
  4. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/agents/extractor.py +16 -30
  5. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/analytics/pricing.py +23 -7
  6. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/analytics/tokens.py +6 -1
  7. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/formatters/table.py +11 -3
  8. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/core/parser.py +27 -23
  9. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/core/session.py +24 -21
  10. agentfluent-0.1.0/PKG-INFO +0 -52
  11. agentfluent-0.1.0/README.md +0 -22
  12. {agentfluent-0.1.0 → agentfluent-0.2.0}/LICENSE +0 -0
  13. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/__init__.py +0 -0
  14. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/agents/__init__.py +0 -0
  15. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/agents/models.py +0 -0
  16. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/analytics/__init__.py +0 -0
  17. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/analytics/agent_metrics.py +0 -0
  18. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/analytics/pipeline.py +0 -0
  19. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/analytics/tools.py +0 -0
  20. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/__init__.py +0 -0
  21. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/commands/__init__.py +0 -0
  22. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/commands/analyze.py +0 -0
  23. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/commands/config_check.py +0 -0
  24. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/commands/list_cmd.py +0 -0
  25. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/exit_codes.py +0 -0
  26. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/formatters/__init__.py +0 -0
  27. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/formatters/helpers.py +0 -0
  28. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/formatters/json_output.py +0 -0
  29. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/cli/main.py +0 -0
  30. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/config/__init__.py +0 -0
  31. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/config/models.py +0 -0
  32. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/config/scanner.py +0 -0
  33. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/config/scoring.py +0 -0
  34. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/core/__init__.py +0 -0
  35. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/core/discovery.py +0 -0
  36. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/diagnostics/__init__.py +0 -0
  37. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/diagnostics/correlator.py +0 -0
  38. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/diagnostics/models.py +0 -0
  39. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/diagnostics/signals.py +0 -0
  40. {agentfluent-0.1.0 → agentfluent-0.2.0}/src/agentfluent/py.typed +0 -0
@@ -0,0 +1,403 @@
1
+ Metadata-Version: 2.4
2
+ Name: agentfluent
3
+ Version: 0.2.0
4
+ Summary: Local-first agent analytics with prompt diagnostics
5
+ Keywords: claude,agent,analytics,cli,llm,diagnostics
6
+ Author: Fred Pearce
7
+ Author-email: Fred Pearce <fpearce@gmail.com>
8
+ License-Expression: MIT
9
+ License-File: LICENSE
10
+ Classifier: Development Status :: 3 - Alpha
11
+ Classifier: Environment :: Console
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Operating System :: OS Independent
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
17
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
18
+ Classifier: Typing :: Typed
19
+ Requires-Dist: typer>=0.15
20
+ Requires-Dist: rich>=14.0
21
+ Requires-Dist: pydantic>=2.0
22
+ Requires-Dist: pyyaml>=6.0
23
+ Requires-Python: >=3.12
24
+ Project-URL: Homepage, https://github.com/frederick-douglas-pearce/agentfluent
25
+ Project-URL: Repository, https://github.com/frederick-douglas-pearce/agentfluent
26
+ Project-URL: Issues, https://github.com/frederick-douglas-pearce/agentfluent/issues
27
+ Project-URL: Changelog, https://github.com/frederick-douglas-pearce/agentfluent/blob/main/CHANGELOG.md
28
+ Description-Content-Type: text/markdown
29
+
30
+ # AgentFluent
31
+
32
+ **Local-first agent analytics with behavior-to-improvement diagnostics. The tools that exist tell you what your agent did — AgentFluent tells you how to make it better.**
33
+
34
+ [![PyPI version](https://img.shields.io/pypi/v/agentfluent.svg)](https://pypi.org/project/agentfluent/)
35
+ [![Python versions](https://img.shields.io/pypi/pyversions/agentfluent.svg)](https://pypi.org/project/agentfluent/)
36
+ [![CI](https://github.com/frederick-douglas-pearce/agentfluent/actions/workflows/ci.yml/badge.svg)](https://github.com/frederick-douglas-pearce/agentfluent/actions/workflows/ci.yml)
37
+ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
38
+
39
+ AI agents are in production at 57% of organizations, and quality is the single top barrier to deployment. When an agent misbehaves — wrong tool choice, retry loops, hallucinated outputs — developers iterate on prompts blind. Existing observability platforms show *what* happened: traces, latency, token counts. They don't tell you *why* the agent misbehaved or *what in its configuration to change*.
40
+
41
+ AgentFluent reads your local [Claude Code](https://code.claude.com) and [Claude Agent SDK](https://code.claude.com/docs/en/agent-sdk/overview) session JSONL, extracts agent invocations and tool patterns, scores each agent's configuration against a best-practice rubric, and correlates observed behavior back to specific fixes — a prompt gap, a missing tool constraint, or a stale model selection. No cloud services, no API keys, no data leaves your machine.
42
+
43
+ Born from [CodeFluent](https://github.com/frederick-douglas-pearce/codefluent) research that identified the agent-quality gap in 2026. See [`docs/AGENT_ANALYTICS_RESEARCH.md`](docs/AGENT_ANALYTICS_RESEARCH.md) for additional market analysis.
44
+
45
+ ## How It Compares
46
+
47
+ The agent observability space is crowded — several tools capture what agents do. None diagnose *why* they misbehave or *what to change* from locally-persisted session data. In the table below, **"What's missing"** is what the tool *does not do* (not what it provides):
48
+
49
+ | Tool | What it measures | What's missing |
50
+ |------|-----------------|----------------|
51
+ | [Langfuse](https://langfuse.com) / [LangSmith](https://smith.langchain.com) / [Arize Phoenix](https://phoenix.arize.com) | Production traces, latency, token counts, errors | Behavior-to-prompt diagnosis; local agent config audit |
52
+ | [Braintrust](https://braintrust.dev) / [Galileo](https://www.galileo.ai) / [DeepEval](https://www.deepeval.com) | LLM-as-judge scoring against rubrics | Requires cloud instrumentation and author-provided test sets; no local agent config audit |
53
+ | [ccusage](https://github.com/ryoppippi/ccusage) / [claude-code-analytics](https://github.com/d-k-patel/claude-code-analytics) / [agents-observe](https://github.com/agents-observe/agents-observe) | Usage stats, token counts, subagent trees | Quality scoring; actionable config recommendations |
54
+ | [claude-code-otel](https://github.com/ColeMurray/claude-code-otel) | OpenTelemetry export of Claude Code sessions | Analysis itself — it's a bridge to other tools |
55
+ | Anthropic Console | Per-request cost, rate-limit tracking | Session-level diagnostics; agent config recommendations |
56
+
57
+ **Where AgentFluent fits.** AgentFluent reads the session JSONL your agent already produced, scores each agent's configuration against a best-practice rubric, and correlates observed behavior back to the specific config line that most likely explains it. It complements the tools above rather than replacing them — use Langfuse/Phoenix for production traces, Braintrust for test-set evals, ccusage for usage dashboards, and AgentFluent for *what in the agent's config to change*. The question *"my Agent SDK agent ran 500 sessions last week — were any of them actually good, and how can I update my agent's configuration to make it better?"* has no answer from the tools above. AgentFluent is built to answer it.
58
+
59
+ ## Why This Is Different
60
+
61
+ - **Research-grounded.** Every diagnostic maps to a specific gap in the agent's prompt, tool list, or model selection — not vibes. See the [research doc](docs/AGENT_ANALYTICS_RESEARCH.md) for the feasibility and positioning analysis.
62
+ - **Behavior-to-improvement, not just traces.** When the agent retries Bash 40% of the time, AgentFluent tells you *which prompt clause is missing* — not just that the retry happened.
63
+ - **The config is the agent.** In interactive sessions, the human course-corrects. In programmatic agents, the prompt and tool setup *are* the agent — a flaw compounds at scale. AgentFluent scores four dimensions of that config today — description, tools (`allowed_tools` / `disallowedTools`), model, and prompt — with hook, MCP, and cross-agent coverage on the roadmap.
64
+ - **Local-first and private.** All analysis runs on your machine. Zero outbound network calls. No API key required.
65
+ - **CLI-native.** `agentfluent analyze --format json | jq ...` — fits agent developer workflows (terminal, CI/CD, PR checks) without a web dashboard dependency.
66
+ - **JSON output envelope is a contract.** A stable `{version, command, data}` schema lets you build PR gates, trend dashboards, and regression detectors on top without tracking AgentFluent's internal refactors.
67
+ - **Correct cost accounting.** Distinguishes pay-per-token API rate from subscription plan flat cost, with per-model pricing that AgentFluent actively maintains ([#80](https://github.com/frederick-douglas-pearce/agentfluent/issues/80) will add per-session historical pricing).
68
+ - **CodeFluent sibling.** Shares the JSONL parsing heritage but asks a different question. CodeFluent scores *human* AI fluency in interactive sessions; AgentFluent scores *agent* quality and tells you what configuration to change. Not forked — two products with a common data source.
69
+
70
+ ## AgentFluent vs CodeFluent
71
+
72
+ Both read `~/.claude/projects/` session JSONL. They answer different questions:
73
+
74
+ | | [CodeFluent](https://github.com/frederick-douglas-pearce/codefluent) | AgentFluent |
75
+ |---|---|---|
76
+ | **Unit of analysis** | Conversations in interactive sessions, plus the supporting `.claude/` config (CLAUDE.md, rules, hooks, commands) | Agent definitions + their observed behavior |
77
+ | **Scoring target** | Developer's AI collaboration fluency and project-config maturity | Agent's prompt, tools, model, hooks |
78
+ | **Feedback loop** | Coaches the human to interact with Claude Code better | Tells the developer what config to change |
79
+ | **Delivery** | VS Code extension + web app | CLI-first (dashboard deferred) |
80
+ | **API calls** | Anthropic API for LLM-as-judge scoring | None — fully local |
81
+
82
+ If you write your own prompts each session, use CodeFluent. If your prompts live in `ClaudeAgentOptions`, `AgentDefinition`, or `.claude/agents/*.md` files, use AgentFluent.
83
+
84
+ ## Screenshots
85
+
86
+ <table>
87
+ <tr><th>Project Discovery</th><th>Execution Analytics</th></tr>
88
+ <tr valign="top">
89
+ <td><img src="images/demo-list.png" alt="agentfluent list"></td>
90
+ <td><img src="images/demo-analyze.png" alt="agentfluent analyze"></td>
91
+ </tr>
92
+ </table>
93
+
94
+ <table>
95
+ <tr><th>Behavior Diagnostics</th><th>Config Assessment</th></tr>
96
+ <tr valign="top">
97
+ <td><img src="images/demo-diagnostics.png" alt="agentfluent analyze --diagnostics"></td>
98
+ <td><img src="images/demo-config-check.png" alt="agentfluent config-check"></td>
99
+ </tr>
100
+ </table>
101
+
102
+ ## Getting Started
103
+
104
+ ### Prerequisites
105
+
106
+ - **Python 3.12 or newer.** Check with `python --version`.
107
+ - **Claude Code or Agent SDK session data.** Generated automatically at `~/.claude/projects/` whenever you use Claude Code or run an Agent SDK script — nothing to configure.
108
+ - **Platforms:** Linux, macOS, Windows. Pure-Python package; the path handling resolves `~/.claude/` on every platform.
109
+
110
+ ### Install
111
+
112
+ ```bash
113
+ # Preferred — isolated tool install via uv (https://docs.astral.sh/uv/)
114
+ uv tool install agentfluent
115
+
116
+ # Fallback — pip into a venv of your choice
117
+ pip install agentfluent
118
+
119
+ # Zero-install one-shot
120
+ uvx agentfluent list
121
+ ```
122
+
123
+ ### First run
124
+
125
+ ```bash
126
+ # Discover which projects have session data
127
+ agentfluent list
128
+
129
+ # Analyze agent behavior + cost in a specific project
130
+ agentfluent analyze --project myproject
131
+
132
+ # Score your agent definitions against the config rubric
133
+ agentfluent config-check
134
+ ```
135
+
136
+ ## Commands
137
+
138
+ ### `agentfluent list` — discover projects and sessions
139
+
140
+ ```bash
141
+ agentfluent list # All projects
142
+ agentfluent list --project codefluent # Sessions in one project
143
+ agentfluent list --format json | jq '.data.projects[].name'
144
+ ```
145
+
146
+ Lists every Claude Code / Agent SDK project found under `~/.claude/projects/`, with session counts, total size, and last-modified timestamp. Pass `--project` to drill into one project and list its individual session files.
147
+
148
+ ### `agentfluent analyze` — token, cost, and behavior metrics
149
+
150
+ ```bash
151
+ agentfluent analyze --project codefluent # Full project analysis
152
+ agentfluent analyze --project codefluent --agent pm # Filter to one subagent
153
+ agentfluent analyze --project codefluent --latest 5 # Last 5 sessions only
154
+ agentfluent analyze --project codefluent --diagnostics # Show behavior diagnostics
155
+ agentfluent analyze --project codefluent --format json | jq '.data.token_metrics.total_cost'
156
+ ```
157
+
158
+ Produces a token-usage table, per-model cost breakdown (labeled as API rate — subscription plans differ), tool usage concentration, and an Agent Invocations table summarizing each subagent's token, duration, and tool-use count. `--diagnostics` surfaces behavior signals (tool errors, token-per-tool-use outliers, duration outliers) with a pointer to the configuration gap most likely responsible.
159
+
160
+ Cost numbers reflect current per-token pricing; historical sessions are priced at today's rates until [#80](https://github.com/frederick-douglas-pearce/agentfluent/issues/80) (time-series pricing) lands.
161
+
162
+ ### `agentfluent config-check` — score agent definitions
163
+
164
+ ```bash
165
+ agentfluent config-check # All user + project agents
166
+ agentfluent config-check --scope user # Only ~/.claude/agents/
167
+ agentfluent config-check --agent pm --verbose # One agent with detailed recs
168
+ agentfluent config-check --format json | jq '.data.scores[] | select(.overall_score < 60)'
169
+ ```
170
+
171
+ Walks `~/.claude/agents/*.md` and `./.claude/agents/*.md`, parses each agent's YAML frontmatter and body, and scores against a 4-dimension rubric (description trigger quality, tool access appropriateness, model selection, prompt completeness). Outputs a score per agent plus ranked recommendations — e.g. "Prompt body doesn't mention error handling."
172
+
173
+ ## Configuration
174
+
175
+ AgentFluent's "configuration" is CLI flags — no config file, no environment variables beyond the defaults. Sensible defaults keep most invocations flagless.
176
+
177
+ | Flag | Default | What it controls |
178
+ |------|---------|-----------------|
179
+ | `--project` | (required on `analyze`) | Filter to a specific project slug or display name |
180
+ | `--scope` | `all` | `config-check` scope: `user`, `project`, or `all` |
181
+ | `--agent` | (none) | Filter `analyze` or `config-check` to one subagent type |
182
+ | `--latest N` | (all sessions) | `analyze` only the N most recent sessions |
183
+ | `--diagnostics` | off | `analyze`: show behavior-correlation signals |
184
+ | `--format` | `table` | Output format: `table` (Rich) or `json` (envelope) |
185
+ | `--verbose` | off | Extra detail (per-session breakdown, per-invocation detail) |
186
+ | `--quiet` | off | Suppress non-essential output (useful in CI) |
187
+
188
+ ## Output formats
189
+
190
+ **Default (table):** Rich-rendered tables in the terminal, designed to be readable at a glance. Colors auto-adapt to terminal theme.
191
+
192
+ **JSON envelope (`--format json`):** Stable schema `{version, command, data}` intended as a contract — pipe to `jq`, integrate with CI, build regression gates on top. Example:
193
+
194
+ ```json
195
+ {
196
+ "version": 1,
197
+ "command": "analyze",
198
+ "data": {
199
+ "token_metrics": { "total_cost": 15.42, "total_tokens": 82940115, ... },
200
+ "by_model": { "claude-opus-4-7": {...}, "claude-sonnet-4-6": {...} },
201
+ "tool_usage": [...],
202
+ "agent_invocations": [...]
203
+ }
204
+ }
205
+ ```
206
+
207
+ No ANSI escapes in JSON output, guaranteed. The key `total_cost` is the pay-per-token equivalent; subscribers on Pro/Max/Team/Enterprise plans see a flat monthly charge regardless.
208
+
209
+ ## How It Works
210
+
211
+ ```mermaid
212
+ flowchart LR
213
+ subgraph Local["Local filesystem — nothing leaves this boundary"]
214
+ S["Session JSONL<br/>~/.claude/projects/"]
215
+ A["Agent definitions<br/>~/.claude/agents/<br/>.claude/agents/"]
216
+ end
217
+
218
+ S --> P[Parser]
219
+ P --> X[Agent Extractor]
220
+ P --> TM[Token &amp; Cost<br/>Metrics]
221
+ P --> TU[Tool Usage<br/>Patterns]
222
+ A --> CS[Config Scanner]
223
+ CS --> SC[Config Scorer]
224
+
225
+ X --> D[Diagnostics<br/>Correlator]
226
+ TM --> D
227
+ TU --> D
228
+ SC --> D
229
+
230
+ D --> OUT["Rich tables<br/>or JSON envelope"]
231
+ TM --> OUT
232
+ TU --> OUT
233
+ SC --> OUT
234
+ ```
235
+
236
+ Step by step:
237
+
238
+ 1. **Parse JSONL** — `core/parser.py` reads each session file into typed `SessionMessage` objects. Handles streaming snapshot deduplication, plain-string vs. array content shapes, and Claude Code's real `toolUseResult` format (see [`CLAUDE.md`](CLAUDE.md) for the format spec).
239
+ 2. **Discover projects and sessions** — `core/discovery.py` enumerates `~/.claude/projects/` and surfaces friendly display names.
240
+ 3. **Extract agent invocations** — `agents/extractor.py` walks messages, pairs Agent `tool_use` blocks with their `tool_result` content blocks, and pulls per-invocation metadata (tokens, duration, tool-use count) from the containing user message's `toolUseResult` sibling.
241
+ 4. **Compute token and cost metrics** — `analytics/tokens.py` aggregates usage per model with `<synthetic>` sentinel filtering; `analytics/pricing.py` applies per-token rates labeled as API rate.
242
+ 5. **Score agent configurations** — `config/scanner.py` parses YAML frontmatter from each `.md` in `.claude/agents/` and `~/.claude/agents/`; `config/scoring.py` scores description, tools, model, and prompt on a 4-dimension rubric.
243
+ 6. **Diagnose behavior** — `diagnostics/signals.py` correlates observed patterns (retry loops, tool errors, zero-invocation agents) with likely configuration root causes and attaches a recommendation.
244
+ 7. **Render** — `cli/formatters/table.py` emits Rich tables; `cli/formatters/json.py` emits the stable JSON envelope. Format is selected by `--format`.
245
+
246
+ Everything runs locally. No outbound network calls, ever. No API key needed.
247
+
248
+ ## Features
249
+
250
+ - **Project and Session Discovery** — Enumerates `~/.claude/projects/`, groups sessions by project, shows per-project session count, total size, and last-modified timestamp. Handles Claude Code subagent sidechain files and Agent SDK sessions uniformly.
251
+ - **Execution Analytics** — Token usage, API-rate cost, cache efficiency, per-model breakdown, tool-call concentration, and per-agent invocation metrics (tokens, duration, tool-use count). Cache creation and cache read tokens are tracked separately so you can see where your prompt caching is working.
252
+ - **Agent Config Assessment** — 4-dimension rubric (description, tools, model, prompt) applied to every `.md` file in `~/.claude/agents/` and `./.claude/agents/`. Produces a 0–100 score plus ranked, specific recommendations ("Prompt body doesn't mention error handling"). Catches agents that are technically valid but miss well-known best practices.
253
+ - **Diagnostics Preview** — `--diagnostics` correlates three behavior signals to configuration gaps: tool errors (caught by keywords like `blocked`, `failed`, `error`) suggesting missing error-handling instructions or over-broad tools; per-tool-use token outliers suggesting an agent that's exploring too broadly or needs a tighter prompt; duration outliers flagging unusually slow invocations. Each signal carries a severity level and a specific recommendation.
254
+ - **JSON Output Envelope** — Stable `{version, command, data}` schema. No ANSI escapes. Intended as a programmatic contract for CI integration, PR gates, and regression tracking.
255
+ - **Quiet and Verbose Modes** — `--quiet` for CI-friendly one-line summaries; `--verbose` for per-session breakdown and per-invocation detail tables. Defaults target interactive humans.
256
+
257
+ ## Privacy and Security
258
+
259
+ AgentFluent is designed so data stays on your machine. The attack surface is small by construction — no web server, no HTML rendering, no webview, no outbound network calls — but this table summarizes the layers that protect it anyway:
260
+
261
+ | Layer | Mechanism | Protects Against |
262
+ |-------|-----------|-----------------|
263
+ | Zero network calls | No outbound connections — all analysis is local | Data exfiltration |
264
+ | Path handling | All paths resolved within `~/.claude/` | Path traversal |
265
+ | Input validation | Pydantic models with strict type constraints | Malformed JSONL crashing the parser |
266
+ | Safe YAML loading | `yaml.safe_load` only | Arbitrary code execution via frontmatter |
267
+ | CI security review | Claude-powered review on every PR | New vulnerabilities |
268
+ | Automated testing | 270+ unit tests incl. security-focused cases | Regressions |
269
+
270
+ ### Secrets handling
271
+
272
+ Claude Code persists every tool output to `~/.claude/projects/<slug>/*.jsonl` — including any `.env`, `credentials.json`, or shell rc file that Claude ever read. `.gitignore` does not protect against this. AgentFluent itself emits only aggregate metrics, so it cannot leak secrets that weren't already on disk — but because the tool *reads* that data, contributors working on AgentFluent risk re-leaking while they work.
273
+
274
+ This repo ships two Claude Code hooks in [`.claude/settings.json`](.claude/settings.json) to reduce that risk:
275
+
276
+ - **PreToolUse block** ([`.claude/hooks/block_secret_reads.py`](.claude/hooks/block_secret_reads.py)) — denies reads of `.env*`, `.envrc`, `credentials.json`, `secrets.{yaml,yml,json}`, `*.pem`, SSH private keys, and shell rc files. Blocks *before* execution, so the file's contents never enter the session transcript.
277
+ - **PostToolUse detect** ([`.claude/hooks/detect_secrets_in_output.py`](.claude/hooks/detect_secrets_in_output.py)) — scans tool output for `sk-ant-*`, `sk-proj-*`, `ghp_*`, `github_pat_*`, `AKIA*`, or `AIza*` patterns. If a match is found, blocks Claude from echoing or summarizing it. The raw value is already on disk at this point, so treat any caught value as compromised and rotate.
278
+
279
+ Any future AgentFluent feature that surfaces raw session content (diff viewers, prompt excerpts, recommendation snippets that quote session text) must re-apply secret-pattern redaction at the display layer — historical JSONL on users' machines may still contain pre-hook leaks.
280
+
281
+ See [`docs/SECURITY.md`](docs/SECURITY.md) for the full policy: leak vector, defense architecture, discipline rules, historical-leak audit one-liner, user-scope deployment, and the bypass surface the hooks do not cover.
282
+
283
+ ## Tech Stack
284
+
285
+ - **Python 3.12+**
286
+ - **[Typer](https://typer.tiangolo.com) + [Rich](https://rich.readthedocs.io)** — CLI framework and terminal formatting
287
+ - **[Pydantic v2](https://docs.pydantic.dev)** — data models across module boundaries
288
+ - **[PyYAML](https://pyyaml.org)** — agent definition frontmatter parsing (`safe_load` only)
289
+ - **[pytest](https://pytest.org) + pytest-cov** — 270+ tests
290
+ - **[mypy](https://mypy.readthedocs.io) strict mode** — full type coverage
291
+ - **[ruff](https://docs.astral.sh/ruff/)** — linting and formatting
292
+ - **[uv](https://docs.astral.sh/uv/)** — package and dependency management
293
+
294
+ ## Project Structure
295
+
296
+ ```
297
+ src/agentfluent/
298
+ ├── cli/ # Typer app, commands, formatters (table + JSON envelope)
299
+ ├── core/ # JSONL parser, session models, project/session discovery
300
+ ├── agents/ # Agent invocation extraction and AgentInvocation model
301
+ ├── analytics/ # Token/cost metrics, tool patterns, model pricing
302
+ ├── config/ # Agent definition scanner and scoring rubric
303
+ └── diagnostics/ # Behavior signals, correlation, recommendations
304
+ ```
305
+
306
+ Full architecture and conventions are documented in [`CLAUDE.md`](CLAUDE.md).
307
+
308
+ ## Development
309
+
310
+ ```bash
311
+ git clone https://github.com/frederick-douglas-pearce/agentfluent.git
312
+ cd agentfluent
313
+ uv sync
314
+ uv run agentfluent --help
315
+ ```
316
+
317
+ ### Testing
318
+
319
+ ```bash
320
+ uv run pytest -m "not integration" # 270+ unit tests (CI default)
321
+ uv run pytest # Full suite incl. integration tests against your real ~/.claude/projects/
322
+ uv run pytest --cov=agentfluent # With coverage
323
+ ```
324
+
325
+ Integration tests (`tests/integration/`) are skipped in CI because they require real session data — they pass on contributor machines with populated `~/.claude/projects/`.
326
+
327
+ ### Lint and type check
328
+
329
+ ```bash
330
+ uv run ruff check src/ tests/
331
+ uv run mypy src/agentfluent/
332
+ ```
333
+
334
+ Both must pass cleanly before a PR merges.
335
+
336
+ ### CI/CD
337
+
338
+ Five GitHub Actions workflows run automatically:
339
+
340
+ - **CI** ([`ci.yml`](.github/workflows/ci.yml)) — Every PR: ruff, mypy strict, full unit-test suite. Must pass to merge.
341
+ - **Security Review** ([`security-review.yml`](.github/workflows/security-review.yml)) — Claude-powered security review of code-changing PRs (markdown and image changes skip it).
342
+ - **Claude Code Review** ([`claude-review.yml`](.github/workflows/claude-review.yml)) — AI-powered PR review, triggered by the `needs-review` label or `@claude` mentions.
343
+ - **Release Please** ([`release-please.yml`](.github/workflows/release-please.yml)) — Auto-generates release PRs with changelog and version bumps from [Conventional Commits](https://www.conventionalcommits.org/).
344
+ - **Dependabot Auto-Merge** ([`dependabot-auto-merge.yml`](.github/workflows/dependabot-auto-merge.yml)) — Auto-merges dependabot PRs once CI passes.
345
+
346
+ ## Roadmap
347
+
348
+ **v0.2 (next release):**
349
+ - Parser fix for real Claude Code `toolUseResult` shape ([#84](https://github.com/frederick-douglas-pearce/agentfluent/issues/84) — merged)
350
+ - Cost label clarity for subscription-plan users ([#76](https://github.com/frederick-douglas-pearce/agentfluent/issues/76) — merged)
351
+ - Pricing data correction + opus-4-7 + synthetic filter ([#75](https://github.com/frederick-douglas-pearce/agentfluent/issues/75) — merged)
352
+
353
+ **v0.3+:**
354
+ - Time-series pricing data structure ([#80](https://github.com/frederick-douglas-pearce/agentfluent/issues/80))
355
+ - Session-timestamp-aware cost calculation ([#81](https://github.com/frederick-douglas-pearce/agentfluent/issues/81))
356
+ - Automated pricing-update service ([#82](https://github.com/frederick-douglas-pearce/agentfluent/issues/82))
357
+ - `--claude-config-dir` flag for non-default session paths ([#90](https://github.com/frederick-douglas-pearce/agentfluent/issues/90))
358
+ - Delegation pattern recognition — cluster `general-purpose` invocations and recommend custom subagents ([#92](https://github.com/frederick-douglas-pearce/agentfluent/issues/92))
359
+ - Deeper diagnostics with per-tool-call evidence
360
+ - Subagent trace parsing (`~/.claude/projects/<session>/subagents/`)
361
+ - Prompt regression detection across agent config versions
362
+ - Retry-pattern and zero-invocation-agent signals (complete the diagnostics surface currently covering tool errors and outliers)
363
+ - Hook and MCP-server coverage in the config rubric
364
+
365
+ **Future:**
366
+ - Webapp dashboard for trend visualization
367
+ - `agentfluent diff` — side-by-side comparison of behavior before/after a prompt change
368
+ - MCP server configuration assessment
369
+ - Closed-loop self-improvement — use AgentFluent's diagnostic output as a feedback signal the agent itself consumes to propose config edits against its own past sessions
370
+ - Agent ROI reporting — roll up cost, usage, and task-completion signals over time so a business can evaluate whether an optimized agent is worth continuing to run
371
+
372
+ Browse [open issues](https://github.com/frederick-douglas-pearce/agentfluent/issues) for the full backlog.
373
+
374
+ ## Troubleshooting
375
+
376
+ | Problem | Solution |
377
+ |---------|----------|
378
+ | **No projects found** | Verify `~/.claude/projects/` exists and contains per-project subdirectories with `.jsonl` session files. Claude Code creates these automatically the first time you use it. |
379
+ | **No agent invocations** | Agent invocation rows require the session to actually call a subagent (`Agent` tool_use with a `subagent_type`). A session that never delegated has no agent data to analyze — this is not an error. |
380
+ | **Zero tokens / dashes in Agent Invocations** | If you're on AgentFluent ≤ 0.1.0, this is the [#84 parser bug](https://github.com/frederick-douglas-pearce/agentfluent/issues/84) — upgrade with `uv tool upgrade agentfluent`. |
381
+ | **Python version error** | AgentFluent requires Python 3.12+. Check with `python --version` and upgrade if needed. |
382
+ | **Non-default session path** | If `~/.claude/` is stored somewhere unusual, AgentFluent currently uses the default path only. Custom path support is planned. |
383
+ | **`Malformed JSON at <file>:<line>` warning** | A session file has a corrupted line — usually null bytes left behind when Claude Code was killed mid-write. The parser skips the line and continues; analytics are unaffected. Safe to ignore, or delete the line with `sed -i '<line>d' <file>` to silence the warning. |
384
+ | **Stale tool install after local build** | If `uv tool install --from <path> agentfluent` seems to reuse cached code, run `uv tool uninstall agentfluent && uv cache clean agentfluent` before reinstalling. |
385
+
386
+ ## Research Foundations
387
+
388
+ AgentFluent's behavior-to-improvement approach is grounded in research on agent quality, observability gaps, and production failure modes:
389
+
390
+ - [`docs/AGENT_ANALYTICS_RESEARCH.md`](docs/AGENT_ANALYTICS_RESEARCH.md) — Full market analysis, competitive landscape (Langfuse, LangSmith, Arize, Braintrust, DeepEval, etc.), and technical feasibility study. This is the document that motivated AgentFluent's existence as a separate product from CodeFluent.
391
+ - [LangChain 2026 State of AI Agents](https://www.langchain.com/stateofaiagents) — 57% of orgs have agents in production; quality is the top blocker.
392
+ - [Anthropic Claude Agent SDK docs](https://code.claude.com/docs/en/agent-sdk/overview) — Agent configuration surface and best practices.
393
+ - [Anthropic Claude Code subagents docs](https://code.claude.com/docs/en/sub-agents) — Subagent definition format and delegation mechanics.
394
+
395
+ ## Contributing
396
+
397
+ Contributions welcome. Start by reading [`CONTRIBUTING.md`](CONTRIBUTING.md) for dev setup, conventions, and the PR checklist. The [architecture overview in `CLAUDE.md`](CLAUDE.md) is the canonical reference for package layout, naming, and the JSONL format.
398
+
399
+ Branching: `feature/<issue>-description` for features, `fix/<issue>-description` for bugs. Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/) — release-please uses them to cut versions and write the changelog automatically.
400
+
401
+ ## License
402
+
403
+ [MIT](LICENSE)