messygit 0.3.2__tar.gz → 0.4.5__tar.gz

messygit-0.4.5/ARCHITECTURE.md

@@ -0,0 +1,93 @@
+# Architecture
+
+This document describes the purpose of each file in the `messygit` project.
+
+The package is organized in layers. `ui/` is the shared presentation foundation
+(console, theme, spinner, banner). `commands/` holds the command handlers grouped
+the same way the `help` screen groups them. `agent/` is a self-contained tool-use
+loop. `cli.py` is a thin shell that wires everything together. Dependencies flow
+one way — `ui/` knows nothing about `commands/`, and `commands/` knows nothing
+about `cli` — so there are no import cycles.
+
+## Root
+
+| File                            | Purpose                                                                                                                        |
+| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
+| `pyproject.toml`                | Package metadata, dependencies (`anthropic`, `click`, `rich`), build system (hatchling), and the `messygit` console script.    |
+| `README.md`                     | User-facing documentation: install, usage, commands, and development instructions.                                             |
+| `ARCHITECTURE.md`               | This file — a per-module map of the codebase.                                                                                   |
+| `CHANGELOG.md`                  | Release notes, generated/updated by the `changelog` command.                                                                   |
+| `.gitignore`                    | Keeps `.venv/`, `__pycache__/`, `dist/`, and `*.egg-info/` out of version control.                                             |
+| `.github/workflows/test.yml`    | Runs `pytest` on every push/PR across Python 3.10–3.13.                                                                         |
+| `.github/workflows/publish.yml` | Builds and publishes to PyPI via trusted publishing on release.                                                                |
+
+## `messygit/` (core package)
+
+| File          | Purpose                                                                                                                                                                                                                                                |
+| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `__init__.py` | Marks the directory as a Python package.                                                                                                                                                                                                              |
+| `cli.py`      | The REPL shell: prints the startup dashboard, runs the prompt loop, owns the `COMMANDS` dispatch table and the `help` screen, and applies the saved theme on launch. Delegates all real work to `commands/`.                                          |
+| `git.py`      | All subprocess calls to `git`. Reads staged diffs (`git diff --cached -U0`), parses them into a compact changed-lines format, filters noise files, handles the large-diff fallback (stat summary + top-N changed files), and runs `add`/`commit`/`push`. Also exposes the primitives incremental `suggest` needs: `get_head_hash`, `get_repo_root`, `is_ancestor` (validate a saved checkpoint), and `get_commits_since`/`get_range_context` (the `saved_hash..HEAD` delta). |
+| `llm.py`      | Anthropic SDK integration for the (non-agentic) `commit` path. Creates the client, calls `messages.create`, extracts text, and maps SDK exceptions (auth, permission, billing/402) into user-friendly error classes. Exposes helpers reused by `agent/agent.py`. |
+| `config.py`   | Reads/writes `~/.messygit/config.json`: API key (plus `ANTHROPIC_API_KEY` env resolution), theme, model, todo, the `verbose` flag, and per-repo `suggest` checkpoints (`{head, suggestions, incremental_runs}`, keyed by repo root). Masks keys for display and defines user-facing error messages and exception classes. |
+| `models.py`   | The selectable Claude models, their labels, and approximate per-million-token pricing; resolves the active model from config.                                                                                                                        |
+| `usage.py`    | Session-local token-usage tracker (`SESSION_USAGE`) and the billing URL. The API exposes no balance endpoint, so usage/cost are accumulated per session.                                                                                              |
+| `prompts.py`  | The system prompts — `COMMIT_SYSTEM_PROMPT`, `SUGGESTION_SYSTEM_PROMPT`, `SUGGESTION_INCREMENTAL_SYSTEM_PROMPT` (delta-only "update the existing list" variant), `CHANGELOG_SYSTEM_PROMPT` — plus `build_user_prompt`. Each prompt includes anti-hallucination and untrusted-input guardrails.                                          |
+
+## `messygit/ui/` (shared presentation)
+
+| File         | Purpose                                                                                                                                                                  |
+| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `theme.py`   | Color palette and theme state. `BRAND`/`BRAND_RGB`/`BANNER_COLOR` are reassigned at runtime by `apply_theme()`, so other modules read them as `theme.BRAND` (never import the name directly). Also `brand_ansi()` and `active_theme()`. |
+| `output.py`  | The shared `console`/`err_console` objects and the small print helpers (`print_error`, `success`, `warn`, `field`).                                                     |
+| `spinner.py` | The threaded `_CharSpinner` loading animation and the `spinner()` factory.                                                                                              |
+| `banner.py`  | The ASCII banner art and its boot animation (`animate_banner`).                                                                                                         |
+
+## `messygit/commands/` (command handlers)
+
+| File              | Purpose                                                                                                                                                  |
+| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `git_cmds.py`     | The git group: `add`, `commit` (AI message + Y/n/e prompt), `push`, `outbox`.                                                                           |
+| `agent_cmds.py`   | The agent group: `suggest` and `changelog`. `_drive()` runs an agent either under a spinner or streaming live (verbose), and records the run's trace. `suggest` is incremental: `_choose_mode` picks a full scan (first run, stale/diverged checkpoint, `suggest full`, or a periodic re-audit every `AUTO_FULL_SCAN_EVERY` runs) vs an incremental run that feeds the prior suggestions plus only the `saved_hash..HEAD` delta (`_incremental_prompt`), then advances the per-repo checkpoint. |
+| `account_cmds.py` | The account group: `config`, `show`, `model`, `tokens`.                                                                                                 |
+| `app_cmds.py`     | The app group: `todo`, `theme`, `verbose`.                                                                                                              |
+| `usage.py`        | Token-usage/cost *display* helpers (`usage_summary`, `print_usage_delta`, `model_pricing`, the high-usage warning). Distinct from the root `usage.py` tracker. |
+| `trace.py`        | The `trace` command. Stores the last agent run's steps and renders them as a panel; also provides `live_reporter()`, the per-step formatter that verbose mode streams. |
+
+## `messygit/agent/` (agentic tool-use subpackage)
+
+| File       | Purpose                                                                                                                                                                                                                            |
+| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `tool.py`  | `Tool` dataclass wrapping a Python callable with a name, description, and parameter schema. `run()` invokes the function; `to_schema()` emits an Anthropic-compatible tool definition.                                            |
+| `tools.py` | Concrete tools: `run_git_tool` (allowlisted read-only git, including tag ranges), `read_file_tool`, `list_directory_tool`, `search_code_tool` (`git grep`), `write_file_tool` (full overwrite), and `edit_file_tool` (unique-match string replace). All file tools reject paths outside the repo root. |
+| `agent.py` | The agentic loop. Sends messages to Claude with tool definitions, dispatches `tool_use` blocks to matching `Tool`s, feeds results back, and iterates up to `max_iterations`. Records each step as a `TraceStep` (optionally streamed via an `on_step` callback) and warns when the iteration cap is hit before finishing. |
+
+## Data flow
+
+```
+User runs `messygit` → cli.py REPL (dispatch table)
+       │
+       ├── "add <files>" / "push" ─────► commands/git_cmds.py ─► git.py
+       ├── "config" / "show" / "model" / "tokens" ─► commands/account_cmds.py ─► config.py / usage.py
+       ├── "todo" / "theme" / "verbose" ─────► commands/app_cmds.py ─► config.py
+       ├── "trace" ─────────────────────► commands/trace.py (render last run)
+       │
+       ├── "commit" ──► commands/git_cmds.py
+       │       │            ├── git.py (staged diff + token budget)
+       │       │            └── llm.py ─► prompts.COMMIT_SYSTEM_PROMPT, config.py (key)
+       │       ▼
+       │   Y/n/e prompt ─► git.py (git commit)
+       │
+       └── "suggest" / "changelog" ──► commands/agent_cmds.py
+               │            ├── config.py: load/save suggest checkpoint (suggest only)
+               │            │      └── full scan on first run / stale hash / `suggest full`
+               │            │          / every AUTO_FULL_SCAN_EVERY runs; else incremental
+               │            │             (git.py: saved_hash..HEAD delta → _incremental_prompt)
+               │            └── agent/agent.py (tool-use loop)
+               │                    ├── config.py (key) + models.py
+               │                    ├── prompts.{SUGGESTION,SUGGESTION_INCREMENTAL,CHANGELOG}_SYSTEM_PROMPT
+               │                    └── agent/tools.py (run_git, read_file, list_directory,
+               │                                        write_file, edit_file)
+               ▼
+           spinner OR live step stream (verbose); steps saved for `trace`
+```

messygit-0.4.5/CHANGELOG.md

@@ -0,0 +1,40 @@
+## [Unreleased]
+### Added
+- Comprehensive evaluation framework (`evals/`) for testing AI model outputs, including:
+  - Generic evaluation harness supporting multiple eval types with deterministic scorers and optional LLM-as-judge grading
+  - Commit message evaluation with test cases for format, semantic accuracy, and adversarial prompt injection
+  - Agentic evaluators for `suggest` and `changelog` commands that build throwaway git repos and grade both output quality and agent process (tool usage, iteration budget, hallucination detection)
+  - Evaluation result reporting with markdown scorecards and aggregated pass rates
+
+## [v0.4.0] - 2026-06-23
+### Added
+- `trace` command to inspect and display the detailed steps of the last agent run (show what tools were called, their inputs, and results)
+- `verbose` mode toggle to stream agent steps live as they execute (replaces the spinner with real-time output during `suggest` and `changelog` operations)
+- `edit_file` tool for the agent to make precise edits to existing files by replacing specific text spans (complements `write_file` for partial updates)
+- `changelog` command to automatically generate and update a CHANGELOG.md file by analyzing git history between tags
+- Iteration limit warning that displays when an agent run hits the maximum iteration limit before completing, alerting users that the task may be incomplete
+
+### Changed
+- CLI code reorganized from a single monolithic module into separate command modules (account_cmds, git_cmds, agent_cmds, app_cmds) and UI modules (banner, spinner, theme, output) for better maintainability
+
+## [v0.3.2] - 2026-06-20
+### Added
+- Error handling for unknown tools—agent now returns an error result instead of crashing when the model requests a non-existent tool
+- Exception handling for tool execution—tool errors are caught and reported back to the model as error results
+- `outbox` command to display committed but unpushed commits, showing how many commits are ahead of the upstream branch
+- Input validation and security checks for `read_file` and `list_directory` tools to prevent directory traversal attacks
+- Enhanced tool descriptions with detailed documentation of parameters and usage examples
+- Support for `required` field in tool schemas for better validation
+- Comprehensive pytest test suite covering agent tool execution, config management, git diff parsing, LLM error handling, and tool schemas
+- GitHub Actions CI workflow for automated testing across Python 3.10–3.13
+
+### Changed
+- Tool schema generation now includes required field declarations
+- Improved error messages throughout the CLI for better user guidance
+- Tool functions now validate inputs for security and proper error reporting
+
+## [v0.3.1] - 2026-06-20
+
+## [v0.3.0] - 2026-06-20
+
+## [v0.2.1] - 2026-06-20

{messygit-0.3.2 → messygit-0.4.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: messygit
-Version: 0.3.2
+Version: 0.4.5
 Summary: An AI-powered interactive git CLI with agentic tools for commits, code suggestions, and workflow automation.
 License-Expression: MIT
 Requires-Python: >=3.10
@@ -13,16 +13,18 @@ Description-Content-Type: text/markdown
 
 # messygit
 
-**messygit** is an interactive CLI that turns messy git workflows into clean Conventional Commits — stage, commit, push, and get AI-powered project suggestions, all from one interface powered by [Claude](https://www.anthropic.com/api).
+**messygit** is an interactive CLI that turns messy git workflows into clean Conventional Commits — stage, commit, push, generate changelogs, and get AI-powered project suggestions, all from one interface powered by [Claude](https://www.anthropic.com/api).
 
 ## Why use it
 
 - **Interactive REPL** — one command drops you into a persistent session where you can stage, commit, push, and more without leaving.
 - **AI commit messages** — sends your staged diff to Claude and suggests a clean Conventional Commits subject line.
-- **Project suggestions** — an AI agent inspects your repo and recommends concrete next steps.
+- **Project suggestions** — an AI agent inspects your repo and recommends concrete next steps. After the first run it only re-analyzes what changed (cheaper) while carrying its previous list forward, and periodically re-audits the whole repo on its own.
+- **Changelog generation** — an agent reads the commits between your two latest tags, drills into unclear ones, categorizes the changes, and writes/updates `CHANGELOG.md`.
+- **See what the agent did** — runs are clean by default; type `trace` to expand the last run's tool calls, or flip `verbose` on to stream each step live.
 - **Token usage & cost** — tracks the tokens each session uses and shows a rough cost estimate, with a one-command jump to billing.
 - **Themed UI** — a colored startup animation and prompt you can recolor with the `theme` command.
-- **Safe by default** — only the staged diff is sent to the model. Your API key is never printed in full.
+- **Safe by default** — only the staged diff is sent for `commit`; agents use read-only git plus repo-scoped file tools that reject paths outside the repository. Your API key is never printed in full.
 
 ## Requirements
 
@@ -96,12 +98,16 @@ Commands are grouped on the `help` screen:
 | `add <file>` or `add .` | Stage files for commit |
 | `commit` | Generate an AI commit message from staged changes, then commit / cancel / edit |
 | `push` | Push commits to remote |
+| `outbox` | Show commits made locally but not yet pushed to the upstream branch |
 
 **messyagent**
 
 | Command | Description |
 |---------|-------------|
-| `suggest` | Get AI-powered next-step suggestions for your project |
+| `suggest` | Get AI-powered next-step suggestions for your project (incremental after the first run) |
+| `suggest full` | Force a fresh full-repo review, ignoring the saved checkpoint |
+| `changelog` | Generate or update `CHANGELOG.md` from the commits between your two latest tags (requires at least one tag) |
+| `trace` | Show what the last agent run actually did — its tool calls and results |
 
 **account**
 
@@ -118,6 +124,7 @@ Commands are grouped on the `help` screen:
 |---------|-------------|
 | `todo` | Open your todo list in `$EDITOR` (saved to `~/.messygit/todo.md`) |
 | `theme` or `theme <name>` | Change the UI color (run `theme` to list presets) |
+| `verbose` or `verbose on\|off` | Toggle live streaming of agent steps (persists; off by default) |
 | `help` | List available commands |
 | `quit` / `exit` | Exit messygit |
 
@@ -136,6 +143,66 @@ messygit > push
 
 > Tip: run `suggest` for AI next-step ideas, or `theme` to recolor the UI.
 
+### Agent commands & transparency
+
+`suggest` and `changelog` are backed by an agent that uses tools (read-only git,
+file reads, and — for `changelog` — repo-scoped file writes) over several steps.
+
+By default a run shows only a spinner and the final result. Two commands let you
+see inside:
+
+```
+messygit > changelog          # generates/updates CHANGELOG.md
+messygit > trace              # expand what that run just did
+
+╭─ trace · changelog · 4 tool calls ─────────────╮
+│ 1. run_git  tag --sort=-creatordate            │
+│    └ v0.4.0  (+3 more lines)                    │
+│ 2. run_git  log v0.3.2..v0.4.0                  │
+│    └ commit a1b2c3 feat: …  (+12 more lines)    │
+│ 3. read_file  CHANGELOG.md                      │
+│    └ ## [v0.3.2] - 2026-06-20                    │
+│ 4. edit_file  CHANGELOG.md                      │
+│    └ File edited successfully.                   │
+╰─────────────────────────────────────────────────╯
+```
+
+Prefer to watch it happen live? Turn on `verbose` — the same steps stream as the
+agent works (no spinner). The setting persists in `~/.messygit/config.json`:
+
+```
+messygit > verbose on
+Verbose on — agent runs will stream their steps live (no spinner)
+```
+
+`changelog` requires at least one git tag; it documents the range between your two
+most recent tags (or the latest tag to `HEAD` when only one exists).
+
+### Incremental suggestions
+
+The first time you run `suggest` in a repo, the agent does a full scan and saves a
+checkpoint — the current `HEAD` hash plus the list it produced — to
+`~/.messygit/config.json`, keyed by repository. Every run after that is
+**incremental**: instead of re-reading the whole tree, the agent is handed only the
+`saved_hash..HEAD` delta (new commits + changed lines) alongside its previous
+suggestions, and updates the list — dropping items the changes resolved, keeping
+standing gaps, and adding follow-ups from what changed. Then it advances the
+checkpoint. This makes repeat runs much cheaper and keeps the advice coherent
+across runs instead of resetting each time.
+
+Because delta-only analysis never re-reads untouched files, a standing issue in a
+corner nothing has touched could otherwise drift out of view. Two things guard
+against that:
+
+- **Automatic re-audit** — after several incremental runs, `suggest` falls back to
+  a full scan on its own to refresh whole-repo coverage.
+- **`suggest full`** — force a fresh full-repo review at any time.
+
+`suggest` also falls back to a full scan automatically when there's no checkpoint
+yet, or when the saved commit is no longer an ancestor of `HEAD` (after a rebase,
+amend, squash, or branch switch), so a stale checkpoint never produces a nonsense
+delta.
+
 ### Commit message style
 
 The model follows **Conventional Commits**: `type(scope): description`
@@ -144,7 +211,7 @@ Allowed types: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `chore`. Subj
 
 ### Token usage & cost
 
-messygit tracks the tokens used by AI commands (`commit`, `suggest`) for the current session and shows a running total after each call. Run `tokens` for a breakdown and a one-key jump to the Anthropic billing console:
+messygit tracks the tokens used by AI commands (`commit`, `suggest`, `changelog`) for the current session and shows a running total after each call. Run `tokens` for a breakdown and a one-key jump to the Anthropic billing console:
 
 ```
 messygit > tokens
@@ -176,6 +243,35 @@ Switch anyway? [y/N]:
 
 Switching to a **more expensive** model prompts for confirmation first. The session cost estimate prices each request at the model used for it, so it stays accurate even if you switch mid-session.
 
+## Development & testing
+
+Install the package with its dev dependencies (pytest), then run the suite:
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+```
+
+The tests live in `tests/` and are pure unit tests — no network calls and no
+API key required (the Anthropic client is simulated). They cover:
+
+| File | What it covers |
+|------|----------------|
+| `tests/config_test.py` | API-key resolution order (env → file), key save/load, theme/model/todo persistence, per-repo `suggest` checkpoints, malformed-config handling |
+| `tests/suggest_test.py` | `suggest` mode selection — full-scan vs incremental, and the periodic auto re-audit cycle |
+| `tests/git_test.py` | The diff parser — noise-file filtering and the compact changed-lines format |
+| `tests/llm_test.py` | Insufficient-balance / billing-error detection and user messaging |
+| `tests/tool_schema_test.py` | Agent tool schemas match the shape the Anthropic Messages API expects |
+| `tests/agent_test.py` | The agent's tool-use loop, driven by a simulated Anthropic client with scripted responses |
+| `tests/trace_test.py` | The `trace` renderer — step numbering, result truncation, empty state, and markup-safety of raw tool output |
+| `tests/verbose_test.py` | The `verbose` setting, the toggle command, and `_drive` choosing live-stream vs. spinner |
+
+### Continuous integration
+
+`.github/workflows/test.yml` runs `pytest` on every push to `main` and on every
+pull request, across Python 3.10–3.13. No secrets are required because the tests
+mock the Anthropic client.
+
 ## Publishing to PyPI
 
 This project uses GitHub Actions with [PyPI trusted publishing](https://docs.pypi.org/trusted-publishers/) — no API tokens needed in your repo.

{messygit-0.3.2 → messygit-0.4.5}/README.md

@@ -1,15 +1,17 @@
 # messygit
 
-**messygit** is an interactive CLI that turns messy git workflows into clean Conventional Commits — stage, commit, push, and get AI-powered project suggestions, all from one interface powered by [Claude](https://www.anthropic.com/api).
+**messygit** is an interactive CLI that turns messy git workflows into clean Conventional Commits — stage, commit, push, generate changelogs, and get AI-powered project suggestions, all from one interface powered by [Claude](https://www.anthropic.com/api).
 
 ## Why use it
 
 - **Interactive REPL** — one command drops you into a persistent session where you can stage, commit, push, and more without leaving.
 - **AI commit messages** — sends your staged diff to Claude and suggests a clean Conventional Commits subject line.
-- **Project suggestions** — an AI agent inspects your repo and recommends concrete next steps.
+- **Project suggestions** — an AI agent inspects your repo and recommends concrete next steps. After the first run it only re-analyzes what changed (cheaper) while carrying its previous list forward, and periodically re-audits the whole repo on its own.
+- **Changelog generation** — an agent reads the commits between your two latest tags, drills into unclear ones, categorizes the changes, and writes/updates `CHANGELOG.md`.
+- **See what the agent did** — runs are clean by default; type `trace` to expand the last run's tool calls, or flip `verbose` on to stream each step live.
 - **Token usage & cost** — tracks the tokens each session uses and shows a rough cost estimate, with a one-command jump to billing.
 - **Themed UI** — a colored startup animation and prompt you can recolor with the `theme` command.
-- **Safe by default** — only the staged diff is sent to the model. Your API key is never printed in full.
+- **Safe by default** — only the staged diff is sent for `commit`; agents use read-only git plus repo-scoped file tools that reject paths outside the repository. Your API key is never printed in full.
 
 ## Requirements
 
@@ -83,12 +85,16 @@ Commands are grouped on the `help` screen:
 | `add <file>` or `add .` | Stage files for commit |
 | `commit` | Generate an AI commit message from staged changes, then commit / cancel / edit |
 | `push` | Push commits to remote |
+| `outbox` | Show commits made locally but not yet pushed to the upstream branch |
 
 **messyagent**
 
 | Command | Description |
 |---------|-------------|
-| `suggest` | Get AI-powered next-step suggestions for your project |
+| `suggest` | Get AI-powered next-step suggestions for your project (incremental after the first run) |
+| `suggest full` | Force a fresh full-repo review, ignoring the saved checkpoint |
+| `changelog` | Generate or update `CHANGELOG.md` from the commits between your two latest tags (requires at least one tag) |
+| `trace` | Show what the last agent run actually did — its tool calls and results |
 
 **account**
 
@@ -105,6 +111,7 @@ Commands are grouped on the `help` screen:
 |---------|-------------|
 | `todo` | Open your todo list in `$EDITOR` (saved to `~/.messygit/todo.md`) |
 | `theme` or `theme <name>` | Change the UI color (run `theme` to list presets) |
+| `verbose` or `verbose on\|off` | Toggle live streaming of agent steps (persists; off by default) |
 | `help` | List available commands |
 | `quit` / `exit` | Exit messygit |
 
@@ -123,6 +130,66 @@ messygit > push
 
 > Tip: run `suggest` for AI next-step ideas, or `theme` to recolor the UI.
 
+### Agent commands & transparency
+
+`suggest` and `changelog` are backed by an agent that uses tools (read-only git,
+file reads, and — for `changelog` — repo-scoped file writes) over several steps.
+
+By default a run shows only a spinner and the final result. Two commands let you
+see inside:
+
+```
+messygit > changelog          # generates/updates CHANGELOG.md
+messygit > trace              # expand what that run just did
+
+╭─ trace · changelog · 4 tool calls ─────────────╮
+│ 1. run_git  tag --sort=-creatordate            │
+│    └ v0.4.0  (+3 more lines)                    │
+│ 2. run_git  log v0.3.2..v0.4.0                  │
+│    └ commit a1b2c3 feat: …  (+12 more lines)    │
+│ 3. read_file  CHANGELOG.md                      │
+│    └ ## [v0.3.2] - 2026-06-20                    │
+│ 4. edit_file  CHANGELOG.md                      │
+│    └ File edited successfully.                   │
+╰─────────────────────────────────────────────────╯
+```
+
+Prefer to watch it happen live? Turn on `verbose` — the same steps stream as the
+agent works (no spinner). The setting persists in `~/.messygit/config.json`:
+
+```
+messygit > verbose on
+Verbose on — agent runs will stream their steps live (no spinner)
+```
+
+`changelog` requires at least one git tag; it documents the range between your two
+most recent tags (or the latest tag to `HEAD` when only one exists).
+
+### Incremental suggestions
+
+The first time you run `suggest` in a repo, the agent does a full scan and saves a
+checkpoint — the current `HEAD` hash plus the list it produced — to
+`~/.messygit/config.json`, keyed by repository. Every run after that is
+**incremental**: instead of re-reading the whole tree, the agent is handed only the
+`saved_hash..HEAD` delta (new commits + changed lines) alongside its previous
+suggestions, and updates the list — dropping items the changes resolved, keeping
+standing gaps, and adding follow-ups from what changed. Then it advances the
+checkpoint. This makes repeat runs much cheaper and keeps the advice coherent
+across runs instead of resetting each time.
+
+Because delta-only analysis never re-reads untouched files, a standing issue in a
+corner nothing has touched could otherwise drift out of view. Two things guard
+against that:
+
+- **Automatic re-audit** — after several incremental runs, `suggest` falls back to
+  a full scan on its own to refresh whole-repo coverage.
+- **`suggest full`** — force a fresh full-repo review at any time.
+
+`suggest` also falls back to a full scan automatically when there's no checkpoint
+yet, or when the saved commit is no longer an ancestor of `HEAD` (after a rebase,
+amend, squash, or branch switch), so a stale checkpoint never produces a nonsense
+delta.
+
 ### Commit message style
 
 The model follows **Conventional Commits**: `type(scope): description`
@@ -131,7 +198,7 @@ Allowed types: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `chore`. Subj
 
 ### Token usage & cost
 
-messygit tracks the tokens used by AI commands (`commit`, `suggest`) for the current session and shows a running total after each call. Run `tokens` for a breakdown and a one-key jump to the Anthropic billing console:
+messygit tracks the tokens used by AI commands (`commit`, `suggest`, `changelog`) for the current session and shows a running total after each call. Run `tokens` for a breakdown and a one-key jump to the Anthropic billing console:
 
 ```
 messygit > tokens
@@ -163,6 +230,35 @@ Switch anyway? [y/N]:
 
 Switching to a **more expensive** model prompts for confirmation first. The session cost estimate prices each request at the model used for it, so it stays accurate even if you switch mid-session.
 
+## Development & testing
+
+Install the package with its dev dependencies (pytest), then run the suite:
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+```
+
+The tests live in `tests/` and are pure unit tests — no network calls and no
+API key required (the Anthropic client is simulated). They cover:
+
+| File | What it covers |
+|------|----------------|
+| `tests/config_test.py` | API-key resolution order (env → file), key save/load, theme/model/todo persistence, per-repo `suggest` checkpoints, malformed-config handling |
+| `tests/suggest_test.py` | `suggest` mode selection — full-scan vs incremental, and the periodic auto re-audit cycle |
+| `tests/git_test.py` | The diff parser — noise-file filtering and the compact changed-lines format |
+| `tests/llm_test.py` | Insufficient-balance / billing-error detection and user messaging |
+| `tests/tool_schema_test.py` | Agent tool schemas match the shape the Anthropic Messages API expects |
+| `tests/agent_test.py` | The agent's tool-use loop, driven by a simulated Anthropic client with scripted responses |
+| `tests/trace_test.py` | The `trace` renderer — step numbering, result truncation, empty state, and markup-safety of raw tool output |
+| `tests/verbose_test.py` | The `verbose` setting, the toggle command, and `_drive` choosing live-stream vs. spinner |
+
+### Continuous integration
+
+`.github/workflows/test.yml` runs `pytest` on every push to `main` and on every
+pull request, across Python 3.10–3.13. No secrets are required because the tests
+mock the Anthropic client.
+
 ## Publishing to PyPI
 
 This project uses GitHub Actions with [PyPI trusted publishing](https://docs.pypi.org/trusted-publishers/) — no API tokens needed in your repo.

messygit-0.4.5/evals/README.md

@@ -0,0 +1,90 @@
+# Evals
+
+Evals measure the **quality of real model output** — distinct from the unit
+tests in `tests/`, which simulate the model and run offline. Evals call the live
+API, so they cost tokens and need an `ANTHROPIC_API_KEY`. They are intentionally
+**not** collected by `pytest`.
+
+## Run it
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...     # or: messygit > config <key>
+python -m evals.commit_eval        # diff -> commit subject
+python -m evals.suggest_eval       # repo state -> next-steps list (agentic)
+python -m evals.changelog_eval     # tagged repo -> written CHANGELOG.md (agentic)
+```
+
+The agentic evals (`suggest`, `changelog`) build a throwaway git repo per case,
+run the real agent inside it, and grade both the output and the trace — so they
+call the API many times over several iterations and take a few minutes each.
+
+Each run writes a markdown report to `evals/results/<eval>/<timestamp>.md`
+(one file per run, grouped by eval) — the terminal only prints the file path.
+Exit code is `0` if the format-check pass rate ≥ `EVAL_PASS_BAR` (default `0.90`),
+else `1` — so it can gate a CI job.
+
+Env knobs:
+
+
+| Var                | Default            | Meaning                                                                                                     |
+| ------------------ | ------------------ | ----------------------------------------------------------------------------------------------------------- |
+| `EVAL_JUDGE_MODEL` | `claude-haiku-4-5` | Model used for LLM-as-judge scoring (set a stronger one like `claude-sonnet-4-6` for more reliable grading) |
+| `EVAL_PASS_BAR`    | `0.9`              | Fraction of applicable checks that must pass                                                                |
+
+
+The model *under test* is whatever `model` is configured in messygit (default
+Haiku) — switch it with `model <name>` to compare models on the same dataset.
+
+## How it's built
+
+
+| File             | Role                                                                                                                                                                    |
+| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `core.py`        | The **generic harness**, shared by every eval: `run_eval`, `render`, the `ScoreResult`/`CaseResult` types, and `make_llm_judge`. Knows nothing about any specific eval. |
+| `commit_eval.py` | **One self-contained eval**: the `CommitCase` dataset, the bespoke scorers, `generate()`, the judge rubric, and `main()`. The reference shape to copy.                  |
+
+
+## Adding a new eval — just write one file
+
+Create `evals/<name>_eval.py` with four things and let `core` do the rest:
+
+```python
+from .core import run_eval, write_report, make_llm_judge, ScoreResult, PASS_BAR
+
+CASES = [...]                              # 1. objects with a `.name`
+def generate(case) -> str: ...            # 2. run the thing under test
+SCORERS = [my_scorer, ...]                # 3. (case, output) -> ScoreResult
+judge = make_llm_judge(system, render_user)  # 4. optional
+
+def main():
+    results = run_eval(CASES, generate, SCORERS, judge=judge)
+    rate = write_report(results, eval_name="<name>")   # writes results/<name>/<ts>.md
+    return 0 if rate >= PASS_BAR else 1
+```
+
+The harness (running cases, catching generation errors, aggregating, the
+scoreboard) is reused as-is. Only the dataset, `generate`, and scorers are
+bespoke — which is inherent: each eval defines its own "what good looks like".
+
+## Adapting this to the agentic evals (`suggest` / `changelog`)
+
+Same one-file shape; only two things grow:
+
+1. **Input/output.** Instead of a diff → subject line, an agentic case is a
+  *repo state + task* → the agent's effect (final text, files written, and the
+   `TraceStep` list the agent already records on `agent.steps`). Run the agent in
+   a throwaway git repo (see how `tests/` build temp repos) so file writes are
+   safe to inspect and discard.
+2. **Scorers get richer.** Beyond output-quality scorers, evaluate the *process*
+  using the trace:
+  - did it call the expected tools (e.g. `changelog` should `git log` a tag
+  range and `git show` at least one commit)?
+  - did it stay within `max_iterations` / a tool-call budget?
+  - did it avoid hallucinated numbers (assert no unverified figures appear) —
+  the exact failure the changelog prompt guardrail targets?
+  - did the written `CHANGELOG.md` parse, and prepend rather than clobber?
+
+Keep deterministic scorers first (cheap, sharp), and reserve the LLM judge for  
+the genuinely subjective properties. Make the judge return structured JSON and  
+give it a rubric, as `scorers.llm_judge` does.
+

messygit-0.4.5/evals/__init__.py

@@ -0,0 +1,11 @@
+"""Model-output evals for messygit.
+
+Unlike the unit tests in `tests/` (which simulate the model and run offline),
+evals call the REAL model and score its output. They cost tokens and need an
+ANTHROPIC_API_KEY, so they are NOT part of the `pytest` run — launch them
+explicitly with `python -m evals.commit_eval`.
+
+`commit_eval` is the worked reference: a graded dataset, programmatic scorers,
+an LLM-as-judge, and an aggregated scoreboard — the same shape you can reuse to
+evaluate the agentic `suggest`/`changelog` commands.
+"""