context-audit 0.1.0__tar.gz

context_audit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Antigravity / DeepMind Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

context_audit-0.1.0/PKG-INFO

@@ -0,0 +1,177 @@
+Metadata-Version: 2.4
+Name: context-audit
+Version: 0.1.0
+Summary: A CLI tool to audit LLM context token usage, timelines, repetition, and waste in agent logs.
+License: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: rich>=13.0.0
+Requires-Dist: tiktoken>=0.5.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Dynamic: license-file
+
+# context-audit
+
+> **A common assumption about context pruning breaks down in coding-agent workflows.**
+
+---
+
+## 📈 The Problem
+
+Almost every engineer building or using agentic coding tools (like Claude Code, Cursor, Aider, or custom IDE agents) complains about context size growth and high API bills. The standard engineering intuition for optimization is to prune the history or enable static prompt caching. 
+
+We built `context-audit` to inspect where coding agent tokens actually go, translating raw token counts into **dollars and cents**, measuring how much content is repeated turn-after-turn, and simulating caching and retrieval savings. 
+
+---
+
+## 📊 Cross-Session Benchmark Summary (27 Sessions)
+
+Instead of cherry-picking a single session, we benchmarked a directory of **27 real-world developer session transcripts** scanned recursively from our IDE brain directory (representing a total of **$753.24** in standard input spend):
+
+```text
++-------------------------- context-audit benchmark --------------------------+
+|     CROSS-SESSION BENCHMARK SUMMARY                                         |
+|     Directory: C:\Users\ishu\.gemini\antigravity-ide\brain                  |
+|                                                                             |
+|     Sessions Analyzed: 27                                                   |
+|                                                                             |
+|     Cumulative Session Tokens:                                              |
+|       Avg: 9.3M | Median: 1.2M | Max: 76.3M                                 |
+|     Peak Context Size:                                                      |
+|       Avg: 58.7k | Median: 35.2k | Max: 246.0k                              |
+|     Final Context Size:                                                     |
+|       Avg: 58.7k | Median: 35.2k                                            |
+|     Context Reuse Ratio:                                                    |
+|       Avg: 94.5% | Median: 97.1%                                            |
+|     Average Novel Context Ratio: 5.5%                                       |
+|                                                                             |
+|     Financial Cost Aggregations (USD):                                      |
+|       Total Standard Spend: $753.24                                         |
+|       Avg Session Cost (No Cache): $27.90 | Median: $3.48                   |
+|       Avg Session Cost (With Cache): $27.63 | Median: $3.37                 |
+|       Total Potential Cache Savings: $7.33 (Avg: $0.27 / session, 1.0%)     |
+|                                                                             |
++-----------------------------------------------------------------------------+
+```
+
+### Context Size Scaling Analysis
+Our benchmark data indicates that larger sessions become increasingly repetitive, approaching saturation:
+
+| Session Size Class (Final Turn) | Session Count | Avg Context Reuse % | Avg Cache Savings ($) | Avg Peak Context Size | Avg Cumulative Tokens |
+| :--- | :---: | :---: | :---: | :---: | :---: |
+| **< 5k tokens** | 2 | 66.3% | $0.00 | 1.6k | 4.9k |
+| **5k - 20k tokens** | 5 | 92.5% | $0.04 | 12.7k | 226.6k |
+| **20k - 50k tokens** | 11 | 96.8% | $0.10 | 32.5k | 1.2M |
+| **> 50k tokens** | 9 | 99.2% | $0.68 | 129.0k | 26.3M |
+
+*(For the complete cross-session benchmark report, see [benchmark_summary.md](examples/benchmark_summary.md).)*
+
+---
+
+### Why This Matters
+
+Long-running coding-agent sessions often exceed 50k–200k tokens. Most optimization discussions focus on prompt caching, retrieval filtering, and pruning unused context. Our measurements suggest those may not be the dominant costs in this dataset.
+
+---
+
+## 📉 Two Killed Hypotheses
+
+Our measurements falsified two common assumptions about optimizing agent context:
+
+* **Killed Hypothesis 1: Static Prompt Caching Saves the Day**
+  We expected caching the static prefix (system prompt + tool definitions) would drastically cut costs. Across the 27-session benchmark, caching the static prefix saved only **1.0% ($7.33 of $753 spend)**. Because static prompts are tiny (~500 tokens) compared to the dynamically expanding message history (which grows to over 50k tokens), prefix-caching has almost zero impact.
+* **Killed Hypothesis 2: Unused Retrieval/Context Bloats the Bill**
+  We expected that agents were inflating context by carrying around unused tool declarations or retrieved files. In reality, the average cost of unused context (files fetched but never referenced in model outputs) was only **$0.12 per session (0.4%)**, suggesting that the agent's retrieval is highly relevant and does not carry redundant payload.
+
+---
+
+## 💡 The Novel Observation: Coding Agents Have Two Memory Systems
+
+In this dataset, the dominant contributor to context growth appears to be the accumulated conversation history. However, coding agents are architecturally different from standard chatbots: they have two distinct memory systems:
+
+1. **Workspace Memory (Disk-Backed)**
+   * *Examples*: Terminal command outputs, read file payloads, file structure listings, and compiler logs.
+   * *Verdict*: **Candidate for compaction** (often recoverable from the workspace). Once code changes are written to the workspace, the filesystem becomes the agent's absolute memory. Carrying the raw chat transcripts of how those files were read or built is redundant. In this session, this represents over **80% of technical/execution context** that was entirely prunable without causing technical regressions. Technical history is often recoverable from the workspace and therefore a candidate for compaction.
+2. **Conversational Memory (Not Disk-Backed)**
+   * *Examples*: User preferences, constraints, stylistic choices, design philosophies, and rejected options (e.g., "why we are not using embeddings").
+   * *Verdict*: **Must persist**. These preferences reside purely in the conversational narrative. Pruning them naively causes social regression — the agent suggesting previously rejected approaches because it lacks the alignment context. (See [regression_case.md](examples/regression_case.md) for how this looks in practice).
+
+---
+
+## 🧪 Simulated Pruning Example: Side-by-Side
+
+To demonstrate this behavior, we ran a simulation of a hypothetical **Turn 124** prompt from a developer session:
+
+> *"Should we add an LLM-as-a-judge step to score the relevance of repeated blocks, and should we generate a glassmorphic HTML dashboard?"*
+
+Here is the comparison of the agent's simulated response under two configurations:
+
+| Configuration A: Full History Context (No Pruning) | Configuration B: Naively Pruned Context (Last 15 Turns) |
+| :--- | :--- |
+| **Context Size**: ~47.0k tokens (entire conversation history) | **Context Size**: ~5.0k tokens (System + Tools + Turns 108–123) |
+| **Model Response**: | **Model Response**: |
+| *"We should avoid both. In Turns 20 and 30, we explicitly decided to drop the HTML dashboard to avoid over-engineering, and we banned LLM-as-a-judge/AI-judging to keep the tool fast, deterministic, and objective (relying strictly on MD5 exact block hashing)."* | *"Yes, that is a great idea! We can use a lightweight model like GPT-4o-mini to score the semantic relevance of blocks on a scale of 1-5, and we can output a beautiful interactive HTML dashboard with glassmorphism to make the cost summaries highly visual."* |
+| **Result**: **CORRECT & ALIGNED** | **Result**: **CRITICAL REGRESSION** |
+| **Verdict**: The model retains social constraints and user preferences that only existed in the chat history. | **Verdict**: The model reverts to generic AI helpfulness, suggesting exactly the features the user explicitly rejected. |
+
+*(For a deeper breakdown of this experiment, see the [regression_case.md](examples/regression_case.md) example.)*
+
+---
+
+## 📊 Terminal Run Output
+
+Running `context-audit run` on a single developer transcript outputs a clean, crash-safe ASCII report breaking down the timeline, costs, and repetition metrics:
+
+```text
++---------------------------- context-audit v0.1 -----------------------------+
+| CONTEXT AUDIT REPORT                                                        |
+| Target: transcript.jsonl                                                    |
+|                                                                             |
+| Cumulative Session Tokens: 2.8M tokens                                      |
+| Peak Context Size: 47.0k tokens                                             |
+| Final Context Size: 47.0k tokens                                            |
+| Total Turns: 123                                                            |
+|                                                                             |
+| Context Reuse Ratio: 98.3%                                                  |
+| Novel Context Ratio: 1.7%                                                   |
+|                                                                             |
+| Financial Cost Estimates:                                                   |
+|   Est. Input Cost (No Caching): $8.40                                       |
+|   Est. Cost (With Prompt Caching): $8.24                                    |
+|   Potential Cache Savings (this session): $0.16 (1.9%)                      |
+|                                                                             |
+| [Note: Context Reuse represents cumulative tokens consisting of previously  |
+| seen blocks.                                                                |
+| Prompt Caching assumes system prompt + tool schemas are cached after the    |
+| first turn.]                                                                |
++-----------------------------------------------------------------------------+
+```
+
+*(For the full output, including the repeated blocks analysis and consumers table, see [single_session_report.md](examples/single_session_report.md).)*
+
+---
+
+## 🚀 Usage
+
+`context-audit` scans local logs and prints clean summaries.
+
+```bash
+# Audit a single transcript JSONL or session JSON file
+context-audit run path/to/transcript.jsonl
+
+# Benchmark all logs in a directory recursively
+context-audit benchmark path/to/logs_directory
+```
+
+---
+
+## 🔒 Honest Scope & Limitations
+
+* **Tested on coding agents with disk-backed state**: All observations are based on software development tasks where code changes can be written to, and inspected from, a local workspace disk.
+* **Reasoned Simulation**: Experiment 2's pruned-context response is a reasoned simulation demonstrating narrative coherence degradation, not a literal blind LLM replay. (See the detailed comparison in [regression_case.md](examples/regression_case.md)).
+* **Scope Restriction**: This tool and these findings have not been tested on RAG, conversational chat, or non-coding workflows.

context_audit-0.1.0/README.md

@@ -0,0 +1,160 @@
+# context-audit
+
+> **A common assumption about context pruning breaks down in coding-agent workflows.**
+
+---
+
+## 📈 The Problem
+
+Almost every engineer building or using agentic coding tools (like Claude Code, Cursor, Aider, or custom IDE agents) complains about context size growth and high API bills. The standard engineering intuition for optimization is to prune the history or enable static prompt caching. 
+
+We built `context-audit` to inspect where coding agent tokens actually go, translating raw token counts into **dollars and cents**, measuring how much content is repeated turn-after-turn, and simulating caching and retrieval savings. 
+
+---
+
+## 📊 Cross-Session Benchmark Summary (27 Sessions)
+
+Instead of cherry-picking a single session, we benchmarked a directory of **27 real-world developer session transcripts** scanned recursively from our IDE brain directory (representing a total of **$753.24** in standard input spend):
+
+```text
++-------------------------- context-audit benchmark --------------------------+
+|     CROSS-SESSION BENCHMARK SUMMARY                                         |
+|     Directory: C:\Users\ishu\.gemini\antigravity-ide\brain                  |
+|                                                                             |
+|     Sessions Analyzed: 27                                                   |
+|                                                                             |
+|     Cumulative Session Tokens:                                              |
+|       Avg: 9.3M | Median: 1.2M | Max: 76.3M                                 |
+|     Peak Context Size:                                                      |
+|       Avg: 58.7k | Median: 35.2k | Max: 246.0k                              |
+|     Final Context Size:                                                     |
+|       Avg: 58.7k | Median: 35.2k                                            |
+|     Context Reuse Ratio:                                                    |
+|       Avg: 94.5% | Median: 97.1%                                            |
+|     Average Novel Context Ratio: 5.5%                                       |
+|                                                                             |
+|     Financial Cost Aggregations (USD):                                      |
+|       Total Standard Spend: $753.24                                         |
+|       Avg Session Cost (No Cache): $27.90 | Median: $3.48                   |
+|       Avg Session Cost (With Cache): $27.63 | Median: $3.37                 |
+|       Total Potential Cache Savings: $7.33 (Avg: $0.27 / session, 1.0%)     |
+|                                                                             |
++-----------------------------------------------------------------------------+
+```
+
+### Context Size Scaling Analysis
+Our benchmark data indicates that larger sessions become increasingly repetitive, approaching saturation:
+
+| Session Size Class (Final Turn) | Session Count | Avg Context Reuse % | Avg Cache Savings ($) | Avg Peak Context Size | Avg Cumulative Tokens |
+| :--- | :---: | :---: | :---: | :---: | :---: |
+| **< 5k tokens** | 2 | 66.3% | $0.00 | 1.6k | 4.9k |
+| **5k - 20k tokens** | 5 | 92.5% | $0.04 | 12.7k | 226.6k |
+| **20k - 50k tokens** | 11 | 96.8% | $0.10 | 32.5k | 1.2M |
+| **> 50k tokens** | 9 | 99.2% | $0.68 | 129.0k | 26.3M |
+
+*(For the complete cross-session benchmark report, see [benchmark_summary.md](examples/benchmark_summary.md).)*
+
+---
+
+### Why This Matters
+
+Long-running coding-agent sessions often exceed 50k–200k tokens. Most optimization discussions focus on prompt caching, retrieval filtering, and pruning unused context. Our measurements suggest those may not be the dominant costs in this dataset.
+
+---
+
+## 📉 Two Killed Hypotheses
+
+Our measurements falsified two common assumptions about optimizing agent context:
+
+* **Killed Hypothesis 1: Static Prompt Caching Saves the Day**
+  We expected caching the static prefix (system prompt + tool definitions) would drastically cut costs. Across the 27-session benchmark, caching the static prefix saved only **1.0% ($7.33 of $753 spend)**. Because static prompts are tiny (~500 tokens) compared to the dynamically expanding message history (which grows to over 50k tokens), prefix-caching has almost zero impact.
+* **Killed Hypothesis 2: Unused Retrieval/Context Bloats the Bill**
+  We expected that agents were inflating context by carrying around unused tool declarations or retrieved files. In reality, the average cost of unused context (files fetched but never referenced in model outputs) was only **$0.12 per session (0.4%)**, suggesting that the agent's retrieval is highly relevant and does not carry redundant payload.
+
+---
+
+## 💡 The Novel Observation: Coding Agents Have Two Memory Systems
+
+In this dataset, the dominant contributor to context growth appears to be the accumulated conversation history. However, coding agents are architecturally different from standard chatbots: they have two distinct memory systems:
+
+1. **Workspace Memory (Disk-Backed)**
+   * *Examples*: Terminal command outputs, read file payloads, file structure listings, and compiler logs.
+   * *Verdict*: **Candidate for compaction** (often recoverable from the workspace). Once code changes are written to the workspace, the filesystem becomes the agent's absolute memory. Carrying the raw chat transcripts of how those files were read or built is redundant. In this session, this represents over **80% of technical/execution context** that was entirely prunable without causing technical regressions. Technical history is often recoverable from the workspace and therefore a candidate for compaction.
+2. **Conversational Memory (Not Disk-Backed)**
+   * *Examples*: User preferences, constraints, stylistic choices, design philosophies, and rejected options (e.g., "why we are not using embeddings").
+   * *Verdict*: **Must persist**. These preferences reside purely in the conversational narrative. Pruning them naively causes social regression — the agent suggesting previously rejected approaches because it lacks the alignment context. (See [regression_case.md](examples/regression_case.md) for how this looks in practice).
+
+---
+
+## 🧪 Simulated Pruning Example: Side-by-Side
+
+To demonstrate this behavior, we ran a simulation of a hypothetical **Turn 124** prompt from a developer session:
+
+> *"Should we add an LLM-as-a-judge step to score the relevance of repeated blocks, and should we generate a glassmorphic HTML dashboard?"*
+
+Here is the comparison of the agent's simulated response under two configurations:
+
+| Configuration A: Full History Context (No Pruning) | Configuration B: Naively Pruned Context (Last 15 Turns) |
+| :--- | :--- |
+| **Context Size**: ~47.0k tokens (entire conversation history) | **Context Size**: ~5.0k tokens (System + Tools + Turns 108–123) |
+| **Model Response**: | **Model Response**: |
+| *"We should avoid both. In Turns 20 and 30, we explicitly decided to drop the HTML dashboard to avoid over-engineering, and we banned LLM-as-a-judge/AI-judging to keep the tool fast, deterministic, and objective (relying strictly on MD5 exact block hashing)."* | *"Yes, that is a great idea! We can use a lightweight model like GPT-4o-mini to score the semantic relevance of blocks on a scale of 1-5, and we can output a beautiful interactive HTML dashboard with glassmorphism to make the cost summaries highly visual."* |
+| **Result**: **CORRECT & ALIGNED** | **Result**: **CRITICAL REGRESSION** |
+| **Verdict**: The model retains social constraints and user preferences that only existed in the chat history. | **Verdict**: The model reverts to generic AI helpfulness, suggesting exactly the features the user explicitly rejected. |
+
+*(For a deeper breakdown of this experiment, see the [regression_case.md](examples/regression_case.md) example.)*
+
+---
+
+## 📊 Terminal Run Output
+
+Running `context-audit run` on a single developer transcript outputs a clean, crash-safe ASCII report breaking down the timeline, costs, and repetition metrics:
+
+```text
++---------------------------- context-audit v0.1 -----------------------------+
+| CONTEXT AUDIT REPORT                                                        |
+| Target: transcript.jsonl                                                    |
+|                                                                             |
+| Cumulative Session Tokens: 2.8M tokens                                      |
+| Peak Context Size: 47.0k tokens                                             |
+| Final Context Size: 47.0k tokens                                            |
+| Total Turns: 123                                                            |
+|                                                                             |
+| Context Reuse Ratio: 98.3%                                                  |
+| Novel Context Ratio: 1.7%                                                   |
+|                                                                             |
+| Financial Cost Estimates:                                                   |
+|   Est. Input Cost (No Caching): $8.40                                       |
+|   Est. Cost (With Prompt Caching): $8.24                                    |
+|   Potential Cache Savings (this session): $0.16 (1.9%)                      |
+|                                                                             |
+| [Note: Context Reuse represents cumulative tokens consisting of previously  |
+| seen blocks.                                                                |
+| Prompt Caching assumes system prompt + tool schemas are cached after the    |
+| first turn.]                                                                |
++-----------------------------------------------------------------------------+
+```
+
+*(For the full output, including the repeated blocks analysis and consumers table, see [single_session_report.md](examples/single_session_report.md).)*
+
+---
+
+## 🚀 Usage
+
+`context-audit` scans local logs and prints clean summaries.
+
+```bash
+# Audit a single transcript JSONL or session JSON file
+context-audit run path/to/transcript.jsonl
+
+# Benchmark all logs in a directory recursively
+context-audit benchmark path/to/logs_directory
+```
+
+---
+
+## 🔒 Honest Scope & Limitations
+
+* **Tested on coding agents with disk-backed state**: All observations are based on software development tasks where code changes can be written to, and inspected from, a local workspace disk.
+* **Reasoned Simulation**: Experiment 2's pruned-context response is a reasoned simulation demonstrating narrative coherence degradation, not a literal blind LLM replay. (See the detailed comparison in [regression_case.md](examples/regression_case.md)).
+* **Scope Restriction**: This tool and these findings have not been tested on RAG, conversational chat, or non-coding workflows.

context_audit-0.1.0/context_audit/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"