tokenaudit 0.2.0__tar.gz → 0.4.0__tar.gz

{tokenaudit-0.2.0 → tokenaudit-0.4.0}/CHANGELOG.md

@@ -1,6 +1,57 @@
 # Changelog
 
-## Unreleased
+## 0.4.0 — 2026-11-12
+
+Upgrading adds findings to traces with these patterns, and changes the
+impact of some existing ones: re-baseline `tokenaudit check` after upgrading.
+
+New detectors, each with a positive corpus fixture and near-miss negatives:
+
+- `misordered_shared_context`: the same reports or documents sent to several
+  agents, each behind its own role text, so the prefix cache never reuses
+  them.
+- `cross_session_drift`: a date, user or ticker near the start of a long
+  prompt, which keeps it from being reused across sessions.
+- `tool_schema_bloat`: tool definitions sent on every request and never
+  called by any request carrying the same tool set.
+- `missing_cache_markers`: a provider that only caches on request
+  (Anthropic), a session with no markers and no cache activity, and requests
+  re-sending a prefix an earlier one sent within the cache lifetime.
+
+The first two find the TradingAgents problems the case study diagnosed by
+hand. The captured runs, previously reported as clean, now show both
+findings and a 25% health floor (was 100%).
+
+Pricing:
+
+- Every finding is priced with its model's read and write rates from the
+  rulebook. A request that wrote the cache is charged the write premium
+  (1.25x on Anthropic's 5-minute cache) instead of full price, and the
+  assumption line names the rates used.
+- `diagnose` and the HTML report show a dollar range when the rulebook has a
+  verified input price for the model.
+
+Ingest:
+
+- Raw provider logs: JSONL or a JSON array of logged request/response pairs
+  from the OpenAI Responses, OpenAI Chat Completions and Anthropic Messages
+  APIs. Anthropic's exclusive token counts are converted; records without
+  timestamps are flagged and time-based detectors abstain.
+- Tool definitions (name and size) and the tools each response called, from
+  OpenInference, `gen_ai.tool.definitions` in current OTel GenAI, and raw
+  logs.
+- Per-line digests kept in memory for the new detectors; `ingest` output
+  leaves them out.
+
+Also:
+
+- `diagnose` and `check` report a cross-session section when given more
+  than one session.
+- Demo: a `render.yaml` blueprint for one-click deploys, and raw provider
+  logs are accepted as uploads.
+- The capture runner records Anthropic runs (`--provider anthropic`);
+  `experiments/tradingagents-claude/` is ready to run with an API key.
+- The corpus has 30 fixtures (was 16).
 
 ## 0.2.0 — 2026-11-10
 

{tokenaudit-0.2.0 → tokenaudit-0.4.0}/DECISIONS.md

@@ -39,3 +39,20 @@ Three calls made together for the 0.1.0 release, all in service of the same prin
 **The cache health score is `realized / achievable` savings** — realized cache savings over realized-plus-what-the-findings-quantify. Both halves are computed the same way, so there are no weights to defend; overlapping findings make the denominator conservative, so the score is presented as a floor and says so.
 
 **Findings carry a corpus record, not a confidence number.** A precision float computed from a 16-fixture corpus would be theater. The package ships the raw record instead — "3/4 planted cases found, 0 false alarms in N negatives" — generated by the eval harness and drift-tested, so what appears next to a finding is exactly what was measured.
+
+## 007 — Line-level digests, and detectors that look across sessions (2026-11-12)
+
+**Decision:** ingest keeps a truncated SHA-256 and length for every line of every message, in memory only, next to the existing whole-message digest. Two detectors use them: `misordered_shared_context` within a session and `cross_session_drift` across sessions.
+
+**Why:** the TradingAgents diagnosis was done by hand, because whole-message hashes can't see the same reports inside two different messages. The tool called those runs clean while the case study showed a 4x improvement. Lines are fine enough to find shared content and to pin a date to one line, and coarse enough to stay fast.
+
+**Boundaries:** line digests never leave the process (`tokenaudit ingest` drops them), because a hash of a short line is easy to guess. Cross-session findings count toward a CI gate's findings but not toward the health floor, since whether sessions share a cache depends on when they ran; the low impact bound only counts sessions that started within the model's cache lifetime of each other. Each new detector defers to an existing one where they overlap: a short edit near the start of a prompt is prefix drift, and a value that changes on every call within a session is prefix drift too.
+
+## 008 — Price against the fix, per model; new detectors yield to older ones (2026-11-12)
+
+**Decision:** a finding's impact is what the affected tokens cost minus what they would have cost under the fix, using the model's read and write multipliers from the rulebook. A request that wrote the cache is charged the write premium, not full price. Dollars appear only when the rulebook has a verified input price; tiered prices (Haiku 5.5) stay in tokens.
+
+**Why:** a flat discount overstated waste on models with cheaper reads and ignored that a miss on Anthropic often costs more than a plain request, because it writes the cache. Keeping the rates in one module means a rulebook update reprices every detector at once.
+
+**Boundaries:** `tool_schema_bloat` prints tool names, which are identifiers a developer chose rather than prompt text. It counts a tool as used if any request with the same tool set called it, so a shared registry where every tool is used by some agent is not flagged; telling agents apart needs more than a trace carries. `missing_cache_markers` only counts block-aligned prefixes repeated within the cache lifetime, since a breakpoint sits at a block's end and a later repeat would have missed anyway. It leaves growing conversations to `context_bloat` and byte-identical re-sends to `retry_storm`, whose fixes fit those cases better.
+

{tokenaudit-0.2.0 → tokenaudit-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tokenaudit
-Version: 0.2.0
+Version: 0.4.0
 Summary: Root-cause diagnosis of wasted LLM tokens, from the traces your agents already emit
 Project-URL: Repository, https://github.com/yashmyeole/tokenaudit
 Project-URL: Issues, https://github.com/yashmyeole/tokenaudit/issues
@@ -43,7 +43,8 @@ It works: diagnosing one of those documented cases produced a 9-file patch that
     tokenaudit diagnose --sample          # diagnose the bundled demo trace
     tokenaudit diagnose mytraces/ --html report.html
 
-Zero runtime dependencies; reads OTLP/JSON files and OTel SDK JSONL. Try it
+Zero runtime dependencies; reads OTLP/JSON files, OTel SDK JSONL, and raw
+OpenAI or Anthropic request/response logs. Try it
 on this repo's real captured runs: `tokenaudit diagnose corpus/captured/runs`.
 Prefer a browser? The [demo app](demo/) diagnoses a dropped trace; uploads
 are parsed in memory and never stored.
@@ -51,7 +52,7 @@ are parsed in memory and never stored.
 The plan, in one line each:
 
 - Ingest standard traces — OpenTelemetry GenAI in all three of its cache-field naming generations, OpenInference, raw provider SDK logs — into one normalized accounting model.
-- Run a library of deterministic detectors for the silent failure modes: prefix drift, sub-minimum cache blocks, model switches, TTL expiry, breakpoint and lookback limits, tool-schema churn and bloat, context bloat, retry storms.
+- Run a library of deterministic detectors for the silent failure modes: prefix drift, sub-minimum cache blocks, model switches, TTL expiry, breakpoint and lookback limits, tool-schema churn and bloat, context bloat, retry storms, shared content sent behind different openings, per-session values that stop a prompt being reused across sessions, and prompts that never ask a cache-on-request provider to cache them.
 - Price each finding against a counterfactual — what the bill would have been under the fix — as a bounded range, never a point guess.
 - Publish per-detector precision and recall against a labeled corpus, so you know how much to trust each finding.
 
@@ -119,10 +120,12 @@ If you know a tool that already does this, please open an issue — I would genu
 - [x] CI cost-regression mode: fail a build that tanks the cache hit rate ([docs/ci.md](docs/ci.md))
 - [x] Demo app: drag a trace in, get the report ([demo/](demo/))
 - [x] Case studies on public agents, with reproduction commands ([TradingAgents: 4.3× cache hit rate](docs/case-studies/tradingagents.md))
-- [ ] Detectors for misordered shared context and cross-run prefix drift
-- [ ] Counterfactual pricing with per-model read rates and dollar ranges
-- [ ] Raw provider SDK log ingest, tool-schema bloat detector
-- [ ] Public demo deployment, case study on an Anthropic-model agent
+- [x] Detectors for misordered shared context and cross-session drift: they now find the TradingAgents problems on their own ([case study, section 6](docs/case-studies/tradingagents.md#6-what-tokenaudit-reports-now))
+- [x] Counterfactual pricing with per-model read and write rates and dollar ranges
+- [x] Raw provider SDK log ingest (OpenAI Responses, Chat Completions, Anthropic Messages), tool-schema bloat detector
+- [x] A detector for prompts that never ask Anthropic to cache them
+- [ ] Public demo deployment (`render.yaml` is ready; see [demo/](demo/))
+- [ ] Case study on an Anthropic-model agent (the harness is ready: [experiments/tradingagents-claude](experiments/tradingagents-claude/))
 
 ## Development
 

{tokenaudit-0.2.0 → tokenaudit-0.4.0}/README.md

@@ -16,7 +16,8 @@ It works: diagnosing one of those documented cases produced a 9-file patch that
     tokenaudit diagnose --sample          # diagnose the bundled demo trace
     tokenaudit diagnose mytraces/ --html report.html
 
-Zero runtime dependencies; reads OTLP/JSON files and OTel SDK JSONL. Try it
+Zero runtime dependencies; reads OTLP/JSON files, OTel SDK JSONL, and raw
+OpenAI or Anthropic request/response logs. Try it
 on this repo's real captured runs: `tokenaudit diagnose corpus/captured/runs`.
 Prefer a browser? The [demo app](demo/) diagnoses a dropped trace; uploads
 are parsed in memory and never stored.
@@ -24,7 +25,7 @@ are parsed in memory and never stored.
 The plan, in one line each:
 
 - Ingest standard traces — OpenTelemetry GenAI in all three of its cache-field naming generations, OpenInference, raw provider SDK logs — into one normalized accounting model.
-- Run a library of deterministic detectors for the silent failure modes: prefix drift, sub-minimum cache blocks, model switches, TTL expiry, breakpoint and lookback limits, tool-schema churn and bloat, context bloat, retry storms.
+- Run a library of deterministic detectors for the silent failure modes: prefix drift, sub-minimum cache blocks, model switches, TTL expiry, breakpoint and lookback limits, tool-schema churn and bloat, context bloat, retry storms, shared content sent behind different openings, per-session values that stop a prompt being reused across sessions, and prompts that never ask a cache-on-request provider to cache them.
 - Price each finding against a counterfactual — what the bill would have been under the fix — as a bounded range, never a point guess.
 - Publish per-detector precision and recall against a labeled corpus, so you know how much to trust each finding.
 
@@ -92,10 +93,12 @@ If you know a tool that already does this, please open an issue — I would genu
 - [x] CI cost-regression mode: fail a build that tanks the cache hit rate ([docs/ci.md](docs/ci.md))
 - [x] Demo app: drag a trace in, get the report ([demo/](demo/))
 - [x] Case studies on public agents, with reproduction commands ([TradingAgents: 4.3× cache hit rate](docs/case-studies/tradingagents.md))
-- [ ] Detectors for misordered shared context and cross-run prefix drift
-- [ ] Counterfactual pricing with per-model read rates and dollar ranges
-- [ ] Raw provider SDK log ingest, tool-schema bloat detector
-- [ ] Public demo deployment, case study on an Anthropic-model agent
+- [x] Detectors for misordered shared context and cross-session drift: they now find the TradingAgents problems on their own ([case study, section 6](docs/case-studies/tradingagents.md#6-what-tokenaudit-reports-now))
+- [x] Counterfactual pricing with per-model read and write rates and dollar ranges
+- [x] Raw provider SDK log ingest (OpenAI Responses, Chat Completions, Anthropic Messages), tool-schema bloat detector
+- [x] A detector for prompts that never ask Anthropic to cache them
+- [ ] Public demo deployment (`render.yaml` is ready; see [demo/](demo/))
+- [ ] Case study on an Anthropic-model agent (the harness is ready: [experiments/tradingagents-claude](experiments/tradingagents-claude/))
 
 ## Development
 

{tokenaudit-0.2.0 → tokenaudit-0.4.0}/corpus/captured/runner.py

@@ -1,10 +1,11 @@
 """Capture one real TradingAgents run as OpenTelemetry spans in a JSONL file.
 
 Zero-modification capture: TradingAgents' code is untouched. Instrumentation
-wraps the OpenAI SDK and LangChain at process level — the same zero-integration
-posture tokenaudit itself takes. Spans are written line-by-line using the OTel
-Python SDK's own ``ReadableSpan.to_json()`` rendering, with no custom mapping,
-so the captured dialect is exactly what the SDK produces in the wild.
+wraps the provider SDK (OpenAI or Anthropic) and LangChain at process level,
+the same zero-integration posture tokenaudit itself takes. Spans are written
+line-by-line using the OTel Python SDK's own ``ReadableSpan.to_json()``
+rendering, with no custom mapping, so the captured dialect is exactly what
+the SDK produces in the wild.
 
 This script is executed by ``capture_tradingagents.sh`` inside a throwaway
 virtualenv on the machine that holds the API key. It is not part of the
@@ -66,16 +67,22 @@ def build_provider(trace_file: Path, run_id: str, ta_commit: str) -> TracerProvi
     return provider
 
 
-def instrument(provider: TracerProvider) -> list[str]:
+def instrument(provider: TracerProvider, llm_provider: str = "openai") -> list[str]:
     """Attach OpenInference instrumentors; return the ones that activated."""
     active: list[str] = []
     try:
-        from openinference.instrumentation.openai import OpenAIInstrumentor
+        if llm_provider == "anthropic":
+            from openinference.instrumentation.anthropic import AnthropicInstrumentor
 
-        OpenAIInstrumentor().instrument(tracer_provider=provider)
-        active.append("openinference-instrumentation-openai")
+            AnthropicInstrumentor().instrument(tracer_provider=provider)
+            active.append("openinference-instrumentation-anthropic")
+        else:
+            from openinference.instrumentation.openai import OpenAIInstrumentor
+
+            OpenAIInstrumentor().instrument(tracer_provider=provider)
+            active.append("openinference-instrumentation-openai")
     except Exception as exc:
-        print(f"warning: OpenAI instrumentor not active: {exc}", file=sys.stderr)
+        print(f"warning: {llm_provider} SDK instrumentor not active: {exc}", file=sys.stderr)
     try:
         from openinference.instrumentation.langchain import LangChainInstrumentor
 
@@ -88,12 +95,14 @@ def instrument(provider: TracerProvider) -> list[str]:
     return active
 
 
-def run_analysis(ticker: str, date: str, deep: str, quick: str, rounds: int) -> str:
+def run_analysis(
+    ticker: str, date: str, deep: str, quick: str, rounds: int, llm_provider: str = "openai"
+) -> str:
     from tradingagents.default_config import DEFAULT_CONFIG
     from tradingagents.graph.trading_graph import TradingAgentsGraph
 
     config = dict(DEFAULT_CONFIG)
-    config["llm_provider"] = "openai"
+    config["llm_provider"] = llm_provider
     config["deep_think_llm"] = deep
     config["quick_think_llm"] = quick
     config["max_debate_rounds"] = rounds
@@ -119,6 +128,7 @@ def main() -> None:
     parser.add_argument("--deep", default="gpt-5.4-mini")
     parser.add_argument("--quick", default="gpt-5.4-mini")
     parser.add_argument("--rounds", type=int, default=1)
+    parser.add_argument("--provider", choices=["openai", "anthropic"], default="openai")
     parser.add_argument("--ta-commit", default="unknown")
     parser.add_argument("--smoke", action="store_true", help="plumbing test, no API calls")
     args = parser.parse_args()
@@ -134,6 +144,7 @@ def main() -> None:
         "smoke": args.smoke,
         "ticker": args.ticker,
         "trade_date": args.date,
+        "llm_provider": args.provider,
         "models": {"deep_think": args.deep, "quick_think": args.quick},
         "max_debate_rounds": args.rounds,
         "tradingagents_commit": args.ta_commit,
@@ -146,8 +157,11 @@ def main() -> None:
                 "tradingagents",
                 "opentelemetry-sdk",
                 "openinference-instrumentation-openai",
+                "openinference-instrumentation-anthropic",
                 "openinference-instrumentation-langchain",
                 "openai",
+                "anthropic",
+                "langchain-anthropic",
                 "langchain-core",
             )
         },
@@ -164,8 +178,10 @@ def main() -> None:
         manifest["instrumentors"] = []
         decision = "smoke"
     else:
-        manifest["instrumentors"] = instrument(provider)
-        decision = run_analysis(args.ticker, args.date, args.deep, args.quick, args.rounds)
+        manifest["instrumentors"] = instrument(provider, args.provider)
+        decision = run_analysis(
+            args.ticker, args.date, args.deep, args.quick, args.rounds, args.provider
+        )
         (args.out / "decision.txt").write_text(decision + "\n", encoding="utf-8")
 
     provider.force_flush()

{tokenaudit-0.2.0 → tokenaudit-0.4.0}/corpus/synthetic/fixtures/breakpoint_limit.label.json

@@ -11,11 +11,15 @@
   ],
   "must_not_contain": [
     "context_bloat",
+    "cross_session_drift",
     "lookback_overflow",
+    "misordered_shared_context",
+    "missing_cache_markers",
     "model_switch",
     "prefix_drift",
     "retry_storm",
     "sub_minimum",
+    "tool_schema_bloat",
     "tool_schema_churn",
     "ttl_expiry"
   ],

{tokenaudit-0.2.0 → tokenaudit-0.4.0}/corpus/synthetic/fixtures/context_bloat.label.json

@@ -11,11 +11,15 @@
   ],
   "must_not_contain": [
     "breakpoint_limit",
+    "cross_session_drift",
     "lookback_overflow",
+    "misordered_shared_context",
+    "missing_cache_markers",
     "model_switch",
     "prefix_drift",
     "retry_storm",
     "sub_minimum",
+    "tool_schema_bloat",
     "tool_schema_churn",
     "ttl_expiry"
   ],

tokenaudit-0.4.0/corpus/synthetic/fixtures/cross_session_distinct.label.json

@@ -0,0 +1,36 @@
+{
+  "fixture": "cross_session_distinct",
+  "schema": "otel-genai-current",
+  "provider_shape": "openai",
+  "models": [
+    "gpt-5.6"
+  ],
+  "content_captured": true,
+  "contains": [],
+  "must_not_contain": [
+    "breakpoint_limit",
+    "context_bloat",
+    "cross_session_drift",
+    "lookback_overflow",
+    "misordered_shared_context",
+    "missing_cache_markers",
+    "model_switch",
+    "prefix_drift",
+    "retry_storm",
+    "sub_minimum",
+    "tool_schema_bloat",
+    "tool_schema_churn",
+    "ttl_expiry"
+  ],
+  "evidence": {
+    "sessions": 3
+  },
+  "notes": "Three sessions run three different agents whose long system prompts share only their last line. Different prompts, not one prompt with a changing value: near-miss negative for cross_session_drift.",
+  "totals": {
+    "calls": 9,
+    "input_tokens": 13713,
+    "cache_read_tokens": 9094,
+    "cache_creation_tokens": 0,
+    "output_tokens": 1440
+  }
+}