costtrace 0.1.0__tar.gz

costtrace-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sanat Bhalla
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

costtrace-0.1.0/PKG-INFO

@@ -0,0 +1,422 @@
+Metadata-Version: 2.4
+Name: costtrace
+Version: 0.1.0
+Summary: A deterministic cost-attribution profiler for LLM traces — an X-ray for where your token spend goes.
+Author: Sanat Bhalla
+License: MIT
+Project-URL: Homepage, https://github.com/sanat77/CostTrace
+Project-URL: Repository, https://github.com/sanat77/CostTrace
+Project-URL: Issues, https://github.com/sanat77/CostTrace/issues
+Keywords: llm,cost,finops,tokens,profiler,observability
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: tiktoken>=0.7
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pyflakes>=3.0; extra == "dev"
+Provides-Extra: examples
+Requires-Dist: openai-agents>=0.18; extra == "examples"
+Dynamic: license-file
+
+# CostTrace
+
+**An X-ray for your LLM token spend.**
+
+CostTrace is a deterministic cost-attribution profiler for LLM applications. You
+add **one line** after each agent run to capture the durable identity of your
+billed calls. Later, CostTrace re-fetches the exact usage from the provider,
+prices it, and tells you three things most dashboards can't:
+
+1. **Where** your money goes — split across system prompt, conversation history,
+   retrieved context, the user's turn, and the model's output.
+2. **What changed** over time — cost-per-turn jumps, model switches, history
+   bloat, longer outputs, cache regressions, or system-prompt edits.
+3. **Why** — an LLM-as-judge explains the most likely driver in plain English,
+   grounded strictly in your real numbers.
+
+> **Integrations:** today CostTrace supports the **OpenAI Agents SDK (Python)**
+> only. The design is provider-tagged, and **support for more SDKs and providers
+> is planned** (see [Roadmap](#roadmap)).
+
+---
+
+## Table of contents
+
+- [Why CostTrace](#why-costtrace)
+- [How it's different](#how-its-different)
+- [Install](#install)
+- [Quickstart](#quickstart)
+- [Core concepts](#core-concepts)
+- [Providers](#providers)
+- [CLI reference](#cli-reference)
+- [Programmatic API](#programmatic-api)
+- [How it works](#how-it-works)
+- [The cost model](#the-cost-model)
+- [Data, privacy & retention](#data-privacy--retention)
+- [Configuration](#configuration)
+- [Development](#development)
+- [Roadmap](#roadmap)
+- [FAQ](#faq)
+- [License](#license)
+
+---
+
+## Why CostTrace
+
+LLM bills are opaque. A provider hands you a token count per call, not dollars,
+and never tells you *which part* of the prompt cost the money. In a multi-turn
+agent, the conversation history is re-sent on every turn, a verbose system
+prompt is billed on every call, and a single model swap can 100× your cost — but
+the invoice just says "tokens."
+
+CostTrace turns that opaque number into an **itemized, timestamped, queryable**
+picture of your spend, so you can answer questions like:
+
+- "What fraction of my bill is just re-sending history?"
+- "Our costs doubled last week — what changed?"
+- "How much cheaper is `nano` than `pro` for the *same* workload?"
+- "Did that system-prompt edit break prompt caching?"
+
+## How it's different
+
+- **Capture is tiny and cheap.** We store only `{provider, conversation_id,
+  response_ids}` — no usage numbers, no message content. Everything else is
+  re-fetched on demand (provider GET reads are **not** billed).
+- **Accounting is deterministic.** Costs are computed by us as `rate × tokens`
+  using live [LiteLLM](https://github.com/BerriAI/litellm) pricing (with a
+  bundled offline fallback) — reproducible, auditable, no guessing.
+- **Detection is general, not single-cause.** Rather than blaming one thing,
+  CostTrace compares a recent window against a prior one across *every*
+  dimension and surfaces each signal that moved.
+- **The LLM only explains, never computes.** The judge receives a compact,
+  number-grounded digest and is instructed to reason over it — it never invents
+  figures.
+- **It's a profiler, not a proxy.** No gateway to route traffic through, no
+  latency added to your requests. You capture ids; analysis happens offline,
+  on demand.
+
+## Install
+
+```bash
+pip install costtrace
+```
+
+This installs the importable **`CostTrace`** package and the **`costtrace`** CLI.
+
+Set your OpenAI key via the environment or a `.env` in your project root:
+
+```bash
+export OPENAI_API_KEY=sk-...
+# or, in ./.env
+# OPENAI_API_KEY=sk-...
+```
+
+Requires Python 3.9+.
+
+## Quickstart
+
+### 1. Capture — one line after each agent run
+
+```python
+from agents import Agent, OpenAIConversationsSession, Runner
+from CostTrace import capture
+
+session = OpenAIConversationsSession()
+result = await Runner.run(agent, user_input, session=session)
+
+capture(result, conversation_id, provider="openai_agents")   # -> ./.costtrace/captures.json
+```
+
+`capture()` extracts the response ids from the run, validates them, de-dupes,
+timestamps them, and appends them under the conversation id. It's idempotent and
+prunes anything older than the retention window on every write.
+
+`provider` is **required** — it records which backend the ids came from so they
+can be re-fetched later. Passing an unknown/missing provider raises an error that
+lists the [available providers](#providers).
+
+You can also pass a plain list of ids instead of a `RunResult`:
+
+```python
+capture(["resp_abc", "resp_def"], conversation_id, provider="openai_agents")
+```
+
+### 2. Analyze — from the CLI
+
+```bash
+costtrace analyze                              # everything captured
+costtrace analyze --limit 5                    # 5 most recent conversations
+costtrace analyze --date 2026/07/20            # a single day
+costtrace analyze --starting-date 2026/07/01   # from that day until today
+costtrace analyze --conversationId conv_abc    # one conversation
+costtrace analyze --no-judge                   # skip the LLM explanation
+```
+
+Example output:
+
+```
+======================================================================
+WHERE THE MONEY GOES
+======================================================================
+sessions: 3   total: $6.143908   avg cache-hit: 53%
+  output      85.9%   $5.275134
+  history     14.0%   $0.857593
+  system       0.1%   $0.009336
+  user         0.0%   $0.001846
+  retrieved    0.0%   $0.000000
+
+======================================================================
+PER-MODEL COMPARISON
+======================================================================
+  model               cost/turn   cache  out tok/turn  turns
+  gpt-5.4-pro        $ 1.118778     0%          5376      5
+  gpt-5.4            $ 0.043458    80%          2305     12
+  gpt-5.4-nano       $ 0.002377    78%          1472     12
+
+  gpt-5.4-pro costs 470.6x more per turn than gpt-5.4-nano
+
+======================================================================
+WHAT CHANGED (deterministic signals)
+======================================================================
+  [1] (medium) dominant_bucket: 'output' is 86% of total spend ($5.275134)
+
+======================================================================
+JUDGE (LLM explanation)
+======================================================================
+Output dominates spend; the cost gap tracks the switch to gpt-5.4-pro
+(higher rate + 3.6x longer outputs + 0% cache vs ~80%). Consider routing
+routine turns to gpt-5.4-nano and capping output length.
+```
+
+## Core concepts
+
+| Term | Meaning |
+|------|---------|
+| **capture** | The dev-side one-liner. Persists `{provider, conversation_id, response_ids}` — the durable *identity* of billed calls, nothing else. |
+| **conversation** | One session, grouped by the provider's conversation id. |
+| **call** | One billed model request == one `response_id`. |
+| **message kind** | Each item in a call is classified as `system`, `history`, `retrieved`, `user`, or `output`. |
+| **bucket** | Cost aggregated by message kind — the "where the money goes" split. |
+| **session metric** | One durable, timestamped row per conversation (cost, buckets, cache rate, cost/turn, model, system-prompt fingerprint). |
+| **provider tag** | Routes fetching to the right backend. Currently only `openai_agents`. |
+
+## Providers
+
+`provider` is a **required** argument to `capture()` — it records which backend
+the response ids belong to, so CostTrace knows where to re-fetch usage from.
+
+| Provider tag | Backend | Status |
+|--------------|---------|--------|
+| `openai_agents` | OpenAI Agents SDK (Python) | ✅ Available |
+| _(more)_ | OpenAI Python SDK, Anthropic, Langfuse/Helicone exports, … | 🚧 Planned ([roadmap](#roadmap)) |
+
+Passing an unknown or missing provider fails fast with an error that lists the
+currently supported tags:
+
+```python
+>>> capture(result, conversation_id)                     # missing
+CaptureSchemaError: provider is required — pass one of the supported providers:
+['openai_agents'] (e.g. capture(result, conversation_id, provider='openai_agents'))
+
+>>> capture(result, conversation_id, provider="anthropic")   # unknown
+CaptureSchemaError: unknown provider 'anthropic'; supported providers: ['openai_agents']
+```
+
+The available tags are also exposed programmatically:
+
+```python
+from CostTrace import Provider
+Provider.values()        # ['openai_agents']
+```
+
+## CLI reference
+
+```
+costtrace analyze [options]
+```
+
+| Option | Description |
+|--------|-------------|
+| `--limit N` | Analyze the N most recent conversations. |
+| `--conversationId conv_xxx` | Analyze a single conversation (alias: `--conversation-id`). |
+| `--date yyyy/mm/dd` | Only conversations from that day. |
+| `--starting-date yyyy/mm/dd` | Conversations from that day until today. |
+| `--no-judge` | Skip the LLM explanation (fully offline). |
+
+`--date` and `--starting-date` are mutually exclusive. Run `costtrace --help` or
+`costtrace analyze --help` for the full, self-documenting listing.
+
+## Programmatic API
+
+```python
+from CostTrace import load_captures, fetch_captures, Pricer
+
+captures = load_captures(".costtrace/captures.json")   # validated ids
+report = fetch_captures(captures)                       # validated traces
+
+report.conversations   # list[ConversationTrace] — ready to analyze
+report.rejections      # ids dropped (expired/invalid), each with a reason
+report.fetched_calls   # count of successfully fetched calls
+```
+
+Build your own analysis with the `analysis` module:
+
+```python
+from CostTrace.analysis import (
+    compute_session_metric, upsert_metrics, load_metrics,
+    summarize_overall, summarize_by_model, detect_all, judge,
+)
+
+pricer = Pricer.default()  # live LiteLLM pricing, cached, yaml fallback
+metrics = [compute_session_metric(c, pricer) for c in report.conversations]
+upsert_metrics(metrics)                       # persist to .costtrace/metrics.jsonl
+
+rows = load_metrics()
+overall = summarize_overall(rows)
+findings = detect_all(overall, rows)
+verdict = judge(overall, findings, by_model=summarize_by_model(rows))
+print(verdict.verdict)
+```
+
+## How it works
+
+```
+                your app
+                   │  capture(result, conversation_id)
+                   ▼
+   .costtrace/captures.json   ── just ids, 15-day retention, auto-pruned
+                   │
+   costtrace analyze:
+                   ▼
+   ingest ──► fetch ──► metrics ──► store ──► aggregate ──► detect ──► judge
+   (validate  (re-fetch  (bucket    (durable   (overall +   (what      (LLM
+    ids)       usage &    split +    JSONL)     per-model +  changed)    explains
+               content)   cost)                 per-prompt)             why)
+```
+
+1. **Ingest** — strict schema + id-format validation; clear errors; duplicate
+   conversations merged.
+2. **Fetch** — for each id, pull the response (model + usage) and its input
+   items from the provider, then classify each message (system/history/
+   retrieved/user/output). **Resilient:** expired or invalid ids are recorded as
+   rejections and skipped; only auth failures are fatal.
+3. **Metrics** — per conversation, compute total cost and split the input cost
+   across buckets by token share (estimated with a tokenizer, then scaled so the
+   parts sum to the provider's real `input_tokens`). Output cost is exact.
+4. **Store** — append/upsert one row per conversation to
+   `.costtrace/metrics.jsonl`; refresh rows that gained turns; prune expired
+   rows. This history survives the id-retention window.
+5. **Aggregate** — overall spend split, per-model comparison, per-prompt-version
+   grouping.
+6. **Detect** — compare a recent window vs a prior one and flag every dimension
+   that moved: `cost_increase`, `model_shift`, `bucket_shift`, `cache_drop`,
+   `output_growth`, `prompt_change`, plus a static `dominant_bucket`.
+7. **Judge** — assemble a compact, grounded digest and ask an LLM to name the
+   likely cause and a fix. It reasons over the signals; it does not compute
+   numbers.
+
+## The cost model
+
+Providers return **tokens, not dollars** — and never a per-call price. CostTrace
+computes cost itself:
+
+```
+cost = uncached_input_tokens × input_rate
+     + cached_input_tokens   × cache_read_rate
+     + output_tokens         × output_rate
+```
+
+Rates come from LiteLLM's community-maintained pricing table (≈3,000 models),
+fetched and cached locally, with a small bundled `pricing.yaml` as an offline
+fallback. Because accounting is pure arithmetic on real token counts, results
+are reproducible and auditable.
+
+The **bucket split** for the input side is an approximation: the provider bills
+one `input_tokens` number per call, so CostTrace estimates each message's share
+with a tokenizer and scales those shares to sum exactly to the real input cost.
+The **output** bucket is exact.
+
+## Data, privacy & retention
+
+- **What's stored on disk:** only response/conversation **ids** in
+  `.costtrace/captures.json`, plus computed **metrics** (numbers, model names,
+  and a hash + copy of the system prompt) in `.costtrace/metrics.jsonl`, plus a
+  cached pricing table. No user/assistant message content is persisted.
+- **Where:** a project-local `.costtrace/` directory, discovered by walking up
+  to the nearest `.git` / `pyproject.toml` (like git finds its root), so
+  `capture()` and the CLI always agree regardless of your working directory.
+  Override with `COSTTRACE_HOME`.
+- **Retention:** ids are only re-fetchable from the provider for a limited
+  window, so captures older than **15 days** are pruned automatically on write —
+  you lose nothing that could still be fetched.
+- **The judge digest** contains aggregate numbers and (only when a prompt change
+  is detected) the system-prompt text — never raw conversation content.
+- **Secrets:** your API key is read from the environment or a `.env`; it is
+  never written to the data directory.
+
+## Configuration
+
+| Variable | Purpose |
+|----------|---------|
+| `OPENAI_API_KEY` | Required for fetching usage and for the LLM judge. |
+| `COSTTRACE_HOME` | Override the `.costtrace/` data directory location. |
+
+## Development
+
+```bash
+git clone https://github.com/sanat77/CostTrace.git
+cd CostTrace
+python -m venv .venv && source .venv/bin/activate
+pip install -e .[dev]
+
+python -m pytest -q                       # run the test suite
+python -m pyflakes src tests conftest.py  # lint
+```
+
+The package source lives in `src/` and is mapped to the import name `CostTrace`
+via `package-dir` in `pyproject.toml`, so the built wheel ships `CostTrace/` and
+`pip install` / `import CostTrace` are unaffected by the on-disk layout. A
+`conftest.py` replicates that mapping so tests import from the source tree
+without installing.
+
+## Roadmap
+
+CostTrace v1 targets the **OpenAI Agents SDK (Python)** (`provider =
+"openai_agents"`). Planned next:
+
+- **More Python LLM SDKs / providers** (e.g. the OpenAI Python SDK directly,
+  Anthropic, and others) behind the same provider-tagged architecture.
+- **Ingesting exports** from existing observability tools (e.g. Langfuse /
+  Helicone) as additional provider tags.
+- **Longer retention & trend history** as an opt-in tier.
+- **Richer detectors** and per-tenant / per-feature attribution.
+
+Contributions and provider requests are welcome — open an issue.
+
+## FAQ
+
+**Does this add latency to my app?** No. You only save ids at runtime; all
+fetching and analysis happens later, offline.
+
+**Do the re-fetches cost money?** No — provider GET reads of past responses are
+not billed. Only your original generations were.
+
+**Why can't I analyze conversations from last month?** Ids expire at the
+provider (~retention window), so they can't be re-fetched. The computed metrics
+you already stored remain, but new deep analysis needs live ids.
+
+**Distribution name vs import name?** You `pip install costtrace` (lowercase)
+and `import CostTrace` (capitalized) — standard PyPI convention.
+
+## License
+
+MIT — see [LICENSE](LICENSE).