swaptrace 0.1.0__tar.gz

swaptrace-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Holden Anderson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

swaptrace-0.1.0/PKG-INFO

@@ -0,0 +1,221 @@
+Metadata-Version: 2.4
+Name: swaptrace
+Version: 0.1.0
+Summary: Trace and compare LLM swap attempts.
+Author: Holden Anderson
+License-Expression: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Provides-Extra: swapllm
+Requires-Dist: swapllm>=0.1.0; extra == "swapllm"
+Dynamic: license-file
+
+# swaptrace
+
+**Fallback-cascade observability for multi-provider LLM routers — see every attempt, not just the final answer.**
+
+## The problem
+
+When a router falls back across providers — Groq rate-limits, so it tries OpenAI,
+which times out, so it tries Anthropic, which answers — the only thing you get
+back is that final answer. The retry history is thrown away the moment a provider
+succeeds. But that history is exactly what answers *"why did this request cost 3×
+the usual"* and *"which provider is actually carrying the traffic."* `swaptrace`
+records every attempt in a cascade — provider, model, outcome, latency, token
+cost — and keeps them whether the trace ends in success or in exhaustion.
+
+## Install
+
+```
+pip install swaptrace
+```
+
+For the [swapLLM](https://github.com/HWalker13/swapllm) integration:
+
+```
+pip install "swaptrace[swapllm]"
+```
+
+`swaptrace`'s core is standard-library only — no runtime dependencies. The
+`swapllm` extra is opt-in.
+
+## Quickstart
+
+### Standalone
+
+Wrap your own retry loop. `swaptrace` defines no exception types and makes no HTTP
+calls — you classify each failure, it does the bookkeeping.
+
+```python
+from swaptrace import Trace
+
+
+class RateLimited(Exception):
+    """Your own exception type — swaptrace defines none of its own."""
+
+
+def call_provider(name):
+    if name == "groq":
+        raise RateLimited("429 Too Many Requests")
+    return {"text": "...", "prompt_tokens": 41, "completion_tokens": 12}
+
+
+with Trace() as trace:
+    for provider in ["groq", "openai"]:
+        with trace.attempt(provider=provider, model="llama-3.1-8b-instant") as attempt:
+            try:
+                resp = call_provider(provider)
+            except RateLimited as err:
+                attempt.record_failure(err, retryable=True)
+                continue
+            attempt.record_success(
+                resp,
+                prompt_tokens=resp["prompt_tokens"],
+                completion_tokens=resp["completion_tokens"],
+            )
+            break
+
+print(trace.final_status)       # success
+print(trace.winning_provider)   # openai
+print(trace.retry_count)        # 1
+for a in trace.attempts:
+    print(a.attempt_index, a.provider, a.outcome, a.error_type, a.cost_usd)
+# 0 groq retryable_error RateLimited None
+# 1 openai success None 3.0100000000000004e-06
+```
+
+`cost_usd` is estimated from a small built-in per-token price table
+(`swaptrace.pricing.DEFAULT_PRICING`). For models it doesn't know, pass
+`pricing_overrides={"model-name": (input_rate, output_rate)}` to
+`record_success` — rates are USD per million tokens.
+
+### swapLLM integration
+
+`traced()` wraps a `swapllm.Router` and records a `Trace` for every `.complete()`
+call — without changing what `.complete()` returns or which exceptions it raises,
+so existing call sites are untouched.
+
+```python
+from swapllm import Router, GroqProvider, OpenAIProvider
+from swaptrace import storage
+from swaptrace.integrations.swapllm import traced
+
+router = Router(
+    providers=[
+        GroqProvider(api_key=..., model="llama-3.1-8b-instant"),
+        OpenAIProvider(api_key=..., model="gpt-4o-mini"),
+    ],
+    fallback_order=["groq", "openai"],
+)
+
+# record every trace to a JSONL file as it completes
+router = traced(
+    router,
+    on_trace=lambda t: storage.append_trace(t, ".swaptrace/traces.jsonl"),
+)
+
+answer = router.complete(messages=[{"role": "user", "content": "..."}])
+
+router.last_trace.winning_provider
+# "openai"
+[(a.provider, a.outcome) for a in router.last_trace.attempts]
+# [('groq', 'retryable_error'), ('openai', 'success')]
+```
+
+`traced()` re-implements swapLLM's fallback loop instead of wrapping
+`Router.complete()`: swapLLM discards per-attempt information once a provider
+succeeds, so the only way to observe it is to drive the loop. `AllProvidersFailedError`,
+`ProviderRequestError`, and schema-validation behaviour are preserved exactly.
+swapLLM's provider adapters return only text, so `cost_usd` stays `None` for
+swapLLM-traced attempts.
+
+## CLI
+
+`swaptrace` never writes trace files on its own — wire `storage.append_trace`
+into `on_trace` as shown above. It reads `.swaptrace/traces.jsonl` (relative to
+the working directory — traces are per-project, like `.git/`) by default;
+override with `--path`.
+
+### `swaptrace query`
+
+List traces, optionally filtered. `--provider` and `--status` match a trace if
+*any* of its attempts matches; `--min-cost` is checked against the trace total.
+Active filters combine with AND.
+
+```
+$ swaptrace query
+2026-09-03T19:35:13.726568+00:00  a60d166a  success    groq          1 attempt(s)  $0.000025  25.0ms
+2026-09-03T19:35:13.752308+00:00  a34ff498  success    openai        2 attempt(s)  $0.000221  45.1ms
+2026-09-03T19:35:13.798053+00:00  4847ffd3  success    anthropic     3 attempt(s)  $0.001980  70.5ms
+2026-09-03T19:35:13.869284+00:00  c627f556  exhausted  -             3 attempt(s)  $0.000000  68.6ms
+2026-09-03T19:35:13.939634+00:00  9647ec87  exhausted  -             1 attempt(s)  $0.000000  22.1ms
+5 of 5 trace(s).
+
+$ swaptrace query --min-cost 0.001
+2026-09-03T19:35:13.798053+00:00  4847ffd3  success    anthropic     3 attempt(s)  $0.001980  70.5ms
+1 of 5 trace(s).
+```
+
+### `swaptrace report --compare-providers`
+
+Flatten every attempt across every trace, grouped by provider — reliability,
+latency, and cost side by side, most reliable first.
+
+```
+$ swaptrace report --compare-providers
+PROVIDER     ATTEMPTS  SUCCESSES  SUCCESS%   AVG LATENCY    TOTAL COST    $/SUCCESS
+anthropic           2          1     50.0%        22.3ms     $0.001980    $0.001980
+groq                4          1     25.0%        23.6ms     $0.000025    $0.000025
+openai              4          1     25.0%        23.1ms     $0.000221    $0.000221
+```
+
+`SUCCESS%` is per-attempt reliability (successes ÷ attempts, across all traces);
+`AVG LATENCY` is over every attempt, successful or not; `$/SUCCESS` is `-` for a
+provider that has never succeeded.
+
+## Instrumentation overhead
+
+Wrapping a provider call in swaptrace's `Trace`/`Attempt` bookkeeping adds a
+measured **7.75 µs per call at the median and 7.96 µs at p95** (CPython 3.12.8,
+macOS arm64; 5 trials × 1000 iterations, warm-up, GC disabled, `time.perf_counter`).
+That overhead is a fixed cost of swaptrace's own work — a couple of `uuid4()` and
+`datetime.now()` calls, the trace rollup, a pricing-table lookup — and does not
+scale with the wrapped call, so against a typical 200 ms–2 s LLM API call it is
+**well under 0.01%** (≈0.008% at 100 ms, ≈0.0004% at 2 s). Reproduce with
+`python benchmarks/benchmark_overhead.py`; the full raw dataset is in
+`benchmarks/results/`.
+
+## What this isn't
+
+- No hosted dashboard or web UI — `swaptrace query` / `report` are the interface.
+- No OpenTelemetry / OTLP exporter yet.
+- No streaming-response support.
+- No multi-agent span tracing (agentrace-ai / spyllm cover that space) — a
+  `swaptrace` trace is scoped to a single provider cascade.
+
+## Development
+
+```
+pip install -e ".[dev]"
+pytest
+```
+
+runs the core suite — **61 passed, 1 skipped** (the swapLLM integration tests
+skip when the extra isn't installed). For the full **73**:
+
+```
+pip install -e ".[dev,swapllm]"
+pytest
+```
+
+Requires Python 3.11+.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

swaptrace-0.1.0/README.md

@@ -0,0 +1,203 @@
+# swaptrace
+
+**Fallback-cascade observability for multi-provider LLM routers — see every attempt, not just the final answer.**
+
+## The problem
+
+When a router falls back across providers — Groq rate-limits, so it tries OpenAI,
+which times out, so it tries Anthropic, which answers — the only thing you get
+back is that final answer. The retry history is thrown away the moment a provider
+succeeds. But that history is exactly what answers *"why did this request cost 3×
+the usual"* and *"which provider is actually carrying the traffic."* `swaptrace`
+records every attempt in a cascade — provider, model, outcome, latency, token
+cost — and keeps them whether the trace ends in success or in exhaustion.
+
+## Install
+
+```
+pip install swaptrace
+```
+
+For the [swapLLM](https://github.com/HWalker13/swapllm) integration:
+
+```
+pip install "swaptrace[swapllm]"
+```
+
+`swaptrace`'s core is standard-library only — no runtime dependencies. The
+`swapllm` extra is opt-in.
+
+## Quickstart
+
+### Standalone
+
+Wrap your own retry loop. `swaptrace` defines no exception types and makes no HTTP
+calls — you classify each failure, it does the bookkeeping.
+
+```python
+from swaptrace import Trace
+
+
+class RateLimited(Exception):
+    """Your own exception type — swaptrace defines none of its own."""
+
+
+def call_provider(name):
+    if name == "groq":
+        raise RateLimited("429 Too Many Requests")
+    return {"text": "...", "prompt_tokens": 41, "completion_tokens": 12}
+
+
+with Trace() as trace:
+    for provider in ["groq", "openai"]:
+        with trace.attempt(provider=provider, model="llama-3.1-8b-instant") as attempt:
+            try:
+                resp = call_provider(provider)
+            except RateLimited as err:
+                attempt.record_failure(err, retryable=True)
+                continue
+            attempt.record_success(
+                resp,
+                prompt_tokens=resp["prompt_tokens"],
+                completion_tokens=resp["completion_tokens"],
+            )
+            break
+
+print(trace.final_status)       # success
+print(trace.winning_provider)   # openai
+print(trace.retry_count)        # 1
+for a in trace.attempts:
+    print(a.attempt_index, a.provider, a.outcome, a.error_type, a.cost_usd)
+# 0 groq retryable_error RateLimited None
+# 1 openai success None 3.0100000000000004e-06
+```
+
+`cost_usd` is estimated from a small built-in per-token price table
+(`swaptrace.pricing.DEFAULT_PRICING`). For models it doesn't know, pass
+`pricing_overrides={"model-name": (input_rate, output_rate)}` to
+`record_success` — rates are USD per million tokens.
+
+### swapLLM integration
+
+`traced()` wraps a `swapllm.Router` and records a `Trace` for every `.complete()`
+call — without changing what `.complete()` returns or which exceptions it raises,
+so existing call sites are untouched.
+
+```python
+from swapllm import Router, GroqProvider, OpenAIProvider
+from swaptrace import storage
+from swaptrace.integrations.swapllm import traced
+
+router = Router(
+    providers=[
+        GroqProvider(api_key=..., model="llama-3.1-8b-instant"),
+        OpenAIProvider(api_key=..., model="gpt-4o-mini"),
+    ],
+    fallback_order=["groq", "openai"],
+)
+
+# record every trace to a JSONL file as it completes
+router = traced(
+    router,
+    on_trace=lambda t: storage.append_trace(t, ".swaptrace/traces.jsonl"),
+)
+
+answer = router.complete(messages=[{"role": "user", "content": "..."}])
+
+router.last_trace.winning_provider
+# "openai"
+[(a.provider, a.outcome) for a in router.last_trace.attempts]
+# [('groq', 'retryable_error'), ('openai', 'success')]
+```
+
+`traced()` re-implements swapLLM's fallback loop instead of wrapping
+`Router.complete()`: swapLLM discards per-attempt information once a provider
+succeeds, so the only way to observe it is to drive the loop. `AllProvidersFailedError`,
+`ProviderRequestError`, and schema-validation behaviour are preserved exactly.
+swapLLM's provider adapters return only text, so `cost_usd` stays `None` for
+swapLLM-traced attempts.
+
+## CLI
+
+`swaptrace` never writes trace files on its own — wire `storage.append_trace`
+into `on_trace` as shown above. It reads `.swaptrace/traces.jsonl` (relative to
+the working directory — traces are per-project, like `.git/`) by default;
+override with `--path`.
+
+### `swaptrace query`
+
+List traces, optionally filtered. `--provider` and `--status` match a trace if
+*any* of its attempts matches; `--min-cost` is checked against the trace total.
+Active filters combine with AND.
+
+```
+$ swaptrace query
+2026-09-03T19:35:13.726568+00:00  a60d166a  success    groq          1 attempt(s)  $0.000025  25.0ms
+2026-09-03T19:35:13.752308+00:00  a34ff498  success    openai        2 attempt(s)  $0.000221  45.1ms
+2026-09-03T19:35:13.798053+00:00  4847ffd3  success    anthropic     3 attempt(s)  $0.001980  70.5ms
+2026-09-03T19:35:13.869284+00:00  c627f556  exhausted  -             3 attempt(s)  $0.000000  68.6ms
+2026-09-03T19:35:13.939634+00:00  9647ec87  exhausted  -             1 attempt(s)  $0.000000  22.1ms
+5 of 5 trace(s).
+
+$ swaptrace query --min-cost 0.001
+2026-09-03T19:35:13.798053+00:00  4847ffd3  success    anthropic     3 attempt(s)  $0.001980  70.5ms
+1 of 5 trace(s).
+```
+
+### `swaptrace report --compare-providers`
+
+Flatten every attempt across every trace, grouped by provider — reliability,
+latency, and cost side by side, most reliable first.
+
+```
+$ swaptrace report --compare-providers
+PROVIDER     ATTEMPTS  SUCCESSES  SUCCESS%   AVG LATENCY    TOTAL COST    $/SUCCESS
+anthropic           2          1     50.0%        22.3ms     $0.001980    $0.001980
+groq                4          1     25.0%        23.6ms     $0.000025    $0.000025
+openai              4          1     25.0%        23.1ms     $0.000221    $0.000221
+```
+
+`SUCCESS%` is per-attempt reliability (successes ÷ attempts, across all traces);
+`AVG LATENCY` is over every attempt, successful or not; `$/SUCCESS` is `-` for a
+provider that has never succeeded.
+
+## Instrumentation overhead
+
+Wrapping a provider call in swaptrace's `Trace`/`Attempt` bookkeeping adds a
+measured **7.75 µs per call at the median and 7.96 µs at p95** (CPython 3.12.8,
+macOS arm64; 5 trials × 1000 iterations, warm-up, GC disabled, `time.perf_counter`).
+That overhead is a fixed cost of swaptrace's own work — a couple of `uuid4()` and
+`datetime.now()` calls, the trace rollup, a pricing-table lookup — and does not
+scale with the wrapped call, so against a typical 200 ms–2 s LLM API call it is
+**well under 0.01%** (≈0.008% at 100 ms, ≈0.0004% at 2 s). Reproduce with
+`python benchmarks/benchmark_overhead.py`; the full raw dataset is in
+`benchmarks/results/`.
+
+## What this isn't
+
+- No hosted dashboard or web UI — `swaptrace query` / `report` are the interface.
+- No OpenTelemetry / OTLP exporter yet.
+- No streaming-response support.
+- No multi-agent span tracing (agentrace-ai / spyllm cover that space) — a
+  `swaptrace` trace is scoped to a single provider cascade.
+
+## Development
+
+```
+pip install -e ".[dev]"
+pytest
+```
+
+runs the core suite — **61 passed, 1 skipped** (the swapLLM integration tests
+skip when the extra isn't installed). For the full **73**:
+
+```
+pip install -e ".[dev,swapllm]"
+pytest
+```
+
+Requires Python 3.11+.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

swaptrace-0.1.0/pyproject.toml

@@ -0,0 +1,31 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "swaptrace"
+version = "0.1.0"
+description = "Trace and compare LLM swap attempts."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Holden Anderson" }]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+
+[project.optional-dependencies]
+dev = ["pytest"]
+swapllm = ["swapllm>=0.1.0"]
+
+[project.scripts]
+swaptrace = "swaptrace.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

swaptrace-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

swaptrace-0.1.0/src/swaptrace/__init__.py

@@ -0,0 +1,5 @@
+"""swaptrace -- trace and compare LLM swap attempts."""
+
+from swaptrace.core import Attempt, AttemptOutcome, Trace, TraceStatus
+
+__all__ = ["Trace", "Attempt", "AttemptOutcome", "TraceStatus"]