llm-tally 0.2.0__tar.gz

llm_tally-0.2.0/.gitignore

@@ -0,0 +1,30 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+dist/
+build/
+*.egg-info/
+.eggs/
+
+# Testing & coverage
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Local LLM Tally runtime logs & generated dashboards
+.llm_tally/
+.ailoghelper/
+
+# IDE & OS files
+.vscode/
+.idea/
+.DS_Store
+

llm_tally-0.2.0/CHANGELOG.md

@@ -0,0 +1,36 @@
+# Changelog
+
+## 0.2.0
+
+### Breaking changes
+- The package was renamed from `django-llm-tally` / `django_llm_tally` to `llm-tally` / `llm_tally`, and the Django helpers (`get_django_llm_tally_settings`, `make_django_dashboard_view`) were removed.
+- **Caching is now opt-in.** `default_cache_enabled` defaults to `False`. Pass `cache=True` per call, or `default_cache_enabled=True` to restore the old behaviour. `@tally.cache`, `cached_call`, and `cached_llm_call` still always cache.
+- **Dashboard auto-updates are throttled** to at most once per `dashboard_update_interval` seconds (default 2.0). Pending updates are written in the background, at exit, or by `tally.flush()`. Set `dashboard_update_interval=0` for the old write-on-every-call behaviour.
+- **HTTP 429 / rate-limit errors are now retried** (honoring `Retry-After`). Set `retry_on_rate_limit=False` to disable this.
+- **The JSONL log is append-only.** A finished stream appends its final record instead of rewriting the file in place, so a stream call takes two lines and readers keep the last line per `id`. `JSONLLogger.compact()` / `llm-tally compact` rewrites the file with one line per call. If other tools read the file directly, de-duplicate by `id`.
+- `log_previews=False` now also hides error message text (exception types and status codes are kept), and `redact` now also applies to error messages, retry-attempt messages, and `config` strings.
+
+### Fixed
+- Functions created by a factory (closures with the same qualified name) no longer share cache entries. Each live closure gets its own cache identity.
+- A *returned* response object with HTTP 429 is now retried like a raised rate-limit error (unless `retry_on_rate_limit=False`), and `Retry-After` is read from dict-style responses too.
+- Passing `None` to `LLMTally(...)` arguments now means "use the default". Before, it silently disabled caching, retries, or the dashboard, logged the model as `"None"`, or raised `TypeError`.
+- A `model=` keyword passed to the wrapped function is now used to price the call when the response doesn't report the model. Before, it was priced as `default_model`.
+- A stream abandoned before it finished was left logged as `success` with 0 output tokens. It is now logged as `incomplete`, with the tokens seen so far.
+- Stream context managers whose `__enter__` / `__aenter__` return a different object (e.g. Anthropic `MessageStreamManager`) are now tracked and iterated correctly. Usage is read from the SDK's message snapshot when you iterate `stream.text_stream`.
+- A stream manager that fails on `with` entry is logged as `failed` with the real error. It used to be logged as `incomplete` / "abandoned".
+- A `redact` function that raises no longer loses the record. The text is stored as `"[redaction failed]"`.
+- On large logs, a stream call took about 110 ms because the whole file was rewritten; it now takes about 1 ms (100k-line log). Reads are incremental, and the dashboard throttle stretches to 10× the rebuild time.
+
+### Added
+- `CallRecord.estimated_tokens` flags costs that were estimated from text length. The report's `overview` has `estimated_calls` / `estimated_cost_usd`, and the dashboard marks these costs with `~`.
+- Per-call `metadata=` on `track` / `run` / `cache` / `cached_call`, instance-wide `metadata=`, and `tally.context(**metadata)` (safe across threads and asyncio). The report's new `metadata_costs` and a new dashboard section show which user, tenant, or prompt version costs the most.
+- `log_previews=False` and `redact=callable` to keep prompt and response text out of logs and the dashboard.
+- `on_record=` sinks / `add_sink()`, and `persist_logs=False`, for custom storage. `async_sinks=True` (with `sink_queue_size`) delivers records on a background thread.
+- `max_log_mb=` / `log_backups=` log rotation, and `llm-tally compact`.
+- `tally.wrap_context(fn)` carries `tally.context(...)` metadata into thread-pool workers.
+- `cache_backend=` for a shared or custom cache.
+- `retry_jitter`, `max_retry_after`, `RetryConfig.compute_wait()`, and `extract_retry_after()`.
+- `pricing_file=` / `tally.load_pricing()` / `PricingRegistry.load_file()` for JSON price lists.
+- `days=` on `get_records` / `get_report` / `render_dashboard` / `generate_dashboard`, and `dashboard_days=`.
+- `llm-tally` command-line tool (`report`, `dashboard`), also runnable as `python -m llm_tally`.
+- `py.typed` marker, and a CI job that builds the wheel and checks it includes the templates.

llm_tally-0.2.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 Fauzi Azzam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

llm_tally-0.2.0/PKG-INFO

@@ -0,0 +1,295 @@
+Metadata-Version: 2.5
+Name: llm-tally
+Version: 0.2.0
+Summary: Zero-dependency Python LLM toolkit with self-evicting in-memory caching, smart 5xx/429/network retries, per-feature cost tracking, and an interactive HTML dashboard.
+Project-URL: Homepage, https://github.com/fozieng/llm_tally
+Project-URL: Repository, https://github.com/fozieng/llm_tally
+Project-URL: Issues, https://github.com/fozieng/llm_tally/issues
+Project-URL: Changelog, https://github.com/fozieng/llm_tally/blob/main/CHANGELOG.md
+Author: Fauzi Azzam
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai,anthropic,cache,cost-tracking,dashboard,gemini,llm,observability,openai,retry
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: build>=1.0.0; extra == 'dev'
+Requires-Dist: pytest>=7.4.0; extra == 'dev'
+Requires-Dist: twine>=5.0.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# llm-tally
+
+Zero-dependency Python toolkit for **self-evicting in-memory LLM caching**, **resilient 5xx / 429 / network retries**, and **per-feature cost & reliability analytics** with a self-contained **HTML dashboard**.
+
+Works in any Python 3.9+ application with zero external runtime dependencies. Fully typed (`py.typed`).
+
+---
+
+## Core Features
+
+### 1. Self-Evicting Local In-Memory Cache (opt-in)
+- **Input + Settings Snapshot**: Hashes the function's arguments (`args`, `kwargs`) together with its settings (`model`, `feature`, `config`) and stores the generated `output`.
+- **Instant Cache Hits**: Calling the function again with the same input and settings returns a copy of the cached output without invoking the LLM.
+- **Lifetime (`ttl`) Auto-Vanish**: Entries expire after `ttl` seconds (default `300s`).
+- **Kill After `X` Requests (`max_requests`)**: An entry removes itself after being fetched `X` times.
+- **Off by default in `@tally.track`.** LLM output with `temperature > 0` is supposed to vary, so caching only happens where you ask for it (`cache=True`, `@tally.cache`, or `default_cache_enabled=True`).
+- **Pluggable**: pass `cache_backend=` to share a cache across workers (e.g. a Redis-backed object with the same `fetch`/`store`/`clear` methods as `InMemoryCache`).
+- Factory-created closures have separate cache identities. These identities last for the function's lifetime in its process, so closure entries are not shared across workers or restarts.
+
+### 2. Smart Retries on HTTP 5xx, 429 Rate Limits & Network Issues
+- Retries **HTTP 500/5xx** errors (raised exceptions or returned response objects), **HTTP 429 / `RateLimitError`**, and **network/timeout** errors (`ConnectionError`, `TimeoutError`, `URLError`, SDK connection/timeout exceptions).
+- Retries **3 times** by default with delays of **`0.5s`, `1.0s`, `1.5s`** (configurable).
+- Honors the server's **`Retry-After` / `retry-after-ms`** header (capped by `max_retry_after`, default 60s). Optional `retry_jitter` spreads out retries from many workers.
+- If all attempts fail, raises `RetryExhaustedError` with every attempt's error (`err.attempts`, `err.errors`).
+
+### 3. Per-Feature Cost Analyzer & Interactive HTML Dashboard
+- **Built-in pricing** for OpenAI, Anthropic, and Google Gemini models, plus `register_model(...)` or a JSON `pricing_file` for anything else.
+- **Accurate model attribution**: uses the decorator's `model=`, then `config["model"]`, then the model reported in the response, then the `model=` keyword your function was called with.
+- **Estimated vs. real tokens**: when a response carries no token usage, tokens are estimated from text length and the record is flagged `estimated_tokens=True` (shown as `~$` in the dashboard).
+- **Per-call metadata**: tag calls with `user_id`, `tenant`, `prompt_version`, ... and see which ones cost the most.
+- **Self-Contained Static HTML Dashboard (`.llm_tally/dashboard.html`)** answers:
+  1. **How much money did I spend this week?**
+  2. **Which feature costs the most?** (per feature and per config)
+  3. **Which user / tenant / prompt version costs the most?** (per metadata key)
+  4. **Which calls fail the most?**
+  5. **Which calls are slow?** (Average, P95, Max)
+  6. **Interactive search** across features, models, configs, metadata, inputs/outputs, statuses, and errors.
+
+---
+
+## Installation
+
+```bash
+pip install llm-tally
+```
+
+---
+
+## Quick Start
+
+```python
+from llm_tally import LLMTally
+
+tally = LLMTally()  # logs to .llm_tally/logs.jsonl, dashboard at .llm_tally/dashboard.html
+
+@tally.track(feature="support_chatbot")
+def ask_llm(prompt: str, model: str = "gpt-4o") -> dict:
+    # Replace with an OpenAI / Anthropic / Gemini SDK call and return its response
+    return {
+        "content": f"Answer to: {prompt}",
+        "usage": {"input_tokens": 120, "output_tokens": 45},
+    }
+
+response = ask_llm("How do I reset my password?", model="gpt-4o-mini")  # priced as gpt-4o-mini
+```
+
+Every constructor argument is optional, and passing `None` always means "use the default", so you can forward optional settings directly:
+
+```python
+tally = LLMTally(
+    default_model=os.getenv("LLM_MODEL"),    # None -> "gpt-4o-mini"
+    log_path=os.getenv("LLM_TALLY_LOG"),     # None -> ".llm_tally/logs.jsonl"
+    default_cache_enabled=True,              # opt in to caching for every tracked call
+    default_ttl=300.0,
+    default_retries=3,
+    retry_delays=(0.5, 1.0, 1.5),
+)
+```
+
+---
+
+## Which function should I use?
+
+| You want to...                                         | Use |
+|--------------------------------------------------------|-----|
+| Track cost + retry (+ optional cache) on a function    | `@tally.track(...)` |
+| Same thing, but for one call without a decorator       | `tally.run(func, *args, **options)` |
+| Cache a function (with cost tracking)                  | `@tally.cache(...)` / `tally.cached_call(func, ...)` |
+| Cache only, no logging, no tally instance              | `cached_llm_call(func, ...)` |
+| Retry only, no logging                                 | `tally.retry_call(func, ...)` |
+
+`run`, `cached_call`, and `retry_call` take their own options (`model`, `config`, `metadata`, `ttl`, ...) as keyword arguments, so those names are not forwarded to `func`. Use the decorator if your function itself needs a `model=` argument.
+
+---
+
+## Caching (Feature 1)
+
+```python
+@tally.track(feature="faq", cache=True, ttl=120.0, max_requests=5, config={"temperature": 0})
+def answer_faq(question: str) -> str:
+    ...
+
+from llm_tally import cached_llm_call
+
+out = cached_llm_call(expensive_prompt, "Quantum Computing", ttl=60.0, max_requests=2,
+                      settings={"temperature": 0})
+```
+
+After `ttl` seconds the entry vanishes on its next fetch; after `max_requests` hits it kills itself, and the next call generates a fresh response.
+
+---
+
+## Retries (Feature 2)
+
+```python
+from llm_tally import LLMTally, RetryExhaustedError
+
+tally = LLMTally(
+    retry_on_rate_limit=True,   # retry HTTP 429 / RateLimitError (default)
+    retry_jitter=0.2,           # add up to +20% random wait per retry
+    max_retry_after=30.0,       # never wait longer than 30s for a Retry-After header
+)
+
+@tally.track(feature="payment_summary", model="claude-3-7-sonnet")
+def flaky_call():
+    raise ConnectionError("Upstream LLM gateway reset connection")
+
+try:
+    flaky_call()
+except RetryExhaustedError as err:
+    for attempt in err.attempts:
+        print(attempt.attempt, attempt.delay_before, attempt.status_code, attempt.message)
+```
+
+Client errors (400, 401, 403, 404, 422) are never retried. Use `extra_exceptions=(MyError,)` to retry your own exception types.
+
+---
+
+## Per-Call Metadata (users, tenants, prompt versions)
+
+```python
+tally = LLMTally(metadata={"env": "prod"})            # added to every record
+
+@tally.track(feature="chat", metadata={"prompt_version": "v3"})
+def chat(prompt: str): ...
+
+with tally.context(user_id=user.id, tenant=org.slug):  # thread- and asyncio-safe
+    chat("hello")
+
+tally.run(other_func, "hi", metadata={"request_id": rid})
+
+report = tally.get_report()
+print(report["metadata_costs"]["user_id"][0])  # most expensive user
+```
+
+Precedence: instance `metadata` < `tally.context(...)` < decorator / `run(metadata=...)`.
+
+Context metadata follows asyncio tasks automatically, but Python does not pass it to `ThreadPoolExecutor` / `threading.Thread` workers. Wrap the function you submit:
+
+```python
+with tally.context(user_id="alice"), ThreadPoolExecutor() as pool:
+    pool.map(tally.wrap_context(chat), prompts)
+```
+
+---
+
+## Privacy: Prompt Previews
+
+By default the first 140 characters of each input and output are stored for search. To turn this off or mask sensitive data:
+
+```python
+import re
+
+# Store no prompt/response text, and hide error message text (exception types and status codes are kept)
+tally = LLMTally(log_previews=False)
+
+# Mask text in previews, error messages, retry-attempt messages, and string values in `config`
+tally = LLMTally(redact=lambda text: re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[email]", text))
+```
+
+If your `redact` function raises, the text is stored as `"[redaction failed]"` and a warning is shown. The call is still logged. `metadata` values are stored exactly as you pass them.
+
+---
+
+## Custom Storage & Sinks
+
+Every finished call is passed as a `CallRecord` to your `on_record` callables. Use them to send records to a database, OpenTelemetry, a queue, etc. A failing sink only warns; it never breaks the LLM call.
+
+```python
+def ship(record):
+    db.insert("llm_calls", record.to_dict())
+
+tally = LLMTally(on_record=ship)                       # JSONL file + your sink
+tally = LLMTally(on_record=ship, persist_logs=False)   # your sink only (no local file)
+tally = LLMTally(on_record=ship, async_sinks=True)     # run sinks on a background thread
+tally.add_sink(another_sink)
+```
+
+By default sinks run inside the call, so a slow sink adds its latency to every LLM call. With `async_sinks=True` records go through a bounded background queue (`sink_queue_size`, default 10,000). If the queue fills up, records are dropped with a warning. `tally.flush()` (also run at exit) waits for queued records to be delivered.
+
+Streams produce exactly one record each, once they finish. A stream that is abandoned before it ends is logged with `status="incomplete"`. An SDK stream manager that fails when the `with` block opens (e.g. Anthropic `messages.stream()` failing to connect) is logged as `failed` with the real error. Retries don't apply to it, because the SDK sends that request on entry.
+
+### The JSONL log file
+- **Append-only and safe across processes**: every write is one appended line under a POSIX file lock, so multiple threads and processes on one machine never lose records. A finished stream appends its final version, and readers keep the **last line for each `id`**. Run `tally.logger.compact()` or `llm-tally compact` to rewrite the file with one line per call.
+- **Incremental reads**: reports and the dashboard only parse what was appended since the last read.
+- **Rotation**: `LLMTally(max_log_mb=100, log_backups=3)` moves the log to `logs.jsonl.1` ... `.3` when it would exceed 100 MB. Reports and the dashboard only read the current file.
+
+---
+
+## Custom Models & Pricing
+
+```python
+tally.register_model("my-company-fine-tune", input_cost_per_1m=1.20, output_cost_per_1m=4.80)
+
+# Or keep prices in a file you can update without a release:
+tally = LLMTally(pricing_file="prices.json")
+tally.load_pricing("more_prices.json")
+```
+
+```json
+{
+  "my-company-fine-tune": {"input_cost_per_1m": 1.2, "output_cost_per_1m": 4.8, "provider": "custom"},
+  "gpt-4o": {"input_cost_per_1m": 2.5, "output_cost_per_1m": 10.0, "cached_input_cost_per_1m": 1.25}
+}
+```
+
+Unknown models fall back to $0.50 / $1.50 per 1M tokens and emit a `UserWarning`.
+
+---
+
+## Dashboard & Reports
+
+The dashboard is regenerated automatically, but **at most once every `dashboard_update_interval` seconds** (default `2.0`), so the overhead per call stays flat as logs grow. If a rebuild is slow (a very large log), the interval stretches to 10× the rebuild time, so rebuilding never uses more than about 10% of a core. A pending update is written in the background when the interval ends, and at interpreter exit.
+
+```python
+tally = LLMTally(dashboard_update_interval=10.0, dashboard_days=30)  # last 30 days only
+
+report = tally.get_report(days=7)
+print("Spent this week ($):", report["spend_summary"]["this_week_usd"])
+print("Most expensive feature:", report["overview"]["top_cost_feature"])
+print("Estimated-token calls:", report["overview"]["estimated_calls"])
+
+tally.flush()                           # write any pending dashboard update now
+path = tally.generate_dashboard()       # or force a full rebuild
+```
+
+Set `dashboard_update_interval=0` to rebuild on every call, or `auto_update_dashboard=False` to only build it yourself.
+
+### Command line
+
+```bash
+llm-tally report                                  # summary of .llm_tally/logs.jsonl
+llm-tally report --days 7 --json                  # full analytics as JSON
+llm-tally dashboard --log logs.jsonl --out dash.html --days 30
+llm-tally compact --log logs.jsonl                # one line per call
+python -m llm_tally report                        # same, without the console script
+```
+
+---
+
+## License
+
+MIT

llm_tally-0.2.0/README.md

@@ -0,0 +1,263 @@
+# llm-tally
+
+Zero-dependency Python toolkit for **self-evicting in-memory LLM caching**, **resilient 5xx / 429 / network retries**, and **per-feature cost & reliability analytics** with a self-contained **HTML dashboard**.
+
+Works in any Python 3.9+ application with zero external runtime dependencies. Fully typed (`py.typed`).
+
+---
+
+## Core Features
+
+### 1. Self-Evicting Local In-Memory Cache (opt-in)
+- **Input + Settings Snapshot**: Hashes the function's arguments (`args`, `kwargs`) together with its settings (`model`, `feature`, `config`) and stores the generated `output`.
+- **Instant Cache Hits**: Calling the function again with the same input and settings returns a copy of the cached output without invoking the LLM.
+- **Lifetime (`ttl`) Auto-Vanish**: Entries expire after `ttl` seconds (default `300s`).
+- **Kill After `X` Requests (`max_requests`)**: An entry removes itself after being fetched `X` times.
+- **Off by default in `@tally.track`.** LLM output with `temperature > 0` is supposed to vary, so caching only happens where you ask for it (`cache=True`, `@tally.cache`, or `default_cache_enabled=True`).
+- **Pluggable**: pass `cache_backend=` to share a cache across workers (e.g. a Redis-backed object with the same `fetch`/`store`/`clear` methods as `InMemoryCache`).
+- Factory-created closures have separate cache identities. These identities last for the function's lifetime in its process, so closure entries are not shared across workers or restarts.
+
+### 2. Smart Retries on HTTP 5xx, 429 Rate Limits & Network Issues
+- Retries **HTTP 500/5xx** errors (raised exceptions or returned response objects), **HTTP 429 / `RateLimitError`**, and **network/timeout** errors (`ConnectionError`, `TimeoutError`, `URLError`, SDK connection/timeout exceptions).
+- Retries **3 times** by default with delays of **`0.5s`, `1.0s`, `1.5s`** (configurable).
+- Honors the server's **`Retry-After` / `retry-after-ms`** header (capped by `max_retry_after`, default 60s). Optional `retry_jitter` spreads out retries from many workers.
+- If all attempts fail, raises `RetryExhaustedError` with every attempt's error (`err.attempts`, `err.errors`).
+
+### 3. Per-Feature Cost Analyzer & Interactive HTML Dashboard
+- **Built-in pricing** for OpenAI, Anthropic, and Google Gemini models, plus `register_model(...)` or a JSON `pricing_file` for anything else.
+- **Accurate model attribution**: uses the decorator's `model=`, then `config["model"]`, then the model reported in the response, then the `model=` keyword your function was called with.
+- **Estimated vs. real tokens**: when a response carries no token usage, tokens are estimated from text length and the record is flagged `estimated_tokens=True` (shown as `~$` in the dashboard).
+- **Per-call metadata**: tag calls with `user_id`, `tenant`, `prompt_version`, ... and see which ones cost the most.
+- **Self-Contained Static HTML Dashboard (`.llm_tally/dashboard.html`)** answers:
+  1. **How much money did I spend this week?**
+  2. **Which feature costs the most?** (per feature and per config)
+  3. **Which user / tenant / prompt version costs the most?** (per metadata key)
+  4. **Which calls fail the most?**
+  5. **Which calls are slow?** (Average, P95, Max)
+  6. **Interactive search** across features, models, configs, metadata, inputs/outputs, statuses, and errors.
+
+---
+
+## Installation
+
+```bash
+pip install llm-tally
+```
+
+---
+
+## Quick Start
+
+```python
+from llm_tally import LLMTally
+
+tally = LLMTally()  # logs to .llm_tally/logs.jsonl, dashboard at .llm_tally/dashboard.html
+
+@tally.track(feature="support_chatbot")
+def ask_llm(prompt: str, model: str = "gpt-4o") -> dict:
+    # Replace with an OpenAI / Anthropic / Gemini SDK call and return its response
+    return {
+        "content": f"Answer to: {prompt}",
+        "usage": {"input_tokens": 120, "output_tokens": 45},
+    }
+
+response = ask_llm("How do I reset my password?", model="gpt-4o-mini")  # priced as gpt-4o-mini
+```
+
+Every constructor argument is optional, and passing `None` always means "use the default", so you can forward optional settings directly:
+
+```python
+tally = LLMTally(
+    default_model=os.getenv("LLM_MODEL"),    # None -> "gpt-4o-mini"
+    log_path=os.getenv("LLM_TALLY_LOG"),     # None -> ".llm_tally/logs.jsonl"
+    default_cache_enabled=True,              # opt in to caching for every tracked call
+    default_ttl=300.0,
+    default_retries=3,
+    retry_delays=(0.5, 1.0, 1.5),
+)
+```
+
+---
+
+## Which function should I use?
+
+| You want to...                                         | Use |
+|--------------------------------------------------------|-----|
+| Track cost + retry (+ optional cache) on a function    | `@tally.track(...)` |
+| Same thing, but for one call without a decorator       | `tally.run(func, *args, **options)` |
+| Cache a function (with cost tracking)                  | `@tally.cache(...)` / `tally.cached_call(func, ...)` |
+| Cache only, no logging, no tally instance              | `cached_llm_call(func, ...)` |
+| Retry only, no logging                                 | `tally.retry_call(func, ...)` |
+
+`run`, `cached_call`, and `retry_call` take their own options (`model`, `config`, `metadata`, `ttl`, ...) as keyword arguments, so those names are not forwarded to `func`. Use the decorator if your function itself needs a `model=` argument.
+
+---
+
+## Caching (Feature 1)
+
+```python
+@tally.track(feature="faq", cache=True, ttl=120.0, max_requests=5, config={"temperature": 0})
+def answer_faq(question: str) -> str:
+    ...
+
+from llm_tally import cached_llm_call
+
+out = cached_llm_call(expensive_prompt, "Quantum Computing", ttl=60.0, max_requests=2,
+                      settings={"temperature": 0})
+```
+
+After `ttl` seconds the entry vanishes on its next fetch; after `max_requests` hits it kills itself, and the next call generates a fresh response.
+
+---
+
+## Retries (Feature 2)
+
+```python
+from llm_tally import LLMTally, RetryExhaustedError
+
+tally = LLMTally(
+    retry_on_rate_limit=True,   # retry HTTP 429 / RateLimitError (default)
+    retry_jitter=0.2,           # add up to +20% random wait per retry
+    max_retry_after=30.0,       # never wait longer than 30s for a Retry-After header
+)
+
+@tally.track(feature="payment_summary", model="claude-3-7-sonnet")
+def flaky_call():
+    raise ConnectionError("Upstream LLM gateway reset connection")
+
+try:
+    flaky_call()
+except RetryExhaustedError as err:
+    for attempt in err.attempts:
+        print(attempt.attempt, attempt.delay_before, attempt.status_code, attempt.message)
+```
+
+Client errors (400, 401, 403, 404, 422) are never retried. Use `extra_exceptions=(MyError,)` to retry your own exception types.
+
+---
+
+## Per-Call Metadata (users, tenants, prompt versions)
+
+```python
+tally = LLMTally(metadata={"env": "prod"})            # added to every record
+
+@tally.track(feature="chat", metadata={"prompt_version": "v3"})
+def chat(prompt: str): ...
+
+with tally.context(user_id=user.id, tenant=org.slug):  # thread- and asyncio-safe
+    chat("hello")
+
+tally.run(other_func, "hi", metadata={"request_id": rid})
+
+report = tally.get_report()
+print(report["metadata_costs"]["user_id"][0])  # most expensive user
+```
+
+Precedence: instance `metadata` < `tally.context(...)` < decorator / `run(metadata=...)`.
+
+Context metadata follows asyncio tasks automatically, but Python does not pass it to `ThreadPoolExecutor` / `threading.Thread` workers. Wrap the function you submit:
+
+```python
+with tally.context(user_id="alice"), ThreadPoolExecutor() as pool:
+    pool.map(tally.wrap_context(chat), prompts)
+```
+
+---
+
+## Privacy: Prompt Previews
+
+By default the first 140 characters of each input and output are stored for search. To turn this off or mask sensitive data:
+
+```python
+import re
+
+# Store no prompt/response text, and hide error message text (exception types and status codes are kept)
+tally = LLMTally(log_previews=False)
+
+# Mask text in previews, error messages, retry-attempt messages, and string values in `config`
+tally = LLMTally(redact=lambda text: re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[email]", text))
+```
+
+If your `redact` function raises, the text is stored as `"[redaction failed]"` and a warning is shown. The call is still logged. `metadata` values are stored exactly as you pass them.
+
+---
+
+## Custom Storage & Sinks
+
+Every finished call is passed as a `CallRecord` to your `on_record` callables. Use them to send records to a database, OpenTelemetry, a queue, etc. A failing sink only warns; it never breaks the LLM call.
+
+```python
+def ship(record):
+    db.insert("llm_calls", record.to_dict())
+
+tally = LLMTally(on_record=ship)                       # JSONL file + your sink
+tally = LLMTally(on_record=ship, persist_logs=False)   # your sink only (no local file)
+tally = LLMTally(on_record=ship, async_sinks=True)     # run sinks on a background thread
+tally.add_sink(another_sink)
+```
+
+By default sinks run inside the call, so a slow sink adds its latency to every LLM call. With `async_sinks=True` records go through a bounded background queue (`sink_queue_size`, default 10,000). If the queue fills up, records are dropped with a warning. `tally.flush()` (also run at exit) waits for queued records to be delivered.
+
+Streams produce exactly one record each, once they finish. A stream that is abandoned before it ends is logged with `status="incomplete"`. An SDK stream manager that fails when the `with` block opens (e.g. Anthropic `messages.stream()` failing to connect) is logged as `failed` with the real error. Retries don't apply to it, because the SDK sends that request on entry.
+
+### The JSONL log file
+- **Append-only and safe across processes**: every write is one appended line under a POSIX file lock, so multiple threads and processes on one machine never lose records. A finished stream appends its final version, and readers keep the **last line for each `id`**. Run `tally.logger.compact()` or `llm-tally compact` to rewrite the file with one line per call.
+- **Incremental reads**: reports and the dashboard only parse what was appended since the last read.
+- **Rotation**: `LLMTally(max_log_mb=100, log_backups=3)` moves the log to `logs.jsonl.1` ... `.3` when it would exceed 100 MB. Reports and the dashboard only read the current file.
+
+---
+
+## Custom Models & Pricing
+
+```python
+tally.register_model("my-company-fine-tune", input_cost_per_1m=1.20, output_cost_per_1m=4.80)
+
+# Or keep prices in a file you can update without a release:
+tally = LLMTally(pricing_file="prices.json")
+tally.load_pricing("more_prices.json")
+```
+
+```json
+{
+  "my-company-fine-tune": {"input_cost_per_1m": 1.2, "output_cost_per_1m": 4.8, "provider": "custom"},
+  "gpt-4o": {"input_cost_per_1m": 2.5, "output_cost_per_1m": 10.0, "cached_input_cost_per_1m": 1.25}
+}
+```
+
+Unknown models fall back to $0.50 / $1.50 per 1M tokens and emit a `UserWarning`.
+
+---
+
+## Dashboard & Reports
+
+The dashboard is regenerated automatically, but **at most once every `dashboard_update_interval` seconds** (default `2.0`), so the overhead per call stays flat as logs grow. If a rebuild is slow (a very large log), the interval stretches to 10× the rebuild time, so rebuilding never uses more than about 10% of a core. A pending update is written in the background when the interval ends, and at interpreter exit.
+
+```python
+tally = LLMTally(dashboard_update_interval=10.0, dashboard_days=30)  # last 30 days only
+
+report = tally.get_report(days=7)
+print("Spent this week ($):", report["spend_summary"]["this_week_usd"])
+print("Most expensive feature:", report["overview"]["top_cost_feature"])
+print("Estimated-token calls:", report["overview"]["estimated_calls"])
+
+tally.flush()                           # write any pending dashboard update now
+path = tally.generate_dashboard()       # or force a full rebuild
+```
+
+Set `dashboard_update_interval=0` to rebuild on every call, or `auto_update_dashboard=False` to only build it yourself.
+
+### Command line
+
+```bash
+llm-tally report                                  # summary of .llm_tally/logs.jsonl
+llm-tally report --days 7 --json                  # full analytics as JSON
+llm-tally dashboard --log logs.jsonl --out dash.html --days 30
+llm-tally compact --log logs.jsonl                # one line per call
+python -m llm_tally report                        # same, without the console script
+```
+
+---
+
+## License
+
+MIT