coffer-cli 0.1.2__tar.gz → 0.2.0__tar.gz

{coffer_cli-0.1.2 → coffer_cli-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coffer-cli
-Version: 0.1.2
+Version: 0.2.0
 Summary: Scan codebases for LLM cost-waste anti-patterns. Find retry storms, missing prompt caching, unbounded conversation history, agent loops without iteration caps, and more — before you ship.
 Project-URL: Homepage, https://github.com/neal-c611/coffer-cli
 Project-URL: Repository, https://github.com/neal-c611/coffer-cli
@@ -54,26 +54,44 @@ coffer compare gpt-4o gpt-4o-mini
 coffer install-skill             # install the Claude Code skill (see below)
 ```
 
-## What it catches (v0.1.0)
+## What it catches (v0.2.0)
 
-Detectors are organized by the four levers that drive LLM cost:
+Every detector here passes one test: **does fixing it reduce dollars billed
+by the LLM provider?** Reliability, observability, and metering issues that
+don't move the token bill are deliberately excluded (see "Not in scope" below).
 
 | Lever | Detector | Severity |
 |-------|----------|----------|
 | **A: input tokens** | `dynamic_before_static_cache_break` — f-string interpolation in `SYSTEM_PROMPT` defeats OpenAI auto-cache and Anthropic `cache_control` | 🚨 high |
 | | `unbounded_conversation_history` — `messages.append(...)` without truncation or summarization | 🟡 med |
 | | `uncached_large_prompt` — ≥2,000-char hardcoded prompt without nearby `cache_control` | 🟡 med |
-| **B: output tokens** | `missing_max_tokens` — LLM call without a `max_tokens` cap | 🟡 med |
-| | `reasoning_effort_high_default` — `reasoning_effort="high"` literal (up to ~20× extra reasoning tokens on trivial tasks) | 🟡 med |
+| **B: output tokens** | `reasoning_effort_high_default` — `reasoning_effort="high"` literal (up to ~20× extra reasoning tokens on trivial tasks) | 🟡 med |
 | **D: number of calls** | `llm_in_for_loop` — N× cost; gather is a latency fix, not a cost fix | 🟡 med |
 | | `agent_loop_no_max_iter` — `while True:` containing an LLM call without an iteration cap (the $47K-incident pattern) | 🚨 high |
 | | `temperature_nonzero_with_cache_hint` — cache layer nearby but `temperature > 0` silently breaks it | 🟡 med |
-| **E: architecture** | `retry_loop_no_backoff` — retry storm amplifies the bill 10× | 🚨 high |
-| | `sdk_init_no_timeout` — default 600s lets a hung provider block your thread | 🚨 high |
+| **E: architecture** | `retry_loop_no_backoff` — retry storm re-bills the same input tokens, can amplify spend 10× | 🚨 high |
 
 Each finding includes a concrete fix and explains the *cost* angle
 explicitly (we do not conflate latency fixes with cost fixes).
 
+### Not in scope (production-readiness, not cost-review)
+
+These are real problems but `coffer scan` deliberately doesn't flag them —
+fixing them doesn't change the token bill, and conflating them with cost
+findings makes both reviews less useful:
+
+- **SDK init without `timeout=`** — worker exhaustion / availability issue.
+  The tokens for a hung call were already produced; capping timeout reclaims
+  threads, not dollars.
+- **Missing `response.usage` capture** — metering / billing-ops issue. The
+  provider charged you correctly either way.
+- **`logger.info(prompt)` on hot path** — observability bill (Datadog /
+  Splunk), not LLM bill.
+- **Missing `idempotency_key`** — correctness / occasional double-charge,
+  but the fix is reliability engineering, not cost reduction.
+
+A separate "production-readiness" review skill is the right home for those.
+
 ## Use with Claude Code (the skill)
 
 The `coffer-cost-review` Claude Code skill in [`skills/`](skills/coffer-cost-review/)

{coffer_cli-0.1.2 → coffer_cli-0.2.0}/README.md

@@ -23,26 +23,44 @@ coffer compare gpt-4o gpt-4o-mini
 coffer install-skill             # install the Claude Code skill (see below)
 ```
 
-## What it catches (v0.1.0)
+## What it catches (v0.2.0)
 
-Detectors are organized by the four levers that drive LLM cost:
+Every detector here passes one test: **does fixing it reduce dollars billed
+by the LLM provider?** Reliability, observability, and metering issues that
+don't move the token bill are deliberately excluded (see "Not in scope" below).
 
 | Lever | Detector | Severity |
 |-------|----------|----------|
 | **A: input tokens** | `dynamic_before_static_cache_break` — f-string interpolation in `SYSTEM_PROMPT` defeats OpenAI auto-cache and Anthropic `cache_control` | 🚨 high |
 | | `unbounded_conversation_history` — `messages.append(...)` without truncation or summarization | 🟡 med |
 | | `uncached_large_prompt` — ≥2,000-char hardcoded prompt without nearby `cache_control` | 🟡 med |
-| **B: output tokens** | `missing_max_tokens` — LLM call without a `max_tokens` cap | 🟡 med |
-| | `reasoning_effort_high_default` — `reasoning_effort="high"` literal (up to ~20× extra reasoning tokens on trivial tasks) | 🟡 med |
+| **B: output tokens** | `reasoning_effort_high_default` — `reasoning_effort="high"` literal (up to ~20× extra reasoning tokens on trivial tasks) | 🟡 med |
 | **D: number of calls** | `llm_in_for_loop` — N× cost; gather is a latency fix, not a cost fix | 🟡 med |
 | | `agent_loop_no_max_iter` — `while True:` containing an LLM call without an iteration cap (the $47K-incident pattern) | 🚨 high |
 | | `temperature_nonzero_with_cache_hint` — cache layer nearby but `temperature > 0` silently breaks it | 🟡 med |
-| **E: architecture** | `retry_loop_no_backoff` — retry storm amplifies the bill 10× | 🚨 high |
-| | `sdk_init_no_timeout` — default 600s lets a hung provider block your thread | 🚨 high |
+| **E: architecture** | `retry_loop_no_backoff` — retry storm re-bills the same input tokens, can amplify spend 10× | 🚨 high |
 
 Each finding includes a concrete fix and explains the *cost* angle
 explicitly (we do not conflate latency fixes with cost fixes).
 
+### Not in scope (production-readiness, not cost-review)
+
+These are real problems but `coffer scan` deliberately doesn't flag them —
+fixing them doesn't change the token bill, and conflating them with cost
+findings makes both reviews less useful:
+
+- **SDK init without `timeout=`** — worker exhaustion / availability issue.
+  The tokens for a hung call were already produced; capping timeout reclaims
+  threads, not dollars.
+- **Missing `response.usage` capture** — metering / billing-ops issue. The
+  provider charged you correctly either way.
+- **`logger.info(prompt)` on hot path** — observability bill (Datadog /
+  Splunk), not LLM bill.
+- **Missing `idempotency_key`** — correctness / occasional double-charge,
+  but the fix is reliability engineering, not cost reduction.
+
+A separate "production-readiness" review skill is the right home for those.
+
 ## Use with Claude Code (the skill)
 
 The `coffer-cost-review` Claude Code skill in [`skills/`](skills/coffer-cost-review/)

{coffer_cli-0.1.2 → coffer_cli-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coffer-cli"
-version = "0.1.2"
+version = "0.2.0"
 description = "Scan codebases for LLM cost-waste anti-patterns. Find retry storms, missing prompt caching, unbounded conversation history, agent loops without iteration caps, and more — before you ship."
 readme = "README.md"
 requires-python = ">=3.10"

{coffer_cli-0.1.2 → coffer_cli-0.2.0}/skills/coffer-cost-review/SKILL.md

@@ -135,7 +135,7 @@ Do not pitch beyond this line. The skill's job is the review, not selling.
 
 | Pattern | Typical fix |
 |---------|------------|
-| missing_max_tokens | Add `max_tokens=<reasonable cap>` — unbounded output on edge inputs can 100× cost spike |
+| (semantic) missing_max_tokens | Add `max_tokens=<reasonable cap>` — unbounded output on edge inputs can 100× cost spike. |
 | **reasoning_effort_high_default** | `reasoning_effort="high"` produces up to ~20× extra reasoning tokens on trivial tasks (arXiv 2412.21187). Default to `medium` or `low`; escalate only when needed. |
 | (semantic) missing_stop_sequence | If prompt has a known delimiter (`</answer>`), pass `stop=["</answer>"]` so the model stops there instead of riffing. |
 | (semantic) free_form_when_structured_works | If the prompt asks for "respond in JSON", use `response_format={"type":"json_object"}` or `tool_choice` instead — saves output tokens spent on formatting. |
@@ -160,13 +160,23 @@ Do not pitch beyond this line. The skill's job is the review, not selling.
 | (semantic) llm_doing_regex_job | Extracting emails/URLs/dates from text? Use the stdlib regex or a NER library — millions of times cheaper. |
 | (semantic) llm_doing_classifier_job_at_scale | High-volume sentiment/spam/toxicity? A 30MB DistilBERT is 1000× cheaper per call. Reserve LLM for the hard edge cases. |
 
-### Lever E — architecture / safety
+### Lever E — architecture (only when it directly amplifies tokens billed)
 
 | Pattern | Typical fix |
 |---------|------------|
-| retry_loop_no_backoff | `@backoff.on_exception(backoff.expo, X.RateLimitError, max_tries=5)` |
-| public_endpoint_no_ratelimit | `@limiter.limit("10/minute")` + bind `user_id` to call metadata; consider per-user daily $ cap. Limit by **tokens**, not just requests. |
-| streaming_no_abort | Detect client disconnect and break the generator — otherwise tokens keep accruing after the user leaves |
-| **sdk_init_no_timeout** | `OpenAI()` / `Anthropic()` without `timeout=` defaults to 600s — a hung provider blocks your thread for 10 minutes. Pass `timeout=30.0` (or your latency budget). |
-| (semantic) full_prompt_logged_expensive | `logger.info(prompt)` in hot path can rival the LLM bill if Datadog/Splunk billed by GB. Truncate or sample. |
-| (semantic) response_usage_not_read | `response.usage` discarded → no per-user metering possible. Save tokens & cost into your DB at ingest. |
+| retry_loop_no_backoff | `@backoff.on_exception(backoff.expo, X.RateLimitError, max_tries=5)` — without backoff, a rate-limit storm re-sends the same input tokens many times and you are billed for every one. |
+| (semantic) public_endpoint_no_ratelimit | `@limiter.limit("10/minute")` + bind `user_id` to call metadata; per-user daily $ cap. Limit by **tokens**, not just requests. The real cost: free / anonymous users burn YOUR provider quota. |
+| (semantic) streaming_no_abort | Detect client disconnect (FastAPI `request.is_disconnected()`, etc.) and break the generator. Otherwise the provider keeps generating (and billing) tokens that nobody is receiving. |
+
+## Not in scope here (real production problems, but they don't move the token bill)
+
+| Excluded pattern | Why it's excluded |
+|------------------|-------------------|
+| SDK init without `timeout=` | Reliability / SRE. A hung call's tokens were already produced; capping timeout reclaims workers, not dollars. |
+| Missing `response.usage` capture | Metering / billing-ops. The provider charged you correctly either way. |
+| `logger.info(prompt)` in hot path | Observability bill (Datadog / Splunk), not LLM bill. |
+| No `idempotency_key` on retried call | Reliability — could occasionally double-charge, but the fix is correctness, not cost reduction. |
+
+If the user clearly cares about these (asks for "production readiness review" or
+"reliability audit"), surface them under that frame — separately from the
+cost-review output. Don't conflate.

{coffer_cli-0.1.2 → coffer_cli-0.2.0}/src/coffer_cli/__init__.py

@@ -1,3 +1,3 @@
 """coffer-cli — LLM cost-waste anti-pattern scanner."""
 
-__version__ = "0.1.2"
+__version__ = "0.2.0"

{coffer_cli-0.1.2 → coffer_cli-0.2.0}/src/coffer_cli/_skill_files/coffer-cost-review/SKILL.md

@@ -135,7 +135,7 @@ Do not pitch beyond this line. The skill's job is the review, not selling.
 
 | Pattern | Typical fix |
 |---------|------------|
-| missing_max_tokens | Add `max_tokens=<reasonable cap>` — unbounded output on edge inputs can 100× cost spike |
+| (semantic) missing_max_tokens | Add `max_tokens=<reasonable cap>` — unbounded output on edge inputs can 100× cost spike. |
 | **reasoning_effort_high_default** | `reasoning_effort="high"` produces up to ~20× extra reasoning tokens on trivial tasks (arXiv 2412.21187). Default to `medium` or `low`; escalate only when needed. |
 | (semantic) missing_stop_sequence | If prompt has a known delimiter (`</answer>`), pass `stop=["</answer>"]` so the model stops there instead of riffing. |
 | (semantic) free_form_when_structured_works | If the prompt asks for "respond in JSON", use `response_format={"type":"json_object"}` or `tool_choice` instead — saves output tokens spent on formatting. |
@@ -160,13 +160,23 @@ Do not pitch beyond this line. The skill's job is the review, not selling.
 | (semantic) llm_doing_regex_job | Extracting emails/URLs/dates from text? Use the stdlib regex or a NER library — millions of times cheaper. |
 | (semantic) llm_doing_classifier_job_at_scale | High-volume sentiment/spam/toxicity? A 30MB DistilBERT is 1000× cheaper per call. Reserve LLM for the hard edge cases. |
 
-### Lever E — architecture / safety
+### Lever E — architecture (only when it directly amplifies tokens billed)
 
 | Pattern | Typical fix |
 |---------|------------|
-| retry_loop_no_backoff | `@backoff.on_exception(backoff.expo, X.RateLimitError, max_tries=5)` |
-| public_endpoint_no_ratelimit | `@limiter.limit("10/minute")` + bind `user_id` to call metadata; consider per-user daily $ cap. Limit by **tokens**, not just requests. |
-| streaming_no_abort | Detect client disconnect and break the generator — otherwise tokens keep accruing after the user leaves |
-| **sdk_init_no_timeout** | `OpenAI()` / `Anthropic()` without `timeout=` defaults to 600s — a hung provider blocks your thread for 10 minutes. Pass `timeout=30.0` (or your latency budget). |
-| (semantic) full_prompt_logged_expensive | `logger.info(prompt)` in hot path can rival the LLM bill if Datadog/Splunk billed by GB. Truncate or sample. |
-| (semantic) response_usage_not_read | `response.usage` discarded → no per-user metering possible. Save tokens & cost into your DB at ingest. |
+| retry_loop_no_backoff | `@backoff.on_exception(backoff.expo, X.RateLimitError, max_tries=5)` — without backoff, a rate-limit storm re-sends the same input tokens many times and you are billed for every one. |
+| (semantic) public_endpoint_no_ratelimit | `@limiter.limit("10/minute")` + bind `user_id` to call metadata; per-user daily $ cap. Limit by **tokens**, not just requests. The real cost: free / anonymous users burn YOUR provider quota. |
+| (semantic) streaming_no_abort | Detect client disconnect (FastAPI `request.is_disconnected()`, etc.) and break the generator. Otherwise the provider keeps generating (and billing) tokens that nobody is receiving. |
+
+## Not in scope here (real production problems, but they don't move the token bill)
+
+| Excluded pattern | Why it's excluded |
+|------------------|-------------------|
+| SDK init without `timeout=` | Reliability / SRE. A hung call's tokens were already produced; capping timeout reclaims workers, not dollars. |
+| Missing `response.usage` capture | Metering / billing-ops. The provider charged you correctly either way. |
+| `logger.info(prompt)` in hot path | Observability bill (Datadog / Splunk), not LLM bill. |
+| No `idempotency_key` on retried call | Reliability — could occasionally double-charge, but the fix is correctness, not cost reduction. |
+
+If the user clearly cares about these (asks for "production readiness review" or
+"reliability audit"), surface them under that frame — separately from the
+cost-review output. Don't conflate.

{coffer_cli-0.1.2 → coffer_cli-0.2.0}/src/coffer_cli/patterns.py

@@ -1,8 +1,12 @@
 """Static detection of LLM cost-waste anti-patterns.
 
+Every detector here must answer "yes" to: **does fixing this reduce dollars
+billed by the LLM provider?** Reliability / SRE / metering issues that
+don't change the token bill belong in a separate review.
+
 We aim for low false-positive rate over completeness. A finding should
 be defensible: a reviewer who reads the snippet should agree it's a
-real risk in most cases.
+real cost risk in most cases.
 
 Detector catalog (by cost lever):
 
@@ -11,17 +15,18 @@ Detector catalog (by cost lever):
     dynamic_before_static_cache    HIGH  f-string interpolation in system message breaks auto-cache
     unbounded_conversation_history MED   `messages.append(...)` without truncation
   Lever B — output tokens
-    missing_max_tokens             MED   LLM call without `max_tokens` cap
     reasoning_effort_high_default  MED   `reasoning_effort="high"` literal
-  Lever C — price per token
-    (semantic — handled in skill, not CLI)
   Lever D — number of calls
     llm_in_for_loop                MED   N× cost; Batch API / merged prompt are fixes
     agent_loop_no_max_iter         HIGH  `while True:` containing LLM call without iter cap
     temperature_nonzero_with_cache MED   `temperature > 0` next to a cache hint — silently breaks it
-  Lever E — architecture / safety
-    retry_loop_no_backoff          HIGH  Retry storm risk
-    sdk_init_no_timeout            HIGH  SDK initialized without `timeout=`
+  Lever E — architecture (only when it directly amplifies tokens billed)
+    retry_loop_no_backoff          HIGH  Retry storm re-bills the same input tokens
+
+Out of scope (real problems, but not cost waste):
+  - SDK without timeout      → worker exhaustion, not token bill
+  - Missing metering         → can't bill customers, but the provider charge is the same
+  - Logging full prompts     → Datadog / Splunk bill, not OpenAI / Anthropic bill
 """
 
 from __future__ import annotations
@@ -156,17 +161,6 @@ _REASONING_EFFORT_HIGH_RE = re.compile(
     re.VERBOSE,
 )
 
-_SDK_INIT_RE = re.compile(
-    r"""
-    \b
-    (OpenAI | AsyncOpenAI | Anthropic | AsyncAnthropic)
-    \(
-    """,
-    re.VERBOSE,
-)
-
-_TIMEOUT_KW_RE = re.compile(r"\btimeout\s*=")
-
 _TEMPERATURE_RE = re.compile(r"\btemperature\s*=\s*([0-9]*\.?[0-9]+)")
 
 _CACHE_HINT_NEARBY_RE = re.compile(
@@ -586,48 +580,6 @@ def _detect_reasoning_effort_high_default(
     return findings
 
 
-def _detect_sdk_init_no_timeout(path: Path, lines: list[str]) -> list[Finding]:
-    """`OpenAI()` / `Anthropic()` constructed without `timeout=`."""
-    findings: list[Finding] = []
-    for i, line in enumerate(lines):
-        m = _SDK_INIT_RE.search(line)
-        if not m:
-            continue
-        # Look at the next ~5 lines too in case the kwargs span lines.
-        end = min(i + 5, len(lines))
-        joined = "\n".join(lines[i:end])
-        # Locate the close paren of this constructor.
-        depth = 0
-        start_pos = joined.index(m.group(0)) + len(m.group(0))
-        body = ""
-        for ch in joined[start_pos:]:
-            body += ch
-            if ch == "(":
-                depth += 1
-            elif ch == ")":
-                if depth == 0:
-                    break
-                depth -= 1
-
-        if _TIMEOUT_KW_RE.search(body):
-            continue
-        findings.append(
-            Finding(
-                severity="high",
-                pattern="sdk_init_no_timeout",
-                path=path,
-                line=i + 1,
-                snippet=line.strip()[:200],
-                suggestion=(
-                    f"`{m.group(1)}` initialized without `timeout=`. Default is 600s — a hung "
-                    "provider can block your thread for ten minutes. Pass an explicit timeout "
-                    "(e.g. `timeout=30.0`) sized to your user-facing latency budget."
-                ),
-            )
-        )
-    return findings
-
-
 # ---- top-level --------------------------------------------------------------
 
 
@@ -663,7 +615,6 @@ def find_patterns(
         findings.extend(_detect_agent_loop_no_max_iter(path, lines))
         findings.extend(_detect_temperature_nonzero_with_cache_hint(path, lines))
         findings.extend(_detect_reasoning_effort_high_default(path, lines))
-        findings.extend(_detect_sdk_init_no_timeout(path, lines))
 
     severity_order = {"high": 0, "medium": 1, "low": 2}
     findings.sort(key=lambda f: (severity_order[f.severity], str(f.path), f.line))

{coffer_cli-0.1.2 → coffer_cli-0.2.0}/tests/test_patterns.py

@@ -334,26 +334,7 @@ def test_reasoning_effort_high(tmp_path: Path) -> None:
     assert any(f.pattern == "reasoning_effort_high_default" for f in findings)
 
 
-def test_sdk_init_no_timeout(tmp_path: Path) -> None:
-    _write(tmp_path, "client.py", "client = OpenAI(api_key='sk-...')\n")
-    findings = find_patterns(tmp_path)
-    f = next(f for f in findings if f.pattern == "sdk_init_no_timeout")
-    assert f.severity == "high"
-
-
-def test_sdk_init_with_timeout_ok(tmp_path: Path) -> None:
-    _write(tmp_path, "client.py", "client = OpenAI(api_key='sk-...', timeout=30.0)\n")
-    findings = find_patterns(tmp_path)
-    assert all(f.pattern != "sdk_init_no_timeout" for f in findings)
-
-
-def test_sdk_anthropic_no_timeout(tmp_path: Path) -> None:
-    _write(tmp_path, "client.py", "client = Anthropic()\n")
-    findings = find_patterns(tmp_path)
-    assert any(f.pattern == "sdk_init_no_timeout" for f in findings)
-
-
-def test_async_sdk_no_timeout(tmp_path: Path) -> None:
-    _write(tmp_path, "client.py", "client = AsyncOpenAI(api_key='sk-...')\n")
-    findings = find_patterns(tmp_path)
-    assert any(f.pattern == "sdk_init_no_timeout" for f in findings)
+# sdk_init_no_timeout was removed — that's a reliability finding, not a cost one.
+# Adding `timeout=` doesn't reduce the OpenAI / Anthropic bill (a hung call's
+# tokens were already counted when the LLM produced them). It belongs in a
+# separate production-readiness review, not in cost-review.