tokenjam 0.6.2__tar.gz → 0.6.4__tar.gz

{tokenjam-0.6.2 → tokenjam-0.6.4}/.github/workflows/ci.yml

@@ -23,6 +23,25 @@ jobs:
             exit 1
           fi
 
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Lint
+        run: ruff check tokenjam/
+
+      - name: Type check
+        run: mypy tokenjam/
+
   test:
     runs-on: ubuntu-latest
     strategy:
@@ -40,12 +59,6 @@ jobs:
       - name: Install dependencies
         run: pip install -e ".[dev,mcp]"
 
-      - name: Lint
-        run: ruff check tokenjam/
-
-      - name: Type check
-        run: mypy tokenjam/
-
       - name: Run tests
         run: pytest tests/unit/ tests/synthetic/ tests/agents/ tests/integration/
 

{tokenjam-0.6.2 → tokenjam-0.6.4}/CLAUDE.md

@@ -203,6 +203,34 @@ Post-ingest hooks run after each span is written to DB:
 
   **A headline figure is the CEILING, not the expectation — and a ceiling still has to be reachable.** Where the arithmetic legitimately supports a range, the number a card leads with is the top of it: the largest figure the derivation honestly bears, not a discounted guess at what a user would actually realize. This is a founder decision and it is deliberate, so do not "correct" a headline downward by applying a realization factor, an adoption rate, or any other haircut invented at the render layer. It is not, however, a licence to publish an unreachable number: a ceiling has to be attainable in *some* real world, so a slice the fix **structurally cannot touch** — spans the remedy does not reach, sessions the analyzer never examined — does not belong inside it, and shaving those out is a correctness fix rather than a haircut. One carve-out: where a card offers a fix the user is about to apply, lead with the realized figure instead, because at that point the honest question has changed from "how large could this be" to "what will this specific action do". This is a convention about which of two defensible numbers to lead with; the numbers themselves belong on the cards and in `estimate_basis`, never in this file.
 
+  **WHERE a permanent rule lands is an INPUT to whether it is offered, not a presentation of the decision.** Four analyzers (`downsize`, `resend`, `subagent`, `relearn`) share one fix shape — a block appended to a `CLAUDE.md` — and each used to offer exactly one destination, the workspace/root file. That is wrong in both directions: a root-level rule is re-sent in every session of every project, including the ones that never exhibited the problem, and it dilutes the file whose dilution is what makes rules start being ignored. Every session records its working directory (`core/transcript.first_recorded_cwd` is the one extractor; `core/summarize/repo_roots.resolve_roots` turns those into boundary-safe roots), so `core/optimize/rule_placement` can resolve which projects incurred a finding and attribute each session's share to a destination. Because the standing cost is `per-session tokens x the sessions that load the file`, confining a rule to the projects that exhibited the behaviour can flip a net-negative verdict to net-positive with no change to the finding at all. Three things fall out and are easy to get wrong. **(a) Two costs, not one.** STANDING cost follows the sessions (what the rule costs to keep, and the only figure the saving is netted against); FOOTPRINT cost follows the FILES (`per-session x destinations written`, and the only figure the write budget is spent in). A three-project split can halve the standing cost while tripling the budget spend; charging either for the other's question makes the placement decision stop being arithmetic. **(b) The growth allowance is per FILE.** `measured_agent_file_tokens` aggregates the `summarize` scan, which is right for "may this window grow the files at all" and wrong for "may this rule grow THIS file" — 10% of an aggregate dominated by a large root file can exceed a small project `CLAUDE.md` entirely, so `measured_agent_file_tokens_by_path` + `WriteBudget.destination_budget` size each destination against its own measured size. **(c) Unifying the FIX surface must not merge the CLAIMS** (Critical Rule 27): the per-session weight maps (`DowngradeFinding.driver_session_tokens`, `ResendFinding.session_weights`) are BREAKDOWNS of each analyzer's own `past_overspend_tokens`, split by the same weights on both the token and the dollar side so every destination's implied rate equals the finding's own (Critical Rule 28). A recorded cwd that no longer exists is counted and surfaced in a `coverage_note` saying the gap is what was **not analysed** (Critical Rule 30), never reconstructed and never redistributed onto the destinations that did resolve.
+
+  **A rule a card writes must ERASE the number the card found.** `past_overspend_usd` is the maximum its analyzer knows: applying the fix should remove the behaviour that produced it. If applying it would leave the number roughly where it was, that is a correctness bug, not a UX gap. The live example: `SUBAGENT_RUBRIC_INTRO` used to end with "a subagent that does little tool work and returns a short result rarely needs the premium tier" — exactly the `output_tokens`/`tool_calls` gate deleted from `analyzers/subagent_rightsizing.py` for inverting on the expensive dispatches (Critical Rule 29), so the rule told the agent to keep doing the thing the card was billing it for. `context_resend.RIGHTSIZE_FIX_TEMPLATE` carried the same sentence in a different costume. Two durable halves: **the shape-based default-down is the lever and stays**, and **a premium escape hatch must be checkable by an outside reader rather than self-assessed** — "unless the subtask genuinely needs deep reasoning" asks the dispatching agent to rate its own task's difficulty, and an agent asked that question answers yes, so the exception swallows the rule. State the exception as conditions that are either written into the dispatch or not.
+
+  **HOW a rule reaches the agent is a DELIVERY MECHANISM, and "rung 3 is free" is FALSE for a context-injecting one.** Appending markdown to a `CLAUDE.md` is one way to put guidance in front of an agent; a Claude Code hook is another, and for several analyzers a better one, because it delivers at the moment of the decision rather than at the top of a long context. `core/rulewrite/delivery.py` is the seam: a `DeliveryKind` owns its own renderer AND its own pricer, and adding a mechanism is a registration plus two functions — never an edit to the staging, diff, apply, undo or budget machinery, none of which may name a CLAUDE.md. The pricing half is the part that bites. `write_budget`'s rung ladder charges rung 3+ ZERO because a settings file is *executed*, never sent as prompt text; that is earned by a formatter or a guard hook, and it is **wrong for a `UserPromptSubmit` re-injection or a `PreToolUse` additionalContext nudge**, which are prompt text on a different SCHEDULE. Their cost is worse-behaved than a rule's, not better: an injected block lands in the conversation and is re-sent every subsequent turn, so it is not even bounded by the number of injections the way a session-start read is bounded by the session count. The ladder grades intervention STRENGTH, not delivery, so the question "does this put tokens in front of the model" is asked of the mechanism (`DeliveryKind.carries_prompt_text`) and never inferred from the rung. Caught in this repo's own first draft of the seam: a rung-3 kind flagged as carrying prompt text still priced to zero, because its pricer routed through the ladder. An unknown mechanism is REFUSED rather than falling back to markdown — a wrong write to a real file is the one failure the lifecycle exists to prevent.
+
+  **A fix the user already dealt with must lose its OFFER and keep its FIGURE.** The rule-write surface consults the apply ledgers (`cost_applied.json` / `applied_fixes.json`, via `cost_apply.applied_signatures` / `relearn_apply.applied_signatures`, matched by `cost_apply.signature_is_applied`) and withdraws the offer for a signature already applied. `past_overspend_usd`/`_tokens` are untouched, and that is Critical Rule 32 in its sharpest form: the waste genuinely happened inside the analyzed window, and the user having since fixed it does not un-spend the money. An action-availability gate establishes that WE have no remedy left to offer, never that the spend was avoidable or free — so it may never reach back and edit a past figure, and a zero there would be the exact "no action for this" / "this was free" conflation. Pin it by asserting the whole record is byte-identical with and without the ledger entry EXCEPT the offer fields; an equal-value check passes a figure that got re-derived through a zero. Two further rules that cost real correctness here: **the row stays listed** carrying an applied state rather than vanishing (a user who applies something and then sees nothing cannot tell "done" from "broken"), and **an unreadable ledger reads as "nothing applied"** — that direction can only waste attention, while the opposite hides fixes the user never made. Reuse `signature_is_applied` rather than writing a second matcher: it already resolves a legacy agent-only mark as covering the later model-qualified signatures for that agent, and a second matcher silently reopens every card it settles.
+
+  **A fix must name an action that EXISTS, for THIS user, through a key the harness actually reads.** Four shipped defects, one framing failure — applying the fix has to be able to erase the number its analyzer found. (a) The right-sizing snippet emitted `reasoning_effort:`, which Claude Code does not read; the field is `effort` (`low`/`medium`/`high`/`xhigh`/`max`). **A silently-ignored key is worse than no line**: it converts an unfixed problem into one the user believes is fixed. It was also hardcoded to `low` for a population that is by construction large and tool-heavy, so the guess recommended making the expensive work worse — a value that cannot be derived from the observation is OMITTED, never guessed, because a plausible line in a frontmatter block is indistinguishable from a measured one. (b) The subagent lane required an agent definition file to already EXIST, and every Claude Code dispatch is a built-in (`general-purpose`, `Explore`, `Plan`, `fork`) which has none — so the branch was unreachable on the dominant corpus and the claim fell through to prose. **A user or project subagent of the same name overrides the built-in and keeps its own `model`, so no-file is not no-fix: offer the CREATE.** The origin belongs in the evidence too — `model` defaults to `inherit`, so an Opus-driven session hands every dispatch an Opus worker unless pinned; that is an unset default, not a decision, which is why it goes unnoticed. Watch the shape checks around this: a lowercase-only slug predicate silently rejected `Explore` and `Plan`, the exact names the fix targets. (c) The per-agent downsize lane told Claude Code users to "redeploy or restart the agent", but there `agent_id` is `claude-code-<cwd-basename>` — a PROJECT DIRECTORY with ephemeral sessions, no process and no model id written down. Gate that with a **persona check at the point of production** (Critical Rule 26(c)), not a dollar threshold, and withhold only the offer. (d) `relearn`'s `edit_before_read` shipped a card whose own fix text says the harness already handles it and no rule is needed. `is_placeholder_fix` cannot see this — the text is a real, specific sentence that happens to say there is nothing to do — so the family carries an explicit `advisory_only` flag and `REASON_ADVISORY_ONLY`, distinct from the placeholder reason because "no template matched" and "none is needed" are different statements. In all four the observation stands and `past_overspend_*` is untouched (Critical Rule 32).
+
+  **Fix text lives in ONE catalog with a lint, because scattered constants are why the above could happen.** `core/fixes/` holds one `FixRecord` per fix type — text, delivery kind, personas, lever, the observation it answers, and the shapes it may not excuse — and every analyzer reads its text from there instead of defining a constant. The proof this is structural, not tidiness: the identical sizing contradiction shipped in BOTH the subagent rubric and the resend right-sizing template in different words, so correcting the reported one left the other live. Two copies of a policy is two policies. `core/fixes/lint.py` checks the properties mechanically — under the published line ceiling, long enough to be an instruction, no escape hatch the agent grades itself against, advisory-only text never offered as a write, fixed-point behaviour escalated to a hook — and above all: **a fix may not re-license the behaviour its own analyzer bills for**, which is the whole D-class and is now caught by a substring check against the record's own `must_not_relicense` set rather than by review. Two dimensions the catalog carries that a bare string cannot: **persona** (a Claude Code user edits instruction files, agent files and hooks; an SDK service edits routing code and redeploys — handing either the other's fix names an action they cannot take) and **lever**, where `model` and `effort` are separate and not substitutes: a worker that did not know enough needs a bigger model, one that did not try hard enough needs more effort.
+
+  **One instruction, one record — and the duplicate check must use CONTAINMENT, not Jaccard.** Three analyzers (`resend`, `downsize`, `relearn`'s `context_overflow`) each authored their own wording of "delegate context-heavy work to a subagent", so three near-identical blocks could land in one `CLAUDE.md`. The write budget's one-block-per-family rule cannot see this: they are three families from three analyzers. The harm is not untidiness — **length and redundancy reduce adherence**, so writing a rule three times makes it LESS likely to be followed than writing it once, and each analyzer's copy actively defeats the others. Their CLAIMS stay disjoint (Critical Rule 27 is untouched — they price different span populations); only the instruction is shared, with per-analyzer framing living in the card's advise text rather than in a second copy of the rule. The lint's metric is load-bearing and the obvious choice is wrong: **Jaccard divides by the union, so it collapses when two texts differ in LENGTH, which is exactly this defect's shape** — measured against the real wordings, the two duplicates scored 24% and 17% Jaccard against the rule they duplicated and would have been certified clean. Containment (`|a ∩ b| / min(|a|,|b|)`) scores the same pairs at 67% and 72%. A check that cannot catch what it was written for is worse than no check, because it reports the problem as absent.
+
+  **Two surfaces rendering one finding must lead with the SAME fix, through the same helper.** The Review inbox card led with the durable offload rule while `tj optimize`'s CLI branch led with `/compact`, for the identical finding — and the CLI led with the weaker one, which `COMPACTION_FIX`'s own text disclaims ("never fixes the pattern going forward; treat it as immediate relief, not the durable fix"). Both now render through `cost_proposals.compound_offload_fix`, so they cannot drift into different wordings again. Related: **`/compact` is never listed as an actionable lever.** It is transient, persists nothing past the session, and users who feel a session filling already do it unprompted — it sits below the actionable ceiling recorded in `.claude/product-state/ANALYZER-PERSONA-MATRIX.md`. Listing it beside durable levers implies it is the same kind of thing, and pads a list whose length is itself a cost to adherence. It survives only in an explicitly-labelled "immediate relief in an already-full session" position.
+
+  **A path-scoped rule is not a cheaper way to say the same thing — it changes WHICH fixes are offerable at all.** A `.claude/rules/*.md` carrying a `paths:` glob loads only when Claude reads a matching file, so only its frontmatter is resident (measured on a real record: ~13% of the same words in a `CLAUDE.md`). Because standing cost is an INPUT to the offer decision, that near-zero rent flips fixes from net-negative to net-positive: the same observation that cannot carry a `CLAUDE.md` rule can carry a scoped one. Three things are load-bearing. **(a) The frontmatter is the entire reason it is cheap** — a rule in the same directory WITHOUT `paths:` loads at launch with `CLAUDE.md` priority, so `load_semantics.classify` takes the file TEXT and decides on `has_paths_scope`, never on the directory; a path-scoped kind that emits no globs has silently become an always-resident rule while still being priced as if it were not, and the renderer refuses rather than degrading because the failure is invisible (the file is written, the rule works, only the cost is wrong). Accept both YAML forms — a block list's inline value is empty, so an inline-only check reads a perfectly good scoped rule as unscoped. **(b) Selection is DERIVED from the LEVER, never preferred.** A rule about the shape of the next action (model, effort, offload, routing) decides something *before any file is read*; scoped to globs it would load only after the decision it governs was made — present, well-formed and useless, which is worse than paying the rent and is invisible. Only file-shaped (awareness) fixes may scope, and only when the record names its own globs; inferring a glob from prose is guessing which files an instruction is about. **(c) Brace expansion has a budget** — past it a pattern is used UNEXPANDED and matches nothing, so the rule reads as applied while never firing; refuse such a glob rather than emitting a silent no-op.
+
+  **Hooks are already written; what was wrong was their PRICE.** `relearn_apply` has shipped hook generation for some time (guard + reactive renderers, the settings patch, the enforcement wiring), so "nothing writes a hook" is not the gap — three of the four rung-3 families that ship are `PostToolUseFailure` reactive hooks that inject `hookSpecificOutput.additionalContext`, and the rung ladder priced all of them at ZERO. For an executing guard that zero is earned; for an injecting hook it is false, and its cost is worse-behaved than a rule's, because the injected block lands in the conversation and is re-sent on every subsequent turn rather than being bounded by session count. Hence two delivery kinds, not one, each with its own pricer. Three rails the generated hook must carry, and they are not interchangeable with a rule's: **`additionalContext` MUST be nested inside `hookSpecificOutput`** (at top level it is silently ignored — the hook runs, exits 0, injects nothing, and looks like it worked); **the per-session cap lives in the script**, keyed on the stdin `session_id`, because only `SessionStart` is structurally once-per-session and the `once` field is honored solely in skill frontmatter, never in a settings file; and **injection is advisory, fail-open, never blocking by default** — a misfiring blocking hook degrades every session until removed, an asymmetry that justifies the rails on its own. Hook entries MERGE across settings layers rather than overriding, so a write cannot clobber the user's hooks but also must not assume it is the only writer.
+
+  **Ground a behavioural rule in the evidence it came from — by REPLACING vague words, never by appending them.** The behavioural fixes (`downsize`'s driver-role card, `resend`'s offload rule, `relearn`'s rung-1 notes) described a HABIT generically while the analyzer that produced them knew which directories the user worked in, which tools drove the tail and which models ran it. That is the shape of instruction the published guidance says gets ignored ("concrete enough to verify": `API handlers live in src/api/handlers/` beats `keep files organized`). Four rules. **(a) Substitution, never generation** — a template carrying "you did X in Y" beats both a generic sentence and a model call: no cost, no latency, no new failure mode, and it cannot hallucinate a directory the user does not have. **(b) SPECIFIC MUST NOT MEAN LONGER.** The guidance is specific *and concise*, and a longer instruction file reduces adherence — the very thing grounding is for — so a substitution REPLACES a vague span and a grounded rendering that came out materially longer is DISCARDED for the generic text (`grounding.MAX_GROUNDED_GROWTH`). On the reference corpus the grounded offload rule is 94% the length of the generic one it replaces, naming three real repos in place of a parenthesised enumeration; the failure mode to guard against is the generic sentence plus an evidence paragraph, which looks like an improvement and is a regression. **(c) Grounding lives in RENDERING, never in the record set** — one catalog record per fix, always; minting a record per user or per finding makes the catalog and its duplicate lint meaningless overnight, so the record declares which spans may be replaced and the renderer fills them. **(d) Degrade per-substitution, and never invent specificity** — an unobserved slot skips its own substitution and keeps that generic span, so a rule can be grounded in its directory even when the tool mix was not observed; an empty field means not observed and is never filled with a plausible-looking value. Mechanical fixes (`deadweight`'s "remove server X", `subagent`'s "set `model:` here") are already concrete by construction and are deliberately left alone — this is specific to rules that describe a habit rather than an edit.
+
+  **A user's DISMISSAL is a decision and belongs server-side — durable, reversible, and figure-safe.** It used to live in the browser (`localStorage`), which means it was not really recorded: a cleared profile, a second browser or a different machine and every dismissed card came back. It now lands in `dismissed.json` beside the DB, the same lock-free sink the applied/reverted ledgers use — and **not** a DuckDB table, for a structural reason rather than a historical one: `tj rules` is in `no_db_commands` and must answer while `tj serve` holds the write lock, and DuckDB permits one writer OR many readers with no concurrent read-only escape, so a dismissal stored in DuckDB could not be READ by the command that has to honour it whenever the daemon is up. Three properties, each a test. **(a) It suppresses the OFFER only** (Critical Rule 32) — the behaviour happened and cost what it cost, and "not this one" is a statement about our recommendation, not about the user's bill; the record is byte-identical with and without the dismissal except the offer fields. **(b) It must be REVERSIBLE, and durability is exactly what makes that non-negotiable** — a dismissal that cannot be undone trades a card that came back on every browser for one that never comes back anywhere, so the row stays listed carrying its state and `undismiss` restores it. **(c) It resolves through the ONE matcher** (`cost_apply.signature_is_applied`) and states its exclusion filter once, like its two sibling ledgers — a dismissal that did not honour the legacy-signature equivalence would let a card dismissed under one signature reappear under its refinement. Browser-local dismissals are not migrated: `localStorage` is unreadable server-side, so they simply lapse, and nothing in the UI pretends otherwise.
+
+  **A flag set UPSTREAM of a pass that rewrites the same field is not a flag, it is a suggestion.** `relearn`'s advisory families set `write_offered=False` at proposal construction, and `_apply_write_budget` then assigned `write_offered=decision.offered` from an allocator that knows nothing about advisory — so the withdrawal held in the unit test (which exercised the constructor) and was silently undone on the live daemon. **Unit tests cannot see this class**: they call the stage that sets the field, not the stage that overwrites it. The fix is to keep such a record out of the later pass entirely rather than to set the field twice. Two general lessons. (a) When a value is written at two stages, the LATER one wins and the earlier one is decoration — either remove the earlier write or exclude the record from the later stage. (b) **A defect that survives a green suite is usually a stage-boundary defect**, so verify through the composed path (here `_apply_write_budget` directly) rather than through the constructor, and boot the daemon against a real corpus before believing a behavioural fix shipped. The same live pass found placement's per-destination session counts computed and never serialized — the payload carried only paths, so every consumer reported `sessions: 0` for destinations the resolver had counted correctly, which is the one figure that justifies placing a rule at all.
+
+  **A gate the user cannot pass is not a gate, it is a dead end — and a ceiling over an AGGREGATE stops being meaningful once the thing it gates can be placed elsewhere.** The write budget's standing ceiling withheld every rule whenever the user's agent files were collectively too large, and told them to compress. Measured on seven real instruction files across six repos, compression returns a ~0.952 prose ratio (much of it whitespace reflow), so the two largest cards this product produces were gated behind a remedy that returns almost nothing. Two fixes, and both generalise. **Name a remedy that works**: these files are long by ACCUMULATION (numbered incident lists, append-only logs), where what frees space is pruning what no longer applies, expiring what was time-bound, and moving file-specific guidance into a path-scoped rule — compression works on prose, and these files are not long because their prose is verbose. **Scope the gate to what it is actually about**: the ceiling now asks whether THIS destination is too large, not whether the corpus is, because one pathological root file was blocking writes into nearly-empty project files with ample headroom. An aggregate gate was correct when there was one destination; placement made it answer a question nobody asked.
+
   **Net-of-standing-cost is part of the contract, not an extra.** A fix that WRITES a permanent artifact into the user's agent files is not free: a rung-1 `CLAUDE.md` rule is re-sent on every future session, forever. Any analyzer whose fix is such a write MUST report `past_overspend_*` **net** of that standing cost and park the pre-net figure on a `gross_*` field; a proposal may never claim a saving larger than its net value. The arithmetic lives in exactly one place, `core/optimize/write_budget.py`, which also owns the write budget (ranked, capped, one block per family, no generic-placeholder rule) and reads the agent-file footprint off the `summarize` analyzer so the product can never recommend adding rules to a file the same report says to compress. Two rules that fall out of it and are easy to get wrong: **rung matters** (rung 2 charges only the always-loaded skill frontmatter, rung 3+ hooks/config charge ZERO because they are executed, never sent as prompt text), and **the budget is spent in PER-SESSION tokens** while the saving is netted against the WINDOW total — charging the window total against the budget would shrink it as a user's session count grows, which is backwards. The 30-day session pace both sides are expressed on is defined once in `core/optimize/projection.py`; never re-derive a projection factor locally.
 
   **The same load semantics govern READING a file, not just writing one.** "Rung matters" is a statement about how Claude Code delivers a file, so it holds identically when an analyzer PRICES an existing file. `core/summarize/load_semantics.py` is the single source of truth both sides classify through: a `CLAUDE.md` / `.claude/rules/*.md` is resident in full on every call of every loading session, while a `.claude/skills/*/SKILL.md`, `.claude/commands/*.md` or `.claude/agents/*.md` surfaces only its **frontmatter** that way and delivers its **body** on invocation. So the window figure is two separately-observed terms — `always-resident reduction x sessions x reads per session` **+** `on-demand reduction x OBSERVED invocations` — never one term applied to the whole file. Getting this wrong in either direction is a real defect: pricing a whole skill body as always-resident made an uninvoked 160 KB skill library the single most expensive prompt file a user owns and pushed the cross-analyzer token rollup to 115% of the window's billed tokens; pricing an invoked body at zero would deny that a long skill that really runs costs anything. The invocation multiplier is OBSERVED from Claude Code transcripts (`core/summarize/invocations.py`: `Skill` tool calls, typed `<command-name>` slash commands, `Task`/`Agent` `subagent_type`), never assumed — and "invoked zero times" (a measurement, prices the body at zero) is a different statement from "no transcript corpus" (not a measurement; Critical Rule 28 corollary (a) degrades BOTH fields).
@@ -228,6 +256,7 @@ Post-ingest hooks run after each span is written to DB:
 - **`tokenjam/api/deps.py`**: `require_api_key` — FastAPI dependency for optional API key auth on GET endpoints. Only enforced when `api.auth.enabled = true` in config.
 - **`tokenjam/api/routes/`**: One file per resource — `spans.py` (OTLP JSON ingest), `traces.py`, `cost.py`, `cost_compare.py`, `tools.py`, `alerts.py`, `drift.py`, `optimize.py`, `reuse.py` (`GET /api/v1/reuse/clusters` — the Reuse finding plus skeleton-rendering extras `planning_texts` + `pricing_mode`; a dedicated endpoint (not bolted onto `/optimize`) so the per-cluster planner text, which can be many KB, isn't paid for on every Overview poll — #154), `budget.py`, `status.py`, `agents.py`, `metrics.py` (Prometheus text format from DB queries), `version.py` (unauthenticated `GET /health` → `{"status":"ok","version":...}` mounted with no prefix, plus `GET /api/v1/version`; the version is derived at runtime via `importlib.metadata.version("tokenjam")` — no hardcoded literal). **The dollar-bearing read routes (`/cost`, `/cost/compare`, `/optimize`, `/budget`) each return a `framing` block** (see `core/framing.py`) so the web UI renders plan-tier-aware figures without re-deriving the rules in JS. `/optimize` takes `?fast=true` to skip the expensive Trim analyzer (returns `skipped_analyzers`) for the polling Overview; `/cost` returns a window-bucketed `series` for the chart (see Web UI below). **Concurrency:** the sync (`def`) read routes (`/optimize`, `/cost/compare`) run in Starlette's threadpool, so concurrent requests reach the DB from multiple threads. `DuckDBBackend.conn` is a **per-thread DuckDB cursor** (`threading.local`) over one shared database — cursors are independent connections safe for concurrent use, so fan-out callers (the Overview) can fetch in parallel. Fixed in #124 (was a single shared connection that aborted under concurrent access); do not collapse `conn` back to one shared connection object.
 - **`tokenjam/core/summarize/`**: Structure-aware prompt summarization (advisory; new feature). `detect.py` classifies prose vs. structure (fenced/inline code, tags, templates, tables); `candidates.py` (+ `catalog.py`/`estimate.py`) powers the `tj summarize list` scan; `wrap.py` is the pure protect→restore algorithm (wrap each structured span behind an id'd `<tj-keep>` marker, restore verbatim by id — structure is a hard guarantee); `session.py` is the no-scratch `prepare`/`check` lifecycle + staging (re-derives the wrap from the live file + a content hash; persists nothing but the staged result); `apply.py` writes a staged rewrite back to the file (default dry-run; `--go` writes) behind an owner + content-hash + symlink guard, with a gzip backup and `undo`; `backup.py` stores the gzipped original + metadata under `~/.tj/summary/backups/`; `delivery.py` is the CLI's automated rewrite step — `claude -p` (subprocess, timeout-guarded) or the Anthropic API (lazy `httpx` + the user's own `TJ_ANTHROPIC_API_KEY`) — plus the "pays for itself" amortization. Pure domain logic — no `tokenjam.cli`/`tokenjam.api` imports (delivery's API path lazily imports `httpx`, its lone outbound dependency).
+- **`tokenjam/core/rulewrite/`**: The ONE rule-write lifecycle every rule-writing analyzer shares (`plan.py` list, `apply.py` stage/check/apply/undo, `store.py` staging + gzip backups, `types.py` shapes). Pure domain — no `tokenjam.cli`/`tokenjam.api` imports. Reuses `core/summarize/apply`'s guard model and `relearn_apply`'s block renderer rather than reimplementing either; the store persists rendered OUTPUT, never a recipe, so what a reviewer approved in the diff is byte-for-byte what apply writes. Surfaced by `tj rules` and `/api/v1/rules/*`. `core/optimize/rule_placement.py` is the WHERE half — named that, not `placement`, because `placement` is already a registered analyzer name for an unrelated question (Batch API lane, `analyzers/batch_placement.py`; Critical Rule 19).
 - **`tokenjam/mcp/server.py`**: FastMCP stdio server exposing observability data to Claude Code (plus the summarize tools — `list_summarize_candidates`, `summarize_prep`, `summarize_check`, `summarize_apply`, `summarize_undo`; see `core/summarize/`). Uses either a read-only DuckDB connection or HTTP proxy to `tj serve`. Initialized via `init()` from `cmd_mcp.py`.
 - **`tokenjam/cli/main.py`**: Root Click group with global options (`--config`, `--json`, `--no-color`, `--db`, `--agent`, `-v`). Registers all subcommands.
 
@@ -248,13 +277,15 @@ Post-ingest hooks run after each span is written to DB:
 
 - **`tj summarize`** (`cmd_summarize.py`) — structure-aware prompt summarization (advisory; new feature). `list` scans for prompt files worth summarizing (catalog-default; any scope-widening flag opens it to `*.md`) and estimates the per-call token saving — read-only. `prep`/`check` are the mechanism: `prep` wraps a prompt's structure behind id'd `<tj-keep>` markers and emits it for rewriting (CLI manual/copy, or the MCP `summarize_prep`/`summarize_check` tools for in-session rewrites); `check` re-reads + hash-guards the file, restores every block verbatim by id, and stages the result **only if structure survives** (a hard gate). No scratch state — the file on disk is the source of truth; `summarize` is in `no_db_commands` (config-only). `apply` writes a staged rewrite back to the file — default **dry-run** (it prints the unified diff), `--go` writes — guarded by an owner + content-hash + symlink check and a gzip backup; `undo` restores from that backup (refusing on drift). `prep --via claude-p` (your local `claude -p`, no key) or `prep --via api` (Anthropic with your `TJ_ANTHROPIC_API_KEY` + the required `[summarize] api_model`, reporting a "pays for itself" amortization) runs the rewrite for you in one shot — both still pass the `check` gate before staging.
 
+- **`tj rules`** (`cmd_rules.py`, `core/rulewrite/`) — the permanent CLAUDE.md rules the analyzers propose, and WHERE each one goes. `list`/`show` read the proposal cache the optimize pass already wrote (`rules` is in `no_db_commands` — config only, no analyzer sweep, works while `tj serve` holds the DuckDB lock); `stage` renders one diff PER DESTINATION; `check` re-verifies each against the file as it stands now; `apply` writes (default **dry-run**, `--go` writes, gzip backup first); `undo` reverts per destination; `applied` lists what can still be undone. The safety model is `core/summarize/apply`'s, reused rather than reimplemented (owner + content-hash + symlink guards, atomic write, refuse-on-drift undo), and the block rendering is `relearn_apply.render_note_content`'s, so a re-apply replaces its own marked block instead of duplicating it and the existing Revert path can still find it. **Partial outcomes are first-class:** three project files written and a fourth skipped because it drifted is the reported result, never rounded up to "applied". Mirrored at `GET/POST /api/v1/rules/*` and on the Lens **Optimize ▸ Rules** screen (`#/optimize/rules`), which exists because a Review-inbox card can express one write and cannot express "this rule, into these 4 of your 11 projects, here is each diff, apply selectively".
+
 - **`tj pricing list`** (`cmd_pricing.py`) — read-only inspection of the resolved model pricing table. Prints one row per `(provider, model)` with `input` / `output` / `cache_read` / `cache_write` rates and a `source` column (`override` / `packaged`) showing which layer the rate resolved from. Supports `--json` and `--model <substring>` (case-insensitive filter). `pricing` is in `no_db_commands` (config-only). The source derivation lives in `core/pricing.load_pricing_sources()` (public) — the CLI reads it rather than re-deriving precedence; a listed row is always `packaged` or `override` (the `DEFAULT_INPUT_PER_MTOK` / `DEFAULT_OUTPUT_PER_MTOK` fallback in `core/pricing.py` applies only to models absent from the table, which never appear here). The `set` half of a future `tj pricing set|list` is not built yet (#282 shipped `list` only).
 
 All commands support `--json` for machine-readable output. Commands that query alerts use exit code 1 if active (unacknowledged, unsuppressed) alerts exist.
 
 **CLI testing pattern:** Tests use `click.testing.CliRunner` with `unittest.mock.patch` on `tokenjam.cli.main.load_config` and `tokenjam.cli.main.open_db` to inject an `InMemoryBackend` and test config. See `tests/integration/test_cli.py`. Note: `cmd_doctor` opens its own DuckDB connection via `config.storage.path` to verify writability — in tests you must set this to a real temp path (e.g. `tmp_path / "test.duckdb"`).
 
-**`no_db_commands` in `cli/main.py`:** Commands that don't open the DB at startup — currently `{stop, uninstall, onboard, mcp, demo, policy, proxy, summarize, pricing}`. New commands that read only from config (or do their own DB connection later) should be added to this set so they work when `tj serve` holds the write lock. Tests for these commands can patch `open_db` with `side_effect=AssertionError(...)` to verify they never touch the DB.
+**`no_db_commands` in `cli/main.py`:** Commands that don't open the DB at startup — read the set in `cli/main.py` rather than trusting an enumeration here. New commands that read only from config (or do their own DB connection later) should be added to this set so they work when `tj serve` holds the write lock. Tests for these commands can patch `open_db` with `side_effect=AssertionError(...)` to verify they never touch the DB.
 
 **Test factories:** `tests/factories.py` provides `make_llm_span(billing_account="anthropic", ...)` and `make_session(plan_tier="api", ...)` with safe defaults that preserve existing test behavior. Tests exercising subscription / local / unknown plan-tier rendering paths should pass the field explicitly.
 
@@ -288,7 +319,7 @@ That routing lives behind **one seam, not per-command duck-typing**: `resolve_da
 
 ### Web UI ("TokenJam Lens")
 
-`tokenjam/ui/index.html` is the served dashboard — a **single-file Preact + htm SPA** (no build step, no TypeScript, no client-side router). "TokenJam Lens" is the **brand only**: it appears in `<title>`, the sidebar wordmark, and the OpenAPI title, but never in module names, route paths, or config keys. Screens: **Overview** (the default landing route — a triage front door), Status, Traces, Cost, Alerts, Drift, Optimize, Budget.
+`tokenjam/ui/index.html` is the served dashboard — a **single-file Preact + htm SPA** (no build step, no TypeScript, no client-side router). "TokenJam Lens" is the **brand only**: it appears in `<title>`, the sidebar wordmark, and the OpenAPI title, but never in module names, route paths, or config keys. Screens: **Overview** (the default landing route — a triage front door), Status, Traces, Cost, Alerts, Drift, Optimize (with **Summarize** and **Rules** sub-views), Budget.
 
 - **Offline-first (Critical Rule 18):** every JS/CSS dep is vendored under `tokenjam/ui/vendor/` — Preact + hooks + htm (ESM via `<script type="importmap">`) and **uPlot** (vendored IIFE global `uPlot` + CSS, pinned in `docs/internal/lens-vendor-versions.md`). No render-time external HTTP. `tests/unit/test_ui_offline.py` enforces this; clickable `<a href>` links are the only allowed external URLs.
 - **Single compute path:** the UI reads everything from the REST API and **never re-implements analysis, aggregation, or plan-tier framing in JS** — it consumes the `framing` block (see `core/framing.py`). If the UI needs a number, extend the endpoint; don't compute it client-side. This applies to gating/filter sets too, not just numbers: if a Python-side map decides which analyzers/findings apply to a persona (e.g. `core/optimize/runner.py`'s `PERSONA_DISABLED_ANALYZERS`), the API already publishes the resolved set (`persona_disabled_analyzers` on `/optimize`) — read that field everywhere the gate is needed, never re-declare the map as a second JS literal. Two call sites deriving the same set from two different sources will silently desync the moment either side edits its copy.
@@ -297,7 +328,9 @@ That routing lives behind **one seam, not per-command duck-typing**: `resolve_da
 - **Charts:** `SpendChart` wraps uPlot, reads CSS custom properties (`--chart-1..5`) so it re-themes, and has a cursor tooltip. The spend chart spans the **full selected window** with zero-fill: `/api/v1/cost` returns a window-bucketed `series` (hourly buckets for ≤2-day windows, daily otherwise; epoch-second `bucket` keys) plus `series_bucket` + `window_start`/`window_end`, and the UI builds a continuous grid + pins the x-scale to the window (issues #133/#136).
 - **Run-rate** is a single linear figure projected to the end of the current calendar cycle (`daily_rate × days-remaining`), captioned "not a forecast". The forecasting boundary is deliberate: linear run-rate only — no EWMA, seasonality, or anomaly detection.
 - **Polling:** the Overview auto-refreshes every 30s only while the tab is visible (`document.visibilityState`) and **fetches its endpoints in parallel** via `Promise.all` (the daemon DB layer is concurrency-safe since #124 — per-thread cursors). The error handling is deliberately asymmetric: `/cost` is load-bearing (no `.catch` — its failure surfaces the error state), while every other panel carries a `.catch` fallback so one failing panel renders empty rather than blanking the Overview. Don't unify them. Detail screens refresh on user action.
-- **Testing the UI (no JS runner in CI):** the Python `test` job can't run JS, so UI fixes are guarded by **static-grep regression tests** in `tests/unit/test_lens_ui_regression.py` (assert buggy patterns are *gone* and new helpers are present) plus `test_ui_offline.py`. When iterating locally, validate syntax with `node --check` on the extracted `<script type="module">` block, and verify visually by running `tj serve` (or a seeded `create_app` + uvicorn on an alt port) and screenshotting with headless Chrome — there is intentionally no Playwright/Cypress.
+- **Spacing comes from the `:root` tokens, never from a hand-picked literal.** Every surface here is one of four shapes — a card/section body, a row inside one, a head strip, a table cell — and they share ONE horizontal inset (`--inset-x`) so a table lines up with the panel head above it; the vertical value varies by density (`--pad-card-y` / `--pad-row-y` / `--pad-head-y`), and `--stack-gap` separates stacked containers. Reach for the shared classes before writing a style attribute: `.panel-body` and `.panel-row` inset a panel's children (`.panel` is deliberately padding-free so it can hold a full-bleed table, which is exactly why its children kept being written with an ad-hoc padding, or none at all), `.opt-tile` is an `.opt-section` used as a flex column, and `.empty` + `.compact`/`.err` and `td.empty-cell` cover every not-here state. The failure mode is not ugliness, it is drift: an inline box and the loading skeleton that stands in for it are edited months apart and stop matching, and a re-declared `border`/`border-radius` in a style attribute hides that the class already owned it.
+- **A cell with `overflow-wrap:anywhere` + `min-width:0` has a min-content width of ONE CHARACTER,** so under auto table-layout its column is the first thing the browser sacrifices and it collapses into a vertical stack of letters rather than wrapping. Any such column needs an explicit `min-width` floor. The mirror image is a `td` left at the default `white-space:nowrap` holding long text: that one column then dictates the whole table's width, and sibling tables on the same page come out visibly different widths with only one of them overflowing. Cap it and let it wrap; `.table-wrap` absorbs the remainder. Both bite on small, unrelated changes (a few px of extra cell padding was enough), so re-check wide tables after any spacing edit.
+- **Testing the UI (no JS runner in CI):** the Python `test` job can't run JS. Guard UI work with tests that **execute** something — `test_lens_dashboard_states.py` and `test_lens_select_all_behaviour.py` extract a pure JS helper out of `index.html` and run it under `node`, and `test_ui_offline.py` asserts file-level invariants (no external hosts, no NUL bytes, and that the module script actually parses). **Do not add static-grep tests that assert on literal substrings of `index.html`.** A 344-test file of those was deleted: it pinned wording and markup structure rather than behaviour, so ordinary visual iteration broke assertions that guarded nothing, and one broke merely because an identifier was added to a `useEffect` dependency array it used as a slice anchor. If a UI behaviour is worth guarding, extract the logic into a pure function and execute it. When iterating locally, verify visually by running `tj serve` (or a seeded `create_app` + uvicorn on an alt port) and screenshotting with headless Chrome — there is intentionally no Playwright/Cypress.
 
 ### Session Continuity
 
@@ -332,7 +365,7 @@ When a span has a `conversation_id` matching an existing session, it's attribute
 23. **When removing something that should never have been user-visible, grep the TESTS for it FIRST — a green suite may be ENFORCING the defect, not protecting against it.** An internal tracker id sat in the `tj doctor` MCP-wiring warning, a string the product prints, and `tests/integration/test_cli.py` asserted that the id was PRESENT in `mcp_checks[0]["message"]` — so the suite actively required the leak to be present, and anyone who did the right thing was told by CI that they had broken something. Most people believe CI, which is why it survived so long. A stale comment is entropy; a test asserting the bad state is an ACTIVE DEFENCE of it, and it is invisible until you touch the thing. **The repair is to INVERT the assertion, not to delete it:** the test now asserts the durable content is present (`"+36% measured"`) AND that no bare `#\d+` remains, so the same pin now defends the correct state. Generalise both halves: before deleting any user-visible string, `grep -rn "<the thing>" tests/`; and when a guard has to go, replace it with its inverse rather than removing coverage. `tests/unit/test_no_tracker_ids_in_printed_strings.py` is the standing version of that inverse for this whole class — it walks every non-docstring string literal under `tokenjam/` and fails on an UNQUALIFIED tracker reference, separating real references from CSS colours and ordinals structurally (markup-shaped literals are skipped; a reference needs two or more digits) rather than by an allowlist, because a guard with exceptions just teaches people to add exceptions. **The sharper half of this lesson, learned by getting it wrong:** the first version of that guard banned EVERY reference, and it immediately pushed a scrub into deleting a correct public issue link out of the spans-column-statistics warning — the reference named the real upstream bug the user had just hit, and removing it destroyed true, useful information. Number-sign references are not the problem. **BARE ones are ambiguous** (they resolve against whatever repo the reader is looking at, and our internal and the public number spaces genuinely collide), and **internal ones must never be public**. Those are two different faults with two different fixes: qualify the first (`Metabuilder-Labs/tokenjam#56`, or a full `https://github.com/...` URL, which is what a printed string should carry since a terminal linkifies nothing and a URL survives being pasted anywhere), and describe-and-delete the second. **Decide which fault you have by EVIDENCE, not by the shape of the token:** open the number and ask whether that artifact actually explains the sentence it sits in. It did for the column-statistics warning, whose public issue is exactly about corrupt spans column statistics, so that one was restored qualified; it did not for the MCP-overhead claim or the schema-self-heal warning, whose numbers name our own queue, so those stayed deleted. A guard that cannot tell those two faults apart will cheerfully destroy good information while looking green.
 24. **A surface is reachable only if a USER has a path to it. "The component mounts", "the route resolves", "the code is invoked" and "it is deliberate" each prove nothing on their own.** Static dead-code analysis cannot find this class: ruff and mypy stayed green through every instance below. Four checks, and a surface must pass all of them: **(a) the inverse direction** — not "does this nav item resolve to a view" but "does anything link TO this view". `case 'sessions'` and `case 'runs'` resolved correctly and had no sidebar entry, so `SessionDetailView` and its Map / Approach / Timeline tabs were reachable only by hand-typed URL. An audit that only checked nav→view reported the UI clean. **(b) does the path carry data** — every click-path into a session came from Status rows, and `/api/v1/status` returned empty because its hand-written SQL knew `'active'` and `'closed'` but not the `'completed'` that `core/backfill.py` stamps on every backfilled session, so EVERY session in the DB was filtered out of both zones and there was nothing on screen to click. One literal missing from two WHERE clauses made every surface downstream of it unreachable at once. **(c) does the destination resolve** — the Review inbox rendered example-session links built from transcript filenames (`analyzers/relearn.py`, `session_id = path.stem`), and only the minority that had actually been ingested had a page; the rest 404'd. Links now render only when the session resolves, checked at READ time so already-cached findings self-correct as ingestion catches up. **(d) does a capability have a name a user can type** — `placement` runs, produces a cost card and has a verify branch, but is absent from `tj optimize`'s Click choices, so it is invisible as a capability and only executes as a side effect of `downsize`. Deliberate design and documentation are NOT a defense against invisibility; the `sessions` view was deliberate too. **The inverse trap, equally real:** absence of a `@register` decorator is not evidence of death. `batch_placement` and `downsize_agents` are invoked by `model_downgrade.py` and documented there, and an auditor applying "no decorator means dead" called a live analyzer dead. **The question that resolves both directions is the same one: what can a user actually do?** Not what the code does. Ask it in both directions before deleting anything or declaring anything fine.
 25. **Removing a feature strands its helper modules — grep every function in the modules it used, not just the files that named it.** When the output-cap hook, its `tj savings` CLI, `core/output_cap.py`, and their tests were deleted, `core/savings_log.py` was left behind untouched: 8 of its 9 functions had zero callers anywhere in the repo, including tests, and its docstring still described `tj hook cap-output`, `tj savings`, and an A/B harness that no longer existed. Only `hooks_dir` was still consumed (by `core/recommendations.py`). The durable rule: after deleting a feature, grep every function in its supporting modules for callers before deciding what happens to the module, and treat a docstring that still describes deleted commands as a live defect, not a stale comment — it actively misleads the next reader. A helper module can legitimately outlive the feature it was built for, but only if it's trimmed to what's still consumed and re-documented for its remaining consumer.
-26. **An analyzer that has no fix for the window's dominant persona must never RUN — see `PERSONA_DISABLED_ANALYZERS` in `core/optimize/runner.py`.** If you are wondering why some analyzer is missing from a report on an interactive coding-agent window, this is why, and it is deliberate — the live gated set is `PERSONA_DISABLED_ANALYZERS['claude-code']` in `runner.py`, where every entry carries an inline comment naming the gate it fails. Read the map; do NOT maintain a second copy of that list in this file, because a doc list drifts from the map silently and then reads as a claim about which analyzers run. Two different "disabled" mechanisms exist in this tree and they must not be conflated: a **fake skip** still runs, still queries, and returns an `enabled: False` finding (what `trim` and `cache-recommend` do when their prerequisites are off), whereas a **true skip** drops the analyzer from the selection before dispatch so it is never invoked at all (what `fast=true` already did for expensive analyzers). The persona gate is a TRUE skip: `build_report` resolves the persona at the top, subtracts `disabled_analyzers_for_persona(persona)` from the selected set right after validation, and the dispatch loop below never sees the name. Three consequences to respect. **(a) The bar is three gates, not one.** Keeping an analyzer requires all of: its output ends in a concrete edit to a file or setting this persona actually controls; the user is better off NET of the fix's own standing cost (a rule written into an always-loaded instruction file is re-sent on every future session, forever, so a 200-token rule that prevents a 1,500-token mistake twice a month is net negative); and the saving does not come from making the agent terser or dumber. "A fix could be described in a text field" is not a bridge to a real edit, and it is the mistake that put every currently-gated analyzer on screen in the first place. **(b) Mirror every gate into `COST_ANALYZERS` (`core/optimize/cost_proposals.py`), which is an INDEPENDENT second analyzer-selection surface** feeding the Review inbox: use `cost_analyzers_for_persona`, and note that `cost_proposals_from_report` gates its adapter table off the same map so a report built elsewhere still can't produce an unapplyable card. Skipping the mirror is how a "disabled" analyzer keeps surfacing findings in Review while looking disabled everywhere else. **(c) A sub-check attached by another analyzer needs its own gate** — `placement` is produced inside `downsize`, not registered under its own name, so the selection gate cannot reach it and `model_downgrade.run` consults the same map itself. Adding a persona-disabled name is therefore three edits, not one, and each entry carries an inline comment stating WHICH gate it fails. The default for `sdk` / `mixed` / `unknown` is to disable nothing: only a window we can positively classify may lose an analyzer. Finally, a disabled analyzer VANISHES; it never renders a placeholder row. The one exception is a name the user typed explicitly, which gets a single "not run, and here is why" line, because silence in response to a typed command reads as a bug.
+26. **An analyzer that has no fix for the window's dominant persona must never RUN — see `PERSONA_DISABLED_ANALYZERS` in `core/optimize/runner.py`.** If you are wondering why some analyzer is missing from a report on an interactive coding-agent window, this is why, and it is deliberate — the gated sets live in `PERSONA_DISABLED_ANALYZERS` in `runner.py` (keyed by persona), where every entry carries an inline comment naming the gate it fails. Read the map; do NOT maintain a second copy of that list in this file, because a doc list drifts from the map silently and then reads as a claim about which analyzers run. Two different "disabled" mechanisms exist in this tree and they must not be conflated: a **fake skip** still runs, still queries, and returns an `enabled: False` finding (what `trim` and `cache-recommend` do when their prerequisites are off), whereas a **true skip** drops the analyzer from the selection before dispatch so it is never invoked at all (what `fast=true` already did for expensive analyzers). The persona gate is a TRUE skip: `build_report` resolves the persona at the top, subtracts `disabled_analyzers_for_persona(persona)` from the selected set right after validation, and the dispatch loop below never sees the name. Three consequences to respect. **(a) The bar is three gates, not one.** Keeping an analyzer requires all of: its output ends in a concrete edit to a file or setting this persona actually controls; the user is better off NET of the fix's own standing cost (a rule written into an always-loaded instruction file is re-sent on every future session, forever, so a 200-token rule that prevents a 1,500-token mistake twice a month is net negative); and the saving does not come from making the agent terser or dumber. "A fix could be described in a text field" is not a bridge to a real edit, and it is the mistake that put every currently-gated analyzer on screen in the first place. **(b) Mirror every gate into `COST_ANALYZERS` (`core/optimize/cost_proposals.py`), which is an INDEPENDENT second analyzer-selection surface** feeding the Review inbox: use `cost_analyzers_for_persona`, and note that `cost_proposals_from_report` gates its adapter table off the same map so a report built elsewhere still can't produce an unapplyable card. Skipping the mirror is how a "disabled" analyzer keeps surfacing findings in Review while looking disabled everywhere else. **(c) A sub-check attached by another analyzer needs its own gate** — `placement` is produced inside `downsize`, not registered under its own name, so the selection gate cannot reach it and `model_downgrade.run` consults the same map itself. Adding a persona-disabled name is therefore three edits, not one, and each entry carries an inline comment stating WHICH gate it fails. Only a window we can positively classify may lose an analyzer: a persona with no key in the map disables nothing. Do not assume which personas have a key — read the map, since more than one does and this sentence is exactly where a stale census would sit. Finally, a disabled analyzer VANISHES; it never renders a placeholder row. The one exception is a name the user typed explicitly, which gets a single "not run, and here is why" line, because silence in response to a typed command reads as a bug.
 27. **Two analyzers that both claim `past_overspend_*` must draw from DISJOINT spans — the rollup cannot save you.** `past_overspend_rollup` (`core/optimize/cost_proposals.py`) dedupes by EXACT signature string, so `cost:downsize:<agent>` and `cost:subagent:<name>` are structurally distinct and both survive even when they price the identical tokens. That is precisely how `downsize` (aggregated per `session_id`) and `subagent` (per `(session_id, sub_agent_id)`) came to double-count a Task dispatch: Claude Code stores a subagent's turns under its PARENT session id, so an unfiltered `GROUP BY session_id` silently absorbs them — and `MODE(model)` could even name a subagent's model as the session's dominant one. The rule for the next analyzer: **before adding a recoverable figure, name the exact span population it claims and prove no shipped analyzer already claims those rows.** Two remedies exist, both in-tree; prefer the first. **(a) Make it disjoint at the source** — `analyze_model_downgrade` filters `sub_agent_id IS NULL`, so main-thread and subagent tokens partition the session. Use this when the overlap is a column filter away; it fixes the derived shape (`MODE(model)`, per-turn tool counts) too, not just the sum. **(b) Subtract what the more specific card already claimed** — `_per_agent_cache_recoverable_by_model` does this for the `cache` family, where the generic per-`(provider, model)` row must stay window-wide by definition and cannot be narrowed. Two traps: **only the CLAIM narrows, never the DENOMINATOR** — `percent_of_sessions` / `percent_of_tokens` are shares of the user's whole window, so scoping their denominator to the same subset would silently inflate both (hence the separate unfiltered totals query); and **any share computed against a narrowed figure has to narrow too** — `thinking_tokens_by_session(..., main_thread_only=True)` exists because a whole-session thinking numerator over a main-thread-only output denominator can exceed 1.0. Pin it with an end-to-end test that runs `build_report` → `cost_proposals_from_report` → `past_overspend_rollup` and asserts the token sum equals the seeded session's real total (`tests/unit/test_rollup_subagent_downsize_dedup.py`); a per-analyzer unit test cannot see an overlap that only exists between two analyzers. The stakes are asymmetric: the product's stance is to show the LARGEST number the derivation legitimately supports, which makes a double-count the one failure that turns a big number into an indefensible one. **A third remedy joined these two when `downsize` grew its driver-role case: (c) partition the session POPULATION behind one shared predicate.** `resend` and `downsize` price the identical mechanism (material re-read by every later main-thread turn), so no column filter separates them — what separates them is *which sessions* each may claim. `analyzers/resend_tail.premium_driver_role` is that partition, imported by both, and the shared tail arithmetic lives in the same module so neither side can reimplement it slightly differently. Use this when two analyzers legitimately measure the same thing about different populations: a single predicate that both call cannot drift, whereas two independently-tuned threshold sets silently will.
 28. **A finding's `past_overspend_tokens` and `past_overspend_usd` must count the SAME events — divide one by the other and the implied per-token rate has to land inside a real price band.** Rule 27 covers two analyzers claiming the same spans; this is the same failure one level down, INSIDE a single finding, and it is the harder one to see because both fields are individually defensible. `summarize` priced its dollars per-call-multiplied (reduction x loading sessions x reads per session, all documented in `estimate_basis`) while leaving its token field on the un-multiplied one-time file reduction — so the card carried a small token count beside a large dollar figure, and dividing them implied a per-token rate orders of magnitude above any rate any provider charges. Nothing was wrong with either number in isolation; they just answered different questions, and the cross-analyzer rollup sums the token fields, so one analyzer's basis leak silently corrupts the product's token floor. The deadweight per-session-vs-per-call bug was this same class. **The check is mechanical, so write it as the test:** assert `usd / tokens` equals the blend the basis string advertises, and assert it stays between the cache-read rate and the input rate. A hardcoded-number assertion passes happily while both fields drift; a rate assertion cannot, because a basis mismatch always throws the implied rate orders of magnitude out of band. Two corollaries. **(a) The "no evidence" degrade has to be symmetric** — if the load count could not be observed, BOTH fields go `None`, never a zero on one side and a number on the other; a candidate contributes to both sums or neither. **(b) If a second basis is genuinely wanted (the curate/diff UI needs the one-time per-file reduction), give it its OWN named field** — `file_reduction_tokens` — and say so in `estimate_basis`. Never overload the aggregate field that cross-analyzer rollups read. Also re-check every derived percentage after a basis change: `reduction_pct` is saved ÷ source tokens, so it must keep the one-time numerator or it reads as >100% the moment sessions and calls multiply in.
 29. **Before "fixing" an analyzer that looks inert, measure whether its gate is unreachable or merely selects worthless sessions — the two have opposite fixes.** `downsize` was contributing a trivial slice of the Review-inbox rollup, and the ticket's hypothesis was that its small-session gate (the `SMALL_INPUT_TOKENS` / `SMALL_OUTPUT_TOKENS` / `SMALL_TOOL_CALLS` ceilings in `model_downgrade.py`) was structurally unreachable for a Claude Code session. Measured on a real corpus, it was not: **a meaningful minority of CC sessions cleared the gate** — they just carried a negligible share of main-thread spend, and only part of even that had a cheaper same-family target at all. The gate fires fine; small sessions are just cheap. Loosening it (the reflex fix for an "unreachable" gate) would have produced MORE near-worthless cards, against the standing don't-fill-the-inbox constraint, while changing the dollars by nothing. The real gap was a question nobody asked: a premium model acting as the DRIVER of a long undelegated session, where the waste GROWS with session size and so is structurally invisible to any summed-per-session gate — *the worse the waste, the less eligible the session became.* Two general lessons. **(a) An inert-analyzer ticket's first deliverable is the gate-clearance measurement, not a rewrite** — sessions-cleared and share-of-spend are different numbers and only the second one tells you whether to touch the threshold. **(b) When a metric is summed per session, waste that compounds with session length inverts the gate.** Check for that inversion before assuming a threshold is miscalibrated.
@@ -356,6 +389,12 @@ When a span has a `conversation_id` matching an existing session, it's attribute
 
 38. **Do not assume a telemetry source carries an id for the thing it describes — check, then derive the key from stored COLUMNS when it does not.** Cross-observer accounting needs a name for the underlying API call, and the obvious design is to stamp the provider's own response id on both sides. That works wherever the observer sees the provider's response — `sdk/integrations/anthropic.py` stamps `gen_ai.response.id`, `core/backfill.py` stamps `tj.call_id` off the assistant message key. It does not work for the pairing that matters most on this product's dominant corpus: **Claude Code's own OTel log exporter emits no per-call identifier at all.** Read `ClaudeCodeEvents` in `otel/semconv.py` for what its `api_request` event actually carries, and confirm there before designing around an id — the absence is the kind of premise that silently invalidates a whole approach. A stamp only ONE side of a pair can carry is worse than no stamp: it yields two different keys for one call, so the pairing never fires and nothing looks broken. **Derive the cross-observer key from columns the store already holds rather than from an attribute an ingest path promises to write** — that also makes it readable on rows written before any stamping existed, which is what lets `tj doctor --repair` clean a database an older build already doubled. Keep stamping the ids that DO exist: they are exact where a derived key is only strong, and they are what a future exporter revision would let you key on directly. If you are about to rely on the absence, re-verify the exporter's attribute set rather than trusting this line.
 
+39. **A flag set UPSTREAM of a pass that rewrites the same field is not a flag, it is a suggestion — and a unit test on the constructor cannot see it.** An analyzer correctly set `advisory_only` on a family and cleared its write offer at construction; a later pass, `_apply_write_budget`, then assigned `write_offered` from an allocator that knew nothing about the flag and quietly restored the offer. Every value on the field was individually correct at the moment it was written, and the last writer won. The unit test passed the whole time because it exercised the CONSTRUCTOR path directly while the daemon ran the COMPOSED path — the test was structurally incapable of observing the defect, so a green suite was evidence of nothing. **The fix shape is not a second guard in the later pass** (that is a race between two writers, which the next pass re-opens); it is to **remove the item from the INPUT the later pass consumes**, so no decision exists to overwrite. Here the advisory family never becomes a `WriteCandidate` at all. Two rules follow. *(a) When you set a field that a downstream pass also writes, either withdraw the row from that pass's input or move the decision into it — never both write it.* Grep for other assignments to the field before assuming your assignment is the final one. *(b) A test that constructs the object under test proves the constructor works and says nothing about the pipeline.* Assert through the composed path a user actually reaches (build the report, run the budget pass, then read the field), which is the same lesson Critical Rule 36 draws for repair passes.
+
+40. **A fixture that is a real domain object can stop being a valid vehicle for what it tests while every assertion still passes — for the wrong reason.** Eight fixtures seeded one particular relearn failure family in order to exercise write-budget behaviour. That family later became advisory-only, so it can no longer demonstrate ANY budget behaviour at all: it is never offered, never priced, never ranked. The tests stayed green, because the assertions they made happened to hold vacuously on a family that had left the population under test. This happened twice on the same branch, with two different families. The hazard is specific to fixtures that are real domain objects rather than stubs: **changing a domain object's semantics silently invalidates every test that used it as scaffolding, and the suite reports success either way.** Three habits. *(a) When you change what a family, analyzer or kind MEANS, grep the tests for its name and re-read each use — asking not "does this still pass" but "can this fixture still exhibit the behaviour the test claims to check".* *(b) Prefer a fixture that would FAIL loudly if it stopped qualifying* — assert the precondition explicitly (`assert cluster.write_offered`) before asserting the behaviour, so the vehicle's own validity is pinned rather than assumed. *(c) When you do have to swap the fixture, say in the docstring which object it replaced and why the old one stopped qualifying* — that note is what stops the next reader from "restoring" it.
+41. **A `list[Any]` field loses its TYPE across a round trip while its VALUE survives intact — and a broad `except` downstream turns that into a whole analyzer silently missing from a published total.** `hydrate_dataclass` dispatches on the annotation, so a field declared `list[Any]` to keep `core/optimize/types.py` free of an analyzer import gives it nothing to work with: the rows come back as plain dicts, correctly shaped and correctly numbered, with the class gone. Nothing fails at the seam. The failure surfaces far away, as `'dict' object has no attribute ...` inside whichever consumer reads a row attribute — and if that consumer sits under a per-item `except Exception: continue`, the item is dropped and the total it fed just gets smaller. That is how one analyzer's entire contribution disappeared from the Review inbox while the Dashboard tile, reading the finding directly and never round-tripping, went on publishing it: two surfaces, one analyzer, no error anywhere. Three rules. **(a) Declare the element type as DATA when you cannot import it**: `field(metadata={"hydrate": "package.module:ClassName"})`, resolved lazily at hydration time (`runner._hydrate_target`), so the layering stays intact and the round trip is lossless in type as well as value. A test asserts every `list[Any]` in `types.py` carries it. **(b) A round-trip test that compares VALUES cannot see this** — it passes on dicts, because the numbers really did survive. Assert `isinstance` of the rehydrated row, not just its fields. **(c) A broad `except` around a per-item transform must record WHAT it skipped**, and the caller must publish that as a known-unknown rather than absorbing it: a skipped item is a hole in every figure summed from the result, and a null-figure disclosure through the `excluded` channel is the difference between a total that says "this analyzer is missing" and one that quietly reads as complete. Continuing on error is right; continuing without a trace is what makes the arithmetic lie.
+
+
 ## Config
 
 Config is TOML, discovered at: `tj.toml` -> `.tj/config.toml` -> `~/.config/tj/config.toml`. Override with `--config` or `TJ_CONFIG` env var. Full config hierarchy is in `tokenjam/core/config.py` (`TjConfig` dataclass).
@@ -437,8 +476,9 @@ The packaged table stays the zero-config default — the override is a *layer*,
 ## CI
 
 GitHub Actions workflow at `.github/workflows/ci.yml` runs on push/PR to `main`:
-- **`test`** job: Python 3.10/3.11/3.12 matrix — `ruff check` and `mypy` (continue-on-error), then `pytest tests/unit/ tests/synthetic/ tests/agents/ tests/integration/` (blocking)
-- **`test-ts`** job: Node 20 — `npm install && npm test` in `sdk-ts/`
+- **`lint`** job: Python 3.12 — `ruff check tokenjam/` and `mypy tokenjam/`
+- **`test`** job: Python 3.10/3.11/3.12 matrix — `pytest tests/unit/ tests/synthetic/ tests/agents/ tests/integration/`
+- **`test-ts`** job: Node 22 — `npm install && npm test` in `sdk-ts/`
 
 All steps are blocking — lint, typecheck, and tests must pass for CI to go green.
 

{tokenjam-0.6.2 → tokenjam-0.6.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tokenjam
-Version: 0.6.2
+Version: 0.6.4
 Summary: TokenJam: a local-first, OTel-native cost-saving utility for AI agents
 Project-URL: Homepage, https://tokenjam.dev
 Project-URL: Repository, https://github.com/Metabuilder-Labs/tokenjam

{tokenjam-0.6.2 → tokenjam-0.6.4}/docs/getting-started.md

@@ -2,15 +2,15 @@
 
 The fastest way onto TokenJam is to install it and run the onboarding wizard for your agent; that
 wires up everything (statusline, backfill, dashboard) in one step. This page walks every entry path
-by persona, most-common first; each rung ends with a way to verify it actually worked before you move
+by persona, most-common first; each step ends with a way to verify it actually worked before you move
 on.
 
 > **Just looking first?** `npx tokenjam` prints a 15-second read-only report over the Claude Code logs
 > you already have on disk (`~/.claude/projects/*.jsonl`); nothing installed, nothing written to disk,
-> no daemon. You should see a "Where your quota goes" panel with a percentage breakdown (or "No Claude
-> Code logs found," which just means you haven't run Claude Code on this machine yet). See
+> no daemon. You should see one line telling you how much of your recent sessions was avoidable (or
+> "No Claude Code logs found," which just means you haven't run Claude Code on this machine yet). See
 > [docs/installation.md](installation.md) for the `uv`/`pipx` runner requirements. When you're ready to
-> keep it, start at rung 1 below.
+> keep it, start at step 1 below.
 
 ## 1. Claude Code / Codex onboarding wizards
 

{tokenjam-0.6.2 → tokenjam-0.6.4}/npm-wrapper/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenjam",
-  "version": "0.6.2",
+  "version": "0.6.4",
   "description": "Zero-install launcher for TokenJam (tj): npx tokenjam runs the Python CLI via uvx/pipx and reports the recurring mistakes your AI agent keeps repeating, no setup required.",
   "keywords": ["claude-code", "ccusage", "tokens", "cost-optimization", "llm", "agents", "observability", "tokenjam", "anthropic", "claude", "token-usage", "agent-behavior", "cli", "statusline", "opentelemetry", "agent-observability"],
   "homepage": "https://tokenjam.dev",

{tokenjam-0.6.2 → tokenjam-0.6.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tokenjam"
-version = "0.6.2"
+version = "0.6.4"
 description = "TokenJam: a local-first, OTel-native cost-saving utility for AI agents"
 readme = "README.md"
 requires-python = ">=3.10"

{tokenjam-0.6.2 → tokenjam-0.6.4}/sdk-ts/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tokenjam/sdk",
-  "version": "0.6.2",
+  "version": "0.6.4",
   "description": "TypeScript SDK for TokenJam — local-first observability for AI agents",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",

{tokenjam-0.6.2 → tokenjam-0.6.4}/tests/integration/test_cli.py

@@ -12,7 +12,7 @@ from click.testing import CliRunner
 
 from tokenjam.cli.main import cli
 from tokenjam.core.config import AgentConfig, BudgetConfig, TjConfig
-from tokenjam.core.db import InMemoryBackend
+from tokenjam.core.db import SPANS_INDEX_SQL, InMemoryBackend
 from tokenjam.core.models import (
     AgentRecord,
     Alert,
@@ -473,6 +473,12 @@ def _duckdb_with_dropped_migration_7(tmp_path):
         backend.conn.execute(f"DROP INDEX IF EXISTS {idx}")
     backend.conn.execute("ALTER TABLE spans DROP COLUMN request_params")
     backend.conn.execute("ALTER TABLE spans DROP COLUMN request_tools")
+    # DuckDB refuses to drop a column while the table carries ART indexes, so
+    # the indexes above come off first — and have to go straight back on. Left
+    # off, this fixture would additionally reproduce the SEPARATE fault doctor's
+    # index-integrity check exists for, and these tests would be asserting an
+    # exit code that has nothing to do with the missing column they are about.
+    backend.conn.execute(SPANS_INDEX_SQL)
     return backend
 
 

{tokenjam-0.6.2 → tokenjam-0.6.4}/tests/integration/test_cost_proposals_api.py

@@ -11,6 +11,8 @@ from datetime import timedelta
 import httpx
 import pytest
 
+from tokenjam.core.rulewrite.kinds import DELIVERY_SKILL
+
 from tokenjam.api.app import create_app
 from tokenjam.core.config import ApiAuthConfig, ApiConfig, StorageConfig, TjConfig
 from tokenjam.core.db import InMemoryBackend
@@ -128,6 +130,73 @@ async def test_mark_cost_applied_round_trip(app, client):
     assert rev.json()["state"] == "reverted"
 
 
+# --- an applied proposal loses its OFFER and keeps its FIGURE ---------------- #
+
+async def test_an_applied_proposal_is_withdrawn_on_the_PAYLOAD(app, client):
+    """Verified WITHOUT consulting a second endpoint — that is the whole point.
+
+    THE DEFECT. This route computed a filtered ``open_proposals`` for its rollup
+    and then returned the UNFILTERED ``proposals`` list, and the rows carried no
+    ``applied``/``applied_at`` field at all. So an already-applied proposal went
+    out advertising ``apply_capable: true`` with nothing on it to say otherwise.
+    Only the browser was safe, and only because it independently re-fetches
+    ``/relearn/cost-applied`` and filters client-side — meaning the CLI,
+    ``--json``, an export and every future surface saw an offer to re-apply
+    something already done, and had to know to cross-reference a second endpoint
+    to avoid it. Measured on a real corpus: the ONE apply-capable row in the
+    whole inbox was a fix that had already been applied.
+
+    Critical Rule 32: the offer is withdrawn, the figure is kept, and the row
+    stays listed CARRYING an applied state.
+    """
+    token = app.state.relearn_write_token
+    hdr = {"X-TJ-Local-Token": token}
+    await client.post("/api/v1/relearn/cost-proposals/refresh", headers=hdr)
+    before = (await client.get("/api/v1/relearn/cost-proposals")).json()["proposals"]
+    prop = next(p for p in before if p["analyzer"] == "cache")
+    assert prop["applied"] is False, "every row carries the field, open ones included"
+    assert prop["applied_at"] is None
+
+    applied = await client.post(
+        "/api/v1/relearn/cost-proposals/apply",
+        json={"proposal_id": prop["proposal_id"]}, headers=hdr,
+    )
+    assert applied.status_code == 200
+
+    after = (await client.get("/api/v1/relearn/cost-proposals")).json()["proposals"]
+    row = next(p for p in after if p["signature"] == prop["signature"])
+    assert row["applied"] is True
+    assert row["applied_at"], "an applied row must carry WHEN"
+    assert row.get("apply_capable") is not True, (
+        "an applied fix is still being offered for application"
+    )
+    # The figure is what the behaviour ALREADY cost. Applying the fix afterwards
+    # does not un-spend the money, and a gate that edits a past figure is the
+    # "this was free" conflation Critical Rule 32 exists to stop.
+    assert row["past_overspend_usd"] == prop["past_overspend_usd"]
+
+
+async def test_a_reverted_proposal_is_offered_again_on_the_payload(app, client):
+    """A revert is the user saying the fix is no longer in place."""
+    token = app.state.relearn_write_token
+    hdr = {"X-TJ-Local-Token": token}
+    await client.post("/api/v1/relearn/cost-proposals/refresh", headers=hdr)
+    proposals = (await client.get("/api/v1/relearn/cost-proposals")).json()["proposals"]
+    prop = next(p for p in proposals if p["analyzer"] == "cache")
+    rec = (await client.post(
+        "/api/v1/relearn/cost-proposals/apply",
+        json={"proposal_id": prop["proposal_id"]}, headers=hdr,
+    )).json()
+    await client.post(
+        f"/api/v1/relearn/cost-applied/{rec['id']}/revert", headers=hdr,
+    )
+
+    after = (await client.get("/api/v1/relearn/cost-proposals")).json()["proposals"]
+    row = next(p for p in after if p["signature"] == prop["signature"])
+    assert row["applied"] is False
+    assert row["applied_at"] is None
+
+
 # --- the marker's numbers come from the STORE, never from the caller -------- #
 
 async def test_mark_applied_refuses_a_caller_supplied_estimate(app, client, config):
@@ -198,7 +267,7 @@ async def test_apply_workspace_refuses_a_caller_supplied_proposed_fix(
 
 
 async def test_cost_apply_workspace_writes_note_and_records_marker(app, client, db, monkeypatch, tmp_path):
-    """A CC-origin subagent proposal routes a reversible rung-1 note through the
+    """A CC-origin subagent proposal routes a reversible CLAUDE.md rule through the
     existing relearn apply path, then records the cost marker for delta-verify."""
     from tokenjam.core.optimize import relearn_apply as pa
 
@@ -253,17 +322,17 @@ async def test_cost_apply_workspace_writes_note_and_records_marker(app, client,
     assert any(r["analyzer"] == "subagent" for r in applied["applied"])
 
 
-async def test_cost_apply_workspace_writes_skill_for_a_rung2_proposal_and_reverts(
+async def test_cost_apply_workspace_writes_skill_for_a_skill_proposal_and_reverts(
     app, client, config, db, monkeypatch, tmp_path,
 ):
     """`apply-workspace` is generic across analyzers, not special-cased to
-    `subagent`: a rung-2 skill-note proposal routes through the SAME path,
+    `subagent`: a skill-note proposal routes through the SAME path,
     writes, and reverts cleanly.
 
     Two facts are pinned here. First, a deterministic tool-call cluster on a
     claude-code window produces NO `script` card any more: the analyzer has no
     fix that survives for that persona and is skipped before it runs. Second,
-    the generic rung-2 write/revert route still works, exercised by seeding the
+    the generic skill write/revert route still works, exercised by seeding the
     proposal the adapter builds directly into the store."""
     from tokenjam.core.optimize import relearn_apply as pa, relearn_store
     from tokenjam.core.optimize.analyzers.workflow_restructure import (
@@ -322,7 +391,7 @@ async def test_cost_apply_workspace_writes_skill_for_a_rung2_proposal_and_revert
     prop = script_props[0]
     assert prop["apply_capable"] is True
     assert prop["advise_only"] is False
-    assert prop["rung"] == 2
+    assert prop["delivery"] == DELIVERY_SKILL
 
     body = {"proposal_id": prop["proposal_id"], "target_path": str(target)}
 

{tokenjam-0.6.2 → tokenjam-0.6.4}/tests/integration/test_db.py

@@ -212,11 +212,76 @@ def test_delete_spans_before_cutoff(db):
     db.insert_span(span_new)
 
     cutoff = now - timedelta(days=90)
-    deleted = db.delete_spans_before(cutoff)
-    assert deleted == 1
+    spans_deleted, sessions_deleted = db.delete_spans_before(cutoff)
+    assert spans_deleted == 1
 
     remaining = db.conn.execute("SELECT COUNT(*) FROM spans").fetchone()
     assert remaining[0] == 1
+    # The session straddles the cutoff and still has a live span, so it stays:
+    # a session is orphaned by the delete only once it has NO spans left.
+    assert sessions_deleted == 0
+    assert db.conn.execute("SELECT COUNT(*) FROM sessions").fetchone()[0] == 1
+
+
+def test_delete_spans_before_cutoff_takes_the_sessions_it_orphans(db):
+    """Deleting only spans left the parent sessions asserting a day had data.
+
+    `core/data_span` unions `sessions.started_at` into the day set it measures
+    the available span from, so every orphan went on claiming a day carried data
+    after the data for that day was destroyed — the deletion skewed the measure
+    of what survived it.
+    """
+    _insert_agent(db)
+    now = utcnow()
+    old = now - timedelta(days=100)
+
+    aged_out = make_session(started_at=old, ended_at=old + timedelta(minutes=1))
+    db.upsert_session(aged_out)
+    db.insert_span(make_llm_span(session_id=aged_out.session_id, start_time=old))
+
+    live = make_session()
+    db.upsert_session(live)
+    db.insert_span(make_llm_span(
+        session_id=live.session_id, start_time=now - timedelta(days=1),
+    ))
+
+    spans_deleted, sessions_deleted = db.delete_spans_before(now - timedelta(days=90))
+    assert (spans_deleted, sessions_deleted) == (1, 1)
+
+    surviving = [
+        r[0] for r in db.conn.execute("SELECT session_id FROM sessions").fetchall()
+    ]
+    assert surviving == [live.session_id]
+
+
+def test_delete_spans_before_cutoff_takes_an_aged_out_session_with_no_spans(db):
+    """A pre-cutoff session with no spans is aged-out history like any other.
+
+    Left behind it would go on asserting, to `core/data_span`, that a day
+    beyond the retention horizon carried data.
+    """
+    _insert_agent(db)
+    now = utcnow()
+    empty = make_session(
+        started_at=now - timedelta(days=100),
+        ended_at=now - timedelta(days=100) + timedelta(minutes=1),
+    )
+    db.upsert_session(empty)
+
+    spans_deleted, sessions_deleted = db.delete_spans_before(now - timedelta(days=90))
+    assert (spans_deleted, sessions_deleted) == (0, 1)
+    assert db.conn.execute("SELECT COUNT(*) FROM sessions").fetchone()[0] == 0
+
+
+def test_delete_spans_before_cutoff_keeps_a_live_session_with_no_spans_yet(db):
+    """An open session that has not written a span is not aged out."""
+    _insert_agent(db)
+    now = utcnow()
+    db.upsert_session(make_session(started_at=now, ended_at=None))
+
+    spans_deleted, sessions_deleted = db.delete_spans_before(now - timedelta(days=90))
+    assert (spans_deleted, sessions_deleted) == (0, 0)
+    assert db.conn.execute("SELECT COUNT(*) FROM sessions").fetchone()[0] == 1
 
 
 # -- Traces --

{tokenjam-0.6.2 → tokenjam-0.6.4}/tests/integration/test_full_pipeline.py

@@ -134,8 +134,12 @@ def full_stack():
     ocw_exporter = TjSpanExporter(pipeline)
 
     # Create a local TracerProvider (not global) and bind the SDK tracer to it
+    # for the duration of this test. Restored in teardown below so a stale
+    # tracer bound to this test's (soon-to-be-shutdown) provider and closed DB
+    # doesn't leak into tests that run later in the same session (#615).
     provider = TracerProvider()
     provider.add_span_processor(SimpleSpanProcessor(ocw_exporter))
+    original_tracer = agent_mod._tracer
     agent_mod._tracer = provider.get_tracer("tokenjam.sdk")
 
     # Seed agent records
@@ -155,6 +159,7 @@ def full_stack():
 
     yield stack
 
+    agent_mod._tracer = original_tracer
     provider.shutdown()
     db.close()