cctally 1.84.1 → 1.85.0

package/CHANGELOG.md

@@ -5,6 +5,13 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [1.85.0] - 2026-07-29
+
+### Fixed
+- Rebuilding `stats.db` now restores only the newest open 5-hour block for each Claude account. A correction that deliberately retired historical block-close facts could previously make the rebuild recreate every older snapshot window as an open, unstamped block; the next dashboard sync then tried to record an obsolete revision over the correction and stayed on `server sync error`. The append-only journal remains unchanged, while rebuilding the disposable index now converges on the corrected history and background sync resumes normally.
+- `cctally db rederive` no longer deletes the history it cannot re-derive. Usage observations are only recorded from the point cctally started journaling them, so everything older survives solely as the rows exported when the journal was first created — and the re-derivation compared those rows against a replay that could never contain them, then retired every one as stale. A single `db rederive --family claude-usage --yes` therefore erased months of weekly usage and cost: a `cctally dollar-per-percent --weeks 12` and the dashboard's `$/1% Trend` card both collapsed to the weeks since the journal began, and because the deletions are durable journal truth every later rebuild reproduced the loss exactly. History from before that point is now held out of the comparison entirely, while anything inside the window a replay does cover is still re-derived and retired as before. On an install that already lost history, the fix is also the cure: the next plan restores those weeks from the journal's own retained lines, so running `cctally db rederive --family claude-usage` and applying it with `--yes` brings the full trend back. A retirement recorded by anything other than this re-derivation is left alone, and `db rederive --json` now reports how many events a plan protected. (#426)
+- The Codex current cycle shows its per-percent milestones again. OpenAI reports the same weekly reset a few seconds apart from one sample to the next, and cctally settles those spellings into a single window — but the dashboard hero was still publishing the raw provider value while the milestone rows were recorded against the settled one, so the two never matched and the cycle read `0 crossed` beneath "No integer-percent crossing has been retained in this cycle yet." while its crossings sat intact and correct in the index. Nothing was lost: the hero, the live quota rows and the milestone ladder now all name the same settled reset, so the crossings reappear on the next load, whether an account is selected or not. A running quota window is also no longer treated as finished the moment its raw reset passes, seconds ahead of the settled one, and the 5-hour percentage a window contributes is held to the same rule. Past cycles were never affected, and an install whose resets do not jitter is unchanged. (#428)
+
 ## [1.84.1] - 2026-07-29
 
 ### Fixed

package/bin/_cctally_dashboard_sources.py

@@ -228,10 +228,16 @@ def _resolve_codex_weekly_cycle(
             continue
         account = history.identity.account_key
         accounts_seen.add(account)
-        if baseline is None or baseline.resets_at <= now_utc:
+        # #428: the CANONICAL anchor, never the raw provider reset. Blocks and
+        # milestones are keyed on the anchor (#416 §4.1), so publishing the raw
+        # value here mints a second spelling of one window — and the client's
+        # current-cycle milestone filter matches `resets_at` exactly, so the
+        # ladder empties for precisely the jittered cycles canonicalization
+        # exists to collapse. Liveness rides the same instant the hero shows.
+        if baseline is None or baseline.canonical_resets_at <= now_utc:
             continue
         state = quota_freshness(history.physical_observations, now_utc).state
-        boundary = (history.identity.window_minutes, baseline.resets_at)
+        boundary = (history.identity.window_minutes, baseline.canonical_resets_at)
         if state == "fresh":
             bucket = fresh_by_account
         elif state == "stale":
@@ -431,7 +437,9 @@ def _codex_next_decision_at(
         baseline = select_baseline(history.observations, now_utc)
         if codex_history_is_model_scoped(history, baseline=baseline):
             continue
-        if baseline is not None and baseline.resets_at > now_utc:
+        # #428: the anchor, so the decision deadline describes the same window
+        # the hero publishes rather than a jitter sibling up to 600s away.
+        if baseline is not None and baseline.canonical_resets_at > now_utc:
             latest = latest_physical_observation(history.physical_observations)
             if latest is not None:
                 candidates.append(
@@ -1747,12 +1755,15 @@ def _quota_read_model(
         history_rows.append(row)
         if _codex_history_row_is_model_scoped(row):
             continue
-        if baseline is not None and baseline.resets_at > context.now_utc:
+        # #428: the client compares `active[].resets_at` against
+        # `hero.cycle.resets_at` (`activeWeeklyKeys`) to decide which weekly
+        # history is the live one, so both must carry the SAME anchor.
+        if baseline is not None and baseline.canonical_resets_at > context.now_utc:
             active_rows.append({
                 "key": dashboard_resource_key("quota", "codex", *key_parts),
                 "current_percent": baseline.used_percent,
                 "captured_at": baseline.captured_at.astimezone(UTC).isoformat(),
-                "resets_at": baseline.resets_at.astimezone(UTC).isoformat(),
+                "resets_at": baseline.canonical_resets_at.astimezone(UTC).isoformat(),
                 "freshness": freshness.state,
                 "stale_after_seconds": freshness.stale_after_seconds,
             })
@@ -2554,7 +2565,9 @@ def _codex_account_five_hour_percent(
         baseline = select_baseline(history.observations, now_utc)
         if codex_history_is_model_scoped(history, baseline=baseline):
             continue
-        if baseline is None or baseline.resets_at <= now_utc:
+        # #428: the anchor — a 5h window whose raw reset has passed but whose
+        # canonical anchor has not is still live and still owns its percent.
+        if baseline is None or baseline.canonical_resets_at <= now_utc:
             continue
         acct = history.identity.account_key
         pct = float(baseline.used_percent)

package/bin/_cctally_five_hour.py

@@ -1526,15 +1526,30 @@ def _backfill_five_hour_blocks(
         # shared physical 5h window observed by two accounts yields two distinct
         # open blocks so rebuild reproduces per-account ownership.
         keys_sql = """
-                SELECT DISTINCT five_hour_window_key, account_key
-                  FROM weekly_usage_snapshots
-                 WHERE five_hour_window_key IS NOT NULL
-                   AND five_hour_percent     IS NOT NULL
+                SELECT DISTINCT snapshots.five_hour_window_key,
+                                snapshots.account_key
+                  FROM weekly_usage_snapshots AS snapshots
+                 WHERE snapshots.five_hour_window_key IS NOT NULL
+                   AND snapshots.five_hour_percent     IS NOT NULL
         """
         if only_missing:
             keys_sql += (
-                "   AND (five_hour_window_key, account_key) NOT IN "
-                "(SELECT five_hour_window_key, account_key FROM five_hour_blocks)"
+                "   AND snapshots.id = ("
+                "       SELECT latest.id"
+                "         FROM weekly_usage_snapshots AS latest"
+                "        WHERE latest.account_key IS snapshots.account_key"
+                "          AND latest.five_hour_window_key IS NOT NULL"
+                "          AND latest.five_hour_percent IS NOT NULL"
+                "        ORDER BY unixepoch(latest.captured_at_utc) DESC,"
+                "                 latest.id DESC"
+                "        LIMIT 1"
+                "   )"
+                "   AND NOT EXISTS ("
+                "       SELECT 1 FROM five_hour_blocks AS blocks"
+                "        WHERE blocks.five_hour_window_key ="
+                "              snapshots.five_hour_window_key"
+                "          AND blocks.account_key IS snapshots.account_key"
+                "   )"
             )
         keys = [(int(r[0]), r[1]) for r in conn.execute(keys_sql).fetchall()]
 
@@ -1784,4 +1799,3 @@ def _backfill_five_hour_blocks(
         eprint(f"[5h-block backfill] failed: {exc}")
         return 0
     return inserted
-

package/bin/_cctally_rederive.py

@@ -397,12 +397,25 @@ def plan_claude_usage(
         )
     with tempfile.TemporaryDirectory(prefix="cctally-rederive-") as tmp:
         desired = _derive_desired_events(records, cache_conn, Path(tmp))
+    # #426: the scratch replay only sees RETAINED sources, so it can never
+    # reproduce the pre-cutover rows the journal exported as `b:<table>:<rowid>`
+    # evt lines. Hold them out of the diff instead of retiring them. Evidence is
+    # counted in OBSERVATIONS alone — an operator record is replay input but
+    # derives nothing on its own, and the cutover re-emits some of them
+    # (`weekly_credit_floor`) carrying their original historical timestamp.
+    preserved = _lib_rederive.preserved_history(
+        records,
+        evidence_retained=any(
+            record.get("t") == "obs" for record in raw_records
+        ),
+    )
     return _lib_rederive.build_claude_usage_plan(
         selection=selection,
         desired_events=desired,
         journal_high_water=journal_high_water,
         cache_fingerprint=cache_fingerprint,
         config_fingerprint=config_fingerprint,
+        preserved_events=preserved.values(),
         conflicted_event_ids=owned_conflicted_event_ids(selection),
     )
 
@@ -868,6 +881,9 @@ def _command_payload(
         "batchId": batch_id,
         "planHash": None if plan is None else plan.plan_hash,
         "actionCounts": counts,
+        # #426: how many owned events the plan held OUT of the diff because no
+        # retained source can re-derive them (pre-cutover exported history).
+        "preservedEventCount": 0 if plan is None else plan.preserved_event_count,
         # `conflicts` is the LEGACY key and keeps its meaning: command-validation
         # failure messages (unsupported family, prod guard, structural journal
         # protocol errors). #374's quarantined same-revision GROUPS ride the new

package/bin/_lib_rederive.py

@@ -20,6 +20,18 @@ from collections.abc import Iterable, Mapping
 
 FAMILY = "claude-usage"
 
+# Prefix of a correction batch this family authored. Only its own destructive
+# batches may be undone by :func:`build_claude_usage_plan` (#426) — a tombstone
+# written by anyone else is a deliberate retirement and stays retired.
+_FAMILY_BATCH_PREFIX = f"rederive:{FAMILY}:"
+
+# Id shape minted by the journal cutover exporter (`_lib_journal.bootstrap_id`,
+# `b:<table>:<rowid>`) for a row that predates the journal. The family's own
+# derivation only ever mints natural-key ids (`sa:`, `wcs:`, `pm:`, …), and the
+# cutover runs once per install, so this prefix identifies exactly the events a
+# replay can never reproduce — independently of any timestamp (#426).
+_CUTOVER_EXPORT_ID_PREFIX = "b:"
+
 
 class RederiveError(RuntimeError):
     """Base error for a plan that cannot be produced truthfully."""
@@ -173,6 +185,74 @@ def validate_claude_cache_contract(tables: Mapping[str, set[str]]) -> None:
         )
 
 
+def preserved_history(records: Iterable[Mapping], *,
+                      evidence_retained: bool) -> dict[str, Mapping]:
+    """Owned events the current re-derivation cannot reach, keyed by event id.
+
+    Two independent, deterministic reasons an event can have no retained source
+    behind it — so a replay-derived desired set can never contain it:
+
+    * its id was minted by the cutover exporter (``b:<table>:<rowid>``). The
+      journal only starts recording observations AT the cutover, so those
+      exported lines ARE the pre-journal truth; the family's own derivation
+      mints natural keys and can never produce them.
+    * nothing is retained at all (``evidence_retained=False``) — every desired
+      set is then empty, so a diff could only ever be destructive.
+
+    Diffing such an event anyway put it in the "current but not desired" branch
+    and TOMBSTONED it: one ``db rederive --yes`` retired every pre-cutover
+    weekly usage/cost snapshot on a real install (#426). Those events are
+    preserved instead — never tombstoned, never rewritten from a re-derivation
+    that does not cover them. Everything the retained observations DO cover
+    still diffs normally, so an obsolete derivation still retires.
+
+    The returned record is the highest-revision non-tombstone line for that id
+    — a rev-0 evt, or a replacement from a committed correction batch — i.e.
+    exactly the payload the selector would choose if no tombstone existed.
+    """
+    committed = {
+        record.get("id") for record in records
+        if record.get("t") == "correction_batch"
+        and record.get("phase") == "commit"
+    }
+    best: dict[str, tuple[int, int, Mapping]] = {}
+    for sequence, record in enumerate(records):
+        if record.get("t") == "evt":
+            candidate = record
+        elif (
+            record.get("t") == "correction"
+            and record.get("action") == "replace"
+            and record.get("batch") in committed
+        ):
+            candidate = {
+                "v": record.get("v"),
+                "t": "evt",
+                "id": record.get("id"),
+                "rev": record.get("rev", 0),
+                "at": record.get("at"),
+                "src": "ingest",
+                "payload": dict(record.get("payload") or {}),
+            }
+        else:
+            continue
+        event_id = candidate.get("id")
+        if not isinstance(event_id, str) or not event_id:
+            continue
+        if not _is_owned_event(candidate):
+            continue
+        if evidence_retained and not event_id.startswith(
+            _CUTOVER_EXPORT_ID_PREFIX
+        ):
+            continue  # the retained observations can re-derive it
+        revision = candidate.get("rev", 0)
+        if not isinstance(revision, int):
+            revision = 0
+        prior = best.get(event_id)
+        if prior is None or (revision, sequence) >= (prior[0], prior[1]):
+            best[event_id] = (revision, sequence, candidate)
+    return {event_id: entry[2] for event_id, entry in best.items()}
+
+
 def _canonical_bytes(value) -> bytes:
     return json.dumps(
         value, sort_keys=True, separators=(",", ":"), ensure_ascii=False,
@@ -266,6 +346,7 @@ class RederivePlan:
     counts: Mapping[str, int]
     actions: tuple[PlanAction, ...]
     retained_event_count: int
+    preserved_event_count: int = 0
 
     def _body(self) -> dict:
         return {
@@ -281,6 +362,7 @@ class RederivePlan:
             "configFingerprint": self.config_fingerprint,
             "counts": dict(self.counts),
             "retainedEventCount": self.retained_event_count,
+            "preservedEventCount": self.preserved_event_count,
             "payloadHashes": sorted(action.payload_hash for action in self.actions),
             "actions": [action.to_dict() for action in self.actions],
         }
@@ -317,9 +399,19 @@ def build_claude_usage_plan(*, selection, desired_events: Iterable[Mapping],
                             journal_high_water: "tuple[str, int] | None",
                             cache_fingerprint: str,
                             config_fingerprint: str,
+                            preserved_events: Iterable[Mapping],
                             conflicted_event_ids=frozenset()) -> RederivePlan:
     """Diff current effective events against one scratch-derived desired set.
 
+    ``preserved_events`` (#426) names the owned events the re-derivation cannot
+    reach — see :func:`preserved_history`. They are held OUT of the diff, so
+    the "current but not desired" branch below can never retire history the
+    replay was never able to reproduce. The keyword is deliberately required:
+    defaulting it to empty is exactly the data-loss bug it exists to prevent.
+    A preserved event is only ever acted on to make it durable again —
+    re-affirmed at ``rev + 1`` when its group is quarantined (#374), or revived
+    at ``rev + 1`` when THIS family's own earlier batch tombstoned it.
+
     ``conflicted_event_ids`` (#374) names the event ids this family owns whose
     same-revision group the selector QUARANTINED at the winning revision. They
     force a ``supersede`` at ``selected.rev + 1`` **even when the provisional
@@ -344,14 +436,43 @@ def build_claude_usage_plan(*, selection, desired_events: Iterable[Mapping],
         and selected.record is not None
         and not _is_owned_event(selected.record)
     )
+    preserved = {}
+    for record in preserved_events:
+        event_id = record.get("id")
+        if isinstance(event_id, str) and event_id and event_id not in desired:
+            preserved[event_id] = record
     counts = {"retain": 0, "supersede": 0, "tombstone": 0, "add": 0}
     actions: list[PlanAction] = []
 
-    for event_id in sorted(set(current) | set(desired)):
+    for event_id in sorted(set(current) | set(desired) | set(preserved)):
         current_record = current.get(event_id)
         desired_record = desired.get(event_id)
         selected = selection.by_id.get(event_id)
-        if current_record is not None and desired_record is not None:
+        preserved_record = preserved.get(event_id)
+        if preserved_record is not None:
+            # Un-re-derivable history: retain it, or restore it when this
+            # family's own earlier plan retired it (#426).
+            revive = (
+                selected is not None
+                and selected.status == "tombstone"
+                and str(selected.batch_id or "").startswith(_FAMILY_BATCH_PREFIX)
+            )
+            reaffirm = (
+                selected is not None
+                and selected.status == "active"
+                and event_id in conflicted_event_ids
+            )
+            if not (revive or reaffirm):
+                counts["retain"] += 1
+                continue
+            source = (
+                selected.record if reaffirm else preserved_record
+            )
+            revision = int(selected.rev) + 1
+            disposition = "supersede"
+            at = str(source["at"])
+            payload = dict(source.get("payload") or {})
+        elif current_record is not None and desired_record is not None:
             desired_record = _preserve_non_derivable_state(
                 current_record, desired_record
             )
@@ -404,4 +525,5 @@ def build_claude_usage_plan(*, selection, desired_events: Iterable[Mapping],
         counts=counts,
         actions=tuple(actions),
         retained_event_count=retained_event_count,
+        preserved_event_count=len(preserved),
     )

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cctally",
-  "version": "1.84.1",
+  "version": "1.85.0",
   "description": "Claude Code usage tracker and local dashboard for Pro/Max subscription limits - weekly cost-per-percent trend, quota forecasts, threshold alerts. ccusage-compatible.",
   "homepage": "https://github.com/omrikais/cctally",
   "repository": {