cctally 1.73.0 → 1.75.0

package/CHANGELOG.md

@@ -5,6 +5,30 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [1.75.0] - 2026-07-20
+
+### Added
+- Add `cctally codex percent-breakdown`, a cached native seven-day Codex milestone command with the same terminal design as `cctally percent-breakdown`, exact root/limit selection, current or retained cycle selection, query-time pricing, and matching five-hour context. The materialized dashboard projection is read by default; `--sync` explicitly refreshes it.
+- Add guided `stats.db` corruption repair and SQLite-native online backups, with verified recovery, preserved usage history/schema version, safe original-file retention, active-writer and production guards, and actionable malformed-database errors. (#314)
+
+### Fixed
+- Dashboard Codex hero now uses the exact Claude hero composition and metric order. Its native seven-day cycle supplies Week Usage, optional 5-hour usage, reset countdown, cycle spend, `$ / 1%`, forecast-at-reset, prior-cycle `$ / 1%` delta, and snapshot age; unavailable cycle accounting stays explicit instead of falling back to token or budget tiles.
+- Force-refresh and automatic OAuth refresh now serialize their complete 429 backoff transitions under the same selected-state lock. Concurrent rate limits preserve both the consecutive-429 count and furthest deadline, while a completed force-refresh can no longer erase a newer automatic-refresh cooldown. (#309)
+- Dashboard Codex modals now populate the canonical Claude-shaped current-cycle, session, period, project, cache, forecast, alert, and **$/1% Trend** views from native session metadata, quota observations, token accounting, and computed cache/model breakdowns instead of sparse placeholders or renamed semantics. The current-cycle modal reuses the complete durable quota breakdown, preserving every 1% crossing even when the dashboard's bounded observation tail begins later. (#319)
+- The current 5-hour block no longer lingers as an approximate (`~`) window in `cctally blocks` and the dashboard Blocks panel while the statusline already shows the correct 5-hour data. When a new 5-hour window opened while your usage percentage was flat, `record-usage` deduped the unchanged tick and never wrote the new window's canonical anchor, so the active block fell back to its heuristic start until the percentage next moved (a lag of up to several minutes at window start). The dedup self-heal now also materializes the rolled-over window's anchor, so the active block reads as API-anchored right away; the statusline was always correct because it renders the live rate limits rather than the stored snapshot.
+- Conversation-transcript retention now reclaims disk automatically and defaults to a tighter window, so `cache.db` no longer grows to multiple gigabytes of transcript text and search index that slow every dashboard sync tick — which, under multi-process contention (a second dashboard plus status-line hooks), could wedge the sync into a "sync paused" state and leave Codex quota/weekly panels blank. A freshly-created `cache.db` now uses SQLite `INCREMENTAL` auto-vacuum, and the once-a-day retention prune returns the freed pages to the OS (physically reclaimed on the next end-of-sync checkpoint) instead of requiring a manual `cctally db vacuum`; a `cache.db` created by an older version keeps the legacy behavior until a one-time `cctally db vacuum` or `cctally cache-sync --rebuild`, after which it too reclaims automatically. The default `conversation.retention_days` is now 90 (was 180) — set your own window or `off` with `cctally config set conversation.retention_days`.
+
+## [1.74.0] - 2026-07-19
+
+### Fixed
+- Dashboard Daily, Weekly, and Monthly details now use the same bounded history density across Claude, Codex, and All sources—30 days, 12 weeks, and 8 months—with compact date labels and contained mobile tables; Codex Blocks remains explicitly unavailable when native 5-hour activity does not exist. (#319)
+- Dashboard Trend, Projects, Cache Report, Forecast, and Alerts details now keep their canonical composed structure for Codex and All sources, show provider-native values or explicit unavailable reasons in every slot, and route provider-qualified project rows through the source-bound detail foundation. (#319)
+- Dashboard source-qualified Session and Project details now preserve Codex accounting totals when project metadata is incomplete; the combined source retains server-issued Claude Block identities and opens the canonical Block modal; source details share the standard focus, Escape, return-focus, and scroll-lock lifecycle; and desktop Share previews retain the captured provider. (#319)
+- Codex accounting no longer double-counts parent token events copied into a forked/subagent rollout before its first model context. Those records remain available to the physical conversation reader but are excluded from accounting; cache migration 027 safely replays retained Codex rollouts to remove existing `unknown` duplicates. Consequently, `cctally pricing-check` and `doctor` no longer report those ingest artifacts as a missing embedded price, while real unknown model IDs remain actionable. The complete scoped OpenAI model set and tracked rates were reverified against the live pricing sources, and the embedded pricing snapshot date is now 2026-07-19.
+- Dashboard Codex Weekly rows now follow observed native seven-day quota-cycle re-anchors instead of calendar weeks, collapsing reset jitter and clipping an older cycle when OpenAI grants an early reset. Codex quota forecasts now use elapsed native-cycle pace rather than activity-only bursts, avoiding misleading projections pinned at 100%. (#312)
+- Dashboard Codex source reads now retain accounting, quota, budget, session, and forensics panels when retained historical rows still lack project attribution. The cache replay path repairs this metadata when possible; until then the source is fresh-but-partial and disables Projects only, with a rebuild remedy. (#312)
+- Dashboard Codex hero cost and token counters now cover the active native seven-day reset cycle, rather than an unrelated calendar slice. A missing or conflicting active seven-day boundary leaves quota and budget visible while clearly marking the hero unavailable; Codex panels regain bounded native chrome and source warnings degrade only the affected domain. (#312)
+
 ## [1.73.0] - 2026-07-18
 
 ### Added

package/bin/_cctally_cache.py

@@ -813,8 +813,30 @@ def _delete_codex_file_derived_rows(
     )
 
 
-def _clear_codex_derived_rows(conn: sqlite3.Connection) -> None:
-    """Clear every re-derivable Codex row family in child-before-root order."""
+def _clear_codex_derived_rows(conn: sqlite3.Connection) -> bool:
+    """Clear every re-derivable Codex row family and report whether state changed.
+
+    The FTS5 ``delete-all`` command can increment SQLite's cumulative change
+    counter even when the semantic Codex surface was already empty.  Callers
+    use this return value, rather than ``Connection.total_changes``, when
+    deciding whether to advance the physical-mutation sequence.
+    """
+    state_changed = any(
+        conn.execute(query).fetchone() is not None
+        for query in (
+            "SELECT 1 FROM codex_session_entries LIMIT 1",
+            "SELECT 1 FROM quota_window_snapshots WHERE source = 'codex' LIMIT 1",
+            "SELECT 1 FROM codex_conversation_threads LIMIT 1",
+            "SELECT 1 FROM codex_conversation_events LIMIT 1",
+            "SELECT 1 FROM codex_session_files LIMIT 1",
+            "SELECT 1 FROM codex_source_roots LIMIT 1",
+            "SELECT 1 FROM codex_conversation_messages LIMIT 1",
+            "SELECT 1 FROM codex_conversation_file_touches LIMIT 1",
+            "SELECT 1 FROM codex_conversation_rollups LIMIT 1",
+            "SELECT 1 FROM cache_meta "
+            "WHERE key='codex_quota_projection_certificate' LIMIT 1",
+        )
+    )
     conn.execute("DELETE FROM codex_session_entries")
     conn.execute("DELETE FROM quota_window_snapshots WHERE source = 'codex'")
     conn.execute("DELETE FROM codex_conversation_threads")
@@ -832,6 +854,7 @@ def _clear_codex_derived_rows(conn: sqlite3.Connection) -> None:
     conn.execute(
         "DELETE FROM cache_meta WHERE key='codex_quota_projection_certificate'"
     )
+    return state_changed
 
 
 def _bump_codex_physical_mutation_seq(conn: sqlite3.Connection) -> None:
@@ -3903,9 +3926,7 @@ def sync_codex_cache(
             # Clear INSIDE the lock — see sync_cache() for the full
             # rationale. Done before the existing SELECT so delta
             # detection sees an empty baseline.
-            before_clear = conn.total_changes
-            _clear_codex_derived_rows(conn)
-            if conn.total_changes != before_clear:
+            if _clear_codex_derived_rows(conn):
                 _bump_codex_physical_mutation_seq(conn)
             conn.commit()
             eprint("[cache-sync] rebuild: cleared Codex cached entries")
@@ -4732,6 +4753,15 @@ def open_cache_db() -> sqlite3.Connection:
     except OSError as exc:
         eprint(f"[cache] could not chmod cache.db 0600 ({exc}); continuing")
 
+    # #313 P3 reclaim: create a fresh cache.db in INCREMENTAL auto-vacuum mode so
+    # the transcript-retention prune can return freed pages to the OS via
+    # `PRAGMA incremental_vacuum` instead of only growing the freelist — the
+    # conversation-transcript store is what bloated cache.db to multiple GB and
+    # slowed every sync tick. This MUST run before the first page is written
+    # (before journal_mode=WAL / any CREATE TABLE) to take effect; it is a
+    # harmless no-op on an already-created DB, which keeps its existing mode
+    # until a full VACUUM (`cctally db vacuum`) or a `cache-sync --rebuild`.
+    conn.execute("PRAGMA auto_vacuum=INCREMENTAL")
     conn.execute("PRAGMA journal_mode=WAL")
     # #297: 15 s (was 5 s) lets a writer wait out a slow-but-normal sync (>5 s)
     # instead of instantly erroring with "database is locked". Covers plain

package/bin/_cctally_config.py

@@ -333,7 +333,7 @@ ALLOWED_CONFIG_KEYS = (
 
 _CODEX_BUDGET_LEAF_PREFIX = "budget.codex."
 
-_DEFAULT_CONVERSATION_RETENTION_DAYS = 180
+_DEFAULT_CONVERSATION_RETENTION_DAYS = 90
 
 
 def _validate_retention_days_value(raw: object) -> int:
@@ -374,9 +374,9 @@ def _validate_retention_days_value(raw: object) -> int:
 def resolve_retention_days(config: dict) -> int:
     """Effective conversation transcript retention in days (F8).
 
-    Default is ``180``; ``0`` disables retention (keep forever). Any malformed
+    Default is ``90``; ``0`` disables retention (keep forever). Any malformed
     persisted value (non-int, boolean, negative, un-parseable string) degrades
-    to the safe 180 default rather than raising.
+    to the safe 90 default rather than raising.
     """
     default = _DEFAULT_CONVERSATION_RETENTION_DAYS
     block = config.get("conversation") if isinstance(config, dict) else None

package/bin/_cctally_core.py

@@ -1098,11 +1098,19 @@ def open_db() -> sqlite3.Connection:
     _log_migration_error = c._log_migration_error
     _clear_migration_error_log_entries = c._clear_migration_error_log_entries
 
+    repair_marker = DB_PATH.with_name("stats.db.repairing")
+    if repair_marker.exists():
+        raise c.StatsDbMaintenanceError()
     ensure_dirs()
+    if repair_marker.exists():
+        raise c.StatsDbMaintenanceError()
     conn = sqlite3.connect(DB_PATH)
+    if repair_marker.exists():
+        conn.close()
+        raise c.StatsDbMaintenanceError()
     conn.row_factory = sqlite3.Row
     # #279 S1 F4: probe connect + initial PRAGMAs so a corrupt stats.db (the
-    # non-re-derivable DB) surfaces as a one-line diagnosis + exit 2 instead of
+    # non-re-derivable DB) surfaces as a one-line diagnosis + staged exit 3 instead of
     # a raw traceback. The catch boundary is DELIBERATELY narrow — ONLY the
     # connect/PRAGMA/probe below. The DDL + `_run_pending_migrations` region
     # further down is NOT wrapped: migration-handler failures have their own
@@ -1133,8 +1141,9 @@ def open_db() -> sqlite3.Connection:
         raise c.StatsDbCorruptError(
             f"stats.db appears corrupt or unreadable ({exc}). path: {DB_PATH}. "
             f"Not auto-recreated — it holds your recorded usage history. "
-            f'Recovery: back up the file, then try `sqlite3 "{DB_PATH}" ".recover"`, '
-            f"or move it aside to start fresh. If this recurs, please file an issue."
+            "Recovery: run `cctally db repair --db stats --yes`; it preserves "
+            "the corrupt original before replacing anything. Do not copy, "
+            "restore, or move the live DB by hand."
         ) from exc
     conn.execute(
         """

package/bin/_cctally_dashboard.py

@@ -483,16 +483,20 @@ def _source_safe_native_detail(row: Mapping, *, source: str,
     common = {"detail_kind": f"{source}_{resource}", "key": key}
     allowed = {
         "session": (
-            "last_activity", "cost_usd", "input_tokens", "cached_input_tokens",
+            "label", "project", "started_at", "duration_min", "last_activity",
+            "cost_usd", "input_tokens", "cached_input_tokens",
             "output_tokens", "reasoning_output_tokens", "total_tokens", "models",
+            "model_breakdowns",
         ),
         "project": (
-            "first_seen", "last_seen", "cost_usd", "input_tokens",
+            "label", "first_seen", "last_seen", "range_start", "range_end",
+            "cost_usd", "input_tokens",
             "cached_input_tokens", "output_tokens", "reasoning_output_tokens",
-            "total_tokens", "session_count", "sessions_count",
+            "total_tokens", "session_count", "sessions_count", "models", "sessions",
         ),
         "block": (
-            "resets_at", "current_percent", "orphaned", "forecast",
+            "label", "start_at", "end_at", "resets_at", "current_percent",
+            "orphaned", "is_active", "cost_usd", "model_breakdowns", "forecast",
             "window_minutes", "observed_slot", "milestones",
         ),
     }[resource]
@@ -570,6 +574,9 @@ def _codex_detail_inputs(snapshot):
             week_start_name=semantics.week_start_name,
             speed=semantics.speed,
             codex_budget=semantics.codex_budget,
+            codex_quota_actual_thresholds=semantics.codex_quota_actual_thresholds,
+            codex_quota_projected_thresholds=semantics.codex_quota_projected_thresholds,
+            cache_report_anomaly_threshold_pp=semantics.cache_report_anomaly_threshold_pp,
         )
         qualified = load_qualified_codex_entries(
             range_start,
@@ -763,6 +770,34 @@ def _build_codex_source_detail(snapshot, *, resource: str, key: str) -> dict[str
     raise SourceResourceNotFound()
 
 
+def _codex_partial_source_detail(snapshot, row: Mapping, *, resource: str,
+                                 key: str) -> dict[str, Any]:
+    """Keep a published cache-only row usable when project metadata is partial."""
+    detail = _source_safe_native_detail(
+        row, source="codex", resource=resource, key=key,
+    )
+    metadata_missing = resource != "session" or not row.get("project")
+    if metadata_missing:
+        detail.update({
+            "metadata_availability": "partial",
+            "metadata_reason": "Project metadata is unavailable for this item.",
+        })
+    if resource == "session":
+        detail.setdefault("models", [])
+        detail.setdefault("model_breakdowns", [])
+    elif resource == "project":
+        generated_at = getattr(snapshot, "generated_at", None)
+        detail.setdefault("range_start", detail.get("first_seen"))
+        detail.setdefault(
+            "range_end",
+            generated_at.astimezone(dt.timezone.utc).isoformat()
+            if isinstance(generated_at, dt.datetime) else detail.get("last_seen"),
+        )
+        detail.setdefault("models", [])
+        detail.setdefault("sessions", [])
+    return detail
+
+
 def _source_safe_claude_session_detail(detail, *, key: str) -> dict[str, Any]:
     """Adapt the existing Claude session detail builder to S4's safe route."""
     payload = _session_detail_to_envelope(detail)
@@ -958,7 +993,24 @@ def build_source_detail(*, snapshot, source: str, resource: str,
     row = source_detail_lookup(snapshot.source_bundle, source, resource, key)
     if source == "claude":
         return _build_claude_source_detail(snapshot, resource=resource, key=key)
-    return _build_codex_source_detail(snapshot, resource=resource, key=key)
+    try:
+        detail = _build_codex_source_detail(snapshot, resource=resource, key=key)
+        if resource == "session":
+            for field in ("label", "project", "started_at", "duration_min"):
+                detail[field] = row.get(field)
+        elif resource == "project":
+            detail["label"] = row.get("label")
+        elif resource == "block":
+            for field in (
+                "label", "start_at", "end_at", "is_active", "cost_usd",
+                "model_breakdowns",
+            ):
+                detail[field] = row.get(field)
+        return detail
+    except sys.modules["_cctally_source_analytics"].QualifiedMetadataUnavailable:
+        return _codex_partial_source_detail(
+            snapshot, row, resource=resource, key=key,
+        )
 
 _ensure_sibling_loaded("_cctally_dashboard_share")
 from _cctally_dashboard_share import (

package/bin/_cctally_dashboard_conversation.py

@@ -39,6 +39,7 @@ edits — the forwarders hit ``sys.modules["_cctally_dashboard"]`` at call time
 from __future__ import annotations
 
 import json
+import re
 import socket
 import sqlite3
 import sys
@@ -224,8 +225,7 @@ _QUALIFIED_SEARCH_ACCEPTED = ("source", "q", "kind", "limit", "cursor")
 _QUALIFIED_FACETS_ACCEPTED = ("source",)
 # A raw browse cursor is a conversation key — printable + URL-safe by construction
 # (§2.2). Syntactic-only validation: reject whitespace/control/empty; echo raw.
-import re as _re_qs  # noqa: E402 — module-level compile for the browse-cursor check
-_BROWSE_CURSOR_RE = _re_qs.compile(r"\A[!-~]+\Z")
+_BROWSE_CURSOR_RE = re.compile(r"\A[!-~]+\Z")
 
 
 def _resolve_effective_speed():

package/bin/_cctally_dashboard_share.py

@@ -1404,6 +1404,8 @@ def _build_codex_source_share_snapshot(ls, *, state, panel: str,
         raise ValueError("source capability unavailable")
     required_domain = {
         "current-week": "hero",
+        "trend": "periods",
+        "forecast": "quota",
         "daily": "periods",
         "monthly": "periods",
         "weekly": "periods",
@@ -1425,14 +1427,47 @@ def _build_codex_source_share_snapshot(ls, *, state, panel: str,
         source_rows = all_rows[-1:] if all_rows else ()
         command = "codex-weekly"
         display_tz = str(weekly.get("display_tz") or "UTC")
-    elif panel in ("daily", "monthly", "weekly"):
+    elif panel in ("daily", "monthly", "weekly", "trend"):
         periods = data.get("periods")
-        panel_data = periods.get(panel) if isinstance(periods, Mapping) else {}
+        period_key = "weekly" if panel == "trend" else panel
+        panel_data = periods.get(period_key) if isinstance(periods, Mapping) else {}
         if not isinstance(panel_data, Mapping):
             raise ValueError("source capability unavailable")
         source_rows = tuple(panel_data.get("rows", ()))
-        command = f"codex-{panel}"
+        command = f"codex-{period_key}"
         display_tz = str(panel_data.get("display_tz") or "UTC")
+    elif panel == "forecast":
+        quota = data.get("quota")
+        panel_data = quota if isinstance(quota, Mapping) else {}
+        source_rows = tuple(panel_data.get("histories", ())) if isinstance(panel_data, Mapping) else ()
+        start, end = _share_codex_period_bounds(
+            state=state, panel="weekly", options=options, rows=source_rows,
+        )
+        rows = []
+        for row in source_rows:
+            if not isinstance(row, Mapping):
+                continue
+            forecast = row.get("forecast") if isinstance(row.get("forecast"), Mapping) else {}
+            current = row.get("current_percent")
+            projected = forecast.get("projected_percent")
+            rows.append(ls.Row(cells={
+                "limit": ls.TextCell(str(row.get("label") or "Codex quota")),
+                "current": ls.TextCell("—" if current is None else f"{float(current):.1f}%"),
+                "projected": ls.TextCell("—" if projected is None else f"{float(projected):.1f}%"),
+            }))
+        return ls.ShareSnapshot(
+            cmd="codex-quota", title="Codex Quota Forecast", subtitle=None,
+            period=ls.PeriodSpec(start=start, end=end, display_tz="UTC", label=None),
+            columns=(
+                ls.ColumnSpec(key="limit", label="Limit"),
+                ls.ColumnSpec(key="current", label="Current", align="right"),
+                ls.ColumnSpec(key="projected", label="Projected", align="right"),
+            ),
+            rows=tuple(rows), chart=None, totals=(), notes=(), generated_at=end,
+            version=sys.modules["cctally"]._share_resolve_version(),
+            template_id=template_id, source="codex", source_label="Codex",
+            availability=availability, availability_reason=reason,
+        )
     elif panel == "sessions":
         panel_data = data.get(panel) if isinstance(data.get(panel), Mapping) else {}
         source_rows = tuple(panel_data.get("rows", ())) if isinstance(panel_data, Mapping) else ()