cctally 1.94.2 → 1.95.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,35 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [1.95.1] - 2026-08-09
9
+
10
+ ### Fixed
11
+ - Corruption recovery now preserves both incidents when two database families are quarantined within the same second. The second incident atomically reserves a distinct retention-compatible directory instead of reusing the first incident's path and failing because the old and new `stats.db` generations both exist.
12
+ - The authoritative test gate now runs its rollback-contention wall-clock checks after the parallel pytest estate, alongside the existing serial rebuild benchmark. Whole-estate CPU contention can no longer consume those latency ceilings while isolated regressions still fail the release.
13
+ - Concurrent usage ingestion no longer derives five-hour blocks or milestones from a snapshot that a retained suppression event deliberately removed. A duplicate observation now respects the absent Model-A target instead of passing `None` into the derivation path and repeatedly failing every ingester behind it.
14
+
15
+ ## [1.95.0] - 2026-08-09
16
+
17
+ ### Changed
18
+ - The test gate refuses to report green while verifying less than it claims. `bin/cctally-test-all` now admits the harness estate from a committed manifest (`tests/authoritative-test-manifest.json`) instead of a two-name list, so a harness that lost its executable bit, went missing, or appeared unexpectedly fails the run instead of quietly leaving the bundle. It also reads each harness's exit status rather than only the counters it printed, requires an audited minimum case count per harness, asserts the runner's capabilities before starting, and refuses a run carrying a golden-regeneration variable (#529).
19
+ - A red run now says which kind of red it was. Every run emits a versioned outcome object whose `failureClass` is `product` (tests genuinely failed), `infrastructure` (the run could not start or the machinery broke), or `incomplete` (the run finished without verifying what it claimed), with every contributing reason retained. The exit band is `0` pass, `1` product failure, `2` usage error, `3` infrastructure or incomplete (#529).
20
+
21
+ ### Fixed
22
+ - `stats.db` now uses SQLite rollback journaling (`DELETE` / `FULL`) instead of WAL. The disposable index no longer depends on live `-wal`/`-shm` generations or checkpoints; corrupt-family replacement requires a cold second drain and preserves the complete main/rollback-journal family before publishing the rebuilt index. Existing WAL installs rebuild once at epoch 1010 and refuse with restart guidance while an older dashboard or process still holds the legacy family. `cctally db checkpoint --db stats` is retired with exit 2; cache and conversations remain WAL-backed (#538).
23
+ - Codex Guardian approval-review usage is now priced through its canonical `gpt-5.5` model instead of the generic `gpt-5` unknown-model fallback. Retained `codex-auto-review` rows no longer trigger a false pricing-coverage advisory, fast-tier reports use the canonical 2.5x multiplier, and genuinely unknown identifiers still warn and remain actionable (#535).
24
+ - Differently pinned worktrees can now share the default remote test workdir. Each agentmem revision has its own selected executable, so provisioning one tree no longer replaces or rejects the other tree's pin and takes the runner fleet out of service (#519).
25
+ - Codex share artifacts now disclose provider availability consistently across all nine dashboard panels. Current-week, trend, daily, monthly, weekly, and sessions exports were deriving availability only from their row count, so a real partial or empty provider state could leave both the rendered artifact and its drift indicator unchanged (#533).
26
+ - Corrupt `stats.db` recovery no longer starves behind continuous hook traffic. The first detector now elects one durable background owner before waiting for maintenance, other callers return promptly and coalesce onto it, and that owner captures forensics and rebuilds after readers drain instead of every caller timing out before recovery is admitted (#530).
27
+ - Escape now closes the share modal after you have clicked into the live preview. The preview is a sandboxed frame, so a key pressed inside it belonged to a different document and never reached the shortcut that closes the modal — the only way out was to tab back out of the frame first. It works from the composer's combined preview too, and it still closes one layer at a time: a section's actions first, then the composer, then the dialog beneath it (#503).
28
+ - The share preview frame is no longer announced to a screen reader as "Report preview (decorative)". It was fully exposed to assistive technology while its only name told you not to bother with it — on the one control the documentation tells you to review before sharing. It now names the panel and template it is showing (#503).
29
+ - Manage presets is now a real dialog. It claimed to be one but rendered as a plain block at the bottom of the share modal: Tab walked straight out of it into the template gallery behind, and closing it left focus nowhere, because it tried to return focus to a menu item that had already disappeared. It now covers the screen, keeps Tab inside itself, fits a phone without sideways scrolling, and returns focus to the presets button you opened it from (#503).
30
+ - Escape inside a composer section's actions menu no longer closes the whole composer and discards your unsaved title, theme and format edits. It closes just the menu and puts focus back on the button that opened it; press it again to close the composer. Opening one section's actions also closes any other section's, which previously could all be open at once. The menu no longer describes itself to screen readers as a menu it never behaved like — both actions remain reachable by Tab exactly as before (#503).
31
+ - Share checkboxes and radio buttons are 44 pixels tall in a narrow browser window, not only on a touch device. The touch sizing keyed on the pointer type rather than the width, so a narrow desktop window got none of it. The `Anon on export` label — the privacy control — was also rendering at 12 pixels on a phone despite a rule meant to set it to 16, which is the threshold below which iOS zooms the page when you tap a control (#503).
32
+ - The share preview on a phone now shows the report instead of its frontmatter. It was capped at 128 pixels, which left about four lines once the privacy notice had taken its share, and the source label was printed twice inside that space. The cap is now proportional to the visible screen, and it shrinks correctly when the on-screen keyboard is up rather than being pushed off the bottom (#503).
33
+ - Gallery text no longer appears to bleed through the pinned preview on a phone. Two separate causes: a 12-pixel band above the preview where scrolling content passed through a gap, and a bottom edge where the preview and the tile beneath it were the same colour with the same hairline border, so two half-cut lines of text read as one overlapping line. The preview now has a visible boundary and sits flush against the top of the scroll area (#503).
34
+ - Rotating a phone or resizing the window with the share modal open no longer resets the preview. The two layouts rendered separate previews, so crossing the breakpoint threw away the rendered report and its privacy answer, blanked the preview for about a second, and briefly showed the wrong statement about whether your export contains real project names. There is now one preview, positioned by layout at both sizes (#503).
35
+ - The keyboard shortcuts in the share documentation now name the gestures that work. Every passage that advertised a bare `S` or `B` has been rewritten; both shortcuts have always required Shift, and an unshifted `s` opens Settings instead. Several passages also told you to click a panel to focus it, which does not focus it — it opens the panel's own modal, which then blocks the shortcut — and two quoted a help message the application has never shown. The correct gesture is to Tab to one of the panel's actions and press Shift+S, which appeared nowhere. The share icon's own tooltip said `Share (S)` and now says `Share (Shift+S)` (#503).
36
+
8
37
  ## [1.94.2] - 2026-08-08
9
38
 
10
39
  ### Fixed
@@ -321,14 +321,6 @@ OAUTH_BACKOFF_CAP_SECONDS = 3600.0
321
321
  _init_paths_from_env()
322
322
 
323
323
 
324
- # stats.db WAL cap (#297). Bounds the persistent WAL file so a resetting
325
- # checkpoint truncates it back down instead of leaving it at its high-water
326
- # size. stats.db writes are small (its -wal was observed at 0 bytes even under
327
- # the contention that bloated cache.db to multi-GB), so a tighter 16 MB cap is
328
- # ample. See docs/superpowers/specs/2026-07-13-cache-db-wal-hardening-design.md.
329
- STATS_WAL_SIZE_LIMIT_BYTES = 16 * 1024 * 1024 # 16777216
330
-
331
-
332
324
  # === stats.db epoch-rebuild versioning (DB journal redesign §7.1/§8) ==
333
325
  #
334
326
  # stats.db is no longer a versioned migration target — it is a DISPOSABLE index
@@ -393,7 +385,12 @@ STATS_WAL_SIZE_LIMIT_BYTES = 16 * 1024 * 1024 # 16777216
393
385
  # before any schema work, so an `add_column_if_missing` would never run on an
394
386
  # upgraded install and the column would simply never appear. The registry stays
395
387
  # frozen at 13 and an epoch mismatch resolves by rebuild.
396
- STATS_INDEX_EPOCH = 1009
388
+ # 1009 -> 1010 (#538 Task A): retire WAL for the disposable stats index. Every
389
+ # stats connection now uses DELETE/FULL rollback journaling, structurally
390
+ # removing the shared-memory WAL page map implicated by the retained corruption
391
+ # bundles. The one-time rebuild is the mode transition; no stats migration is
392
+ # added and cache/conversations remain WAL/NORMAL.
393
+ STATS_INDEX_EPOCH = 1010
397
394
  LEGACY_STATS_HEAD = 13
398
395
 
399
396
  #: #496 S1 F1. A NEW branch, for a state that cannot occur before the
@@ -773,8 +770,8 @@ def ensure_dirs() -> None:
773
770
  # bin/_cctally_store.py _heal_flock_blocking, reached through
774
771
  # _acquire_stats_maintenance_reentrant by the epoch
775
772
  # resolver
776
- # bin/_cctally_store.py _acquire_stats_maintenance_for_heal, the corruption
777
- # heal's ownership-first BOUNDED acquire (#496 S3)
773
+ # bin/_cctally_store.py detached corruption-heal worker maintenance acquire
774
+ # (the detector itself never acquires it; #530)
778
775
  # bin/_cctally_db.py cmd_db_rebuild, _acquire_db_admin_writer_flocks
779
776
  # (db skip / db unskip), _cmd_db_repair_exclusive,
780
777
  # _vacuum_one_db
@@ -1647,11 +1644,12 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
1647
1644
  # #453: probe the live index before the maintenance-shared opener. During
1648
1645
  # a large detached rebuild that opener can wait for its bounded 5-second
1649
1646
  # guard; a statusline arriving mid-replay would otherwise pay that delay
1650
- # (and can reach more than one stats opener in a single render). The raw
1651
- # probe is read-only and deliberately recognizes only a readable,
1652
- # post-legacy wrong epoch. Missing/unreadable/legacy indexes retain their
1653
- # established guarded-open paths. The post-open epoch gate below remains a
1654
- # race-closing defense if the version changes after this probe.
1647
+ # (and can reach more than one stats opener in a single render). The probe
1648
+ # reads only the fixed user_version field in the main-file header; it never
1649
+ # opens SQLite or bypasses the maintenance opener fence. Missing, invalid-
1650
+ # header and legacy indexes retain their established guarded-open paths.
1651
+ # The post-open epoch gate below remains a race-closing defense if the
1652
+ # version changes after this probe.
1655
1653
  if (
1656
1654
  _target_path is None
1657
1655
  and _cctally_store.stats_epoch_enabled()
@@ -1682,18 +1680,16 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
1682
1680
  # (corruption that only surfaces mid-DDL stays a raw error; the probe
1683
1681
  # catches the common case).
1684
1682
  try:
1685
- # §6.1 PRAGMA policy via the shared table (stats: WAL / NORMAL /
1683
+ # §6.1 PRAGMA policy via the shared table (stats: DELETE / FULL /
1686
1684
  # busy_timeout 15000 / journal_size_limit 16 MiB / auto_vacuum unset).
1687
1685
  # #297: busy_timeout 15000 lets a writer wait out a slow-but-normal sync
1688
- # (>5 s) instead of erroring "database is locked"; it does NOT absorb
1689
- # SQLITE_BUSY_SNAPSHOT (defended by BEGIN IMMEDIATE / write-first at the
1690
- # write sites, cctally-dev#87). journal_size_limit bounds the persistent
1691
- # WAL. The corruption probe (SELECT 1) stays here inside the narrow
1686
+ # (>5 s) instead of erroring "database is locked". The corruption probe
1687
+ # (SELECT 1) stays here inside the narrow
1692
1688
  # StatsDbCorruptError boundary.
1693
1689
  _cctally_store.apply_policy(conn, "stats")
1694
1690
  conn.execute("SELECT 1").fetchone()
1695
- # §9.2 (#496 S6 F23). AFTER `apply_policy`, because that is what sets
1696
- # journal_mode=WAL and so what can materialize the sidecars.
1691
+ # §9.2 (#496 S6 F23). AFTER `apply_policy`, so any rollback journal
1692
+ # materialized during the transition is included in family hardening.
1697
1693
  _cctally_store._harden_stats_family(db_path)
1698
1694
  except sqlite3.DatabaseError as exc:
1699
1695
  try:
@@ -1714,7 +1710,15 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
1714
1710
  # it — all of which fall through to the original guided
1715
1711
  # StatsDbCorruptError so the manual path still applies.
1716
1712
  heal = getattr(_cctally_store, "HEAL_HOOK", None)
1717
- if _target_path is None and heal is not None and heal("stats", exc):
1713
+ heal_enabled = (
1714
+ os.environ.get("CCTALLY_TEST_DISABLE_STATS_AUTO_HEAL") != "1"
1715
+ )
1716
+ if (
1717
+ _target_path is None
1718
+ and heal is not None
1719
+ and heal_enabled
1720
+ and heal("stats", exc)
1721
+ ):
1718
1722
  # #386: the post-heal retry is an opener too. The heal released the
1719
1723
  # maintenance lock before returning, so another maintenance path can
1720
1724
  # legitimately own the family by now.
@@ -1763,9 +1763,15 @@ def _build_codex_source_share_snapshot(ls, *, state, panel: str,
1763
1763
  display_tz_label=display_tz,
1764
1764
  )
1765
1765
  codex_module = sys.modules["cctally"]._load_sibling("_cctally_codex")
1766
+ # The canonical CLI adapter derives availability from its row count, but
1767
+ # this source-aware path has an authoritative provider state. Project the
1768
+ # normalized state after adapting the rows so all nine panels carry the
1769
+ # same source chrome and built-snapshot digest signal (#533).
1766
1770
  return replace(
1767
1771
  codex_module._build_codex_share_snapshot(command, view, normalized_rows),
1768
1772
  template_id=template_id,
1773
+ availability=availability,
1774
+ availability_reason=reason,
1769
1775
  )
1770
1776
 
1771
1777
 
@@ -373,10 +373,11 @@ class StatsHealDeferred(StatsRebuildDeferred):
373
373
  quarantine and rebuild while the caller waited (#496 S3 §6). The caller
374
374
  degrades where it already degrades; it never blocks on the rebuild.
375
375
 
376
- ``heal_id`` correlates this signal with the durable heal event the hook
377
- recorded at detection, and ``forensics_path`` is the absolute bundle the
378
- user was told about. Both are attributes rather than message text so a
379
- consumer can use them without parsing.
376
+ ``heal_id`` correlates this signal with the durable admitted occurrence.
377
+ ``forensics_path`` is optional: since #530 the fresh worker captures and
378
+ persists that bundle after maintenance drains, so the detector normally
379
+ raises before a path exists. Both remain attributes rather than parsed
380
+ message text.
380
381
  """
381
382
 
382
383
  def __init__(
@@ -1042,7 +1043,7 @@ def write_corruption_forensics(
1042
1043
  "truncated": False,
1043
1044
  },
1044
1045
  }
1045
- for suffix in ("", "-wal", "-shm"):
1046
+ for suffix in ("", "-journal", "-wal", "-shm"):
1046
1047
  p = pathlib.Path(str(db_path) + suffix)
1047
1048
  try:
1048
1049
  st = p.stat()
@@ -1276,6 +1277,7 @@ def _load_pending_quarantine(db_path: pathlib.Path) -> "dict[str, Any] | None":
1276
1277
  isinstance(name, str)
1277
1278
  and name in {
1278
1279
  db_path.name,
1280
+ f"{db_path.name}-journal",
1279
1281
  f"{db_path.name}-wal",
1280
1282
  f"{db_path.name}-shm",
1281
1283
  }
@@ -1298,6 +1300,37 @@ def _load_pending_quarantine(db_path: pathlib.Path) -> "dict[str, Any] | None":
1298
1300
  _QUARANTINE_MISSING_CONTEXT_WARNED = False # one-shot warn flag
1299
1301
 
1300
1302
 
1303
+ def _reserve_quarantine_incident(
1304
+ root: pathlib.Path, db_name: str, timestamp: str
1305
+ ) -> pathlib.Path:
1306
+ """Atomically reserve a retention-compatible incident directory."""
1307
+ root.mkdir(parents=True, exist_ok=True)
1308
+ primary = root / f"{db_name}-{timestamp}"
1309
+ try:
1310
+ primary.mkdir()
1311
+ return primary
1312
+ except FileExistsError:
1313
+ pass
1314
+
1315
+ match = re.fullmatch(r"(\d{8}T\d{6})(?:Z|_(\d{6}))", timestamp)
1316
+ if match is None:
1317
+ stem = dt.datetime.now(dt.timezone.utc).strftime("%Y%m%dT%H%M%S")
1318
+ first = 1
1319
+ else:
1320
+ stem = match.group(1)
1321
+ first = int(match.group(2) or "0") + 1
1322
+ for microsecond in range(first, 1_000_000):
1323
+ candidate = root / f"{db_name}-{stem}_{microsecond:06d}"
1324
+ try:
1325
+ candidate.mkdir()
1326
+ return candidate
1327
+ except FileExistsError:
1328
+ continue
1329
+ raise OSError(
1330
+ f"could not allocate a unique quarantine incident for {db_name}"
1331
+ )
1332
+
1333
+
1301
1334
  def _warn_quarantine_created_without_context(incident: pathlib.Path) -> None:
1302
1335
  """Report a creation that supplied no `QuarantineContext` (#496 S6 §4.2).
1303
1336
 
@@ -1350,8 +1383,7 @@ def _quarantine_db_family_strict(
1350
1383
  if state is None:
1351
1384
  ts = ts or _db_backup_timestamp()
1352
1385
  root = _cctally_core.APP_DIR / "quarantine"
1353
- incident = root / f"{db_path.name}-{ts}"
1354
- incident.mkdir(parents=True, exist_ok=True)
1386
+ incident = _reserve_quarantine_incident(root, db_path.name, ts)
1355
1387
  for directory in (root, incident):
1356
1388
  try:
1357
1389
  os.chmod(directory, 0o700)
@@ -1363,6 +1395,7 @@ def _quarantine_db_family_strict(
1363
1395
  members = [
1364
1396
  candidate.name
1365
1397
  for candidate in (
1398
+ pathlib.Path(f"{db_path}-journal"),
1366
1399
  pathlib.Path(f"{db_path}-wal"),
1367
1400
  pathlib.Path(f"{db_path}-shm"),
1368
1401
  db_path,
@@ -1448,7 +1481,7 @@ def quarantine_db_family(
1448
1481
  strict: bool = False,
1449
1482
  context: "QuarantineContext | None" = None,
1450
1483
  ) -> pathlib.Path:
1451
- """Move a damaged DB + its ``-wal``/``-shm`` sidecars into a single
1484
+ """Move a damaged DB and its SQLite journals into a single
1452
1485
  timestamped incident directory under ``quarantine/`` with a manifest (spec
1453
1486
  §6.3). NEVER deletes evidence — three renames under the caller's exclusion
1454
1487
  locks, not pretending to be one atomic op. ``0o700`` dir / ``0o600`` files.
@@ -1475,7 +1508,7 @@ def quarantine_db_family(
1475
1508
  except OSError:
1476
1509
  pass
1477
1510
  moved: list = []
1478
- for suffix in ("", "-wal", "-shm"):
1511
+ for suffix in ("", "-journal", "-wal", "-shm"):
1479
1512
  src = pathlib.Path(str(db_path) + suffix)
1480
1513
  if not src.exists():
1481
1514
  continue
@@ -9415,10 +9448,10 @@ def _release_repair_marker(path: pathlib.Path, claim: RepairMarkerClaim) -> None
9415
9448
 
9416
9449
 
9417
9450
  def _db_family_open_pids(path: pathlib.Path) -> "set[int] | None":
9418
- """Return processes with main/WAL/SHM open; None means unverifiable."""
9451
+ """Return processes with any SQLite family member open; None if unknown."""
9419
9452
  family = [
9420
9453
  pathlib.Path(str(path) + suffix)
9421
- for suffix in ("", "-wal", "-shm")
9454
+ for suffix in ("", "-journal", "-wal", "-shm")
9422
9455
  if pathlib.Path(str(path) + suffix).exists()
9423
9456
  ]
9424
9457
  if not family:
@@ -9478,7 +9511,7 @@ def _unique_sibling_path(path: pathlib.Path) -> pathlib.Path:
9478
9511
  def family_exists(candidate: pathlib.Path) -> bool:
9479
9512
  return any(
9480
9513
  pathlib.Path(str(candidate) + suffix).exists()
9481
- for suffix in ("", "-wal", "-shm")
9514
+ for suffix in ("", "-journal", "-wal", "-shm")
9482
9515
  )
9483
9516
 
9484
9517
  if not family_exists(path):
@@ -9510,9 +9543,9 @@ def _copy_db_family(
9510
9543
  source: pathlib.Path,
9511
9544
  destination: pathlib.Path,
9512
9545
  *,
9513
- suffixes: "tuple[str, ...]" = ("", "-wal", "-shm"),
9546
+ suffixes: "tuple[str, ...]" = ("", "-journal", "-wal", "-shm"),
9514
9547
  ) -> None:
9515
- """Copy main/WAL/SHM bytes while the caller holds SQLite's writer lock."""
9548
+ """Copy main and journal bytes while the caller owns the family."""
9516
9549
  for suffix in suffixes:
9517
9550
  src = pathlib.Path(str(source) + suffix)
9518
9551
  if not src.exists():
@@ -9523,6 +9556,53 @@ def _copy_db_family(
9523
9556
  _fsync_file(dst)
9524
9557
 
9525
9558
 
9559
+ def _reserve_post_drain_evidence_path(
9560
+ backup: pathlib.Path, suffix: str
9561
+ ) -> pathlib.Path:
9562
+ """Reserve a unique sibling without ever overwriting retained evidence."""
9563
+ for number in range(1, 10_000):
9564
+ infix = ".post-drain" if number == 1 else f".post-drain-{number}"
9565
+ candidate = pathlib.Path(f"{backup}{infix}{suffix}")
9566
+ try:
9567
+ fd = os.open(
9568
+ candidate,
9569
+ os.O_WRONLY | os.O_CREAT | os.O_EXCL,
9570
+ 0o600,
9571
+ )
9572
+ except FileExistsError:
9573
+ continue
9574
+ os.close(fd)
9575
+ return candidate
9576
+ raise OSError(f"could not allocate post-drain evidence beside {backup}")
9577
+
9578
+
9579
+ def _preserve_post_drain_sidecars(
9580
+ source: pathlib.Path, backup: pathlib.Path
9581
+ ) -> "tuple[pathlib.Path, ...]":
9582
+ """Move late sidecars aside without changing the classified backup."""
9583
+ moved = []
9584
+ for suffix in ("-journal", "-wal", "-shm"):
9585
+ member = pathlib.Path(str(source) + suffix)
9586
+ if not member.exists():
9587
+ continue
9588
+ destination = _reserve_post_drain_evidence_path(backup, suffix)
9589
+ try:
9590
+ # The O_EXCL reservation owns this pathname, so replace cannot
9591
+ # destroy pre-existing user or pre-checkpoint evidence.
9592
+ os.replace(member, destination)
9593
+ except OSError:
9594
+ try:
9595
+ os.unlink(destination)
9596
+ except OSError:
9597
+ pass
9598
+ raise
9599
+ os.chmod(destination, 0o600)
9600
+ _fsync_file(destination)
9601
+ _fsync_directory(destination.parent)
9602
+ moved.append(destination)
9603
+ return tuple(moved)
9604
+
9605
+
9526
9606
  #: The classification sidecar's own schema version, matching the incident
9527
9607
  #: `classification.json` the correlator writes.
9528
9608
  _BACKUP_CLASSIFICATION_SCHEMA_VERSION = 1
@@ -9554,7 +9634,7 @@ def _write_backup_classification_sidecar(
9554
9634
  safe direction.
9555
9635
  """
9556
9636
  members = []
9557
- for suffix in ("", "-wal", "-shm"):
9637
+ for suffix in ("", "-journal", "-wal", "-shm"):
9558
9638
  member = pathlib.Path(str(stem) + suffix)
9559
9639
  try:
9560
9640
  info = member.stat()
@@ -9680,7 +9760,23 @@ def _repair_preflight_and_copy(
9680
9760
  *,
9681
9761
  timeout_ms: int,
9682
9762
  ) -> "tuple[int, dict[str, int | None], int | None, sqlite3.Connection | None, str]":
9683
- """Preserve forensic bytes, drain WAL, and return a held writer guard."""
9763
+ """Preserve bytes, drain only proven-coherent legacy WAL, and hold writer."""
9764
+ import _lib_stats_wal
9765
+
9766
+ wal_evidence = _lib_stats_wal.inspect_wal_index_family(path)
9767
+ wal_verdict = wal_evidence.get("verdict")
9768
+ if _lib_stats_wal.is_incoherent_wal_index(wal_evidence):
9769
+ return (
9770
+ 3, {}, None, None,
9771
+ "incoherent legacy WAL family; refusing to open or checkpoint it",
9772
+ )
9773
+ if wal_verdict not in {"coherent", "wal_absent", "wal_empty"}:
9774
+ return (
9775
+ 3, {}, None, None,
9776
+ "could not prove the legacy WAL family coherent "
9777
+ f"({wal_verdict or 'unknown'}); refusing repair",
9778
+ )
9779
+ checkpoint_legacy_wal = wal_verdict == "coherent"
9684
9780
  conn = sqlite3.connect(
9685
9781
  f"file:{path}?mode=rw", uri=True, timeout=max(timeout_ms, 0) / 1000
9686
9782
  )
@@ -9717,20 +9813,23 @@ def _repair_preflight_and_copy(
9717
9813
  _write_backup_classification_sidecar(backup)
9718
9814
  conn.rollback()
9719
9815
 
9720
- try:
9721
- checkpoint = conn.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()
9722
- except sqlite3.DatabaseError as exc:
9723
- conn.close()
9724
- return 3, {}, None, None, f"WAL checkpoint failed: {exc}"
9725
- wal_path = pathlib.Path(str(path) + "-wal")
9726
- wal_bytes = wal_path.stat().st_size if wal_path.exists() else 0
9727
- if checkpoint is None or int(checkpoint[0]) != 0 or wal_bytes != 0:
9728
- conn.close()
9729
- return 3, {}, None, None, "WAL could not be fully checkpointed"
9816
+ if checkpoint_legacy_wal:
9817
+ try:
9818
+ checkpoint = conn.execute(
9819
+ "PRAGMA wal_checkpoint(TRUNCATE)"
9820
+ ).fetchone()
9821
+ except sqlite3.DatabaseError as exc:
9822
+ conn.close()
9823
+ return 3, {}, None, None, f"WAL checkpoint failed: {exc}"
9824
+ wal_path = pathlib.Path(str(path) + "-wal")
9825
+ wal_bytes = wal_path.stat().st_size if wal_path.exists() else 0
9826
+ if checkpoint is None or int(checkpoint[0]) != 0 or wal_bytes != 0:
9827
+ conn.close()
9828
+ return 3, {}, None, None, "WAL could not be fully checkpointed"
9730
9829
 
9731
- # Hold this one write exclusion continuously through .recover and the
9732
- # main-file replace. Since the WAL is empty, replacement failure leaves
9733
- # the old main file coherent and no committed frames can be lost.
9830
+ # Hold this write exclusion through source capture and `.recover`.
9831
+ # Final replacement releases it only to perform the mandatory second
9832
+ # whole-family drain under the repair marker + maintenance fence.
9734
9833
  try:
9735
9834
  conn.execute("BEGIN IMMEDIATE")
9736
9835
  except sqlite3.DatabaseError as exc:
@@ -10047,18 +10146,34 @@ def _cmd_db_repair_under_retention(
10047
10146
  try:
10048
10147
  os.chmod(recovered_path, 0o600)
10049
10148
  _fsync_file(recovered_path)
10050
- # WAL is already fully checkpointed and this same guard has blocked
10051
- # every writer since capture. Replace the coherent main file first;
10052
- # a failed replace therefore leaves the old coherent main + empty
10053
- # sidecars intact. New cctally opens remain blocked by the marker.
10054
- os.replace(recovered_path, path)
10055
- _fsync_directory(path.parent)
10149
+ # Release our own SQLite handle before the destructive-edge drain.
10150
+ # The repair marker and maintenance-exclusive hold still fence every
10151
+ # normal cctally opener. A second whole-family scan closes the gap
10152
+ # between the admission scan and this physical replacement.
10056
10153
  close_guard()
10057
- for suffix in ("-wal", "-shm"):
10058
- try:
10059
- pathlib.Path(str(path) + suffix).unlink()
10060
- except FileNotFoundError:
10061
- pass
10154
+ open_pids = _db_family_open_pids(path)
10155
+ if open_pids is None:
10156
+ eprint(
10157
+ "cctally: final stats.db repair declined: the second cold "
10158
+ "drain could not verify open handles. Nothing was moved."
10159
+ )
10160
+ return 3
10161
+ if open_pids:
10162
+ eprint(
10163
+ "cctally: final stats.db repair declined after a new reader "
10164
+ "appeared in process(es) "
10165
+ + ", ".join(str(pid) for pid in sorted(open_pids))
10166
+ + ". Stop the dashboard and older cctally processes, then "
10167
+ "retry; nothing was moved."
10168
+ )
10169
+ return 3
10170
+
10171
+ # The old family was copied, fsync'd, and classified before
10172
+ # recovery. Preserve any later survivors under unique post-drain
10173
+ # names; overwriting the exact pre-checkpoint members would destroy
10174
+ # evidence and make their inode/size classification stale.
10175
+ _preserve_post_drain_sidecars(path, backup)
10176
+ os.replace(recovered_path, path)
10062
10177
  _fsync_directory(path.parent)
10063
10178
  except OSError as exc:
10064
10179
  close_guard()
@@ -10128,11 +10243,11 @@ def cmd_db_backup(args: argparse.Namespace) -> int:
10128
10243
  prefix=f".{output.name}.tmp-", dir=output.parent
10129
10244
  ) as scratch_raw:
10130
10245
  temp_path = pathlib.Path(scratch_raw) / output.name
10131
- # #386: `db backup --db stats` holds this handle across the entire
10132
- # `source.backup(destination)` loop a long-lived READ TRANSACTION,
10133
- # which Stage 1 measured as the only thing that pins the stats WAL —
10134
- # so it participates in the replacement protocol. The `mode=ro` open
10135
- # is preserved verbatim through the `connect` seam.
10246
+ # #386/#538: `db backup --db stats` holds this handle across the
10247
+ # entire `source.backup(destination)` loop. In rollback mode that is
10248
+ # a long-lived reader, so it participates in the replacement
10249
+ # protocol. The `mode=ro` open is preserved verbatim through the
10250
+ # `connect` seam.
10136
10251
  def _backup_source_connect(_p, _timeout_ms=timeout_ms):
10137
10252
  return sqlite3.connect(
10138
10253
  f"file:{_p}?mode=ro", uri=True,
@@ -10183,8 +10298,7 @@ def cmd_db_backup(args: argparse.Namespace) -> int:
10183
10298
 
10184
10299
 
10185
10300
  def cmd_db_checkpoint(args: argparse.Namespace) -> int:
10186
- """Fast, non-destructive WAL drain (TRUNCATE checkpoint) for cache.db /
10187
- stats.db (#297).
10301
+ """Fast, non-destructive WAL drain (TRUNCATE checkpoint) for cache.db.
10188
10302
 
10189
10303
  Opens a RAW existing-file-only connection — NOT open_cache_db()/open_db(),
10190
10304
  which apply schema, run the migration dispatcher, can DELETE Codex rows on a
@@ -10200,17 +10314,29 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
10200
10314
  was not fully truncated through the timeout — an actionable "something is
10201
10315
  still holding it" signal.
10202
10316
  """
10203
- import _cctally_cache
10204
10317
  from _lib_json_envelope import stamp_schema_version
10205
10318
 
10206
10319
  which = args.db # "cache" | "stats"
10207
- if which == "cache":
10208
- path, label = _cctally_core.CACHE_DB_PATH, "cache.db"
10209
- else:
10210
- path, label = _cctally_core.DB_PATH, "stats.db"
10320
+ as_json = bool(getattr(args, "json", False))
10321
+ if which == "stats":
10322
+ reason = (
10323
+ "stats.db uses rollback journaling; no WAL checkpoint is applicable"
10324
+ )
10325
+ if as_json:
10326
+ print(json.dumps(stamp_schema_version({
10327
+ "db": "stats.db",
10328
+ "status": "notApplicable",
10329
+ "reason": reason,
10330
+ }, version=1)))
10331
+ else:
10332
+ eprint(f"cctally: {reason}.")
10333
+ return 2
10334
+
10335
+ import _cctally_cache
10336
+
10337
+ path, label = _cctally_core.CACHE_DB_PATH, "cache.db"
10211
10338
  timeout = int(getattr(args, "busy_timeout_ms", None)
10212
10339
  or _cctally_cache.CHECKPOINT_CMD_BUSY_TIMEOUT_MS)
10213
- as_json = bool(getattr(args, "json", False))
10214
10340
 
10215
10341
  # Absent file → nothing to drain; a missing re-derivable cache is not an
10216
10342
  # error (mirrors cmd_db_recover / cmd_db_unskip). Do NOT connect — mode=rw
@@ -10227,20 +10353,10 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
10227
10353
 
10228
10354
  from _lib_cache_writer_lock import acquire_ordered_flocks
10229
10355
 
10230
- # #386: the stats leg previously took NO advisory lock — the flock branch was
10231
- # gated `if which == "cache"` — while running a real
10232
- # `wal_checkpoint(TRUNCATE)` against the live family. Maintenance SHARED is
10233
- # the right strength: a checkpoint is not a physical replacement, it just
10234
- # must not overlap one.
10235
- lock_plan = {
10236
- "cache": [
10237
- (_cctally_core.CACHE_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
10238
- (_cctally_core.CACHE_LOCK_PATH, fcntl.LOCK_EX),
10239
- ],
10240
- "stats": [
10241
- (_cctally_core.STATS_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
10242
- ],
10243
- }[which]
10356
+ lock_plan = [
10357
+ (_cctally_core.CACHE_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
10358
+ (_cctally_core.CACHE_LOCK_PATH, fcntl.LOCK_EX),
10359
+ ]
10244
10360
 
10245
10361
  held: list[int] = []
10246
10362
  try:
@@ -10286,7 +10402,7 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
10286
10402
 
10287
10403
  # VACUUM writes a full fresh copy of the database into a temporary file and then
10288
10404
  # swaps it in, so it transiently needs roughly the DB's own size on top of the
10289
- # existing file, plus room for the drained WAL. A short busy_timeout keeps a
10405
+ # existing file, plus room for any WAL on WAL-backed stores. A short busy_timeout keeps a
10290
10406
  # contended VACUUM from hanging (F13).
10291
10407
  _VACUUM_BUSY_TIMEOUT_MS = 250
10292
10408
 
@@ -10313,7 +10429,7 @@ def _vacuum_required_free_bytes(path) -> int:
10313
10429
 
10314
10430
 
10315
10431
  def _run_vacuum_exclusive(path, label: str) -> int:
10316
- """Checkpoint + VACUUM ``path`` under a real SQLite EXCLUSIVE lock (F13).
10432
+ """VACUUM ``path`` under EXCLUSIVE; checkpoint WAL-backed stores first.
10317
10433
 
10318
10434
  ``locking_mode=EXCLUSIVE`` + a short ``busy_timeout`` make a concurrent
10319
10435
  reader/writer FAIL PROMPTLY (no TOCTOU gap — the exclusion is the DB's own
@@ -10326,7 +10442,8 @@ def _run_vacuum_exclusive(path, label: str) -> int:
10326
10442
  before = conn.execute("PRAGMA page_count").fetchone()[0]
10327
10443
  try:
10328
10444
  conn.execute("PRAGMA locking_mode=EXCLUSIVE")
10329
- conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
10445
+ if label != "stats.db":
10446
+ conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
10330
10447
  conn.execute("VACUUM")
10331
10448
  except sqlite3.OperationalError as exc:
10332
10449
  if "lock" in str(exc).lower() or "busy" in str(exc).lower():
@@ -10424,6 +10541,17 @@ def _vacuum_one_db(path, label: str, provider_locked: bool) -> int:
10424
10541
  )
10425
10542
  return 3
10426
10543
  held.append(fh)
10544
+ if stats_store:
10545
+ import _cctally_store
10546
+
10547
+ if _cctally_store._stats_legacy_wal_family_present(path):
10548
+ eprint(
10549
+ "cctally: stats.db VACUUM declined: the index still uses "
10550
+ "legacy WAL. Stop and restart older cctally processes, "
10551
+ "then run a normal command to complete the epoch rebuild "
10552
+ "before retrying VACUUM."
10553
+ )
10554
+ return 3
10427
10555
  return _run_vacuum_exclusive(path, label)
10428
10556
  finally:
10429
10557
  for fh in held: