cctally 1.94.1 → 1.95.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,33 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [1.95.0] - 2026-08-09
9
+
10
+ ### Changed
11
+ - The test gate refuses to report green while verifying less than it claims. `bin/cctally-test-all` now admits the harness estate from a committed manifest (`tests/authoritative-test-manifest.json`) instead of a two-name list, so a harness that lost its executable bit, went missing, or appeared unexpectedly fails the run instead of quietly leaving the bundle. It also reads each harness's exit status rather than only the counters it printed, requires an audited minimum case count per harness, asserts the runner's capabilities before starting, and refuses a run carrying a golden-regeneration variable (#529).
12
+ - A red run now says which kind of red it was. Every run emits a versioned outcome object whose `failureClass` is `product` (tests genuinely failed), `infrastructure` (the run could not start or the machinery broke), or `incomplete` (the run finished without verifying what it claimed), with every contributing reason retained. The exit band is `0` pass, `1` product failure, `2` usage error, `3` infrastructure or incomplete (#529).
13
+
14
+ ### Fixed
15
+ - `stats.db` now uses SQLite rollback journaling (`DELETE` / `FULL`) instead of WAL. The disposable index no longer depends on live `-wal`/`-shm` generations or checkpoints; corrupt-family replacement requires a cold second drain and preserves the complete main/rollback-journal family before publishing the rebuilt index. Existing WAL installs rebuild once at epoch 1010 and refuse with restart guidance while an older dashboard or process still holds the legacy family. `cctally db checkpoint --db stats` is retired with exit 2; cache and conversations remain WAL-backed (#538).
16
+ - Codex Guardian approval-review usage is now priced through its canonical `gpt-5.5` model instead of the generic `gpt-5` unknown-model fallback. Retained `codex-auto-review` rows no longer trigger a false pricing-coverage advisory, fast-tier reports use the canonical 2.5x multiplier, and genuinely unknown identifiers still warn and remain actionable (#535).
17
+ - Differently pinned worktrees can now share the default remote test workdir. Each agentmem revision has its own selected executable, so provisioning one tree no longer replaces or rejects the other tree's pin and takes the runner fleet out of service (#519).
18
+ - Codex share artifacts now disclose provider availability consistently across all nine dashboard panels. Current-week, trend, daily, monthly, weekly, and sessions exports were deriving availability only from their row count, so a real partial or empty provider state could leave both the rendered artifact and its drift indicator unchanged (#533).
19
+ - Corrupt `stats.db` recovery no longer starves behind continuous hook traffic. The first detector now elects one durable background owner before waiting for maintenance, other callers return promptly and coalesce onto it, and that owner captures forensics and rebuilds after readers drain instead of every caller timing out before recovery is admitted (#530).
20
+ - Escape now closes the share modal after you have clicked into the live preview. The preview is a sandboxed frame, so a key pressed inside it belonged to a different document and never reached the shortcut that closes the modal — the only way out was to tab back out of the frame first. It works from the composer's combined preview too, and it still closes one layer at a time: a section's actions first, then the composer, then the dialog beneath it (#503).
21
+ - The share preview frame is no longer announced to a screen reader as "Report preview (decorative)". It was fully exposed to assistive technology while its only name told you not to bother with it — on the one control the documentation tells you to review before sharing. It now names the panel and template it is showing (#503).
22
+ - Manage presets is now a real dialog. It claimed to be one but rendered as a plain block at the bottom of the share modal: Tab walked straight out of it into the template gallery behind, and closing it left focus nowhere, because it tried to return focus to a menu item that had already disappeared. It now covers the screen, keeps Tab inside itself, fits a phone without sideways scrolling, and returns focus to the presets button you opened it from (#503).
23
+ - Escape inside a composer section's actions menu no longer closes the whole composer and discards your unsaved title, theme and format edits. It closes just the menu and puts focus back on the button that opened it; press it again to close the composer. Opening one section's actions also closes any other section's, which previously could all be open at once. The menu no longer describes itself to screen readers as a menu it never behaved like — both actions remain reachable by Tab exactly as before (#503).
24
+ - Share checkboxes and radio buttons are 44 pixels tall in a narrow browser window, not only on a touch device. The touch sizing keyed on the pointer type rather than the width, so a narrow desktop window got none of it. The `Anon on export` label — the privacy control — was also rendering at 12 pixels on a phone despite a rule meant to set it to 16, which is the threshold below which iOS zooms the page when you tap a control (#503).
25
+ - The share preview on a phone now shows the report instead of its frontmatter. It was capped at 128 pixels, which left about four lines once the privacy notice had taken its share, and the source label was printed twice inside that space. The cap is now proportional to the visible screen, and it shrinks correctly when the on-screen keyboard is up rather than being pushed off the bottom (#503).
26
+ - Gallery text no longer appears to bleed through the pinned preview on a phone. Two separate causes: a 12-pixel band above the preview where scrolling content passed through a gap, and a bottom edge where the preview and the tile beneath it were the same colour with the same hairline border, so two half-cut lines of text read as one overlapping line. The preview now has a visible boundary and sits flush against the top of the scroll area (#503).
27
+ - Rotating a phone or resizing the window with the share modal open no longer resets the preview. The two layouts rendered separate previews, so crossing the breakpoint threw away the rendered report and its privacy answer, blanked the preview for about a second, and briefly showed the wrong statement about whether your export contains real project names. There is now one preview, positioned by layout at both sizes (#503).
28
+ - The keyboard shortcuts in the share documentation now name the gestures that work. Every passage that advertised a bare `S` or `B` has been rewritten; both shortcuts have always required Shift, and an unshifted `s` opens Settings instead. Several passages also told you to click a panel to focus it, which does not focus it — it opens the panel's own modal, which then blocks the shortcut — and two quoted a help message the application has never shown. The correct gesture is to Tab to one of the panel's actions and press Shift+S, which appeared nowhere. The share icon's own tooltip said `Share (S)` and now says `Share (Shift+S)` (#503).
29
+
30
+ ## [1.94.2] - 2026-08-08
31
+
32
+ ### Fixed
33
+ - Release validation's segment-elision oracle no longer flakes when two equivalent rebuilds cross a one-second boundary: it still compares the complete migration set, but ignores each rebuild's wall-clock migration timestamp.
34
+
8
35
  ## [1.94.1] - 2026-08-08
9
36
 
10
37
  ### Fixed
@@ -321,14 +321,6 @@ OAUTH_BACKOFF_CAP_SECONDS = 3600.0
321
321
  _init_paths_from_env()
322
322
 
323
323
 
324
- # stats.db WAL cap (#297). Bounds the persistent WAL file so a resetting
325
- # checkpoint truncates it back down instead of leaving it at its high-water
326
- # size. stats.db writes are small (its -wal was observed at 0 bytes even under
327
- # the contention that bloated cache.db to multi-GB), so a tighter 16 MB cap is
328
- # ample. See docs/superpowers/specs/2026-07-13-cache-db-wal-hardening-design.md.
329
- STATS_WAL_SIZE_LIMIT_BYTES = 16 * 1024 * 1024 # 16777216
330
-
331
-
332
324
  # === stats.db epoch-rebuild versioning (DB journal redesign §7.1/§8) ==
333
325
  #
334
326
  # stats.db is no longer a versioned migration target — it is a DISPOSABLE index
@@ -393,7 +385,12 @@ STATS_WAL_SIZE_LIMIT_BYTES = 16 * 1024 * 1024 # 16777216
393
385
  # before any schema work, so an `add_column_if_missing` would never run on an
394
386
  # upgraded install and the column would simply never appear. The registry stays
395
387
  # frozen at 13 and an epoch mismatch resolves by rebuild.
396
- STATS_INDEX_EPOCH = 1009
388
+ # 1009 -> 1010 (#538 Task A): retire WAL for the disposable stats index. Every
389
+ # stats connection now uses DELETE/FULL rollback journaling, structurally
390
+ # removing the shared-memory WAL page map implicated by the retained corruption
391
+ # bundles. The one-time rebuild is the mode transition; no stats migration is
392
+ # added and cache/conversations remain WAL/NORMAL.
393
+ STATS_INDEX_EPOCH = 1010
397
394
  LEGACY_STATS_HEAD = 13
398
395
 
399
396
  #: #496 S1 F1. A NEW branch, for a state that cannot occur before the
@@ -773,8 +770,8 @@ def ensure_dirs() -> None:
773
770
  # bin/_cctally_store.py _heal_flock_blocking, reached through
774
771
  # _acquire_stats_maintenance_reentrant by the epoch
775
772
  # resolver
776
- # bin/_cctally_store.py _acquire_stats_maintenance_for_heal, the corruption
777
- # heal's ownership-first BOUNDED acquire (#496 S3)
773
+ # bin/_cctally_store.py detached corruption-heal worker maintenance acquire
774
+ # (the detector itself never acquires it; #530)
778
775
  # bin/_cctally_db.py cmd_db_rebuild, _acquire_db_admin_writer_flocks
779
776
  # (db skip / db unskip), _cmd_db_repair_exclusive,
780
777
  # _vacuum_one_db
@@ -1647,11 +1644,12 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
1647
1644
  # #453: probe the live index before the maintenance-shared opener. During
1648
1645
  # a large detached rebuild that opener can wait for its bounded 5-second
1649
1646
  # guard; a statusline arriving mid-replay would otherwise pay that delay
1650
- # (and can reach more than one stats opener in a single render). The raw
1651
- # probe is read-only and deliberately recognizes only a readable,
1652
- # post-legacy wrong epoch. Missing/unreadable/legacy indexes retain their
1653
- # established guarded-open paths. The post-open epoch gate below remains a
1654
- # race-closing defense if the version changes after this probe.
1647
+ # (and can reach more than one stats opener in a single render). The probe
1648
+ # reads only the fixed user_version field in the main-file header; it never
1649
+ # opens SQLite or bypasses the maintenance opener fence. Missing, invalid-
1650
+ # header and legacy indexes retain their established guarded-open paths.
1651
+ # The post-open epoch gate below remains a race-closing defense if the
1652
+ # version changes after this probe.
1655
1653
  if (
1656
1654
  _target_path is None
1657
1655
  and _cctally_store.stats_epoch_enabled()
@@ -1682,18 +1680,16 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
1682
1680
  # (corruption that only surfaces mid-DDL stays a raw error; the probe
1683
1681
  # catches the common case).
1684
1682
  try:
1685
- # §6.1 PRAGMA policy via the shared table (stats: WAL / NORMAL /
1683
+ # §6.1 PRAGMA policy via the shared table (stats: DELETE / FULL /
1686
1684
  # busy_timeout 15000 / journal_size_limit 16 MiB / auto_vacuum unset).
1687
1685
  # #297: busy_timeout 15000 lets a writer wait out a slow-but-normal sync
1688
- # (>5 s) instead of erroring "database is locked"; it does NOT absorb
1689
- # SQLITE_BUSY_SNAPSHOT (defended by BEGIN IMMEDIATE / write-first at the
1690
- # write sites, cctally-dev#87). journal_size_limit bounds the persistent
1691
- # WAL. The corruption probe (SELECT 1) stays here inside the narrow
1686
+ # (>5 s) instead of erroring "database is locked". The corruption probe
1687
+ # (SELECT 1) stays here inside the narrow
1692
1688
  # StatsDbCorruptError boundary.
1693
1689
  _cctally_store.apply_policy(conn, "stats")
1694
1690
  conn.execute("SELECT 1").fetchone()
1695
- # §9.2 (#496 S6 F23). AFTER `apply_policy`, because that is what sets
1696
- # journal_mode=WAL and so what can materialize the sidecars.
1691
+ # §9.2 (#496 S6 F23). AFTER `apply_policy`, so any rollback journal
1692
+ # materialized during the transition is included in family hardening.
1697
1693
  _cctally_store._harden_stats_family(db_path)
1698
1694
  except sqlite3.DatabaseError as exc:
1699
1695
  try:
@@ -1714,7 +1710,15 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
1714
1710
  # it — all of which fall through to the original guided
1715
1711
  # StatsDbCorruptError so the manual path still applies.
1716
1712
  heal = getattr(_cctally_store, "HEAL_HOOK", None)
1717
- if _target_path is None and heal is not None and heal("stats", exc):
1713
+ heal_enabled = (
1714
+ os.environ.get("CCTALLY_TEST_DISABLE_STATS_AUTO_HEAL") != "1"
1715
+ )
1716
+ if (
1717
+ _target_path is None
1718
+ and heal is not None
1719
+ and heal_enabled
1720
+ and heal("stats", exc)
1721
+ ):
1718
1722
  # #386: the post-heal retry is an opener too. The heal released the
1719
1723
  # maintenance lock before returning, so another maintenance path can
1720
1724
  # legitimately own the family by now.
@@ -1763,9 +1763,15 @@ def _build_codex_source_share_snapshot(ls, *, state, panel: str,
1763
1763
  display_tz_label=display_tz,
1764
1764
  )
1765
1765
  codex_module = sys.modules["cctally"]._load_sibling("_cctally_codex")
1766
+ # The canonical CLI adapter derives availability from its row count, but
1767
+ # this source-aware path has an authoritative provider state. Project the
1768
+ # normalized state after adapting the rows so all nine panels carry the
1769
+ # same source chrome and built-snapshot digest signal (#533).
1766
1770
  return replace(
1767
1771
  codex_module._build_codex_share_snapshot(command, view, normalized_rows),
1768
1772
  template_id=template_id,
1773
+ availability=availability,
1774
+ availability_reason=reason,
1769
1775
  )
1770
1776
 
1771
1777
 
@@ -373,10 +373,11 @@ class StatsHealDeferred(StatsRebuildDeferred):
373
373
  quarantine and rebuild while the caller waited (#496 S3 §6). The caller
374
374
  degrades where it already degrades; it never blocks on the rebuild.
375
375
 
376
- ``heal_id`` correlates this signal with the durable heal event the hook
377
- recorded at detection, and ``forensics_path`` is the absolute bundle the
378
- user was told about. Both are attributes rather than message text so a
379
- consumer can use them without parsing.
376
+ ``heal_id`` correlates this signal with the durable admitted occurrence.
377
+ ``forensics_path`` is optional: since #530 the fresh worker captures and
378
+ persists that bundle after maintenance drains, so the detector normally
379
+ raises before a path exists. Both remain attributes rather than parsed
380
+ message text.
380
381
  """
381
382
 
382
383
  def __init__(
@@ -1042,7 +1043,7 @@ def write_corruption_forensics(
1042
1043
  "truncated": False,
1043
1044
  },
1044
1045
  }
1045
- for suffix in ("", "-wal", "-shm"):
1046
+ for suffix in ("", "-journal", "-wal", "-shm"):
1046
1047
  p = pathlib.Path(str(db_path) + suffix)
1047
1048
  try:
1048
1049
  st = p.stat()
@@ -1276,6 +1277,7 @@ def _load_pending_quarantine(db_path: pathlib.Path) -> "dict[str, Any] | None":
1276
1277
  isinstance(name, str)
1277
1278
  and name in {
1278
1279
  db_path.name,
1280
+ f"{db_path.name}-journal",
1279
1281
  f"{db_path.name}-wal",
1280
1282
  f"{db_path.name}-shm",
1281
1283
  }
@@ -1363,6 +1365,7 @@ def _quarantine_db_family_strict(
1363
1365
  members = [
1364
1366
  candidate.name
1365
1367
  for candidate in (
1368
+ pathlib.Path(f"{db_path}-journal"),
1366
1369
  pathlib.Path(f"{db_path}-wal"),
1367
1370
  pathlib.Path(f"{db_path}-shm"),
1368
1371
  db_path,
@@ -1448,7 +1451,7 @@ def quarantine_db_family(
1448
1451
  strict: bool = False,
1449
1452
  context: "QuarantineContext | None" = None,
1450
1453
  ) -> pathlib.Path:
1451
- """Move a damaged DB + its ``-wal``/``-shm`` sidecars into a single
1454
+ """Move a damaged DB and its SQLite journals into a single
1452
1455
  timestamped incident directory under ``quarantine/`` with a manifest (spec
1453
1456
  §6.3). NEVER deletes evidence — three renames under the caller's exclusion
1454
1457
  locks, not pretending to be one atomic op. ``0o700`` dir / ``0o600`` files.
@@ -1475,7 +1478,7 @@ def quarantine_db_family(
1475
1478
  except OSError:
1476
1479
  pass
1477
1480
  moved: list = []
1478
- for suffix in ("", "-wal", "-shm"):
1481
+ for suffix in ("", "-journal", "-wal", "-shm"):
1479
1482
  src = pathlib.Path(str(db_path) + suffix)
1480
1483
  if not src.exists():
1481
1484
  continue
@@ -9415,10 +9418,10 @@ def _release_repair_marker(path: pathlib.Path, claim: RepairMarkerClaim) -> None
9415
9418
 
9416
9419
 
9417
9420
  def _db_family_open_pids(path: pathlib.Path) -> "set[int] | None":
9418
- """Return processes with main/WAL/SHM open; None means unverifiable."""
9421
+ """Return processes with any SQLite family member open; None if unknown."""
9419
9422
  family = [
9420
9423
  pathlib.Path(str(path) + suffix)
9421
- for suffix in ("", "-wal", "-shm")
9424
+ for suffix in ("", "-journal", "-wal", "-shm")
9422
9425
  if pathlib.Path(str(path) + suffix).exists()
9423
9426
  ]
9424
9427
  if not family:
@@ -9478,7 +9481,7 @@ def _unique_sibling_path(path: pathlib.Path) -> pathlib.Path:
9478
9481
  def family_exists(candidate: pathlib.Path) -> bool:
9479
9482
  return any(
9480
9483
  pathlib.Path(str(candidate) + suffix).exists()
9481
- for suffix in ("", "-wal", "-shm")
9484
+ for suffix in ("", "-journal", "-wal", "-shm")
9482
9485
  )
9483
9486
 
9484
9487
  if not family_exists(path):
@@ -9510,9 +9513,9 @@ def _copy_db_family(
9510
9513
  source: pathlib.Path,
9511
9514
  destination: pathlib.Path,
9512
9515
  *,
9513
- suffixes: "tuple[str, ...]" = ("", "-wal", "-shm"),
9516
+ suffixes: "tuple[str, ...]" = ("", "-journal", "-wal", "-shm"),
9514
9517
  ) -> None:
9515
- """Copy main/WAL/SHM bytes while the caller holds SQLite's writer lock."""
9518
+ """Copy main and journal bytes while the caller owns the family."""
9516
9519
  for suffix in suffixes:
9517
9520
  src = pathlib.Path(str(source) + suffix)
9518
9521
  if not src.exists():
@@ -9523,6 +9526,53 @@ def _copy_db_family(
9523
9526
  _fsync_file(dst)
9524
9527
 
9525
9528
 
9529
+ def _reserve_post_drain_evidence_path(
9530
+ backup: pathlib.Path, suffix: str
9531
+ ) -> pathlib.Path:
9532
+ """Reserve a unique sibling without ever overwriting retained evidence."""
9533
+ for number in range(1, 10_000):
9534
+ infix = ".post-drain" if number == 1 else f".post-drain-{number}"
9535
+ candidate = pathlib.Path(f"{backup}{infix}{suffix}")
9536
+ try:
9537
+ fd = os.open(
9538
+ candidate,
9539
+ os.O_WRONLY | os.O_CREAT | os.O_EXCL,
9540
+ 0o600,
9541
+ )
9542
+ except FileExistsError:
9543
+ continue
9544
+ os.close(fd)
9545
+ return candidate
9546
+ raise OSError(f"could not allocate post-drain evidence beside {backup}")
9547
+
9548
+
9549
+ def _preserve_post_drain_sidecars(
9550
+ source: pathlib.Path, backup: pathlib.Path
9551
+ ) -> "tuple[pathlib.Path, ...]":
9552
+ """Move late sidecars aside without changing the classified backup."""
9553
+ moved = []
9554
+ for suffix in ("-journal", "-wal", "-shm"):
9555
+ member = pathlib.Path(str(source) + suffix)
9556
+ if not member.exists():
9557
+ continue
9558
+ destination = _reserve_post_drain_evidence_path(backup, suffix)
9559
+ try:
9560
+ # The O_EXCL reservation owns this pathname, so replace cannot
9561
+ # destroy pre-existing user or pre-checkpoint evidence.
9562
+ os.replace(member, destination)
9563
+ except OSError:
9564
+ try:
9565
+ os.unlink(destination)
9566
+ except OSError:
9567
+ pass
9568
+ raise
9569
+ os.chmod(destination, 0o600)
9570
+ _fsync_file(destination)
9571
+ _fsync_directory(destination.parent)
9572
+ moved.append(destination)
9573
+ return tuple(moved)
9574
+
9575
+
9526
9576
  #: The classification sidecar's own schema version, matching the incident
9527
9577
  #: `classification.json` the correlator writes.
9528
9578
  _BACKUP_CLASSIFICATION_SCHEMA_VERSION = 1
@@ -9554,7 +9604,7 @@ def _write_backup_classification_sidecar(
9554
9604
  safe direction.
9555
9605
  """
9556
9606
  members = []
9557
- for suffix in ("", "-wal", "-shm"):
9607
+ for suffix in ("", "-journal", "-wal", "-shm"):
9558
9608
  member = pathlib.Path(str(stem) + suffix)
9559
9609
  try:
9560
9610
  info = member.stat()
@@ -9680,7 +9730,23 @@ def _repair_preflight_and_copy(
9680
9730
  *,
9681
9731
  timeout_ms: int,
9682
9732
  ) -> "tuple[int, dict[str, int | None], int | None, sqlite3.Connection | None, str]":
9683
- """Preserve forensic bytes, drain WAL, and return a held writer guard."""
9733
+ """Preserve bytes, drain only proven-coherent legacy WAL, and hold writer."""
9734
+ import _lib_stats_wal
9735
+
9736
+ wal_evidence = _lib_stats_wal.inspect_wal_index_family(path)
9737
+ wal_verdict = wal_evidence.get("verdict")
9738
+ if _lib_stats_wal.is_incoherent_wal_index(wal_evidence):
9739
+ return (
9740
+ 3, {}, None, None,
9741
+ "incoherent legacy WAL family; refusing to open or checkpoint it",
9742
+ )
9743
+ if wal_verdict not in {"coherent", "wal_absent", "wal_empty"}:
9744
+ return (
9745
+ 3, {}, None, None,
9746
+ "could not prove the legacy WAL family coherent "
9747
+ f"({wal_verdict or 'unknown'}); refusing repair",
9748
+ )
9749
+ checkpoint_legacy_wal = wal_verdict == "coherent"
9684
9750
  conn = sqlite3.connect(
9685
9751
  f"file:{path}?mode=rw", uri=True, timeout=max(timeout_ms, 0) / 1000
9686
9752
  )
@@ -9717,20 +9783,23 @@ def _repair_preflight_and_copy(
9717
9783
  _write_backup_classification_sidecar(backup)
9718
9784
  conn.rollback()
9719
9785
 
9720
- try:
9721
- checkpoint = conn.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()
9722
- except sqlite3.DatabaseError as exc:
9723
- conn.close()
9724
- return 3, {}, None, None, f"WAL checkpoint failed: {exc}"
9725
- wal_path = pathlib.Path(str(path) + "-wal")
9726
- wal_bytes = wal_path.stat().st_size if wal_path.exists() else 0
9727
- if checkpoint is None or int(checkpoint[0]) != 0 or wal_bytes != 0:
9728
- conn.close()
9729
- return 3, {}, None, None, "WAL could not be fully checkpointed"
9786
+ if checkpoint_legacy_wal:
9787
+ try:
9788
+ checkpoint = conn.execute(
9789
+ "PRAGMA wal_checkpoint(TRUNCATE)"
9790
+ ).fetchone()
9791
+ except sqlite3.DatabaseError as exc:
9792
+ conn.close()
9793
+ return 3, {}, None, None, f"WAL checkpoint failed: {exc}"
9794
+ wal_path = pathlib.Path(str(path) + "-wal")
9795
+ wal_bytes = wal_path.stat().st_size if wal_path.exists() else 0
9796
+ if checkpoint is None or int(checkpoint[0]) != 0 or wal_bytes != 0:
9797
+ conn.close()
9798
+ return 3, {}, None, None, "WAL could not be fully checkpointed"
9730
9799
 
9731
- # Hold this one write exclusion continuously through .recover and the
9732
- # main-file replace. Since the WAL is empty, replacement failure leaves
9733
- # the old main file coherent and no committed frames can be lost.
9800
+ # Hold this write exclusion through source capture and `.recover`.
9801
+ # Final replacement releases it only to perform the mandatory second
9802
+ # whole-family drain under the repair marker + maintenance fence.
9734
9803
  try:
9735
9804
  conn.execute("BEGIN IMMEDIATE")
9736
9805
  except sqlite3.DatabaseError as exc:
@@ -10047,18 +10116,34 @@ def _cmd_db_repair_under_retention(
10047
10116
  try:
10048
10117
  os.chmod(recovered_path, 0o600)
10049
10118
  _fsync_file(recovered_path)
10050
- # WAL is already fully checkpointed and this same guard has blocked
10051
- # every writer since capture. Replace the coherent main file first;
10052
- # a failed replace therefore leaves the old coherent main + empty
10053
- # sidecars intact. New cctally opens remain blocked by the marker.
10054
- os.replace(recovered_path, path)
10055
- _fsync_directory(path.parent)
10119
+ # Release our own SQLite handle before the destructive-edge drain.
10120
+ # The repair marker and maintenance-exclusive hold still fence every
10121
+ # normal cctally opener. A second whole-family scan closes the gap
10122
+ # between the admission scan and this physical replacement.
10056
10123
  close_guard()
10057
- for suffix in ("-wal", "-shm"):
10058
- try:
10059
- pathlib.Path(str(path) + suffix).unlink()
10060
- except FileNotFoundError:
10061
- pass
10124
+ open_pids = _db_family_open_pids(path)
10125
+ if open_pids is None:
10126
+ eprint(
10127
+ "cctally: final stats.db repair declined: the second cold "
10128
+ "drain could not verify open handles. Nothing was moved."
10129
+ )
10130
+ return 3
10131
+ if open_pids:
10132
+ eprint(
10133
+ "cctally: final stats.db repair declined after a new reader "
10134
+ "appeared in process(es) "
10135
+ + ", ".join(str(pid) for pid in sorted(open_pids))
10136
+ + ". Stop the dashboard and older cctally processes, then "
10137
+ "retry; nothing was moved."
10138
+ )
10139
+ return 3
10140
+
10141
+ # The old family was copied, fsync'd, and classified before
10142
+ # recovery. Preserve any later survivors under unique post-drain
10143
+ # names; overwriting the exact pre-checkpoint members would destroy
10144
+ # evidence and make their inode/size classification stale.
10145
+ _preserve_post_drain_sidecars(path, backup)
10146
+ os.replace(recovered_path, path)
10062
10147
  _fsync_directory(path.parent)
10063
10148
  except OSError as exc:
10064
10149
  close_guard()
@@ -10128,11 +10213,11 @@ def cmd_db_backup(args: argparse.Namespace) -> int:
10128
10213
  prefix=f".{output.name}.tmp-", dir=output.parent
10129
10214
  ) as scratch_raw:
10130
10215
  temp_path = pathlib.Path(scratch_raw) / output.name
10131
- # #386: `db backup --db stats` holds this handle across the entire
10132
- # `source.backup(destination)` loop a long-lived READ TRANSACTION,
10133
- # which Stage 1 measured as the only thing that pins the stats WAL —
10134
- # so it participates in the replacement protocol. The `mode=ro` open
10135
- # is preserved verbatim through the `connect` seam.
10216
+ # #386/#538: `db backup --db stats` holds this handle across the
10217
+ # entire `source.backup(destination)` loop. In rollback mode that is
10218
+ # a long-lived reader, so it participates in the replacement
10219
+ # protocol. The `mode=ro` open is preserved verbatim through the
10220
+ # `connect` seam.
10136
10221
  def _backup_source_connect(_p, _timeout_ms=timeout_ms):
10137
10222
  return sqlite3.connect(
10138
10223
  f"file:{_p}?mode=ro", uri=True,
@@ -10183,8 +10268,7 @@ def cmd_db_backup(args: argparse.Namespace) -> int:
10183
10268
 
10184
10269
 
10185
10270
  def cmd_db_checkpoint(args: argparse.Namespace) -> int:
10186
- """Fast, non-destructive WAL drain (TRUNCATE checkpoint) for cache.db /
10187
- stats.db (#297).
10271
+ """Fast, non-destructive WAL drain (TRUNCATE checkpoint) for cache.db.
10188
10272
 
10189
10273
  Opens a RAW existing-file-only connection — NOT open_cache_db()/open_db(),
10190
10274
  which apply schema, run the migration dispatcher, can DELETE Codex rows on a
@@ -10200,17 +10284,29 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
10200
10284
  was not fully truncated through the timeout — an actionable "something is
10201
10285
  still holding it" signal.
10202
10286
  """
10203
- import _cctally_cache
10204
10287
  from _lib_json_envelope import stamp_schema_version
10205
10288
 
10206
10289
  which = args.db # "cache" | "stats"
10207
- if which == "cache":
10208
- path, label = _cctally_core.CACHE_DB_PATH, "cache.db"
10209
- else:
10210
- path, label = _cctally_core.DB_PATH, "stats.db"
10290
+ as_json = bool(getattr(args, "json", False))
10291
+ if which == "stats":
10292
+ reason = (
10293
+ "stats.db uses rollback journaling; no WAL checkpoint is applicable"
10294
+ )
10295
+ if as_json:
10296
+ print(json.dumps(stamp_schema_version({
10297
+ "db": "stats.db",
10298
+ "status": "notApplicable",
10299
+ "reason": reason,
10300
+ }, version=1)))
10301
+ else:
10302
+ eprint(f"cctally: {reason}.")
10303
+ return 2
10304
+
10305
+ import _cctally_cache
10306
+
10307
+ path, label = _cctally_core.CACHE_DB_PATH, "cache.db"
10211
10308
  timeout = int(getattr(args, "busy_timeout_ms", None)
10212
10309
  or _cctally_cache.CHECKPOINT_CMD_BUSY_TIMEOUT_MS)
10213
- as_json = bool(getattr(args, "json", False))
10214
10310
 
10215
10311
  # Absent file → nothing to drain; a missing re-derivable cache is not an
10216
10312
  # error (mirrors cmd_db_recover / cmd_db_unskip). Do NOT connect — mode=rw
@@ -10227,20 +10323,10 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
10227
10323
 
10228
10324
  from _lib_cache_writer_lock import acquire_ordered_flocks
10229
10325
 
10230
- # #386: the stats leg previously took NO advisory lock — the flock branch was
10231
- # gated `if which == "cache"` — while running a real
10232
- # `wal_checkpoint(TRUNCATE)` against the live family. Maintenance SHARED is
10233
- # the right strength: a checkpoint is not a physical replacement, it just
10234
- # must not overlap one.
10235
- lock_plan = {
10236
- "cache": [
10237
- (_cctally_core.CACHE_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
10238
- (_cctally_core.CACHE_LOCK_PATH, fcntl.LOCK_EX),
10239
- ],
10240
- "stats": [
10241
- (_cctally_core.STATS_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
10242
- ],
10243
- }[which]
10326
+ lock_plan = [
10327
+ (_cctally_core.CACHE_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
10328
+ (_cctally_core.CACHE_LOCK_PATH, fcntl.LOCK_EX),
10329
+ ]
10244
10330
 
10245
10331
  held: list[int] = []
10246
10332
  try:
@@ -10286,7 +10372,7 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
10286
10372
 
10287
10373
  # VACUUM writes a full fresh copy of the database into a temporary file and then
10288
10374
  # swaps it in, so it transiently needs roughly the DB's own size on top of the
10289
- # existing file, plus room for the drained WAL. A short busy_timeout keeps a
10375
+ # existing file, plus room for any WAL on WAL-backed stores. A short busy_timeout keeps a
10290
10376
  # contended VACUUM from hanging (F13).
10291
10377
  _VACUUM_BUSY_TIMEOUT_MS = 250
10292
10378
 
@@ -10313,7 +10399,7 @@ def _vacuum_required_free_bytes(path) -> int:
10313
10399
 
10314
10400
 
10315
10401
  def _run_vacuum_exclusive(path, label: str) -> int:
10316
- """Checkpoint + VACUUM ``path`` under a real SQLite EXCLUSIVE lock (F13).
10402
+ """VACUUM ``path`` under EXCLUSIVE; checkpoint WAL-backed stores first.
10317
10403
 
10318
10404
  ``locking_mode=EXCLUSIVE`` + a short ``busy_timeout`` make a concurrent
10319
10405
  reader/writer FAIL PROMPTLY (no TOCTOU gap — the exclusion is the DB's own
@@ -10326,7 +10412,8 @@ def _run_vacuum_exclusive(path, label: str) -> int:
10326
10412
  before = conn.execute("PRAGMA page_count").fetchone()[0]
10327
10413
  try:
10328
10414
  conn.execute("PRAGMA locking_mode=EXCLUSIVE")
10329
- conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
10415
+ if label != "stats.db":
10416
+ conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
10330
10417
  conn.execute("VACUUM")
10331
10418
  except sqlite3.OperationalError as exc:
10332
10419
  if "lock" in str(exc).lower() or "busy" in str(exc).lower():
@@ -10424,6 +10511,17 @@ def _vacuum_one_db(path, label: str, provider_locked: bool) -> int:
10424
10511
  )
10425
10512
  return 3
10426
10513
  held.append(fh)
10514
+ if stats_store:
10515
+ import _cctally_store
10516
+
10517
+ if _cctally_store._stats_legacy_wal_family_present(path):
10518
+ eprint(
10519
+ "cctally: stats.db VACUUM declined: the index still uses "
10520
+ "legacy WAL. Stop and restart older cctally processes, "
10521
+ "then run a normal command to complete the epoch rebuild "
10522
+ "before retrying VACUUM."
10523
+ )
10524
+ return 3
10427
10525
  return _run_vacuum_exclusive(path, label)
10428
10526
  finally:
10429
10527
  for fh in held: