cctally 1.94.1 → 1.95.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -0
- package/bin/_cctally_core.py +28 -24
- package/bin/_cctally_dashboard_share.py +6 -0
- package/bin/_cctally_db.py +166 -68
- package/bin/_cctally_journal.py +164 -288
- package/bin/_cctally_store.py +460 -292
- package/bin/_lib_doctor.py +2 -2
- package/bin/_lib_pricing.py +26 -7
- package/dashboard/static/assets/index-BvCbpJJA.css +1 -0
- package/dashboard/static/assets/{index-CaUQziq_.js → index-CRZMxeYI.js} +55 -55
- package/dashboard/static/dashboard.html +2 -2
- package/package.json +1 -1
- package/dashboard/static/assets/index-DUQjFSX7.css +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,33 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [1.95.0] - 2026-08-09
|
|
9
|
+
|
|
10
|
+
### Changed
|
|
11
|
+
- The test gate refuses to report green while verifying less than it claims. `bin/cctally-test-all` now admits the harness estate from a committed manifest (`tests/authoritative-test-manifest.json`) instead of a two-name list, so a harness that lost its executable bit, went missing, or appeared unexpectedly fails the run instead of quietly leaving the bundle. It also reads each harness's exit status rather than only the counters it printed, requires an audited minimum case count per harness, asserts the runner's capabilities before starting, and refuses a run carrying a golden-regeneration variable (#529).
|
|
12
|
+
- A red run now says which kind of red it was. Every run emits a versioned outcome object whose `failureClass` is `product` (tests genuinely failed), `infrastructure` (the run could not start or the machinery broke), or `incomplete` (the run finished without verifying what it claimed), with every contributing reason retained. The exit band is `0` pass, `1` product failure, `2` usage error, `3` infrastructure or incomplete (#529).
|
|
13
|
+
|
|
14
|
+
### Fixed
|
|
15
|
+
- `stats.db` now uses SQLite rollback journaling (`DELETE` / `FULL`) instead of WAL. The disposable index no longer depends on live `-wal`/`-shm` generations or checkpoints; corrupt-family replacement requires a cold second drain and preserves the complete main/rollback-journal family before publishing the rebuilt index. Existing WAL installs rebuild once at epoch 1010 and refuse with restart guidance while an older dashboard or process still holds the legacy family. `cctally db checkpoint --db stats` is retired with exit 2; cache and conversations remain WAL-backed (#538).
|
|
16
|
+
- Codex Guardian approval-review usage is now priced through its canonical `gpt-5.5` model instead of the generic `gpt-5` unknown-model fallback. Retained `codex-auto-review` rows no longer trigger a false pricing-coverage advisory, fast-tier reports use the canonical 2.5x multiplier, and genuinely unknown identifiers still warn and remain actionable (#535).
|
|
17
|
+
- Differently pinned worktrees can now share the default remote test workdir. Each agentmem revision has its own selected executable, so provisioning one tree no longer replaces or rejects the other tree's pin and takes the runner fleet out of service (#519).
|
|
18
|
+
- Codex share artifacts now disclose provider availability consistently across all nine dashboard panels. Current-week, trend, daily, monthly, weekly, and sessions exports were deriving availability only from their row count, so a real partial or empty provider state could leave both the rendered artifact and its drift indicator unchanged (#533).
|
|
19
|
+
- Corrupt `stats.db` recovery no longer starves behind continuous hook traffic. The first detector now elects one durable background owner before waiting for maintenance, other callers return promptly and coalesce onto it, and that owner captures forensics and rebuilds after readers drain instead of every caller timing out before recovery is admitted (#530).
|
|
20
|
+
- Escape now closes the share modal after you have clicked into the live preview. The preview is a sandboxed frame, so a key pressed inside it belonged to a different document and never reached the shortcut that closes the modal — the only way out was to tab back out of the frame first. It works from the composer's combined preview too, and it still closes one layer at a time: a section's actions first, then the composer, then the dialog beneath it (#503).
|
|
21
|
+
- The share preview frame is no longer announced to a screen reader as "Report preview (decorative)". It was fully exposed to assistive technology while its only name told you not to bother with it — on the one control the documentation tells you to review before sharing. It now names the panel and template it is showing (#503).
|
|
22
|
+
- Manage presets is now a real dialog. It claimed to be one but rendered as a plain block at the bottom of the share modal: Tab walked straight out of it into the template gallery behind, and closing it left focus nowhere, because it tried to return focus to a menu item that had already disappeared. It now covers the screen, keeps Tab inside itself, fits a phone without sideways scrolling, and returns focus to the presets button you opened it from (#503).
|
|
23
|
+
- Escape inside a composer section's actions menu no longer closes the whole composer and discards your unsaved title, theme and format edits. It closes just the menu and puts focus back on the button that opened it; press it again to close the composer. Opening one section's actions also closes any other section's, which previously could all be open at once. The menu no longer describes itself to screen readers as a menu it never behaved like — both actions remain reachable by Tab exactly as before (#503).
|
|
24
|
+
- Share checkboxes and radio buttons are 44 pixels tall in a narrow browser window, not only on a touch device. The touch sizing keyed on the pointer type rather than the width, so a narrow desktop window got none of it. The `Anon on export` label — the privacy control — was also rendering at 12 pixels on a phone despite a rule meant to set it to 16, which is the threshold below which iOS zooms the page when you tap a control (#503).
|
|
25
|
+
- The share preview on a phone now shows the report instead of its frontmatter. It was capped at 128 pixels, which left about four lines once the privacy notice had taken its share, and the source label was printed twice inside that space. The cap is now proportional to the visible screen, and it shrinks correctly when the on-screen keyboard is up rather than being pushed off the bottom (#503).
|
|
26
|
+
- Gallery text no longer appears to bleed through the pinned preview on a phone. Two separate causes: a 12-pixel band above the preview where scrolling content passed through a gap, and a bottom edge where the preview and the tile beneath it were the same colour with the same hairline border, so two half-cut lines of text read as one overlapping line. The preview now has a visible boundary and sits flush against the top of the scroll area (#503).
|
|
27
|
+
- Rotating a phone or resizing the window with the share modal open no longer resets the preview. The two layouts rendered separate previews, so crossing the breakpoint threw away the rendered report and its privacy answer, blanked the preview for about a second, and briefly showed the wrong statement about whether your export contains real project names. There is now one preview, positioned by layout at both sizes (#503).
|
|
28
|
+
- The keyboard shortcuts in the share documentation now name the gestures that work. Every passage that advertised a bare `S` or `B` has been rewritten; both shortcuts have always required Shift, and an unshifted `s` opens Settings instead. Several passages also told you to click a panel to focus it, which does not focus it — it opens the panel's own modal, which then blocks the shortcut — and two quoted a help message the application has never shown. The correct gesture is to Tab to one of the panel's actions and press Shift+S, which appeared nowhere. The share icon's own tooltip said `Share (S)` and now says `Share (Shift+S)` (#503).
|
|
29
|
+
|
|
30
|
+
## [1.94.2] - 2026-08-08
|
|
31
|
+
|
|
32
|
+
### Fixed
|
|
33
|
+
- Release validation's segment-elision oracle no longer flakes when two equivalent rebuilds cross a one-second boundary: it still compares the complete migration set, but ignores each rebuild's wall-clock migration timestamp.
|
|
34
|
+
|
|
8
35
|
## [1.94.1] - 2026-08-08
|
|
9
36
|
|
|
10
37
|
### Fixed
|
package/bin/_cctally_core.py
CHANGED
|
@@ -321,14 +321,6 @@ OAUTH_BACKOFF_CAP_SECONDS = 3600.0
|
|
|
321
321
|
_init_paths_from_env()
|
|
322
322
|
|
|
323
323
|
|
|
324
|
-
# stats.db WAL cap (#297). Bounds the persistent WAL file so a resetting
|
|
325
|
-
# checkpoint truncates it back down instead of leaving it at its high-water
|
|
326
|
-
# size. stats.db writes are small (its -wal was observed at 0 bytes even under
|
|
327
|
-
# the contention that bloated cache.db to multi-GB), so a tighter 16 MB cap is
|
|
328
|
-
# ample. See docs/superpowers/specs/2026-07-13-cache-db-wal-hardening-design.md.
|
|
329
|
-
STATS_WAL_SIZE_LIMIT_BYTES = 16 * 1024 * 1024 # 16777216
|
|
330
|
-
|
|
331
|
-
|
|
332
324
|
# === stats.db epoch-rebuild versioning (DB journal redesign §7.1/§8) ==
|
|
333
325
|
#
|
|
334
326
|
# stats.db is no longer a versioned migration target — it is a DISPOSABLE index
|
|
@@ -393,7 +385,12 @@ STATS_WAL_SIZE_LIMIT_BYTES = 16 * 1024 * 1024 # 16777216
|
|
|
393
385
|
# before any schema work, so an `add_column_if_missing` would never run on an
|
|
394
386
|
# upgraded install and the column would simply never appear. The registry stays
|
|
395
387
|
# frozen at 13 and an epoch mismatch resolves by rebuild.
|
|
396
|
-
|
|
388
|
+
# 1009 -> 1010 (#538 Task A): retire WAL for the disposable stats index. Every
|
|
389
|
+
# stats connection now uses DELETE/FULL rollback journaling, structurally
|
|
390
|
+
# removing the shared-memory WAL page map implicated by the retained corruption
|
|
391
|
+
# bundles. The one-time rebuild is the mode transition; no stats migration is
|
|
392
|
+
# added and cache/conversations remain WAL/NORMAL.
|
|
393
|
+
STATS_INDEX_EPOCH = 1010
|
|
397
394
|
LEGACY_STATS_HEAD = 13
|
|
398
395
|
|
|
399
396
|
#: #496 S1 F1. A NEW branch, for a state that cannot occur before the
|
|
@@ -773,8 +770,8 @@ def ensure_dirs() -> None:
|
|
|
773
770
|
# bin/_cctally_store.py _heal_flock_blocking, reached through
|
|
774
771
|
# _acquire_stats_maintenance_reentrant by the epoch
|
|
775
772
|
# resolver
|
|
776
|
-
# bin/_cctally_store.py
|
|
777
|
-
#
|
|
773
|
+
# bin/_cctally_store.py detached corruption-heal worker maintenance acquire
|
|
774
|
+
# (the detector itself never acquires it; #530)
|
|
778
775
|
# bin/_cctally_db.py cmd_db_rebuild, _acquire_db_admin_writer_flocks
|
|
779
776
|
# (db skip / db unskip), _cmd_db_repair_exclusive,
|
|
780
777
|
# _vacuum_one_db
|
|
@@ -1647,11 +1644,12 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
|
|
|
1647
1644
|
# #453: probe the live index before the maintenance-shared opener. During
|
|
1648
1645
|
# a large detached rebuild that opener can wait for its bounded 5-second
|
|
1649
1646
|
# guard; a statusline arriving mid-replay would otherwise pay that delay
|
|
1650
|
-
# (and can reach more than one stats opener in a single render). The
|
|
1651
|
-
#
|
|
1652
|
-
#
|
|
1653
|
-
# established guarded-open paths.
|
|
1654
|
-
# race-closing defense if the
|
|
1647
|
+
# (and can reach more than one stats opener in a single render). The probe
|
|
1648
|
+
# reads only the fixed user_version field in the main-file header; it never
|
|
1649
|
+
# opens SQLite or bypasses the maintenance opener fence. Missing, invalid-
|
|
1650
|
+
# header and legacy indexes retain their established guarded-open paths.
|
|
1651
|
+
# The post-open epoch gate below remains a race-closing defense if the
|
|
1652
|
+
# version changes after this probe.
|
|
1655
1653
|
if (
|
|
1656
1654
|
_target_path is None
|
|
1657
1655
|
and _cctally_store.stats_epoch_enabled()
|
|
@@ -1682,18 +1680,16 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
|
|
|
1682
1680
|
# (corruption that only surfaces mid-DDL stays a raw error; the probe
|
|
1683
1681
|
# catches the common case).
|
|
1684
1682
|
try:
|
|
1685
|
-
# §6.1 PRAGMA policy via the shared table (stats:
|
|
1683
|
+
# §6.1 PRAGMA policy via the shared table (stats: DELETE / FULL /
|
|
1686
1684
|
# busy_timeout 15000 / journal_size_limit 16 MiB / auto_vacuum unset).
|
|
1687
1685
|
# #297: busy_timeout 15000 lets a writer wait out a slow-but-normal sync
|
|
1688
|
-
# (>5 s) instead of erroring "database is locked"
|
|
1689
|
-
#
|
|
1690
|
-
# write sites, cctally-dev#87). journal_size_limit bounds the persistent
|
|
1691
|
-
# WAL. The corruption probe (SELECT 1) stays here inside the narrow
|
|
1686
|
+
# (>5 s) instead of erroring "database is locked". The corruption probe
|
|
1687
|
+
# (SELECT 1) stays here inside the narrow
|
|
1692
1688
|
# StatsDbCorruptError boundary.
|
|
1693
1689
|
_cctally_store.apply_policy(conn, "stats")
|
|
1694
1690
|
conn.execute("SELECT 1").fetchone()
|
|
1695
|
-
# §9.2 (#496 S6 F23). AFTER `apply_policy`,
|
|
1696
|
-
#
|
|
1691
|
+
# §9.2 (#496 S6 F23). AFTER `apply_policy`, so any rollback journal
|
|
1692
|
+
# materialized during the transition is included in family hardening.
|
|
1697
1693
|
_cctally_store._harden_stats_family(db_path)
|
|
1698
1694
|
except sqlite3.DatabaseError as exc:
|
|
1699
1695
|
try:
|
|
@@ -1714,7 +1710,15 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
|
|
|
1714
1710
|
# it — all of which fall through to the original guided
|
|
1715
1711
|
# StatsDbCorruptError so the manual path still applies.
|
|
1716
1712
|
heal = getattr(_cctally_store, "HEAL_HOOK", None)
|
|
1717
|
-
|
|
1713
|
+
heal_enabled = (
|
|
1714
|
+
os.environ.get("CCTALLY_TEST_DISABLE_STATS_AUTO_HEAL") != "1"
|
|
1715
|
+
)
|
|
1716
|
+
if (
|
|
1717
|
+
_target_path is None
|
|
1718
|
+
and heal is not None
|
|
1719
|
+
and heal_enabled
|
|
1720
|
+
and heal("stats", exc)
|
|
1721
|
+
):
|
|
1718
1722
|
# #386: the post-heal retry is an opener too. The heal released the
|
|
1719
1723
|
# maintenance lock before returning, so another maintenance path can
|
|
1720
1724
|
# legitimately own the family by now.
|
|
@@ -1763,9 +1763,15 @@ def _build_codex_source_share_snapshot(ls, *, state, panel: str,
|
|
|
1763
1763
|
display_tz_label=display_tz,
|
|
1764
1764
|
)
|
|
1765
1765
|
codex_module = sys.modules["cctally"]._load_sibling("_cctally_codex")
|
|
1766
|
+
# The canonical CLI adapter derives availability from its row count, but
|
|
1767
|
+
# this source-aware path has an authoritative provider state. Project the
|
|
1768
|
+
# normalized state after adapting the rows so all nine panels carry the
|
|
1769
|
+
# same source chrome and built-snapshot digest signal (#533).
|
|
1766
1770
|
return replace(
|
|
1767
1771
|
codex_module._build_codex_share_snapshot(command, view, normalized_rows),
|
|
1768
1772
|
template_id=template_id,
|
|
1773
|
+
availability=availability,
|
|
1774
|
+
availability_reason=reason,
|
|
1769
1775
|
)
|
|
1770
1776
|
|
|
1771
1777
|
|
package/bin/_cctally_db.py
CHANGED
|
@@ -373,10 +373,11 @@ class StatsHealDeferred(StatsRebuildDeferred):
|
|
|
373
373
|
quarantine and rebuild while the caller waited (#496 S3 §6). The caller
|
|
374
374
|
degrades where it already degrades; it never blocks on the rebuild.
|
|
375
375
|
|
|
376
|
-
``heal_id`` correlates this signal with the durable
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
376
|
+
``heal_id`` correlates this signal with the durable admitted occurrence.
|
|
377
|
+
``forensics_path`` is optional: since #530 the fresh worker captures and
|
|
378
|
+
persists that bundle after maintenance drains, so the detector normally
|
|
379
|
+
raises before a path exists. Both remain attributes rather than parsed
|
|
380
|
+
message text.
|
|
380
381
|
"""
|
|
381
382
|
|
|
382
383
|
def __init__(
|
|
@@ -1042,7 +1043,7 @@ def write_corruption_forensics(
|
|
|
1042
1043
|
"truncated": False,
|
|
1043
1044
|
},
|
|
1044
1045
|
}
|
|
1045
|
-
for suffix in ("", "-wal", "-shm"):
|
|
1046
|
+
for suffix in ("", "-journal", "-wal", "-shm"):
|
|
1046
1047
|
p = pathlib.Path(str(db_path) + suffix)
|
|
1047
1048
|
try:
|
|
1048
1049
|
st = p.stat()
|
|
@@ -1276,6 +1277,7 @@ def _load_pending_quarantine(db_path: pathlib.Path) -> "dict[str, Any] | None":
|
|
|
1276
1277
|
isinstance(name, str)
|
|
1277
1278
|
and name in {
|
|
1278
1279
|
db_path.name,
|
|
1280
|
+
f"{db_path.name}-journal",
|
|
1279
1281
|
f"{db_path.name}-wal",
|
|
1280
1282
|
f"{db_path.name}-shm",
|
|
1281
1283
|
}
|
|
@@ -1363,6 +1365,7 @@ def _quarantine_db_family_strict(
|
|
|
1363
1365
|
members = [
|
|
1364
1366
|
candidate.name
|
|
1365
1367
|
for candidate in (
|
|
1368
|
+
pathlib.Path(f"{db_path}-journal"),
|
|
1366
1369
|
pathlib.Path(f"{db_path}-wal"),
|
|
1367
1370
|
pathlib.Path(f"{db_path}-shm"),
|
|
1368
1371
|
db_path,
|
|
@@ -1448,7 +1451,7 @@ def quarantine_db_family(
|
|
|
1448
1451
|
strict: bool = False,
|
|
1449
1452
|
context: "QuarantineContext | None" = None,
|
|
1450
1453
|
) -> pathlib.Path:
|
|
1451
|
-
"""Move a damaged DB
|
|
1454
|
+
"""Move a damaged DB and its SQLite journals into a single
|
|
1452
1455
|
timestamped incident directory under ``quarantine/`` with a manifest (spec
|
|
1453
1456
|
§6.3). NEVER deletes evidence — three renames under the caller's exclusion
|
|
1454
1457
|
locks, not pretending to be one atomic op. ``0o700`` dir / ``0o600`` files.
|
|
@@ -1475,7 +1478,7 @@ def quarantine_db_family(
|
|
|
1475
1478
|
except OSError:
|
|
1476
1479
|
pass
|
|
1477
1480
|
moved: list = []
|
|
1478
|
-
for suffix in ("", "-wal", "-shm"):
|
|
1481
|
+
for suffix in ("", "-journal", "-wal", "-shm"):
|
|
1479
1482
|
src = pathlib.Path(str(db_path) + suffix)
|
|
1480
1483
|
if not src.exists():
|
|
1481
1484
|
continue
|
|
@@ -9415,10 +9418,10 @@ def _release_repair_marker(path: pathlib.Path, claim: RepairMarkerClaim) -> None
|
|
|
9415
9418
|
|
|
9416
9419
|
|
|
9417
9420
|
def _db_family_open_pids(path: pathlib.Path) -> "set[int] | None":
|
|
9418
|
-
"""Return processes with
|
|
9421
|
+
"""Return processes with any SQLite family member open; None if unknown."""
|
|
9419
9422
|
family = [
|
|
9420
9423
|
pathlib.Path(str(path) + suffix)
|
|
9421
|
-
for suffix in ("", "-wal", "-shm")
|
|
9424
|
+
for suffix in ("", "-journal", "-wal", "-shm")
|
|
9422
9425
|
if pathlib.Path(str(path) + suffix).exists()
|
|
9423
9426
|
]
|
|
9424
9427
|
if not family:
|
|
@@ -9478,7 +9481,7 @@ def _unique_sibling_path(path: pathlib.Path) -> pathlib.Path:
|
|
|
9478
9481
|
def family_exists(candidate: pathlib.Path) -> bool:
|
|
9479
9482
|
return any(
|
|
9480
9483
|
pathlib.Path(str(candidate) + suffix).exists()
|
|
9481
|
-
for suffix in ("", "-wal", "-shm")
|
|
9484
|
+
for suffix in ("", "-journal", "-wal", "-shm")
|
|
9482
9485
|
)
|
|
9483
9486
|
|
|
9484
9487
|
if not family_exists(path):
|
|
@@ -9510,9 +9513,9 @@ def _copy_db_family(
|
|
|
9510
9513
|
source: pathlib.Path,
|
|
9511
9514
|
destination: pathlib.Path,
|
|
9512
9515
|
*,
|
|
9513
|
-
suffixes: "tuple[str, ...]" = ("", "-wal", "-shm"),
|
|
9516
|
+
suffixes: "tuple[str, ...]" = ("", "-journal", "-wal", "-shm"),
|
|
9514
9517
|
) -> None:
|
|
9515
|
-
"""Copy main
|
|
9518
|
+
"""Copy main and journal bytes while the caller owns the family."""
|
|
9516
9519
|
for suffix in suffixes:
|
|
9517
9520
|
src = pathlib.Path(str(source) + suffix)
|
|
9518
9521
|
if not src.exists():
|
|
@@ -9523,6 +9526,53 @@ def _copy_db_family(
|
|
|
9523
9526
|
_fsync_file(dst)
|
|
9524
9527
|
|
|
9525
9528
|
|
|
9529
|
+
def _reserve_post_drain_evidence_path(
|
|
9530
|
+
backup: pathlib.Path, suffix: str
|
|
9531
|
+
) -> pathlib.Path:
|
|
9532
|
+
"""Reserve a unique sibling without ever overwriting retained evidence."""
|
|
9533
|
+
for number in range(1, 10_000):
|
|
9534
|
+
infix = ".post-drain" if number == 1 else f".post-drain-{number}"
|
|
9535
|
+
candidate = pathlib.Path(f"{backup}{infix}{suffix}")
|
|
9536
|
+
try:
|
|
9537
|
+
fd = os.open(
|
|
9538
|
+
candidate,
|
|
9539
|
+
os.O_WRONLY | os.O_CREAT | os.O_EXCL,
|
|
9540
|
+
0o600,
|
|
9541
|
+
)
|
|
9542
|
+
except FileExistsError:
|
|
9543
|
+
continue
|
|
9544
|
+
os.close(fd)
|
|
9545
|
+
return candidate
|
|
9546
|
+
raise OSError(f"could not allocate post-drain evidence beside {backup}")
|
|
9547
|
+
|
|
9548
|
+
|
|
9549
|
+
def _preserve_post_drain_sidecars(
|
|
9550
|
+
source: pathlib.Path, backup: pathlib.Path
|
|
9551
|
+
) -> "tuple[pathlib.Path, ...]":
|
|
9552
|
+
"""Move late sidecars aside without changing the classified backup."""
|
|
9553
|
+
moved = []
|
|
9554
|
+
for suffix in ("-journal", "-wal", "-shm"):
|
|
9555
|
+
member = pathlib.Path(str(source) + suffix)
|
|
9556
|
+
if not member.exists():
|
|
9557
|
+
continue
|
|
9558
|
+
destination = _reserve_post_drain_evidence_path(backup, suffix)
|
|
9559
|
+
try:
|
|
9560
|
+
# The O_EXCL reservation owns this pathname, so replace cannot
|
|
9561
|
+
# destroy pre-existing user or pre-checkpoint evidence.
|
|
9562
|
+
os.replace(member, destination)
|
|
9563
|
+
except OSError:
|
|
9564
|
+
try:
|
|
9565
|
+
os.unlink(destination)
|
|
9566
|
+
except OSError:
|
|
9567
|
+
pass
|
|
9568
|
+
raise
|
|
9569
|
+
os.chmod(destination, 0o600)
|
|
9570
|
+
_fsync_file(destination)
|
|
9571
|
+
_fsync_directory(destination.parent)
|
|
9572
|
+
moved.append(destination)
|
|
9573
|
+
return tuple(moved)
|
|
9574
|
+
|
|
9575
|
+
|
|
9526
9576
|
#: The classification sidecar's own schema version, matching the incident
|
|
9527
9577
|
#: `classification.json` the correlator writes.
|
|
9528
9578
|
_BACKUP_CLASSIFICATION_SCHEMA_VERSION = 1
|
|
@@ -9554,7 +9604,7 @@ def _write_backup_classification_sidecar(
|
|
|
9554
9604
|
safe direction.
|
|
9555
9605
|
"""
|
|
9556
9606
|
members = []
|
|
9557
|
-
for suffix in ("", "-wal", "-shm"):
|
|
9607
|
+
for suffix in ("", "-journal", "-wal", "-shm"):
|
|
9558
9608
|
member = pathlib.Path(str(stem) + suffix)
|
|
9559
9609
|
try:
|
|
9560
9610
|
info = member.stat()
|
|
@@ -9680,7 +9730,23 @@ def _repair_preflight_and_copy(
|
|
|
9680
9730
|
*,
|
|
9681
9731
|
timeout_ms: int,
|
|
9682
9732
|
) -> "tuple[int, dict[str, int | None], int | None, sqlite3.Connection | None, str]":
|
|
9683
|
-
"""Preserve
|
|
9733
|
+
"""Preserve bytes, drain only proven-coherent legacy WAL, and hold writer."""
|
|
9734
|
+
import _lib_stats_wal
|
|
9735
|
+
|
|
9736
|
+
wal_evidence = _lib_stats_wal.inspect_wal_index_family(path)
|
|
9737
|
+
wal_verdict = wal_evidence.get("verdict")
|
|
9738
|
+
if _lib_stats_wal.is_incoherent_wal_index(wal_evidence):
|
|
9739
|
+
return (
|
|
9740
|
+
3, {}, None, None,
|
|
9741
|
+
"incoherent legacy WAL family; refusing to open or checkpoint it",
|
|
9742
|
+
)
|
|
9743
|
+
if wal_verdict not in {"coherent", "wal_absent", "wal_empty"}:
|
|
9744
|
+
return (
|
|
9745
|
+
3, {}, None, None,
|
|
9746
|
+
"could not prove the legacy WAL family coherent "
|
|
9747
|
+
f"({wal_verdict or 'unknown'}); refusing repair",
|
|
9748
|
+
)
|
|
9749
|
+
checkpoint_legacy_wal = wal_verdict == "coherent"
|
|
9684
9750
|
conn = sqlite3.connect(
|
|
9685
9751
|
f"file:{path}?mode=rw", uri=True, timeout=max(timeout_ms, 0) / 1000
|
|
9686
9752
|
)
|
|
@@ -9717,20 +9783,23 @@ def _repair_preflight_and_copy(
|
|
|
9717
9783
|
_write_backup_classification_sidecar(backup)
|
|
9718
9784
|
conn.rollback()
|
|
9719
9785
|
|
|
9720
|
-
|
|
9721
|
-
|
|
9722
|
-
|
|
9723
|
-
|
|
9724
|
-
|
|
9725
|
-
|
|
9726
|
-
|
|
9727
|
-
|
|
9728
|
-
|
|
9729
|
-
|
|
9786
|
+
if checkpoint_legacy_wal:
|
|
9787
|
+
try:
|
|
9788
|
+
checkpoint = conn.execute(
|
|
9789
|
+
"PRAGMA wal_checkpoint(TRUNCATE)"
|
|
9790
|
+
).fetchone()
|
|
9791
|
+
except sqlite3.DatabaseError as exc:
|
|
9792
|
+
conn.close()
|
|
9793
|
+
return 3, {}, None, None, f"WAL checkpoint failed: {exc}"
|
|
9794
|
+
wal_path = pathlib.Path(str(path) + "-wal")
|
|
9795
|
+
wal_bytes = wal_path.stat().st_size if wal_path.exists() else 0
|
|
9796
|
+
if checkpoint is None or int(checkpoint[0]) != 0 or wal_bytes != 0:
|
|
9797
|
+
conn.close()
|
|
9798
|
+
return 3, {}, None, None, "WAL could not be fully checkpointed"
|
|
9730
9799
|
|
|
9731
|
-
# Hold this
|
|
9732
|
-
#
|
|
9733
|
-
#
|
|
9800
|
+
# Hold this write exclusion through source capture and `.recover`.
|
|
9801
|
+
# Final replacement releases it only to perform the mandatory second
|
|
9802
|
+
# whole-family drain under the repair marker + maintenance fence.
|
|
9734
9803
|
try:
|
|
9735
9804
|
conn.execute("BEGIN IMMEDIATE")
|
|
9736
9805
|
except sqlite3.DatabaseError as exc:
|
|
@@ -10047,18 +10116,34 @@ def _cmd_db_repair_under_retention(
|
|
|
10047
10116
|
try:
|
|
10048
10117
|
os.chmod(recovered_path, 0o600)
|
|
10049
10118
|
_fsync_file(recovered_path)
|
|
10050
|
-
#
|
|
10051
|
-
#
|
|
10052
|
-
#
|
|
10053
|
-
#
|
|
10054
|
-
os.replace(recovered_path, path)
|
|
10055
|
-
_fsync_directory(path.parent)
|
|
10119
|
+
# Release our own SQLite handle before the destructive-edge drain.
|
|
10120
|
+
# The repair marker and maintenance-exclusive hold still fence every
|
|
10121
|
+
# normal cctally opener. A second whole-family scan closes the gap
|
|
10122
|
+
# between the admission scan and this physical replacement.
|
|
10056
10123
|
close_guard()
|
|
10057
|
-
|
|
10058
|
-
|
|
10059
|
-
|
|
10060
|
-
|
|
10061
|
-
|
|
10124
|
+
open_pids = _db_family_open_pids(path)
|
|
10125
|
+
if open_pids is None:
|
|
10126
|
+
eprint(
|
|
10127
|
+
"cctally: final stats.db repair declined: the second cold "
|
|
10128
|
+
"drain could not verify open handles. Nothing was moved."
|
|
10129
|
+
)
|
|
10130
|
+
return 3
|
|
10131
|
+
if open_pids:
|
|
10132
|
+
eprint(
|
|
10133
|
+
"cctally: final stats.db repair declined after a new reader "
|
|
10134
|
+
"appeared in process(es) "
|
|
10135
|
+
+ ", ".join(str(pid) for pid in sorted(open_pids))
|
|
10136
|
+
+ ". Stop the dashboard and older cctally processes, then "
|
|
10137
|
+
"retry; nothing was moved."
|
|
10138
|
+
)
|
|
10139
|
+
return 3
|
|
10140
|
+
|
|
10141
|
+
# The old family was copied, fsync'd, and classified before
|
|
10142
|
+
# recovery. Preserve any later survivors under unique post-drain
|
|
10143
|
+
# names; overwriting the exact pre-checkpoint members would destroy
|
|
10144
|
+
# evidence and make their inode/size classification stale.
|
|
10145
|
+
_preserve_post_drain_sidecars(path, backup)
|
|
10146
|
+
os.replace(recovered_path, path)
|
|
10062
10147
|
_fsync_directory(path.parent)
|
|
10063
10148
|
except OSError as exc:
|
|
10064
10149
|
close_guard()
|
|
@@ -10128,11 +10213,11 @@ def cmd_db_backup(args: argparse.Namespace) -> int:
|
|
|
10128
10213
|
prefix=f".{output.name}.tmp-", dir=output.parent
|
|
10129
10214
|
) as scratch_raw:
|
|
10130
10215
|
temp_path = pathlib.Path(scratch_raw) / output.name
|
|
10131
|
-
# #386: `db backup --db stats` holds this handle across the
|
|
10132
|
-
# `source.backup(destination)` loop
|
|
10133
|
-
#
|
|
10134
|
-
#
|
|
10135
|
-
#
|
|
10216
|
+
# #386/#538: `db backup --db stats` holds this handle across the
|
|
10217
|
+
# entire `source.backup(destination)` loop. In rollback mode that is
|
|
10218
|
+
# a long-lived reader, so it participates in the replacement
|
|
10219
|
+
# protocol. The `mode=ro` open is preserved verbatim through the
|
|
10220
|
+
# `connect` seam.
|
|
10136
10221
|
def _backup_source_connect(_p, _timeout_ms=timeout_ms):
|
|
10137
10222
|
return sqlite3.connect(
|
|
10138
10223
|
f"file:{_p}?mode=ro", uri=True,
|
|
@@ -10183,8 +10268,7 @@ def cmd_db_backup(args: argparse.Namespace) -> int:
|
|
|
10183
10268
|
|
|
10184
10269
|
|
|
10185
10270
|
def cmd_db_checkpoint(args: argparse.Namespace) -> int:
|
|
10186
|
-
"""Fast, non-destructive WAL drain (TRUNCATE checkpoint) for cache.db
|
|
10187
|
-
stats.db (#297).
|
|
10271
|
+
"""Fast, non-destructive WAL drain (TRUNCATE checkpoint) for cache.db.
|
|
10188
10272
|
|
|
10189
10273
|
Opens a RAW existing-file-only connection — NOT open_cache_db()/open_db(),
|
|
10190
10274
|
which apply schema, run the migration dispatcher, can DELETE Codex rows on a
|
|
@@ -10200,17 +10284,29 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
|
|
|
10200
10284
|
was not fully truncated through the timeout — an actionable "something is
|
|
10201
10285
|
still holding it" signal.
|
|
10202
10286
|
"""
|
|
10203
|
-
import _cctally_cache
|
|
10204
10287
|
from _lib_json_envelope import stamp_schema_version
|
|
10205
10288
|
|
|
10206
10289
|
which = args.db # "cache" | "stats"
|
|
10207
|
-
|
|
10208
|
-
|
|
10209
|
-
|
|
10210
|
-
|
|
10290
|
+
as_json = bool(getattr(args, "json", False))
|
|
10291
|
+
if which == "stats":
|
|
10292
|
+
reason = (
|
|
10293
|
+
"stats.db uses rollback journaling; no WAL checkpoint is applicable"
|
|
10294
|
+
)
|
|
10295
|
+
if as_json:
|
|
10296
|
+
print(json.dumps(stamp_schema_version({
|
|
10297
|
+
"db": "stats.db",
|
|
10298
|
+
"status": "notApplicable",
|
|
10299
|
+
"reason": reason,
|
|
10300
|
+
}, version=1)))
|
|
10301
|
+
else:
|
|
10302
|
+
eprint(f"cctally: {reason}.")
|
|
10303
|
+
return 2
|
|
10304
|
+
|
|
10305
|
+
import _cctally_cache
|
|
10306
|
+
|
|
10307
|
+
path, label = _cctally_core.CACHE_DB_PATH, "cache.db"
|
|
10211
10308
|
timeout = int(getattr(args, "busy_timeout_ms", None)
|
|
10212
10309
|
or _cctally_cache.CHECKPOINT_CMD_BUSY_TIMEOUT_MS)
|
|
10213
|
-
as_json = bool(getattr(args, "json", False))
|
|
10214
10310
|
|
|
10215
10311
|
# Absent file → nothing to drain; a missing re-derivable cache is not an
|
|
10216
10312
|
# error (mirrors cmd_db_recover / cmd_db_unskip). Do NOT connect — mode=rw
|
|
@@ -10227,20 +10323,10 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
|
|
|
10227
10323
|
|
|
10228
10324
|
from _lib_cache_writer_lock import acquire_ordered_flocks
|
|
10229
10325
|
|
|
10230
|
-
|
|
10231
|
-
|
|
10232
|
-
|
|
10233
|
-
|
|
10234
|
-
# must not overlap one.
|
|
10235
|
-
lock_plan = {
|
|
10236
|
-
"cache": [
|
|
10237
|
-
(_cctally_core.CACHE_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
|
|
10238
|
-
(_cctally_core.CACHE_LOCK_PATH, fcntl.LOCK_EX),
|
|
10239
|
-
],
|
|
10240
|
-
"stats": [
|
|
10241
|
-
(_cctally_core.STATS_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
|
|
10242
|
-
],
|
|
10243
|
-
}[which]
|
|
10326
|
+
lock_plan = [
|
|
10327
|
+
(_cctally_core.CACHE_LOCK_MAINTENANCE_PATH, fcntl.LOCK_SH),
|
|
10328
|
+
(_cctally_core.CACHE_LOCK_PATH, fcntl.LOCK_EX),
|
|
10329
|
+
]
|
|
10244
10330
|
|
|
10245
10331
|
held: list[int] = []
|
|
10246
10332
|
try:
|
|
@@ -10286,7 +10372,7 @@ def cmd_db_checkpoint(args: argparse.Namespace) -> int:
|
|
|
10286
10372
|
|
|
10287
10373
|
# VACUUM writes a full fresh copy of the database into a temporary file and then
|
|
10288
10374
|
# swaps it in, so it transiently needs roughly the DB's own size on top of the
|
|
10289
|
-
# existing file, plus room for
|
|
10375
|
+
# existing file, plus room for any WAL on WAL-backed stores. A short busy_timeout keeps a
|
|
10290
10376
|
# contended VACUUM from hanging (F13).
|
|
10291
10377
|
_VACUUM_BUSY_TIMEOUT_MS = 250
|
|
10292
10378
|
|
|
@@ -10313,7 +10399,7 @@ def _vacuum_required_free_bytes(path) -> int:
|
|
|
10313
10399
|
|
|
10314
10400
|
|
|
10315
10401
|
def _run_vacuum_exclusive(path, label: str) -> int:
|
|
10316
|
-
"""
|
|
10402
|
+
"""VACUUM ``path`` under EXCLUSIVE; checkpoint WAL-backed stores first.
|
|
10317
10403
|
|
|
10318
10404
|
``locking_mode=EXCLUSIVE`` + a short ``busy_timeout`` make a concurrent
|
|
10319
10405
|
reader/writer FAIL PROMPTLY (no TOCTOU gap — the exclusion is the DB's own
|
|
@@ -10326,7 +10412,8 @@ def _run_vacuum_exclusive(path, label: str) -> int:
|
|
|
10326
10412
|
before = conn.execute("PRAGMA page_count").fetchone()[0]
|
|
10327
10413
|
try:
|
|
10328
10414
|
conn.execute("PRAGMA locking_mode=EXCLUSIVE")
|
|
10329
|
-
|
|
10415
|
+
if label != "stats.db":
|
|
10416
|
+
conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
|
10330
10417
|
conn.execute("VACUUM")
|
|
10331
10418
|
except sqlite3.OperationalError as exc:
|
|
10332
10419
|
if "lock" in str(exc).lower() or "busy" in str(exc).lower():
|
|
@@ -10424,6 +10511,17 @@ def _vacuum_one_db(path, label: str, provider_locked: bool) -> int:
|
|
|
10424
10511
|
)
|
|
10425
10512
|
return 3
|
|
10426
10513
|
held.append(fh)
|
|
10514
|
+
if stats_store:
|
|
10515
|
+
import _cctally_store
|
|
10516
|
+
|
|
10517
|
+
if _cctally_store._stats_legacy_wal_family_present(path):
|
|
10518
|
+
eprint(
|
|
10519
|
+
"cctally: stats.db VACUUM declined: the index still uses "
|
|
10520
|
+
"legacy WAL. Stop and restart older cctally processes, "
|
|
10521
|
+
"then run a normal command to complete the epoch rebuild "
|
|
10522
|
+
"before retrying VACUUM."
|
|
10523
|
+
)
|
|
10524
|
+
return 3
|
|
10427
10525
|
return _run_vacuum_exclusive(path, label)
|
|
10428
10526
|
finally:
|
|
10429
10527
|
for fh in held:
|