easy_sync 0.0.5 → 2.0.0

data/docs/parallel-scrub.md

@@ -0,0 +1,280 @@
+# Parallel scrub: `easy_sync scrub --jobs N` (built and verified on real hardware)
+
+The build spec for scrubbing several drives at once. Every decision below was
+already argued out; implement it as written, and if the code forces a change,
+update this doc in the same PR. Read `docs/integrity-scan.md` first: this only
+changes *how many* drives `scrub` works on at a time, never what it does to
+one drive.
+
+## Why
+
+A full scrub of the fleet (~35 TB) takes about 2 days, because `scrub --all`
+does one drive at a time and a drive reads at 170-200 MB/s. The drives are
+not sharing a bottleneck. Measured on 2026-09-21 with Scrubber's own read
+path (8 MB `File#read` chunks, `F_NOCACHE`), cold files, all in the ThunderBay 8:
+
+| drives at once | per-drive MB/s | aggregate MB/s |
+|---|---|---|
+| 1 | 169-191 | ~180 |
+| 2 | 169, 202 | ~350 |
+| 4 (7-9 GB files each) | 184, 180, 173, 172 | **641** (90% of the sum of solo speeds) |
+
+Each bay has its own AHCI controller on a 40 Gb/s Thunderbolt link
+(docs/performance.md, "Enclosure bandwidth"), so 8-way should scale too, but
+only 4-way is measured. SHA-256 runs at ~2,800 MB/s on one core, so CPU is
+never the limit.
+
+This is the opposite of `sync`, which is sequential on purpose (one NAS, one
+network link; CLAUDE.md invariants). Do not use this doc as a reason to
+parallelize `sync`.
+
+## Decisions (do not re-litigate)
+
+1. **Threads, not processes.** The work is I/O-bound; MRI releases the GVL
+   during blocking `File#read`, so threads overlap the reads. Processes would
+   mean result marshaling and redoing the process-group signal forwarding
+   CLAUDE.md warns about. No new gems.
+2. **SQLite: WAL + one connection per worker.** Not a write queue. Details
+   in "Manifest" below.
+3. **One drive per worker.** Never two workers on the same drive (two
+   readers on one spindle just seek against each other). Rows are
+   partitioned by `drive_serial`, so workers never touch each other's rows.
+4. **Default 4 jobs**, the largest count actually measured. `--jobs 1` must
+   behave exactly like today's sequential `scrub --all`.
+
+## Behavior
+
+    easy_sync scrub [NAME ...] [--all] [--jobs N] [--for DURATION] [--dry-run] [--no-keep-awake]
+
+- `--jobs N` (N >= 1; reject 0/negative/non-integer with an `Error`).
+  Default comes from config `scrub_jobs` (default 4), added to
+  `Config::DEFAULTS`, `KEY_COMMENTS` and `SAMPLE` the same way
+  `scrub_stale_days` was.
+- Plain `scrub` (no names, no `--all`) still picks one drive, so jobs is
+  irrelevant there. Named targets and `--all` go through the pool.
+- Work order is unchanged: targets are resolved exactly as
+  `CLI#resolve_scrub_targets` does today (stalest first for `--all`, given
+  order for names) and put on a `Queue`. `min(jobs, targets.size)` worker
+  threads each pop the next target, scrub it, and pop again until the queue
+  is empty. With 8 drives and 4 jobs, the 4 stalest start first and the rest
+  start as slots free up.
+- `--for`: the deadline is shared. A worker that finds `clock.now >=
+  deadline` before popping stops without starting a new drive (a new drive
+  would otherwise run its whole walk phase before the hash loop notices the
+  deadline). A drive already mid-hash stops at its next file, as today.
+- A drive that gets unmounted stops only its own worker's drive; that worker
+  moves on to the next target.
+- `--dry-run` goes through the same pool (it only reads), so output lines
+  must be attributable to a drive (see "Output").
+- Exit code, the "Scrub finished" / "Scrub stopped early (...)" final line,
+  and `findings`/`stopped` aggregation are unchanged; build them from the
+  collected `Result`s in **target order**, not completion order, so the log
+  summary is deterministic.
+
+## Code changes
+
+### `Jbod::ScrubPool` (new, `lib/easy_sync/jbod/scrub_pool.rb`)
+
+Owns the threads so `CLI#scrub` stays readable and the threading is testable
+without the CLI. Suggested shape:
+
+    ScrubPool.new(jobs:, open_manifest:, scrubber_options:, lock:, clock:, deadline:, out:)
+    #run(targets) -> [Result, ...] in target order
+
+- `open_manifest` is a lambda returning a new `Manifest` (the CLI passes
+  `-> { Jbod::Manifest.open(settings[:manifest_path]) }`).
+- **Open all worker manifests on the calling thread before starting any
+  thread.** `Manifest#initialize` runs `migrate!` and the WAL pragma; doing
+  that from N threads at once invites lock errors on the first run.
+- Each worker: own `Manifest`, own `Scrubber.new(manifest, **scrubber_options)`.
+  `Scrubber` itself does not know it is in a pool.
+- Close every worker manifest in an `ensure` after joining.
+- Track the set of drives currently being scrubbed (a `Set` under a `Mutex`);
+  after every add/remove call `lock.note(*active_names)` (see RunLock).
+- **Ctrl-C.** `Interrupt` arrives on the main thread, which is blocked in
+  `join`. Rescue it there, `t.raise(Interrupt)` every live worker (this
+  interrupts a blocked `File#read`, matching today's "stop mid-file, leave
+  that row untouched" behavior), join them all (swallowing their
+  `Interrupt`), then re-raise so `CLI#scrub`'s existing `rescue Interrupt`
+  logs the interrupted line. Waiting for workers to reach a file boundary is
+  wrong: one file can be 58 GB (several minutes).
+- Any other exception in a worker: let it propagate out of `join`/`value`
+  after stopping the other workers the same way (so they flush). Don't
+  swallow it.
+
+### `Scrubber#hash_phase`: buffer outcomes, flush in short transactions
+
+Today `hash_phase` opens a transaction, keeps it open, and commits and
+reopens every `COMMIT_INTERVAL` (30 s). In WAL, a connection holds the single
+write lock from its first write until commit, so one worker would hold the
+lock ~30 s at a time and every other worker's write would time out. So:
+
+- Collect outcomes in an array (`[args, kwargs]` for each `checksum_hashed`
+  call that `record_hash`/`record_read_error`/the vanished branch make today)
+  instead of writing immediately.
+- Flush every `COMMIT_INTERVAL` and in `ensure`: one
+  `db.transaction(:immediate) { pending.each { ... checksum_hashed ... } }`,
+  then clear the array. Lock held for milliseconds.
+- Wrap the `ensure` flush in `Thread.handle_interrupt(Interrupt => :never)`
+  so a `Thread#raise` from the pool can't abort it half-way. Outcomes already
+  in the buffer are for files that were fully read and are valid to record.
+- Keep "one `@clock.now` read per file" (the deadline spec counts calls);
+  flushing must not read the clock.
+- Print the progress line right after each interval flush, as now.
+
+This is the only `Scrubber` change besides output prefixes. It applies to
+`--jobs 1` too; the single-drive path must keep passing every existing spec
+unchanged.
+
+### `Manifest`
+
+- In `Manifest.open` (not `new`, so `:memory:` specs are untouched), before
+  `migrate!`: `db.execute('PRAGMA journal_mode = WAL')` unless path is
+  `':memory:'`, and on every connection `db.busy_timeout = 30_000`
+  (sqlite3 2.9.6 has `Database#busy_timeout=`). WAL is sticky in the file,
+  so the real manifest switches the first time any command opens it.
+  Consequence: `manifest.sqlite3-wal` and `-shm` appear next to it; nothing
+  copies the manifest file directly (`backup_to` uses the online backup API,
+  which is WAL-safe), so that is cosmetic.
+- Change every `db.transaction do` in `Manifest` to
+  `db.transaction(:immediate) do`. A deferred transaction that reads and then
+  writes can fail in WAL with `SQLITE_BUSY` *without* the busy handler
+  retrying (another connection committed after its read snapshot). Immediate
+  takes the write lock at `BEGIN`, where `busy_timeout` does apply. Scrub uses
+  `reconcile_checksums` (walk phase) this way, but changing all of them is
+  simpler than auditing which ones can race and costs nothing single-threaded.
+
+### `RunLock#note`: plural
+
+Today the lock file is `pid\nkind\n` plus one optional third line (the drive
+`scrub` is on). Change `note(text)` to `note(*names)`, writing each name on
+its own line from line 3 on (friendly names are not validated, but a newline
+in one is not realistic). `Status#current` becomes an `Array` of names
+(empty when none). Update callers:
+
+- `CLI#print_scrub_status`: `run.current.include?(d.friendly_name)`.
+- `Dashboard#scrubbing?`: same.
+- `CLI#print_run_status`: for a scrub, append `on backup-01-8tb, backup-03-8tb`
+  when `current` is non-empty.
+- Existing `run_lock_spec` note/current specs move to the array form.
+
+Keep: `note` preserves the lock file's mtime (the run's start time) and is a
+no-op unless this process holds the lock.
+
+### Output
+
+- `RunLog#puts` and `#print` wrap their two writes in a `Mutex`, so lines
+  from different threads never interleave mid-line.
+- Every line a `Scrubber` prints must name its drive. Progress lines, the
+  per-drive summary and `stopped:` lines already do. Prefix the two
+  `WARNING:` lines (walk "could not walk it completely" and hash_one
+  "skipped, left as it was") with `"#{result.drive}: "`, and the dry-run
+  line already starts with the drive.
+- Header line gains `· jobs N` when N > 1.
+
+### CLI / docs
+
+- `CLI#scrub`: parse `--jobs`, build the `ScrubPool`, keep the RunLock /
+  RunLog / KeepAwake / final-line / exit-code code as is around it.
+- `COMMAND_GROUPS` usage line and README: document `--jobs N`, the default,
+  and that it is scrub-only. README config example gains `scrub_jobs: 4`.
+- CLAUDE.md: replace the "first full pass ... about 2 days" wording in Open
+  items with the measured parallel number once the hardware check below is
+  done; add the 4-way scaling result under "Things learned on real hardware".
+- docs/performance.md "Enclosure bandwidth": add the table from "Why" above.
+- Flip this doc's title to "(built, not yet verified on real hardware)" when
+  code lands, and to "(built and verified on real hardware)" with a Results
+  section after the checklist passes, as integrity-scan.md did.
+
+## Tests
+
+`ScrubPool` and anything using more than one connection needs a
+**file-backed** manifest (`Manifest.open(File.join(temp_dir, 'm.sqlite3'))`):
+every `:memory:` connection is its own separate database. Real files, no
+FakeShell, like `scrubber_spec`.
+
+- Manifest: `Manifest.open` on a file reports `journal_mode` `wal` and a
+  30 s busy timeout; `:memory:` still works.
+- Scrubber: outcomes are written in batches (stub/spy the transaction: with
+  a ticking clock crossing `COMMIT_INTERVAL` once, exactly two flushes; with a
+  static clock, one flush at the end); a run interrupted by `Interrupt`
+  raised from inside `File.open` for file N still records files 1..N-1 and
+  leaves file N untouched. All existing scrubber specs pass unmodified.
+- ScrubPool, 3 drives on disk (reuse `write_marker`-style setup):
+  - `jobs: 2` baselines every file on all 3 drives and returns results in
+    target order.
+  - never more than `jobs` `Scrubber#run` calls in flight at once, and never
+    two on the same drive (instrument `run` with a counter under a mutex).
+  - past the deadline, no new drive is started.
+  - `lock.note` receives the active set as drives start and finish
+    (spy on `note`).
+  - `Interrupt` on the calling thread stops all workers and re-raises; rows
+    for completed files are present.
+  - an exception in one worker propagates, and other workers' completed
+    rows are still flushed.
+- CLI: `--jobs 0`/`--jobs x` rejected; `--jobs 2 --all` scrubs everything;
+  `status` shows "scrubbing now" for every name in a multi-line lock file;
+  `print_run_status` lists the active drives.
+- Dashboard: two tiles show "scrubbing now" when `current` has two names.
+- RunLog: many threads calling `puts` concurrently produce only whole lines.
+
+## Verify on real hardware before merging
+
+Use a scratch `--config` whose `manifest_path`, `log_dir`, `lock_path` and
+`dashboard_path` point into a scratch directory, so the real
+`~/.easy_sync/` is never touched (CLAUDE.md, "Verify live"). Scrub only reads
+the drives, so the real fleet can be the target. The scratch manifest needs
+the fleet registered: copy the real manifest into the scratch dir first
+(`sqlite3 ~/.easy_sync/manifest.sqlite3 ".backup <scratch>/manifest.sqlite3"`),
+then use `--for` so each step is bounded.
+
+1. `scrub --all --jobs 4 --for 5m`: aggregate throughput (sum of bytes read /
+   wall time from the per-drive summary lines) within ~15% of 4x one drive.
+   `iostat -w 5` shows 4 disks busy.
+2. While step 1 runs: `status --config <scratch>` lists all 4 drives as
+   "scrubbing now"; `dashboard --config <scratch>` shows 4 tiles likewise.
+   A 5th drive starts as soon as one finishes (use a small drive, e.g.
+   backup-08-2tb, among the first 4 to see it).
+3. `--jobs 8 --for 5m`: record the 8-way aggregate in docs/performance.md
+   whether or not it scales. If it doesn't, leave the default at 4.
+4. Ctrl-C mid-run: every worker stops within a few seconds (not at the end of
+   its current file), the log says "interrupted", and a follow-up
+   `scrub --all --jobs 4 --for 2m` resumes with no row reset and no findings.
+5. `sqlite3 <scratch>/manifest.sqlite3 "PRAGMA integrity_check;"` -> `ok`,
+   and `PRAGMA journal_mode;` -> `wal`.
+6. The WAL switch touches `sync`'s database too: run `sync` against the
+   `jbod-test` drives with a scratch config (CLAUDE.md, "Verify live") and
+   confirm it completes and `status`/dashboard read back normally.
+7. No output line in the scrub log is interleaved or lacks a drive name.
+
+## Results (2026-09-21, against the real ThunderBay 8 fleet)
+
+All 7 checks above passed on a scratch config against the real fleet (all 8
+drives, none of the two `jbod-test` drives, which are retired):
+
+1. `--jobs 4 --for 5m`: 4 drives (backup-01/02/03/04) read concurrently,
+   ~782 MB/s aggregate (docs/performance.md), `iostat -w 2` showed exactly 4
+   physical disks busy the whole time.
+2. `status --config <scratch>` (run from a second process, reading the same
+   lock file) listed all 4 as "scrubbing now" in one line; the dashboard
+   showed 4 tiles the same way.
+3. `--jobs 8 --for 3m`: all 8 active drives ran at once (`iostat` confirmed
+   8 disks busy), ~1175 MB/s aggregate - default stays 4 (the largest
+   *measured before this pass*; 8 works too, but nothing changes the
+   default without more runs to be sure it's not the two 2 TB/3 TB drives
+   inflating the average).
+4. `kill -INT` on the process 8s in: exited in well under a second, log said
+   "Scrub interrupted (Ctrl-C)", the lock file was removed, and a follow-up
+   `--jobs 4` resumed on the same 4 drives with new baselines only for files
+   not yet reached - no repeats, no reset.
+5. `PRAGMA integrity_check` -> `ok`, `PRAGMA journal_mode` -> `wal`, after
+   every run above including the Ctrl-C'd one.
+6. A real `sync` (fresh scratch manifest, `jbod-test-1` registered with its
+   real marker serial, a two-folder scratch NAS source) placed and copied
+   both folders onto `jbod-test-1`, `status` and the dashboard read the WAL
+   manifest back with no errors. Cleaned up afterward (test folders and the
+   scratch `.easy_sync/manifest.sqlite3`/`config.yml` copies removed from the
+   drive; its real `drive.json` marker was left untouched).
+7. Confirmed by grep across every log produced above: every line either
+   starts with a drive name or is one of the run's own fixed header/footer
+   lines.

data/docs/performance.md

@@ -0,0 +1,146 @@
+# Measured performance
+
+These numbers were measured, not estimated. Re-measure rather than eyeball.
+The fleet is 8 drives, 44.59 TB (4 × 7.28, 2 × 5.46, 1 × 2.73, 1 × 1.82 TB),
+against a library of about 31.9 TB (tv 17.7, movies 12.3, synology 1.9,
+pro 0.04).
+
+## Sync throughput: 62.9 MB/s aggregate
+
+The source is the `sync_runs` table of the real manifest, for the first
+full-library run (2026-09-13 to 2026-09-17). That covers 154 folder copies of
+5 GB or more: 11.3 TB in 49.9 hours of transfer time. Runs under 5 GB are
+left out so that per-folder setup time doesn't skew the rate.
+
+```sql
+SELECT folder_path, bytes_transferred,
+       strftime('%s',finished_at) - strftime('%s',started_at) AS secs
+  FROM sync_runs
+ WHERE exit_status = 0 AND bytes_transferred >= 5e9
+   AND strftime('%s',finished_at) > strftime('%s',started_at);
+```
+
+| statistic | MB/s |
+|---|---|
+| **aggregate (total bytes / total time)** | **62.9** |
+| median folder | 67.7 |
+| p25 – p75 | 57.8 – 79.8 |
+| p10 – p90 | 50.2 – 87.6 |
+| min / max | 16.6 / 98.3 |
+
+- **Use the aggregate for planning.** It is the only figure that predicts
+  wall-clock time. The median looks better because slow folders take up more
+  of the clock.
+- **Don't read the rate off an rsync log.** The rate `--info=progress2` prints
+  is a cumulative average (bytes ÷ elapsed). It changes slowly and hides how
+  much the real rate varies.
+- **The floor for any change to `sync`: the aggregate must stay at or above
+  50 MB/s.** It is a floor on the aggregate because 10.7% of transfer time
+  already runs below 50. Hashing inside the copy path costs 25–33%, which
+  would bring the aggregate down to 42–47 MB/s. That is why `scrub` is a
+  separate command (see `docs/integrity-scan.md`).
+- **Time Machine skews the numbers.** It backs up to a sparsebundle on the
+  same Synology (DS1019) that serves the media shares, so it competes for the
+  same array and SMB link. It was running during part of this run. Run
+  `sudo tmutil disable` before re-measuring, and check it first whenever a
+  sync looks slow.
+- **The NAS read side is not measured yet**, and it is the part that limits
+  speed. It needs a quiet NAS: no sync and no Time Machine running.
+
+## Drives are not the limit
+
+Sync throughput per target drive: backup-01-8tb 66.5, backup-03-8tb 66.8,
+backup-04-8tb 66.4, backup-06-8tb 68.8, backup-02-6tb 67.8 MB/s. They are within 2.4 MB/s of each other,
+because none of them is the constraint.
+
+Each drive was also measured directly: 8 GB sequential, buffer cache bypassed
+with `fcntl(F_NOCACHE)`, closing `fsync` included in the timing. The "used"
+column is how full the drive was when it was measured.
+
+| drive | model | used then | write MB/s | read MB/s |
+|---|---|---|---|---|
+| backup-04-8tb | ST8000VN004 | 2.2 TB | **203** | 213 |
+| backup-06-8tb | ST8000VN004 | 4.1 TB | 182 | 190 |
+| backup-01-8tb | ST8000VN004 | 2.5 TB | 178 | 199 |
+| backup-07-6tb | HDWE160 | empty | 143 | 179 |
+| backup-08-2tb | ST2000LM015 | empty | 116 | 126 |
+| backup-05-3tb | WD30EFRX | empty | 115 | 128 |
+
+The 8 TB figures are the mean of three runs; the others are single runs.
+Runs vary by about ±7%, so smaller differences than that aren't real.
+backup-04 really is about 14% faster than the identical backup-01, across
+three runs. Even the slowest drive writes at 1.8× the sync rate.
+
+`easy_sync benchmark` measures each drive this way (8 GB by default) and
+keeps the last 25 runs per drive, so these numbers can be re-measured and
+tracked rather than read off this table once.
+
+How to benchmark a drive so the result is real:
+
+- With 24 GB of RAM, a test smaller than RAM measures the buffer cache, not
+  the disk.
+- macOS `dd` has no `oflag=direct`. `io.fcntl(48, 1)` (`F_NOCACHE`) on the
+  file descriptor is the only way to bypass the cache.
+- A spinning drive that benchmarks in GB/s means the flag didn't take effect.
+- Use random data, and include the closing `fsync` in the timing.
+- Never benchmark a drive that a sync is writing to at the same time.
+
+## Enclosure bandwidth
+
+The OWC ThunderBay 8 is Thunderbolt, not USB-C. Each bay has its own AHCI
+controller at 6 Gb/s (about 600 MB/s), on a 40 Gb/s link (about 5,000 MB/s).
+`system_profiler SPSerialATADataType SPThunderboltDataType` shows the layout.
+All 8 drives reading at about 190 MB/s together would use about 1,500 MB/s,
+roughly 30% of the link. So scrubbing several drives in parallel would not be
+limited by bandwidth, if that is ever wanted.
+
+Measured on 2026-09-21 with `scrub --jobs N` (docs/parallel-scrub.md), Scrubber's
+own read path (8 MB `File#read` chunks, `F_NOCACHE`), cold files, all in the
+ThunderBay 8:
+
+| drives at once | per-drive MB/s | aggregate MB/s |
+|---|---|---|
+| 1 | 169-191 | ~180 |
+| 2 | 169, 202 | ~350 |
+| 4 (7-9 GB files each) | 184, 180, 173, 172 | **641** (90% of the sum of solo speeds) |
+
+Re-measured on 2026-09-21 against the real ThunderBay 8 fleet (all 8 active
+drives, `--jobs 4`/`--jobs 8 --for 3m`, resuming from wherever each drive's
+last check left off - so file sizes and cache state vary by drive):
+
+| drives at once | per-drive MB/s | aggregate MB/s |
+|---|---|---|
+| 4 | 205, 181, 194, 202 | ~782 |
+| 8 | 120, 154, 147, 147, 162, 135, 172, 137 | **~1175** (6.5x one drive; confirmed with `iostat -w`) |
+
+8-way scales as well as 4-way did, with the enclosure's bandwidth headroom
+above easily covering it. CPU is never the limit (see "Hash speed" below).
+
+## `easy_sync benchmark` (sequential write/read per drive)
+
+Checked on `jbod-test-1` on 2026-09-22: a 1.5 GB test file wrote at
+86-99 MB/s and read at ~163 MB/s. That is close to scrub's ~150 MB/s on the
+same drive, so the page cache was bypassed. Ctrl-C removed the test file.
+
+First real-fleet `benchmark --all`, 2026-09-22 14:54-15:08, 8 GB file,
+write/read in MiB/s (as `benchmark` and `scrub` report):
+
+| drive | write | read |
+|---|---|---|
+| backup-01-8tb | 224 | 247 |
+| backup-07-6tb | 180 | 196 |
+| backup-03-8tb | 158 | 163 |
+| backup-04-8tb | 136 | 154 |
+| backup-02-6tb | 137 | 142 |
+| backup-06-8tb | 130 | 136 |
+| backup-08-2tb | 107 | 123 |
+| backup-05-3tb | 106 | 112 |
+
+Each drive had one run then; a SLOWER flag needs 3 earlier runs. The 8 GB
+table under "Drives are not the limit" doesn't say whether it used MB or
+MiB (~5% apart), so compare the two loosely.
+
+## Hash speed (Apple Silicon)
+
+SHA-256 2514 MB/s, SHA-1 2486, MD5 763. Apple Silicon speeds up SHA but not
+MD5. All of these are faster than any drive can read.

data/docs/tripwire.md

@@ -0,0 +1,232 @@
+# Tripwire: stop a sync that would overwrite the backup wholesale (built, report-only)
+
+Built 2026-09-24 (`Jbod::Tripwire`, `Mirror#check`, Runner phase 1b). It
+ships **report-only** (`tripwire_enforce: false`, Rollout step 1). See
+"As built" at the end for where the code differs from this design, and for
+what was checked on real hardware.
+
+## The gap
+
+A mirror keeps one copy. If ransomware or a bad app rewrites files on the
+NAS, the next `sync` copies the damage over the good backup. Nothing
+covers that today:
+
+- The deletion grace period only protects files that disappear. rsync
+  still overwrites files that changed.
+- `scrub` trusts the NAS. It treats a new size or mtime as a legitimate
+  re-sync and makes the new hash the baseline, and its repair path fetches
+  the file again from the NAS (docs/integrity-scan.md, "What is already
+  covered").
+- Synology snapshots and Backblaze's 1-year version history can recover
+  after the fact. But Backblaze uploads from the drives, so every
+  overwritten file on a drive also becomes the current version offsite.
+
+The tripwire stops the copy *before* it starts when a run would change far
+more existing files than the library ever changes normally. A media library
+adds files, and now and then one is replaced (a better rip, a re-tag). It
+almost never rewrites hundreds of existing files at once. Ransomware always
+does.
+
+This replaces an earlier design (keeping every replaced file on the drive
+for `grace_days`). That design was dropped: Synology snapshots and
+Backblaze history already cover recovery, and it cost drive space on every
+run. The tripwire is cheap and prevents the damage instead of storing it.
+
+## What counts as a change
+
+rsync's itemized output already classifies every file. In a dry run
+(`-n --itemize-changes`):
+
+| Line | Meaning | Counts? |
+|---|---|---|
+| `>f+++++++++ path` | new file | no: nothing on the drive is lost |
+| `>f.st...... path`, `>f..t...... path`, any other `>f` | existing file would be overwritten | **yes, replaced** |
+| `*deleting path` | on the drive, gone from the NAS | **yes, missing** |
+| `cd+++`, `.d..t`, symlinks, etc. | directories/metadata | no |
+
+Missing files count as well, because most ransomware writes
+`photo.jpg.locked` and deletes `photo.jpg`. That shows up as one new file
+plus one deletion, with nothing replaced. Counting the deletion means a
+rename counts once. Deletions are still protected by the grace period. The
+tripwire just stops the new junk being copied and says why.
+
+A file whose size and mtime stay the same is never copied by rsync's quick
+check, so it can't reach the drive. Content-only ransomware that preserves
+both is harmless to the backup.
+
+## Where it runs: one probe, moved before the copy
+
+Today each folder gets a copy pass, then a read-only deletion probe
+(`Mirror#probe`, `rsync -an --itemize-changes --delete [--delete-excluded]`).
+That probe's output already has every line the table above needs. The
+change is to run it *before* the copy instead of after it, for every
+planned folder, as a new phase between placement and copying:
+
+1. **Phase 1 (unchanged):** measure, place, record `source_inventory`.
+2. **Phase 1b, check:** for every planned folder that has synced before
+   (`sync_runs` has a successful row), run the probe with `--stats` added.
+   Keep per folder: replaced count, missing list (already parsed by
+   `parse_extraneous`), and the files-on-drive count (see Thresholds).
+   A folder placed this run has nothing on the drive yet, so it isn't checked.
+3. **Decide.** Apply the thresholds below. If the whole run trips, stop:
+   no copy phase and no purge. If only some folders trip, skip those and
+   carry on with the rest.
+4. **Phase 2, copy:** as today, minus the post-copy probe. After a
+   folder's copy succeeds, feed its phase-1b missing list to
+   `note_missing`. The old rule stays: a folder whose copy failed starts no
+   deletion clocks.
+
+**Cost:** the same two walks per folder as today, in a different order,
+so the 50 MB/s aggregate floor (docs/performance.md) should not move. The
+difference is that about 2,700 probes (~0.15 s each for an unchanged folder,
+so ~7 min) now run before the first byte is copied. Verify both numbers on
+the real fleet before merging.
+
+**Staleness:** the missing list can be up to a copy phase old (days, on a
+first sync) by the time it is recorded. That is harmless.
+`reconcile_pending` already removes a pending deletion when the file turns
+up again. A file deleted during the copy is found on the next run. Purger
+still needs `grace_days` and `grace_runs`.
+
+**Probe failure:** a folder whose check probe fails is not copied this run
+(warn, status `skipped_check_failed`). Nothing can vouch for it, and a
+failing probe over SMB almost always means the copy would fail too. This
+is a change from today, where a failed post-copy probe only warns.
+
+## Thresholds
+
+Three config keys. Defaults are guesses to be tuned from real runs (see
+Rollout):
+
+    :tripwire_run_files: 500       # whole run: stop everything at this many changed files
+    :tripwire_folder_files: 50     # one folder: needs at least this many changed files...
+    :tripwire_folder_ratio: 0.25   # ...AND at least this share of its files on the drive
+
+- **Files on the drive** for a folder = the NAS regular-file count from
+  the probe's `--stats` ("Number of files: N (reg: R, ...)"), minus new
+  files, plus missing files. No extra walk of the drive.
+- **Per folder:** trips when changed ≥ `tripwire_folder_files` AND
+  changed / files on drive ≥ `tripwire_folder_ratio`. The ratio stops a big
+  share with a routine 60-file edit from tripping. The minimum stops a
+  3-file folder with one replaced file from tripping.
+- **Whole run:** trips when the sum of changed files across all checked
+  folders ≥ `tripwire_run_files`. This is the check that catches ransomware
+  spread across thousands of single-file movie folders, none of which
+  trips on its own. Because every folder is checked before any copy
+  starts, the run stops before *anything* is overwritten.
+- `tripwire_run_files: 0` turns the tripwire off entirely (report-only
+  logging still runs).
+
+## When it trips
+
+- **Tripped folder:** not copied, no deletion clocks, no refetch of scrub
+  flags. `mark_folder_status(key, 'tripped')`. The trip is sticky for
+  free: nothing was copied, so the next run sees the same changes and
+  trips again until you accept them.
+- **Tripped run:** nothing copied, **Purger does not run** (ransomware that
+  deletes could otherwise ride out the grace period while you're away),
+  and every planned folder is marked `tripped`. `copy_state_to_drives` and
+  the dashboard write still happen, so the trip shows up.
+- **Output:** per tripped folder, the counts plus up to 10 sample paths.
+  Seeing `*.locked` or unreadable names is usually enough to tell ransomware
+  from a re-tag. `sync` exits non-zero.
+- **Dashboard:** the verdict goes red ("Sync stopped: 2,314 files would
+  change on the NAS side") above everything else. The tripped folders are
+  listed with their counts and samples. Store what the dashboard needs in
+  a `tripwire_trips` table (run_started_at, folder_path, replaced, missing,
+  files_on_drive, samples JSON, accepted_at). Add it with
+  `CREATE TABLE IF NOT EXISTS`, per the manifest rule in CLAUDE.md.
+
+## Accepting a legitimate bulk change
+
+    easy_sync sync --accept-changes              # accept every trip this run
+    easy_sync sync --accept-changes music/Jazz   # accept only these folders
+
+The acceptance applies to that run only. It is never saved, so a later
+unrelated trip still stops. `--accept-changes` with a folder that doesn't
+trip is fine (no-op). The accepted trip is still recorded (`accepted_at`
+set), so the dashboard's history shows that a big change went through and
+who allowed it (the flag).
+
+## Dry run
+
+`--dry-run` runs phase 1b in full (the probes are read-only already) and
+prints what would trip. Per CLAUDE.md, it writes nothing: no
+`tripwire_trips` row, no folder status. Running `sync --dry-run` is how you
+look before accepting.
+
+## Rollout
+
+1. **Report-only first.** Build the probe move and the counting, and log the
+   per-folder and per-run counts on every sync, but enforce nothing
+   (`tripwire_enforce: false` until step 2). Run the real fleet for a couple
+   of weeks and read the largest legitimate counts. Set the defaults
+   comfortably above them.
+2. **Enforce.** Flip the default, and keep the report-only key for anyone
+   whose library churns.
+
+## Edge cases to test
+
+- Ransomware-style rename across 1,000 single-file folders: no folder
+  trips, the run does, nothing is copied, Purger does not run.
+- One folder with 80 of 100 files replaced: that folder trips, the others
+  sync, Purger runs for them.
+- A share-wide re-tag of 5,000 music files: trips; `--accept-changes`
+  syncs it; the following run is quiet.
+- Root-files unit (`root_only`): its probe has no `--delete-excluded`, so
+  subfolders are never counted as missing, same as today.
+- A folder placed this run: not checked, copied normally.
+- Probe fails for one folder: that folder is skipped, the rest continue.
+- `--dry-run` with a trip: prints it, manifest byte-for-byte unchanged.
+- Interrupted during phase 1b: nothing copied, nothing recorded beyond
+  phase 1 (as today).
+- Real hardware: measure phase 1b's duration and the run's aggregate MB/s
+  against docs/performance.md. Then simulate ransomware on a scratch
+  source (rename + rewrite a few hundred files) against `jbod-test-1`.
+
+## As built
+
+Differences from the design above, all found while building it:
+
+- **Files already pending don't count again.** A file missing from the NAS
+  stays on the drive for `grace_days`, and every run's probe reports it
+  again. As designed, an accepted bulk delete or rename would trip again on
+  every run until the purge, about a week later. Found live on `jbod-test-1`.
+  `Check#known` holds the folder's `pending_deletions` paths, and only
+  newly missing files count.
+- **Excluded junk doesn't count.** `--delete-excluded` reports
+  already-copied junk (`.DS_Store`, `#recycle`) as missing. Adding a pattern
+  to `exclude_folders` would otherwise trip on the next run. Paths with any
+  component matching `exclude_folders` are left out of the count. They are
+  still fed to the purge as before.
+- **"Synced before"** means the folder's `last_synced_at` is set. It is only
+  set by a successful sync_run, the same condition as in the design.
+- `tripwire_trips` has a `scope` column: `folder` when the folder tripped on
+  its own, `run` when it only counted towards a run-wide trip.
+- `--accept-changes FOLDER` also leaves that folder out of the run total. A
+  name that isn't in the sync gets a warning.
+- Dashboard: an unaccepted trip from the latest run shows as a critical
+  issue plus a Tripwire section (counts and sample paths). It stays until a
+  later run syncs. A stopped run also replaces the headline ("Sync stopped:
+  N files would change on the NAS side"). Accepted trips are listed under
+  "Accepted bulk changes". Folder statuses: `tripped`, `skipped_check_failed`.
+- A stopped run still prints and records the pending-deletion count. Only
+  the purge is skipped.
+
+Checked on `jbod-test-1` (2026-09-24, scratch config, enforce on,
+200-file album + ten single-file movie folders):
+
+| Step | Result |
+|---|---|
+| 120 album files rewritten as `*.locked`, originals removed; `sync --dry-run` | prints the trip with 10 samples, exit 0, manifest `.dump` identical |
+| same, `sync` | album held back (`tripped`), other 10 synced, exit 1, no `.locked` file on the drive, no deletion clock started |
+| `sync --accept-changes tripsrc/Album` | copied, 120 pending deletions, trip recorded with `accepted_at` |
+| `tripwire_run_files: 8`, one file rewritten in each movie folder | whole run stopped: nothing copied or purged, all 11 `tripped`, dashboard headline "Sync stopped: 10 files would change" (the 120 already-pending files not counted), exit 1 |
+| `sync --accept-changes`, then a plain `sync` | copied; the following run reports 0 changed files |
+
+**Still to do before enforcing (Rollout):** measure phase 1b's duration on
+the real fleet (~2,700 probes, estimated ~7 min) and confirm the sync
+aggregate stays at or above 50 MB/s (docs/performance.md). Then read the
+largest legitimate counts from a couple of weeks of logs (`grep -A3
+'tripwire would have tripped' ~/.easy_sync/logs/*` and each run's `Checked in`
+line) before setting `tripwire_enforce: true`.