@wipcomputer/wip-ldm-os 0.4.86 → 0.4.87-alpha.2

package/docs/backup/TECHNICAL.md

@@ -1,112 +1,129 @@
 # Backup: Technical Details
 
-## Config Schema
-
-All backup settings are in `~/.ldm/config.json`. The backup script reads these at runtime.
-
-```json
-{
-  "org": "wipcomputerinc",
-  "paths": {
-    "workspace": "~/wipcomputerinc",
-    "ldm": "~/.ldm",
-    "claude": "~/.claude",
-    "openclaw": "~/.openclaw",
-    "icloudBackup": "~/Library/Mobile Documents/com~apple~CloudDocs/wipcomputerinc-icloud/backups"
-  },
-  "backup": {
-    "keep": 7,
-    "includeSecrets": false
-  }
-}
-```
+## Installed components
 
-| Key | Type | Default | Description |
-|-----|------|---------|-------------|
-| `paths.workspace` | string | required | Root workspace directory to back up |
-| `paths.icloudBackup` | string | optional | iCloud destination for offsite copies |
-| `backup.keep` | number | 7 | Days of backups to keep before rotation |
-| `backup.includeSecrets` | boolean | false | Whether to include `~/.ldm/secrets/` |
-| `org` | string | required | Used as prefix in iCloud tar filenames |
-
-## Script Location
-
-- **Source:** `scripts/ldm-backup.sh` in the wip-ldm-os-private repo
-- **Deployed to:** `~/.ldm/bin/ldm-backup.sh`
-- **Deployed by:** `deployScripts()` in `bin/ldm.js`, called during both `ldm init` and `ldm install`
-- **Restore script:** `scripts/ldm-restore.sh` deployed to `~/.ldm/bin/ldm-restore.sh`
-
-All `.sh` files in the repo's `scripts/` directory are deployed to `~/.ldm/bin/` on every `ldm install`. This means script fixes land automatically on the next update without requiring a full `ldm init`.
-
-## LaunchAgent
-
-**Label:** `ai.openclaw.ldm-backup`
-**Plist source:** `shared/launchagents/ai.openclaw.ldm-backup.plist`
-**Deployed to:** `~/Library/LaunchAgents/ai.openclaw.ldm-backup.plist`
-
-```xml
-<key>StartCalendarInterval</key>
-<dict>
-  <key>Hour</key>
-  <integer>3</integer>
-  <key>Minute</key>
-  <integer>0</integer>
-</dict>
-```
+| Source | Installed path | Purpose |
+|---|---|---|
+| `scripts/ldm-backup.sh` | `~/.ldm/bin/ldm-backup.sh` | Stable shell entry point |
+| `scripts/ldm-backup-engine.py` | `~/.ldm/bin/ldm-backup-engine.py` | Policy, planning, capture, validation, state, and retention engine |
+| `scripts/ldm-backup-policy.json` | `~/.ldm/bin/ldm-backup-policy.json` | Canonical source and exclusion policy |
+
+The package bin manifest owns all three files. `ldm install` deploys code only. It does not run a backup, create a destination, change the LaunchAgent, enable a schedule, or write a containment marker.
+
+The engine runtime is Python 3 standard library only. The shell entry point checks for `python3` and emits the structured failure code `python3-missing` before execution when it is unavailable. No pip package or third-party Python dependency is introduced.
+
+## Canonical policy schema
+
+Policy schema version 1 contains:
+
+- `rule_version`
+- Explicit `evidence_status_values`
+- Disk floors and start reserve
+- Source ID, root, relative path, safe display path, destination, type, classification, required flag, and action
+- Block reason when encryption or reconstruction proof is missing
+- Exclusion pattern, reason, reconstruction source, and rule version
+
+Legal actions are `capture`, `blocked`, `manifest`, and `exclude`. Legal types are `file`, `tree`, and `sqlite`. The placeholder string `complete-or-deferred-after-write-failure` is rejected by policy and runtime state validation.
+
+The production policy has an explicit ID for every controlling Slice 1 source row. A fixture asserts exact set equality, not subset membership, so either an omission or an unreviewed addition fails loudly.
+
+`manifest` requires a named evidence generator. The implemented `path-inventory` generator records the source type, exact members, member count, logical bytes, and SHA-256 of the serialized inventory. A missing source, generator failure, malformed record, or missing required evidence prevents completion. Required evidence without a complete Slice 2 generator uses `blocked`, including repo-aware Git refs and worktree state, macOS system facts, package inventories, and out-of-band 1Password bootstrap proof.
+
+## Planning and dry run
+
+Planning resolves the configured roots, expands each policy entry, selects files using its exclusion rules, and computes checksums and logical sizes. `estimated_new_bytes` is the exact logical payload plus the serialized manifest and completion marker for an unchanged source set. The disk reserve is evaluated against that plan-time value.
 
-The plist uses `{{HOME}}` placeholders that are replaced at deploy time by `ldm init`.
+`--dry-run` stops after planning and preflight. It does not acquire the persistent lock or call any atomic writer. JSON output is available with `--dry-run --json` for reviewers and tests.
 
-**Logs:** stdout and stderr both go to `~/.ldm/logs/backup.log`.
+Configuration loading is fail-closed. A missing config permits documented defaults. An unreadable file, malformed JSON, a non-object root, or non-object `paths` or `backup` field produces `config-invalid`.
 
-**No Full Disk Access (FDA):** The LaunchAgent runs at 3:00 AM without FDA. Some paths (like `~/Library/Messages/`) are inaccessible without FDA. The target is to move the trigger to midnight via LDMDevTools.app (which has FDA) once the PID error is resolved.
+## Mutation-during-run contract
 
-### Dead Triggers (Cleaned Automatically)
+Planning is not treated as a filesystem freeze. Immediately before each source capture, the engine rebuilds that source observation from the same canonical policy. Plan-to-capture changes are allowed and recorded through planned size, modification time, checksum, capture-time size, capture-time modification time, destination checksum, and `changed_since_plan`.
 
-The `cleanDeadBackupTriggers()` function in `ldm.js` removes old competing triggers on every `ldm init`:
-- Old cron entries referencing `LDMDevTools.app`
-- `com.wipcomputer.daily-backup` LaunchAgent
-- OpenClaw `backup-verify` cron entries
+For regular files and trees, the capture-time observation is the source correspondence proof. Destination bytes are compared to that observation, and the manifest records per-source member counts, payload bytes, and inventory fingerprints for both planning and capture. After copying the full source, the engine observes it again. Any member addition, removal, type change, size change, modification-time change, or checksum change during capture fails with `source-changed-during-capture`. Same-size content changes are therefore detected. A plan-time change does not force failure merely because an active source changed before its capture began.
 
-Only `ai.openclaw.ldm-backup` should exist.
+SQLite uses a separate contract. Python's SQLite backup API produces a transactionally consistent destination snapshot. The manifest labels these records `sqlite-backup-api-transaction-snapshot`; it does not claim that the destination checksum equals a raw live database file checksum.
 
-## Rotation Logic
+Before copying each source, the engine refreshes size and modification-time estimates for every capture source still pending without hashing each remaining tree. It then performs the full member and checksum observation for the source about to be copied. The engine recalculates the reserve from bytes already staged, refreshed remaining payload, projected manifest and completion metadata, current available space, and the required reserve. Metadata projection updates only changed source and evidence records instead of rebuilding the complete manifest for every source. A failed recheck raises `capture-reserve-gate` before the changed source is copied and records the updated estimate and reserve in `result.json`.
 
-The backup script handles rotation after a successful backup:
+Manifest actions are gated before any source path is inspected or walked. An entry without an implemented generator is reported as blocked immediately. This prevents generator-less entries from hashing large trees or failing on special files that the unused source path happens to contain.
 
-1. List all dated directories in `~/.ldm/backups/` (format: `YYYY-MM-DD--HH-MM-SS`)
-2. Sort by name (which sorts chronologically)
-3. Skip any directory containing a `.pinned` marker file
-4. Delete directories beyond the `keep` count (oldest first)
-5. Same rotation logic applies to iCloud tars at `paths.icloudBackup`
+## Run lifecycle
 
-**Pinning:** `ldm backup --pin "reason"` creates a `.pinned` file in the latest backup directory. Pinned backups are never rotated.
+1. Create the lock with `O_CREAT | O_EXCL`.
+2. Reject an active lock. Report a dead-PID lock as stale without replacing it.
+3. Build the plan from the canonical policy.
+4. Atomically write `schedule.json` and `attempt.json` before staging.
+5. Fail preflight before any staging directory when a required gate, destination, or reserve check fails.
+6. Create `.in-progress-<snapshot-id>`.
+7. Generate and validate required manifest evidence.
+8. Re-observe each capture source and all remaining capture sources.
+9. Recalculate the disk reserve before copying the current source.
+10. Capture SQLite databases with Python's SQLite backup API and run `PRAGMA quick_check` on each destination.
+11. Re-observe regular-file and tree sources after capture and fail if they changed during capture.
+12. Atomically write `manifest.json`.
+13. Validate every required capture record and manifest-evidence record.
+14. Atomically write `completion.json` last.
+15. Verify completion proof, then atomically promote staging.
+16. Write `result.json` and `success-local.json`.
+17. Rotate only eligible verified snapshots.
 
-## iCloud Offsite Details
+Failures do not promote staging and do not print `Backup complete`. A process crash can leave only an unmistakable in-progress directory and stale lock.
 
-After the local backup completes:
+## Manifest fields
 
-1. Tar + gzip the entire dated backup directory
-2. Filename format: `<org>-<machine>-<timestamp>.tar.gz`
-3. Copy to `paths.icloudBackup` (from config.json)
-4. Apply the same rotation (keep N, skip pinned)
-5. iCloud syncs the file to all devices automatically
+Each captured member records:
 
-The iCloud path must exist. The script does not create it. `ldm init` does not create it either. Create it manually if it does not exist.
+- Source ID and resolved source path
+- Relative destination
+- Type
+- Logical size
+- Modification time in nanoseconds
+- SHA-256 checksum
+- Plan-time size, modification time, and checksum where meaningful
+- Whether the source changed between planning and capture
+- The source consistency contract
 
-## SQLite Safety
+The manifest also records schema version, UTC snapshot ID, tool version, policy rule version, timestamps, planned and captured payload bytes, required manifest evidence, exclusions, and terminal result. Each exclusion records source ID, path or pattern, reason, reconstruction source, rule version, and matched count.
 
-SQLite files are backed up using `sqlite3 .backup`, not `cp`. This ensures a consistent snapshot even if the database is being written to. The script checks for the `sqlite3` binary and skips database backup with a warning if it is not found.
+`completion.json` records snapshot ID, `status: complete`, completion time, and manifest checksum. Its presence alone is insufficient unless the manifest checksum and terminal result validate.
 
-Files backed up this way:
-- `~/.ldm/memory/crystal.db`
-- `~/.openclaw/memory/main.sqlite`
-- `~/.openclaw/memory/context-embeddings.sqlite`
+## Structured state
 
-## Excludes
+All state files use same-directory temporary files, `fsync`, and atomic rename.
+
+| File | Meaning |
+|---|---|
+| `schedule.json` | Schedule ID, timezone, expected slot, containment status, and marker-derived evidence status or `unknown` |
+| `attempt.json` | Run ID, PID, tool version, start time, and preflight values |
+| `result.json` | Terminal result, failure code, estimates, and per-tier status |
+| `success-local.json` | Last verified internal success |
+| `success-vault.json` | Reserved for Slice 3 |
+| `success-icloud.json` | Reserved for Slice 4 |
+
+These records are diagnostic. They do not replace snapshot completion proof.
+
+## Retention
+
+Retention considers a directory eligible only when:
+
+- It is not named `.in-progress-*`.
+- `manifest.json` has `terminal_result: success`.
+- `completion.json` has `status: complete`.
+- The completion marker matches the manifest SHA-256.
+- `.pinned` is absent.
+
+The engine keeps the configured number of unpinned eligible snapshots plus every pinned snapshot. Unverified directories are preserved for later disposition.
+
+## Fixture isolation
+
+Tests set `LDM_BACKUP_REQUIRE_FIXTURE=1` and provide a unique `LDM_BACKUP_FIXTURE_ROOT` whose basename starts with `ldm-backup-fixture-`. Before planning or writing, the engine proves that all injected home, LDM, OpenClaw, Claude, Codex, workspace, destination, state, iCloud substitute, and lock paths resolve beneath that root.
+
+The entire fixture suite runs with:
+
+```bash
+npm run test:backup-engine
+```
 
-The workspace tar excludes:
-- `node_modules/` ... reconstructable via npm install
-- `.git/objects/` ... reconstructable via git fetch
-- `backups/` ... avoids recursive backup
-- `_trash/` ... already deleted content
-- `*.tar.gz` ... avoids backing up old backup archives
+It covers success, dry-run parity, exact policy IDs, required manifest evidence and generator failures, generator-less manifest short-circuiting over a tree containing a Unix socket, plan-to-capture size and same-size changes, capture-time reserve success and refusal, fatal during-capture mutation, concurrent and stale locks, missing and blocked required sources, SQLite failure, simulated ENOSPC, preflight failure, crash staging, atomic state, completion proof, configuration failures, UTC IDs, evidence status, retention, and path escape refusal.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wipcomputer/wip-ldm-os",
-  "version": "0.4.86",
+  "version": "0.4.87-alpha.2",
   "type": "module",
   "description": "LDM OS: identity, memory, and sovereignty infrastructure for AI agents",
   "engines": {
@@ -18,7 +18,7 @@
   "scripts": {
     "build:bridge": "cd src/bridge && npm install && npx tsup core.ts mcp-server.ts cli.ts openclaw.ts --format esm --dts --clean --outDir ../../dist/bridge",
     "build": "npm run build:bridge",
-    "prepublishOnly": "npm run build:bridge && npm run validate:bin-manifest && npm run test:install-prompt-policy && npm run test:readme-install-prompt && npm run test:ldm-status-timeout && npm run test:ldm-status-concurrency && npm run test:legacy-npm-sources-migration",
+    "prepublishOnly": "npm run build:bridge && npm run validate:bin-manifest && npm run test:install-prompt-policy && npm run test:readme-install-prompt && npm run test:ldm-status-timeout && npm run test:ldm-status-concurrency && npm run test:legacy-npm-sources-migration && npm run test:ldm-install-bin-shim && npm run test:backup-engine",
     "validate:bin-manifest": "node scripts/validate-bin-manifest.mjs",
     "test:skill-frontmatter": "node scripts/test-skill-frontmatter.mjs",
     "test:install-prompt-policy": "node scripts/test-install-prompt-policy.mjs",
@@ -50,7 +50,9 @@
     "test:boot-hook-registration": "node scripts/test-boot-hook-registration.mjs",
     "test:doctor-hook-dedupe": "node scripts/test-doctor-hook-dedupe.mjs",
     "test:boot-dir-truth": "node scripts/test-boot-dir-truth.mjs",
-    "test:deploy-hook-ownership": "node scripts/test-deploy-hook-ownership.mjs"
+    "test:deploy-hook-ownership": "node scripts/test-deploy-hook-ownership.mjs",
+    "test:doctor-commit-deployed": "node scripts/test-doctor-commit-deployed.mjs",
+    "test:backup-engine": "PYTHONDONTWRITEBYTECODE=1 python3 scripts/test-backup-engine.py"
   },
   "claudeCode": {
     "hook": {
@@ -72,6 +74,17 @@
         "source": "scripts/ldm-backup.sh",
         "purpose": "daily backup; LaunchAgent at 03:00"
       },
+      {
+        "name": "ldm-backup-engine.py",
+        "source": "scripts/ldm-backup-engine.py",
+        "purpose": "policy-driven backup engine"
+      },
+      {
+        "name": "ldm-backup-policy.json",
+        "source": "scripts/ldm-backup-policy.json",
+        "executable": false,
+        "purpose": "canonical backup source policy"
+      },
       {
         "name": "ldm-restore.sh",
         "source": "scripts/ldm-restore.sh",