joplin-plugin-incremental-backup 1.0.3 → 1.0.5

package/README.md

@@ -4,7 +4,7 @@ A Joplin desktop plugin that creates **deduplicated incremental snapshots** of y
 
 Each snapshot is a complete, self-contained copy of your data. Consecutive snapshots share unchanged files via hardlinks (rsnapshot-style), so storing dozens of snapshots costs only a fraction of the space of full copies. If hardlinks aren't supported on your backup drive (e.g. some network shares or FAT/exFAT drives), the plugin transparently falls back to plain copies.
 
-> **Status:** in development. The plugin requires Joplin 3.7+ and desktop only — Joplin's mobile apps do not support plugins.
+> **Status:** in development. The plugin requires Joplin 3.2.11+ and desktop only — Joplin's mobile apps do not support plugins.
 
 ## What it does
 
@@ -13,8 +13,9 @@ Each snapshot is a complete, self-contained copy of your data. Consecutive snaps
 - **Content-addressed matching** — files are matched by SHA-256 hash, so even renamed or moved notes don't consume extra space.
 - **Retention pruning** — keeps a configurable number of snapshots (default: 5) and automatically deletes the oldest ones. Deleting an old snapshot never damages newer ones, because hardlinked files are only removed when no snapshot references them.
 - **Skip unchanged backups** — if nothing has changed since the last snapshot (notes, attachments, notebooks, tags, or which notes carry which tags), the backup is skipped entirely (can be disabled).
-- **Integrity verification** — every snapshot includes a `manifest.json` with the size and SHA-256 hash of every file. A verify command re-checks the whole snapshot and reports any missing or corrupted files.
+- **Integrity verification** — every snapshot includes a `manifest.json` with the size and SHA-256 hash of every file. A verify command re-checks the whole snapshot and reports any missing or corrupted files, and warns about files the manifest doesn't list. While a verification runs, backups can still run, but retention pruning skips the snapshot being verified (it is pruned by a later run instead). A verification started while a backup is running says so and asks to be run again once it finishes.
 - **Crash-safe** — each backup run stages its files in a unique temporary directory (e.g. `_joplin_active_20261021T090000_12345_abcdef`) that is atomically renamed into place when complete. An exclusive lock prevents overlapping runs; the lock's mtime is refreshed periodically so a long export is never mistaken for an abandoned one. Stale locks are recovered atomically (rename-steal), and leftover staging data is cleaned up on startup unless another backup holds a fresh lock.
+- **Optional local staging** — with **Stage the export on this computer** on (under Advanced), the export goes to a temporary folder in the Joplin profile instead of the backup drive. Unchanged files are then hardlinked from the previous snapshot and only new or changed files are copied to the drive, so a daily backup of a large vault writes only what changed. See [Local staging](#local-staging).
 - **Disk-space pre-flight** — checks available space before starting a backup and aborts the run if the backup drive is below the minimum free-space threshold, so a full disk can't produce a truncated snapshot. (If free space can't be measured on a platform, the backup proceeds with a warning.)
 - **Export warnings surfaced** — if Joplin's exporter skips any item (e.g. still-encrypted notes), the warnings are logged and shown in the backup dialog so an incomplete snapshot isn't mistaken for a complete one.
 - **Export completeness check** — after each export, the plugin compares the note/folder/resource counts from Joplin's database against the exported files. Missing items (often due to still-encrypted content) are reported as warnings.
@@ -29,6 +30,7 @@ Each snapshot is a complete, self-contained copy of your data. Consecutive snaps
    - **Backup interval (hours)** — `0` disables automatic backups.
    - **Snapshots to keep** — oldest snapshots beyond this count are pruned.
    - **Skip if nothing changed** — on by default.
+   - **Stage the export on this computer** (Advanced) — off by default. See [Local staging](#local-staging).
 4. Backups now run automatically on the interval. You can also trigger one any time via **Tools > Create incremental backup**, and check snapshot health via **Tools > Verify incremental backup**.
 
 ### Backup layout
@@ -44,13 +46,35 @@ Each snapshot is a complete, self-contained copy of your data. Consecutive snaps
 ├── 20261021T090000/              ← next snapshot (unchanged files hardlinked)
 ├── _joplin_active_20261021T090000_12345_abcdef/  ← staging dir (only while running)
 ├── latest.txt                    ← name of the newest snapshot
-└── .backup.lock                  ← present only while a backup is running
+├── .joplin-incremental-backup    ← marker: this is a backup directory
+├── .backup.lock                  ← present only while a backup is running
+└── .verifying.<snapshot>.<id>    ← present only while a snapshot is being verified
 ```
 
 Notes, notebooks, tags and note-tags are each written as a single `<id>.md` file directly in the snapshot root (this mirrors Joplin's Raw export exactly); only attachments live under `resources/`.
 
 The staging directory is unique per run (embedding the start time, process id, and a random suffix) so two concurrent runs can never write into the same directory. On startup, any leftover staging directories older than the stale-lock threshold are cleaned up — unless another Joplin instance holds a fresh lock, in which case they are left untouched.
 
+### Disconnected backup drives
+
+The first backup to a directory creates it and writes a marker file, `.joplin-incremental-backup`. After that, the plugin refuses to back up to that directory if the marker is missing, instead of recreating the directory. This protects against an unmounted drive: without the check, a path like `/mnt/backup/joplin` would be recreated on the internal disk, the snapshots would go there, and they would be hidden under the mount point once the drive came back.
+
+If you moved or deleted the backups on purpose, create an empty `.joplin-incremental-backup` file in the directory, or choose a different directory.
+
+When upgrading from a version without the marker, the first backup adopts your existing backup directory. If that directory does not exist at that point but backups have been made before, the plugin assumes the drive is not mounted and refuses rather than creating it. If it really is a new location, create the directory yourself and the next backup will use it.
+
+### Local staging
+
+By default each run exports the whole vault into the staging directory **on the backup drive**, hashes it there, and then replaces unchanged files with hardlinks. That saves space but not writes: a 5 GB vault backed up daily writes about 5 GB to the drive every day, even if one note changed. On a USB stick or a slow drive that also makes every run slow.
+
+With **Stage the export on this computer** on, a run instead:
+
+1. Exports into `plugin-data/<plugin id>/staging/_joplin_active_…` in the Joplin profile, and hashes the files there.
+2. Builds the staging directory on the backup drive by hardlinking files that are identical to the previous snapshot and copying only new or changed files.
+3. Renames the staging directory into place and deletes the local export, as before.
+
+The trade-off is temporary local disk space about the size of the vault. Before exporting, the plugin checks that the local disk has room for the previous snapshot plus the usual minimum free space. If it doesn't, the folder can't be created, or the local export fails (for example because the disk fills up on a first backup), the run stages on the backup drive as before and reports a warning. If the backup directory is on the same disk as the Joplin profile, staging locally would only add writes, so the run stages on the backup drive without a warning. Local staging left behind by a crash is removed the next time Joplin starts.
+
 ### Scheduler catch-up behaviour
 
 When Joplin starts, the plugin checks when the last successful backup finished:
@@ -62,7 +86,7 @@ When Joplin starts, the plugin checks when the last successful backup finished:
 
 If strict 24 h spacing matters more than avoiding an export at launch, this is a one-line change in `src/scheduler.ts` (replace the "next interval boundary" branch with `now + intervalMs`).
 
-After a successful backup, the next one is scheduled exactly one interval later. After a failed backup, the next attempt is scheduled after 5 minutes rather than waiting a full interval.
+After a successful backup, the next one is scheduled exactly one interval later. After a failed backup, the next attempt is scheduled sooner than a full interval, backing off while failures repeat: 5 minutes, then 15 minutes, then 1 hour, then the regular interval (never longer than the interval). Retries after a failure run quietly: only the first failure in a row shows toasts, and later ones are logged and recorded in **Show last backup result**. A successful backup (including a manual one) or a change to the backup directory or interval starts over. If another backup is already running, the scheduled one is retried after 5 minutes and does not count as a failure.
 
 ### Encrypted items
 
@@ -90,11 +114,15 @@ This makes the backups a safe, vendor-independent archive: even without the plug
 
 ## Building from source
 
+Needs Node.js 22.12 or later.
+
 ```bash
 npm install
 npm run dist     # builds the plugin into publish/
 ```
 
+The project started from the `generator-joplin` scaffold, but don't refresh it with the generator: its template pins older dev dependencies and would overwrite `tsconfig.json`. Update dependencies directly with npm.
+
 Other scripts:
 
 ```bash
@@ -103,6 +131,22 @@ npm run format:check  # check formatting (CI)
 npm run updateVersion # sync version from package.json into the manifest
 ```
 
+### End-to-end tests
+
+`e2e/` runs the built plugin inside a real Joplin desktop app with a throwaway profile. The tests add notes through Joplin's Data API, let the scheduler take the backups, and check the snapshots on disk (Raw layout, manifest hashes, hardlinks between snapshots, change summary, skipped runs). The scenario runs twice: once staging on the backup drive, and once with **Stage the export on this computer** on. For the second run the backups go on another filesystem (`/dev/shm` by default, override with `JOPLIN_E2E_OTHER_DEVICE`), since local staging is skipped when both are on the same disk.
+
+A second scenario (`e2e/startup.e2e.test.ts`) covers what happens at startup: automatic backups paused without a usable directory, the upgrade guard for a missing drive, cleanup after a crashed run (stale lock and staging directories), backing off from another instance's lock, **Only back up when something changed** turned off, and the schedule not running a backup when it is turned off or not yet due. It also checks how moved notes, renamed notebooks, trashed notes and new attachments show up in a snapshot.
+
+```bash
+npm run fetch-joplin          # download + verify + extract Joplin (default 3.7.22) into .cache/joplin/
+npm run test:e2e              # builds the plugin, then runs e2e/ in a visible Joplin window
+npm run test:e2e:headless     # same, on a virtual display (as in CI); needs xvfb
+```
+
+For headless runs, install Xvfb first (`sudo apt-get install xvfb`). The harness unsets `WAYLAND_DISPLAY` for Joplin, so on WSLg or a Wayland desktop no window appears.
+
+CI runs the suite on the oldest supported Joplin (3.2.11) and on 3.7.22. To test another version locally, run `npm run fetch-joplin -- 3.2.11`; it becomes the default for `npm run test:e2e`. Set `JOPLIN_BIN` to use a different Joplin executable, `JOPLIN_E2E_WORKDIR` to choose where the profile, logs and backups go, and `JOPLIN_E2E_KEEP=1` to keep them after a passing run. They are always kept when a test fails. The tests never touch your own Joplin profile, and they run alongside a Joplin you already have open.
+
 For information on publishing the plugin, see [GENERATOR_DOC.md](./GENERATOR_DOC.md).
 
 ## License

package/package.json

@@ -1,16 +1,22 @@
 {
   "name": "joplin-plugin-incremental-backup",
-  "version": "1.0.3",
+  "version": "1.0.5",
   "scripts": {
     "dist": "webpack --env joplin-plugin-config=buildMain && webpack --env joplin-plugin-config=buildExtraScripts && webpack --env joplin-plugin-config=createArchive",
     "prepare": "npm run dist",
     "test": "vitest run",
     "test:coverage": "vitest run --coverage",
-    "lint": "oxlint src tests",
+    "pretest:e2e": "npm run dist",
+    "test:e2e": "vitest run --config vitest.e2e.config.ts",
+    "test:e2e:headless": "xvfb-run -a npm run test:e2e",
+    "fetch-joplin": "scripts/fetch-joplin.sh",
+    "lint": "oxlint src tests e2e",
     "format": "prettier --write .",
     "format:check": "prettier --check .",
-    "updateVersion": "webpack --env joplin-plugin-config=updateVersion",
-    "update": "npm install -g generator-joplin && yo joplin --node-package-manager npm --update --force"
+    "updateVersion": "webpack --env joplin-plugin-config=updateVersion"
+  },
+  "engines": {
+    "node": ">=22.12"
   },
   "license": "MIT",
   "keywords": [
@@ -24,17 +30,17 @@
     "@types/node": "^22.20.5",
     "@vitest/coverage-v8": "^5.0.5",
     "chalk": "^4.1.0",
-    "copy-webpack-plugin": "^11.0.0",
-    "fs-extra": "^10.1.0",
-    "glob": "^8.0.3",
+    "copy-webpack-plugin": "^14.0.0",
+    "fs-extra": "^11.4.1",
+    "glob": "^14.0.4",
     "memfs": "^4.80.0",
     "oxlint": "^1.89.0",
     "prettier": "3.9.10",
-    "tar": "^6.1.11",
-    "ts-loader": "^9.3.1",
-    "typescript": "^4.8.2",
+    "tar": "^7.5.22",
+    "ts-loader": "^9.6.2",
+    "typescript": "^6.0.3",
     "vitest": "^5.0.5",
-    "webpack": "^5.74.0",
-    "webpack-cli": "^4.10.0"
+    "webpack": "^5.112.2",
+    "webpack-cli": "^7.3.1"
   }
 }

package/publish/io.github.raytray.joplin-incremental-backup.json

@@ -1,8 +1,8 @@
 {
 	"manifest_version": 1,
 	"id": "io.github.raytray.joplin-incremental-backup",
-	"app_min_version": "3.7",
-	"version": "1.0.3",
+	"app_min_version": "3.2.11",
+	"version": "1.0.5",
 	"name": "Incremental Backup",
 	"description": "Deduplicated incremental snapshots of your Joplin data using the native Raw export format.",
 	"author": "raytray",
@@ -20,6 +20,6 @@
 	"screenshots": [],
 	"icons": {},
 	"promo_tile": {},
-	"_publish_hash": "sha256:cfeee5c68acb2a7ddf31f3f1981dea03ff960cefde01932ef16ccaedc48611d2",
-	"_publish_commit": "main:24c6b70de862802fe79d3c67b2afa903b4662630"
+	"_publish_hash": "sha256:85d623a1ab68d6c28d340e1b7319e547f6ac7517083102c447c0eab44fd815a0",
+	"_publish_commit": "main:7330557655bf68105b530da5d1829d75e77c2c83"
 }