ruborg 0.8.1 → 0.9.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 74bbfee5ede7b87c5a99ddec057c90430ee66a5667e6a933455baf4478f569d2
-  data.tar.gz: f7ddb9aa319573d39888fad0933de0ce99fcc08069eae459550605eba9ff1082
+  metadata.gz: 8f3c9f239a72ec2f321419eb6f4349c25fe4cec4ca5174c156e2f1d3fe84ec68
+  data.tar.gz: 4930dbbc8dc2d4e2040628a8c57e62d7bf3ac20ca58a0e7e363a649f6f8d6546
 SHA512:
-  metadata.gz: 285d04844f53fc87e5af8b28d24fc9dfe410e1610e2c79845ea0b1a5ea8a7d60ed76c5df2f667068b0c2b55269870d9af2c30f64ed9f8efc63312d1f99acd60a
-  data.tar.gz: e72be5e589955c1e3832a5f38236d485d61db23a8f60ece3a147a6ecf21eb1b66fd9e1eddbc77c4720c4dc69f675491eb4c7eb5dc22b93b2a1856757af4e7a96
+  metadata.gz: ba9bd867f83d24c88abf435f4b64368b9c414080290747f107a6b5e94db6e82ddba0a59b0f700a4625e689b54724aefceecccc56f5e51f3fc8271be74601bc5e
+  data.tar.gz: e95dd232d73c7c63968c0105170928d0ab28d689342ad875a81fbc30ac56d508179a1866b476b79125fe43a85fc69cc68baef14a45cd220019be5612ce458bea

data/CHANGELOG.md

@@ -7,6 +7,91 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.9.3] - 2026-05-09
+
+### Added
+- **`ruborg lock` command**: Check for and optionally break stale Borg repository locks
+  - `ruborg lock --repository NAME` — exits 0 if no lock, exits 1 if lock detected
+  - `ruborg lock --repository NAME --break --yes` — breaks the lock via `borg break-lock`
+  - `ruborg lock --repository NAME --force --yes` — force-removes lock files directly without invoking Borg (useful when Borg itself can't run)
+  - `--break` and `--force` are mutually exclusive; both require `--yes` as a safety guard
+  - `Repository#locked?` — pure filesystem check on `lock.exclusive` / `lock.roster`, no Borg invocation or passphrase required
+  - `Repository#break_lock` — delegates to `borg break-lock`; requires Borg >= 1.4.0
+  - `Repository#force_break_lock` — direct filesystem removal of lock files/dirs; no Borg needed
+  - Status output (lock present/absent) goes to stdout for scriptability; warning messages go to `$stderr`
+- **Pre-flight lock detection during backup**: If a repository is locked when `backup` starts, ruborg waits and retries instead of failing immediately
+  - Polls every 5 seconds, prints elapsed time via the spinner
+  - Aborts with a clear error message after `lock_wait` seconds (default 300), suggesting `ruborg lock` to inspect or clear
+- **`lock_wait` config key**: Optional integer (seconds). When set, also passed as `--lock-wait` to all Borg commands so Borg itself waits for mid-operation locks. Omitting the key leaves Borg at its own default (1 second)
+- **Minimum Borg version**: Raised to 1.4.0; `break_lock` verifies this before invoking `borg break-lock`
+- **`CLI::DEFAULT_LOCK_WAIT = 300`**: Named constant for the pre-flight wait timeout
+- Fixes [#8](https://github.com/mpantel/ruborg/issues/8)
+
+## [0.9.2] - 2026-05-09
+
+### Added
+- **CLI progress display**: Real-time feedback during backup operations
+  - Named stages printed to `$stderr`: `[1/3] Verifying repository`, `[2/3] Backing up files`, `[3/3] Pruning`
+  - Animated spinner for indeterminate operations (cache loading, `borg create`, pruning)
+  - Inline progress bar for per-file backup mode: `[=========>     ] 42/120  filename.jpg`
+  - Stage count adapts to the operation: 2 stages without pruning, 3 with
+  - Degrades gracefully to plain text lines when output is piped or redirected (non-TTY)
+  - All progress output goes to `$stderr` — `--json` stdout and piped output remain clean
+  - No external gem dependencies — pure Ruby with ANSI `\r` rewrite
+  - Fixes [#6](https://github.com/mpantel/ruborg/issues/6)
+
+## [0.9.1] - 2026-05-09
+
+### Added
+- **Archive metadata cache**: New `ArchiveCache` class eliminates N+1 `borg info` calls during per-file backup runs
+  - Metadata cached in `<repo_path>.ruborg_cache.json` — sibling file to the repository
+  - Cache is shared across machines: any host with access to the repo path shares the same cache
+  - Local repos use `File::LOCK_EX` for safe concurrent reads/writes with merge-on-conflict
+  - SSH repos (`user@host:/path`, `ssh://user@host/path`) fetch/push the cache via `scp` with optimistic locking (fetch-fresh → merge → push), avoiding deadlocks on process crashes
+  - Only archives not yet in cache trigger a `borg info` call; warm runs reduce subprocess overhead from O(n) to O(new archives)
+  - Fixes [#4](https://github.com/mpantel/ruborg/issues/4)
+- **Catalog command**: New `ruborg catalog` command for fast, offline browsing of backed-up files
+  - Reads the local cache file — no `borg` subprocess calls needed
+  - `--search PATTERN` — filter entries by file path using a regex
+  - `--stats` — show aggregate statistics (total archives, unique files, total size, source dirs)
+  - `--json` — machine-readable JSON output; default is a human-friendly text table
+  - Works per-repository like all other commands (`--repository`)
+  - Supports SSH repos transparently via the same `scp`-based cache fetch
+- **Bug fix**: `ArchiveCache` now normalises all loaded metadata to symbol keys, ensuring cache hits and cache misses return identical key types
+
+## [0.9.0] - 2025-10-14
+
+### Changed
+- **Command Consolidation**: Unified validation commands for consistency
+  - `ruborg validate` renamed to `ruborg validate config` (validates YAML configuration)
+  - `ruborg check` renamed to `ruborg validate repo` (validates repository compatibility and integrity)
+  - Both commands now use consistent `validate` terminology
+  - `--verify-data` option remains available for `validate repo` to run full integrity checks
+  - Eliminates confusion between `validate` (config) and `check` (repository)
+  - Updated README with new command syntax and examples
+  - Updated all tests to use new command format
+
+### Added
+- **Skip Hash Check Option**: New `skip_hash_check` configuration option for faster per-file backups
+  - Skips expensive SHA256 content hash calculation when file already exists
+  - Trusts file path, size, and modification time matching for duplicate detection
+  - Significantly speeds up backups with many unchanged files
+  - Configurable globally or per-repository
+  - Default: `false` (paranoid mode - always verify content hash)
+  - Use case: Large directories where mtime changes are reliable (most filesystems)
+  - Example: `skip_hash_check: true` in YAML configuration
+- **Migration Help**: `ruborg check` now displays a helpful deprecation notice
+  - Shows clear message explaining the command has been renamed
+  - Provides examples of the new `ruborg validate repo` syntax
+  - Exits with error to prevent confusion
+  - Logs deprecation warning for audit trail
+- **Enhanced Version Command**: `ruborg version` now shows both Ruborg and Borg versions with path
+  - Displays Ruborg version (gem version)
+  - Displays installed Borg version and executable path
+  - Example output: `borg 1.2.8 (/usr/local/bin/borg)`
+  - Gracefully handles missing Borg installation
+  - Helps users verify both tool versions and location at a glance
+
 ## [0.8.1] - 2025-10-09
 
 ### Added

data/README.md

@@ -25,13 +25,15 @@ A friendly Ruby frontend for [Borg Backup](https://www.borgbackup.org/). Ruborg
 - 📈 **Summary View** - Quick overview of all repositories and their configurations
 - 🔧 **Custom Borg Path** - Support for custom Borg executable paths per repository
 - 🏠 **Hostname Validation** - NEW! Restrict backups to specific hosts (global or per-repository)
-- ✅ **Well-tested** - Comprehensive test suite with RSpec (294 examples, 0 failures)
+- 🔒 **Lock Management** - Detect and break stale Borg repository locks with `ruborg lock`
+- ⏳ **Lock-aware Backups** - Pre-flight lock detection with configurable wait timeout before backup
+- ✅ **Well-tested** - Comprehensive test suite with RSpec (412 examples, 0 failures)
 - 🔒 **Security-focused** - Path validation, safe YAML loading, command injection protection
 
 ## Prerequisites
 
 - Ruby >= 3.2.0
-- [Borg Backup](https://www.borgbackup.org/) installed and available in PATH
+- [Borg Backup](https://www.borgbackup.org/) >= 1.4.0 installed and available in PATH
 - [Passbolt CLI](https://github.com/passbolt/go-passbolt-cli) (optional, for password management)
 
 ### Installing Borg Backup
@@ -163,13 +165,15 @@ repositories:
 
 **Configuration Features:**
 - **Automatic Type Validation**: Configuration is validated on startup to catch type errors early
-- **Validation Command**: Run `ruborg validate` to check configuration files for errors
+- **Validation Command**: Run `ruborg validate config` to check configuration files for errors
 - **Descriptions**: Add `description` field to document each repository's purpose
 - **Hostname Validation**: Optional `hostname` field to restrict backups to specific hosts (global or per-repository)
 - **Source Deletion Safety**: `allow_remove_source` flag to explicitly enable `--remove-source` option (default: disabled)
+- **Skip Hash Check**: Optional `skip_hash_check` flag to skip content hash verification for faster backups (per-file mode only)
 - **Type-Safe Booleans**: Strict boolean validation prevents configuration errors (must use `true`/`false`, not strings)
-- **Global Settings**: Hostname, compression, encryption, auto_init, allow_remove_source, log_file, borg_path, borg_options, and retention apply to all repositories
-- **Per-Repository Overrides**: Any global setting can be overridden at the repository level (including hostname, allow_remove_source, and custom borg_path)
+- **Lock Wait Timeout**: Optional `lock_wait` (integer, seconds) — how long ruborg waits for a locked repository before aborting. Also passed as `--lock-wait` to Borg when set. Default: 300 seconds (pre-flight), Borg default: 1 second (when not configured)
+- **Global Settings**: Hostname, compression, encryption, auto_init, allow_remove_source, skip_hash_check, lock_wait, log_file, borg_path, borg_options, and retention apply to all repositories
+- **Per-Repository Overrides**: Any global setting can be overridden at the repository level (including hostname, allow_remove_source, skip_hash_check, lock_wait, and custom borg_path)
 - **Custom Borg Path**: Specify a custom Borg executable path if borg is not in PATH or to use a specific version
 - **Retention Policies**: Define how many backups to keep (hourly, daily, weekly, monthly, yearly)
 - **Multiple Sources**: Each repository can have multiple backup sources with their own exclude patterns
@@ -184,7 +188,7 @@ Ruborg automatically validates your configuration on startup. All commands check
 Check your configuration file for errors:
 
 ```bash
-ruborg validate --config ruborg.yml
+ruborg validate config --config ruborg.yml
 ```
 
 **Validation checks:**
@@ -471,20 +475,52 @@ Group: postgres
 Type: regular file
 ```
 
-### Check Repository Compatibility
+### Manage Repository Locks
+
+Borg uses lock files to prevent concurrent access. If a backup crashes, stale locks can block all subsequent operations. Use `ruborg lock` to inspect and clear them.
+
+```bash
+# Check if a repository is locked (exits 0 = no lock, 1 = locked)
+ruborg lock --repository documents
+
+# Break the lock via borg break-lock (requires Borg >= 1.4.0)
+ruborg lock --repository documents --break --yes
+
+# Force-remove lock files directly (no Borg required, last resort)
+ruborg lock --repository documents --force --yes
+```
+
+**Lock-aware backups:** When `ruborg backup` starts and detects a lock, it waits up to `lock_wait` seconds (default 300) for the lock to clear before aborting:
+
+```
+[1/2] Verifying repository: documents
+  Repository locked — waiting for lock to clear (5s / 300s)…
+  Repository locked — waiting for lock to clear (10s / 300s)…
+  ✓ Lock cleared
+[2/2] Creating archive
+```
+
+Configure the timeout in `ruborg.yml` (also passed as `--lock-wait` to Borg when set):
+
+```yaml
+# Wait up to 60s for a lock before aborting; also passes --lock-wait 60 to borg commands
+lock_wait: 60
+```
+
+### Validate Repository Compatibility
 
 ```bash
 # Check specific repository compatibility with installed Borg version
-ruborg check --repository documents
+ruborg validate repo --repository documents
 
 # Check all repositories
-ruborg check --all
+ruborg validate repo --all
 
 # Check with data integrity verification (slower)
-ruborg check --repository documents --verify-data
+ruborg validate repo --repository documents --verify-data
 ```
 
-The `check` command verifies:
+The `validate repo` command verifies:
 - Installed Borg version
 - Repository format version
 - Compatibility between Borg and repository versions
@@ -494,11 +530,11 @@ The `check` command verifies:
 ```
 Borg version: 1.2.8
 
---- Checking repository: documents ---
+--- Validating repository: documents ---
   Repository version: 1
   ✓ Compatible with Borg 1.2.8
 
---- Checking repository: databases ---
+--- Validating repository: databases ---
   Repository version: 2
   ✗ INCOMPATIBLE with Borg 1.2.8
     Repository version 2 cannot be read by Borg 1.2.8
@@ -652,26 +688,27 @@ See [SECURITY.md](SECURITY.md) for detailed security information and best practi
 | Command | Description | Options |
 |---------|-------------|---------|
 | `init REPOSITORY` | Initialize a new Borg repository | `--passphrase`, `--passbolt-id`, `--log` |
-| `validate` | Validate configuration file for type errors | `--config`, `--log` |
+| `validate config` | Validate configuration file for type errors | `--config`, `--log` |
+| `validate repo` | Validate repository compatibility and integrity | `--config`, `--repository`, `--all`, `--verify-data`, `--log` |
 | `backup` | Create a backup using config file | `--config`, `--repository`, `--all`, `--name`, `--remove-source`, `--log` |
 | `list` | List archives or files in repository | `--config`, `--repository`, `--archive`, `--log` |
 | `restore ARCHIVE` | Restore files from archive | `--config`, `--repository`, `--destination`, `--path`, `--log` |
 | `metadata ARCHIVE` | Get file metadata from archive | `--config`, `--repository`, `--file`, `--log` |
+| `lock` | Check for and optionally break a repository lock | `--config`, `--repository`, `--break`, `--force`, `--yes`, `--log` |
 | `info` | Show repository information | `--config`, `--repository`, `--log` |
-| `check` | Check repository integrity and compatibility | `--config`, `--repository`, `--all`, `--verify-data`, `--log` |
 | `version` | Show ruborg version | None |
 
 ### Options
 
 - `--config`: Path to configuration file (default: `ruborg.yml`)
 - `--log`: Path to log file (overrides config, default: `~/.ruborg/logs/ruborg.log`)
-- `--repository` / `-r`: Repository name (optional for info, required for backup/list/restore/check unless --all)
-- `--all`: Process all repositories (backup and check commands)
+- `--repository` / `-r`: Repository name (optional for info, required for backup/list/restore/validate repo unless --all)
+- `--all`: Process all repositories (backup and validate repo commands)
 - `--name`: Custom archive name (backup command only)
 - `--remove-source`: Remove source files after successful backup (backup command only)
 - `--destination`: Destination directory for restore (restore command only)
 - `--path`: Specific file or directory to restore (restore command only)
-- `--verify-data`: Run full data integrity check (check command only, slower)
+- `--verify-data`: Run full data integrity check (validate repo command only, slower)
 
 ## Retention Policies
 
@@ -822,6 +859,60 @@ repositories:
 
 **Backup vs Retention:** The per-file `retention_mode` only affects how archives are created and pruned. Traditional backup commands still work normally - you can list, restore, and check per-file archives just like standard archives.
 
+### Skip Hash Check for Faster Backups
+
+**NEW:** In per-file backup mode, you can optionally skip content hash verification for faster duplicate detection:
+
+```yaml
+repositories:
+  - name: project-files
+    path: /mnt/backup/project-files
+    retention_mode: per_file
+    skip_hash_check: true    # Skip SHA256 content hash verification
+    sources:
+      - name: projects
+        paths:
+          - /home/user/projects
+```
+
+**How it works:**
+- **Default (paranoid mode)**: Ruborg calculates SHA256 hash of file content to verify files haven't changed (even when size and mtime are identical)
+- **With skip_hash_check: true**: Ruborg trusts file path, size, and modification time for duplicate detection (skips hash calculation)
+
+**When to use:**
+- ✅ **Large directories** with thousands of files where hash calculation is slow
+- ✅ **Reliable filesystems** where modification time changes are trustworthy
+- ✅ **Regular backups** where files are unlikely to be manually modified with `touch -t`
+
+**When NOT to use:**
+- ❌ **Security-critical data** where you want maximum verification
+- ❌ **Untrusted sources** where files might be tampered with
+- ❌ **Systems with unreliable mtime** (rare, but some network filesystems)
+
+**Performance impact:**
+```yaml
+# Example: 10,000 unchanged files, average 50KB each
+# With skip_hash_check: false (default) - ~30 seconds (read + hash all files)
+# With skip_hash_check: true            - ~3 seconds  (read metadata only)
+```
+
+**Console output:**
+```
+# With skip_hash_check: true
+[1/10000] Backing up: /home/user/file1.txt - Archive already exists (skipped hash check)
+[2/10000] Backing up: /home/user/file2.txt - Archive already exists (skipped hash check)
+...
+✓ Per-file backup completed: 50 file(s) backed up, 9950 skipped (hash check skipped)
+
+# With skip_hash_check: false (default)
+[1/10000] Backing up: /home/user/file1.txt - Archive already exists (file unchanged)
+[2/10000] Backing up: /home/user/file2.txt - Archive already exists (file unchanged)
+...
+✓ Per-file backup completed: 50 file(s) backed up, 9950 skipped (unchanged)
+```
+
+**Security note:** Even with `skip_hash_check: true`, files are still verified by path, size, and mtime. The only difference is skipping the SHA256 content hash verification, which catches rare edge cases like manual file tampering with preserved timestamps.
+
 ### Automatic Pruning
 
 Enable **automatic pruning** to remove old backups after each backup operation:

data/lib/ruborg/archive_cache.rb

@@ -0,0 +1,189 @@
+# frozen_string_literal: true
+
+require "json"
+require "open3"
+require "tempfile"
+
+module Ruborg
+  # Persistent cache of per-archive metadata, stored as a JSON file sibling to the
+  # Borg repository. Eliminates repeated `borg info` calls across runs.
+  #
+  # Supports local paths (File::LOCK_EX) and SSH paths (optimistic merge via scp).
+  # All metadata is stored and returned with symbol keys (:path, :size, :hash, :source_dir).
+  class ArchiveCache
+    SSH_PATTERN = %r{\A(?:ssh://|[^\s/]+@[^\s:]+:)}
+
+    def initialize(repo_path)
+      @repo_path = repo_path
+      @data = {}
+      @snapshot = {}
+      @loaded = false
+    end
+
+    def fetch
+      return self if @loaded
+
+      if ssh?
+        load_remote
+      else
+        load_local
+      end
+
+      @snapshot = snapshot(@data)
+      @loaded = true
+      self
+    end
+
+    def [](archive_name)
+      @data[archive_name]
+    end
+
+    def store(archive_name, metadata)
+      @data[archive_name] = symbolize(metadata)
+    end
+
+    # Returns all cached entries as an array of hashes, each including :archive_name.
+    def entries
+      @data.map { |archive_name, metadata| metadata.merge(archive_name: archive_name) }
+    end
+
+    def save_if_changed
+      return unless dirty?
+
+      if ssh?
+        save_remote
+      else
+        save_local
+      end
+    end
+
+    private
+
+    def dirty?
+      @data != @snapshot
+    end
+
+    def snapshot(hash)
+      hash.transform_values(&:dup)
+    end
+
+    def ssh?
+      SSH_PATTERN.match?(@repo_path)
+    end
+
+    def cache_path_for(path)
+      "#{path}.ruborg_cache.json"
+    end
+
+    def symbolize(metadata)
+      metadata.transform_keys(&:to_sym)
+    end
+
+    def normalize_archives(raw)
+      (raw || {}).transform_values { |v| symbolize(v) }
+    end
+
+    def load_local
+      path = cache_path_for(@repo_path)
+      return unless File.exist?(path)
+
+      File.open(path, "r") do |f|
+        f.flock(File::LOCK_SH)
+        parsed = JSON.parse(f.read)
+        @data = normalize_archives(parsed["archives"])
+      end
+    rescue JSON::ParserError
+      @data = {}
+    end
+
+    def save_local
+      path = cache_path_for(@repo_path)
+      File.open(path, File::RDWR | File::CREAT, 0o600) do |f|
+        f.flock(File::LOCK_EX)
+        existing = read_existing_local(f)
+        merged = existing.merge(@data)
+        f.rewind
+        f.write(JSON.generate({ "version" => 1, "archives" => stringify_for_storage(merged) }))
+        f.truncate(f.pos)
+      end
+    end
+
+    def read_existing_local(file)
+      content = file.read
+      return {} if content.empty?
+
+      normalize_archives(JSON.parse(content)["archives"])
+    rescue JSON::ParserError
+      {}
+    end
+
+    # JSON requires string keys; convert symbol keys back before writing.
+    def stringify_for_storage(data)
+      data.transform_values { |v| v.transform_keys(&:to_s) }
+    end
+
+    def parse_ssh
+      if @repo_path.start_with?("ssh://")
+        require "uri"
+        uri = URI.parse(@repo_path)
+        host = uri.user ? "#{uri.user}@#{uri.host}" : uri.host
+        host = "#{host}:#{uri.port}" if uri.port && uri.port != 22
+        [host, uri.path]
+      else
+        match = @repo_path.match(%r{\A([^\s/]+@[^\s:]+):(.+)\z})
+        return [nil, nil] unless match
+
+        [match[1], match[2]]
+      end
+    end
+
+    def load_remote
+      host, path = parse_ssh
+      return unless host
+
+      remote = "#{host}:#{cache_path_for(path)}"
+      loaded = nil
+      Tempfile.create(["ruborg_cache", ".json"]) do |tmp|
+        _, status = Open3.capture2e("scp", "-q", "-B", remote, tmp.path)
+        next unless status.success?
+
+        begin
+          loaded = normalize_archives(JSON.parse(File.read(tmp.path))["archives"])
+        rescue JSON::ParserError
+          loaded = {}
+        end
+      end
+      @data = loaded if loaded
+    end
+
+    def save_remote
+      host, path = parse_ssh
+      return unless host
+
+      remote = "#{host}:#{cache_path_for(path)}"
+      fresh = fetch_remote_fresh(remote)
+      merged = fresh.merge(@data)
+
+      Tempfile.create(["ruborg_cache_upload", ".json"]) do |tmp|
+        tmp.write(JSON.generate({ "version" => 1, "archives" => stringify_for_storage(merged) }))
+        tmp.flush
+        Open3.capture2e("scp", "-q", "-B", tmp.path, remote)
+      end
+    end
+
+    def fetch_remote_fresh(remote)
+      result = {}
+      Tempfile.create(["ruborg_cache_fresh", ".json"]) do |tmp|
+        _, status = Open3.capture2e("scp", "-q", "-B", remote, tmp.path)
+        next unless status.success?
+
+        begin
+          result = normalize_archives(JSON.parse(File.read(tmp.path))["archives"])
+        rescue JSON::ParserError
+          result = {}
+        end
+      end
+      result
+    end
+  end
+end