bitferry 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 07536a8e52281f9dbe3a085aa9bedff6164f116babcb163149e8a13e2214f771
+  data.tar.gz: a81dbb898d67e9c9e2083871c69bfd6396e56caa12904b085cf8e7c5e85248d5
+SHA512:
+  metadata.gz: c8025822e4520ec87254036acd3b7c6f3933949b3a0737cc9d12d91d5ae4d5fa5d98487a9f2d60a18c15cfb9d4009212b35ead99ce415ececdc42b48c03bcd7f
+  data.tar.gz: 1af8e4d44dd64818f78eadb27b62233e33bd4f7f1cc493dfafbcb7ca4defd77e5ea868ad5060ee28e6eefc3f4c122842e7774b4627506305ee553e12157ea3c0

data/README.md

@@ -0,0 +1,233 @@
+# Bitferry - file synchronization/backup automation tool
+
+<div align="right"><i>Ein Backup ist kein Backup</i></div><br><br>
+
+The [Bitferry](https://github.com/okhlybov/bitferry) is aimed at establishing the automated file synchronization/replication/backup routes between multiple endpoints where the latter can be the local directories, online cloud remotes or portable offline storages.
+
+The intended usage ranges from maintaining simple directory copy to another location (disk, mount point) to complex many-to-many (online/offline) data replication/backup solution employing portable media as additional data storage and a means of data propagation between the offsites.
+
+Bitferry is a frontend to the [Rclone](https://rclone.org) and [Restic](https://restic.net) utilities.
+
+
+## Features
+
+* Multiplatform (Windows / UNIX / macOSX) operation
+
+* Automated task-based data processing
+
+* One way / two way data synchronization
+
+* Recursive directory copy / update / synchronize
+
+* Incremental directory backup with snapshotting
+
+* File/repository password-based end-to-end encryption
+
+* Online cloud storage relay
+
+* Offline portable storage (USB flash, HDDs, SSDs etc.) relay
+
+
+## Use cases
+
+* Maintain an update-only files copy in a separate location on the same site
+
+* Maintain offline secure two way file synchronization between two offsites
+
+* Maintain an incremental files backup on a portable medium with multiple offsite copies of the repository
+
+
+## Implementation
+
+The Bitferry itself is written in [Ruby](https://www.ruby-lang.org) programming language. Being a Ruby code, the Bitferry requires the platform-specific Ruby runtime, version 3.0 or higher. 
+
+The source code is hosted on [GitHub](https://github.com/okhlybov/bitferry) and the binary releases in form of a GEM package are distributed through the [RubyGems](https://rubygems.org/gems/bitferry) repository channel.
+
+In addition, the platform-specific [Rclone](https://github.com/rclone/rclone/releases) and [Restic](https://github.com/restic/restic/releases) executables are required to be accessible through the `PATH` directory list or through the respective `RCLONE` and `RESTIC` environment variables.
+
+
+## Kickstart
+
+Install Bitferry
+
+```shell
+gem install bitferry
+```
+
+Prepare source Bitferry volume for a mounted local filesystem
+
+```shell
+bitferry create volume /data
+```
+
+Prepare destination Bitferry volume for a mounted portable storage
+
+```shell
+bitferry create volume /mnt/usb-drive
+```
+
+Ensure the volumes are intact
+
+```shell
+bitferry show
+```
+
+```
+# Intact volumes
+
+  d2f10024    /data
+  e42f2d8c    /mnt/usb-drive
+```
+
+Create a (Rclone) sync task with data encryption
+
+```shell
+bitferry create task sync -e /data /mnt/usb-drive/backup
+```
+
+Review the changes
+
+```shell
+bitferry
+```
+
+```
+# Intact volumes
+
+  d2f10024    /data
+  e42f2d8c    /mnt/usb-drive
+
+
+# Intact tasks
+
+  89e1c119    encrypt+synchronize :d2f10024: --> :e42f2d8c:backup
+```
+
+Perform a dry run of the specific task
+
+```shell
+bitferry process -vn 89e
+```
+
+<details>
+<summary>...</summary>
+
+```
+rclone sync --filter -\ .bitferry --filter -\ .bitferry\~ --verbose --progress --dry-run --metadata --crypt-filename-encoding base32 --crypt-filename-encryption standard --crypt-remote /mnt/usb-drive/backup /data :crypt:
+2024/03/05 11:46:45 NOTICE: README.md: Skipped copy as --dry-run is set (size 3.073Ki)
+2024/03/05 11:46:45 NOTICE: LICENSE: Skipped copy as --dry-run is set (size 1.467Ki)
+2024/03/05 11:46:45 NOTICE: bitferry.gemspec: Skipped copy as --dry-run is set (size 996)
+Transferred:        5.513 KiB / 5.513 KiB, 100%, 0 B/s, ETA -
+Transferred:            3 / 3, 100%
+Elapsed time:         0.0s
+2024/03/05 11:46:45 NOTICE: 
+Transferred:        5.513 KiB / 5.513 KiB, 100%, 0 B/s, ETA -
+Transferred:            3 / 3, 100%
+Elapsed time:         0.0s
+```
+
+</details>
+
+Process all intact tasks in sequence
+
+```shell
+bitferry -v x
+```
+
+<details>
+<summary>...</summary>
+
+```
+rclone sync --filter -\ .bitferry --filter -\ .bitferry\~ --verbose --progress --metadata --crypt-filename-encoding base32 --crypt-filename-encryption standard --crypt-remote /mnt/usb-drive/backup /data :crypt:
+2024/03/05 11:44:31 INFO  : LICENSE: Copied (new)
+2024/03/05 11:44:31 INFO  : README.md: Copied (new)
+2024/03/05 11:44:31 INFO  : bitferry.gemspec: Copied (new)
+Transferred:        5.653 KiB / 5.653 KiB, 100%, 0 B/s, ETA -
+Transferred:            3 / 3, 100%
+Elapsed time:         0.0s
+2024/03/05 11:44:31 INFO  : 
+Transferred:        5.653 KiB / 5.653 KiB, 100%, 0 B/s, ETA -
+Transferred:            3 / 3, 100%
+Elapsed time:         0.0s
+```
+
+</details>
+
+Observe the result
+
+```shell
+ls -l /mnt/usb-drive/backup
+```
+
+<details>
+<summary>...</summary>
+
+```
+-rw-r--r-- 1 user user 1044 feb 27 17:09 0u1vi7ka5p88u62kof9k6mf2z00354g6fa0c9a0g6di2f0ocds80
+-rw-r--r-- 1 user user 1550 jan 29 11:57 21dgu5vs2c4rjfkieeemjvaf78
+-rw-r--r-- 1 user user 3195 mar  5 11:43 m9rhq3q2m5h2q5l1ke00u0gdjc
+```
+
+</details>
+
+Examine the detailed usage instructions
+
+```shell
+bitferry c t s -h
+```
+
+<details>
+<summary>...</summary>
+
+```
+Usage:
+    bitferry c t s [OPTIONS] SOURCE DESTINATION
+
+  Create source --> destination one way file synchronization task.
+
+  The task operates recursively on two specified endpoints.
+  This task copies newer source files while skipping unchanged files in destination.
+  Also, it deletes destination files which are non-existent in source.
+
+    The endpoint may be one of:
+    * directory -- absolute or relative local directory (/data, ../source, c:\data)
+    * local:directory, :directory -- absolute local directory (:/data, local:c:\data)
+    * :tag:directory -- path relative to the intact volume matched by (partial) tag (:fa2c:source/data)
+
+    The former case resolves specified directory againt an intact volume to make it volume-relative.
+    It is an error if there is no intact volume that encompasses specified directory.
+    The local: directory is left as is (not resolved against volumes).
+    The :tag: directory is bound to the specified volume.
+
+
+
+    The encryption mode is controlled by --encrypt or --decrypt options.
+    The mandatory password will be read from the standard input channel (pipe or keyboard).
+
+  This task employs the Rclone worker.
+
+Parameters:
+    SOURCE                   Source endpoint specifier
+    DESTINATION              Destination endpoint specifier
+
+Options:
+    -e                       Encrypt files in destination using default profile (alias for -E default)
+    -d                       Decrypt source files using default profile (alias for -D default)
+    -x                       Use extended encryption profile options (applies to -e, -d)
+    --process, -X OPTIONS    Extra task processing profile/options
+    --encrypt, -E OPTIONS    Encrypt files in destination using specified profile/options
+    --decrypt, -D OPTIONS    Decrypt source files using specified profile/options
+    --version                Print version
+    --verbose, -v            Extensive logging
+    --quiet, -q              Disable logging
+    --dry-run, -n            Simulation mode (make no on-disk changes)
+    -h, --help               print help
+```
+
+</details>
+
+## The rest is about to come
+
+*Cheers!*
+
+Oleg A. Khlybov <fougas@mail.ru>

data/bin/bitferry

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+require 'bitferry/cli'
+#

data/lib/bitferry/cli.rb

@@ -0,0 +1,344 @@
+require 'clamp'
+require 'bitferry'
+require 'io/console'
+
+
+Endpoint = %{
+  The endpoint may be one of:
+  * directory -- absolute or relative local directory (/data, ../source, c:\\data)
+  * local:directory, :directory -- absolute local directory (:/data, local:c:\\data)
+  * :tag:directory -- path relative to the intact volume matched by (partial) tag (:fa2c:source/data)
+
+  The former case resolves specified directory againt an intact volume to make it volume-relative.
+  It is an error if there is no intact volume that encompasses specified directory.
+  The local: directory is left as is (not resolved against volumes).
+  The :tag: directory is bound to the specified volume.
+}
+
+
+Encryption = %{
+  The encryption mode is controlled by --encrypt or --decrypt options.
+  The mandatory password will be read from the standard input channel (pipe or keyboard).
+}
+
+
+def setup_rclone_task(x)
+  x.parameter 'SOURCE', 'Source endpoint specifier'
+  x.parameter 'DESTINATION', 'Destination endpoint specifier'
+  x.option '-e', :flag, 'Encrypt files in destination using default profile (alias for -E default)', attribute_name: :e do
+    $encryption = Bitferry::Rclone::Encrypt
+    $profile = :default
+  end
+  x.option '-d', :flag, 'Decrypt source files using default profile (alias for -D default)', attribute_name: :d do
+    $encryption = Bitferry::Rclone::Decrypt
+    $profile = :default
+  end
+  x.option '-x', :flag, 'Use extended encryption profile options (applies to -e, -d)', attribute_name: :x do
+    $extended = true
+  end
+  x.option ['--process', '-X'], 'OPTIONS', 'Extra task processing profile/options' do |opts|
+    $process = opts
+  end
+  x.option ['--encrypt', '-E'], 'OPTIONS', 'Encrypt files in destination using specified profile/options' do |opts|
+    $encryption = Bitferry::Rclone::Encrypt
+    $profile = opts
+  end
+  x.option ['--decrypt', '-D'], 'OPTIONS', 'Decrypt source files using specified profile/options' do |opts|
+    $encryption = Bitferry::Rclone::Decrypt
+    $profile = opts
+  end
+end
+
+
+def create_rclone_task(task, *args, **opts)
+  task.new(*args,
+    process: $process,
+    encryption: $encryption&.new(obtain_password, process: $extended ? :extended : $profile),
+    **opts
+  )
+end
+
+
+def bitferry(&code)
+  begin
+    Bitferry.restore
+    result = yield
+    exit(Bitferry.commit && result ? 0 : 1)
+  rescue => e
+    Bitferry.log.fatal(e.message)
+    exit(1)
+  end
+end
+
+
+def obtain_password
+  if $stdin.tty?
+    p1 = IO.console.getpass 'Enter password:'
+    p2 = IO.console.getpass 'Repeat password:'
+    raise 'passwords do not match' unless p1 == p2
+    p1
+  else
+    $stdin.readline.strip!
+  end
+end
+
+
+Bitferry.log.level = Logger::DEBUG if $DEBUG
+
+
+Clamp do
+
+
+  self.default_subcommand = 'show'
+
+
+  option '--version', :flag, 'Print version' do
+    puts Bitferry::VERSION
+    exit
+  end
+
+
+  option ['--verbose', '-v'], :flag, 'Extensive logging' do
+    Bitferry.verbosity = :verbose
+  end
+
+
+  option ['--quiet', '-q'], :flag, 'Disable logging' do
+    Bitferry.verbosity = :quiet
+  end
+
+
+  option ['--dry-run', '-n'], :flag, 'Simulation mode (make no on-disk changes)' do
+    Bitferry.simulate = true
+  end
+
+
+  subcommand ['show', 'info', 'i'], 'Print state' do
+    def execute
+      Bitferry.restore
+      unless (xs = Bitferry::Volume.intact).empty?
+        puts '# Intact volumes'
+        puts
+        xs.each do |volume|
+          puts "  #{volume.tag}    #{volume.root}"
+        end
+      end
+      unless (xs = Bitferry::Task.intact).empty?
+        puts
+        puts '# Intact tasks'
+        puts
+        xs.each do |task|
+          puts "  #{task.tag}    #{task.show_status}"
+        end
+      end
+      unless (xs = Bitferry::Task.stale).empty?
+        puts
+        puts '# Stale tasks'
+        puts
+        xs.each do |task|
+          puts "  #{task.tag}    #{task.show_status}"
+        end
+      end
+    end
+  end
+
+
+  subcommand ['create', 'c'], 'Create entity' do
+
+
+    subcommand ['volume', 'v'], 'Create volume' do
+      banner %{
+        Create new volume in specified directory. Create directory if it does not exist.
+        Refuse to overwrite existing volume storage unless --force is specified.
+      }
+      option '--force', :flag, 'Overwrite existing volume storage in target directory'
+      parameter 'DIRECTORY', 'Target volume directory'
+      def execute
+        bitferry { Bitferry::Volume.new(directory, overwrite: force?) }
+      end
+    end
+
+
+    subcommand ['task', 't'], 'Create task' do
+
+
+      subcommand ['copy', 'c'], 'Create copy task' do
+        banner %{
+          Create source --> destination file copy task.
+      
+          The task operates recursively on two specified endpoints.
+          This task unconditionally copies all source files overwriting existing files in destination.
+      
+          #{Endpoint}
+      
+          #{Encryption}
+          
+          This task employs the Rclone worker.
+        }
+        setup_rclone_task(self)
+        def execute
+          bitferry { create_rclone_task(Bitferry::Rclone::Copy, source, destination) }
+        end
+      end
+
+
+      subcommand ['update', 'u'], 'Create update task' do
+        banner %{
+          Create source --> destination file update (freshen) task.
+      
+          The task operates recursively on two specified endpoints.
+          This task copies newer source files while skipping unchanged files in destination.
+      
+          #{Endpoint}
+      
+          #{Encryption}
+
+          This task employs the Rclone worker.
+        }
+        setup_rclone_task(self)
+        def execute
+          bitferry { create_rclone_task(Bitferry::Rclone::Update, source, destination) }
+        end
+      end
+
+
+      subcommand ['synchronize', 'sync', 's'], 'Create one way sync task' do
+        banner %{
+          Create source --> destination one way file synchronization task.
+      
+          The task operates recursively on two specified endpoints.
+          This task copies newer source files while skipping unchanged files in destination.
+          Also, it deletes destination files which are non-existent in source.
+      
+          #{Endpoint}
+
+          #{Encryption}
+      
+          This task employs the Rclone worker.
+        }
+        setup_rclone_task(self)
+        def execute
+          bitferry { create_rclone_task(Bitferry::Rclone::Synchronize, source, destination) }
+        end
+      end
+
+
+      subcommand ['equalize', 'bisync', 'e'], 'Create two way sync task' do
+        banner %{
+          Create source <-> destination two way file synchronization task.
+      
+          The task operates recursively on two specified endpoints.
+          This task retains only the most recent versions of files on both endpoints.
+          Opon execution both endpoints are left identical.
+      
+          #{Endpoint}
+
+          #{Encryption}
+      
+          This task employs the Rclone worker.
+        }
+        setup_rclone_task(self)
+        def execute
+          bitferry { create_rclone_task(Bitferry::Rclone::Equalize, source, destination) }
+        end
+      end
+
+
+      subcommand ['backup', 'b'], 'Create backup task' do
+        banner %{
+          Create source --> repository incremental backup task.
+          This task employs the Restic worker.
+        }
+        option '--force', :flag, 'Force overwriting existing repository' do $format = true end
+        option ['--attach', '-a'], :flag, 'Attach to existing repository' do $format = false end
+        option '-f', :flag, 'Rig for application of the snapshot retention policy (alias for -F default)', attribute_name: :f do $forget = :default end
+        option '-c', :flag, 'Rig for repository checking (alias for -C default)', attribute_name: :c do $check = :default end
+        option ['--process', '-X'], 'OPTIONS', 'Extra task processing profile/options' do |opts| $process = opts end
+        option ['--forget', '-F'], 'OPTIONS', 'Rig for snapshot retention policy with profile/options' do |opts| $forget = opts end
+        option ['--check', '-C'], 'OPTIONS', 'Rig for repository checking with profile/options' do |opts| $check = opts end
+        parameter 'SOURCE', 'Source endpoint specifier'
+        parameter 'REPOSITORY', 'Destination repository endpoint specifier'
+        def execute
+          bitferry {
+            Bitferry::Restic::Backup.new(
+              source, repository, obtain_password,
+              format: $format,
+              process: $process,
+              check: $check,
+              forget: $forget
+            )
+          }
+        end
+      end
+
+
+      subcommand ['restore', 'r'], 'Create restore task' do
+        banner %{
+          Create repository --> destination restore task.
+          This task employs the Restic worker.
+        }
+        option ['--process', '-X'], 'OPTIONS', 'Extra task processing profile/options' do |opts| $process = opts end
+        parameter 'REPOSITORY', 'Source repository endpoint specifier'
+        parameter 'DESTINATION', 'Destination endpoint specifier'
+        def execute
+          bitferry {
+            Bitferry::Restic::Restore.new(
+              destination, repository, obtain_password,
+              process: $process,
+            )
+          }
+        end
+      end
+
+
+    end
+
+
+  end
+
+
+  subcommand ['delete', 'd'], 'Delete entity' do
+
+
+    subcommand ['volume', 'v'], 'Delete volume' do
+      banner %{
+        Delete volumes matched by specified (partial) tags.
+        There may be multiple tags but each tag must match at most one volume.
+        This command deletes the volume storage file only with the rest of data left intact.
+      }
+      option '--wipe', :flag, 'Wipe target directory upon deletion'
+      parameter 'TAG ...', 'Volume tags', attribute_name: :tags
+      def execute
+        bitferry { Bitferry::Volume.delete(*tags, wipe: wipe?) }
+      end
+    end
+
+
+    subcommand ['task', 't'], 'Delete task' do
+      banner %{
+        Delete tasks matched by specified (partial) tags.
+        There may be multiple tags but each tag must match at most one task.
+      }
+      parameter 'TAG ...', 'Task tags', attribute_name: :tags
+      def execute
+        bitferry { Bitferry::Task.delete(*tags) }
+      end
+    end
+
+
+  end
+
+
+  subcommand ['process', 'x'], 'Process tasks' do
+    banner %{
+      Process tasks matched by specified (partial) tags.
+      If no tags are given, process all intact tasks.
+    }
+    parameter '[TAG] ...', 'Task tags', attribute_name: :tags
+    def execute
+      bitferry { Bitferry.process(*tags) }
+    end
+  end
+
+
+end