imap-backup 11.1.0 → 12.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fdcbbac2458c28626fdf8be328db6f949be5cf981fde3438df03356b28849ebf
-  data.tar.gz: 1ed5fc6db436f57c30e03a5036a61c2fa21853cd2d7ae08b861b491b04b6d12b
+  metadata.gz: ed7a9fd869f7b1ea66f5dc0816a4ce35d77c150bcf8170e2d778e893fda43e9d
+  data.tar.gz: e75c535f849eb326906c1965a1e86507676209cf41cae5b98ad62e894912d9b8
 SHA512:
-  metadata.gz: a60997a5fdee3274d726e54c413c7b0e38df97246f4619358128a6786f5a148414d7050a6d498b805cb5376fca3d555f232e90c10d15d092ea49dce6007272e7
-  data.tar.gz: 41b3038e096c34486670e423e6231cb4dd6eb8c39b035f8ce7580fb406bc6210ba8e436a51d51c61e67f9b5c3af3a971ad6ce016e1904850bdb14bc52e150f9c
+  metadata.gz: 7ced0838fae828c771e831441fc3caf71b0465f4da630c3580e1ff842c687e51ad92d7e1913e7f8d48e9bb2c2db2eeee60eea3e2dcb464fda2b51ffc9252d8ea
+  data.tar.gz: d751a6fb0e2eebc107f539cb80609d353bb0c07126fb09e5001bea69476ca9eaa5f1ffce7e6bab27f7efdc149be213997d20c741cc2b60bf3ccf6a3fd0bd810c

data/README.md

@@ -113,7 +113,7 @@ These are activated via two settings:
 As with all performance tweaks, there are trade-offs.
 If you are using a small virtual server or Raspberry Pi
 to run your backups, you will probably want to leave
-the deafult settings.
+the default settings.
 If, on the other hand, you are using a computer with a
 fair bit of RAM, and you are dealing with a *lot* of email,
 then changing these settings may be worthwhile.
@@ -122,16 +122,17 @@ then changing these settings may be worthwhile.
 
 This setting affects all account backups.
 
-When not set, each message is written to disk, one at a time.
-Doing so means the message itself is appended to the MBox file,
-but more importantly, the JSON metadata is rewritten to disk
-from scratch.
+By default, `imap-backup` uses the "delay metadata" strategy.
+As messages are being backed-up, the message *text*
+is written to disk, while the related metadata is stored in memory.
 
-When in use, all of a mailboxes unbackupped messages are
-downloaded first, and then written to disk just once.
+While this uses a little more memory, it avoids rewiting a growing JSON
+file for every message, speeding things up and reducing disk wear.
 
-This speeds up backup as the metadata file is not rewritten
-after each message is added, but it potentially uses much more memory.
+The alternative strategy, called "direct", writes everything to disk
+as it is received. This method is slower, but has the advantage
+of using slightly less memory, which may be important on very
+resource-limited systems, like Raspberry Pis.
 
 ## Multi-fetch Size
 
@@ -140,12 +141,11 @@ By default, during backup, each message is downloaded one-by-one.
 Using this setting, you can download chunks of emails at a time,
 potentially speeding up the process.
 
-If you're not using "Delayed downlaod writes",
-using multi-fetch *will* mean that the backup process will use
-more memory - equivalent to the size of the greater number
-of messages downloaded at a time.
+Using multi-fetch *will* mean that the backup process will use
+more memory - equivalent to the size of the groups of messages
+that are downloaded.
 
-This behaviour may also exceed limits on your email provider,
+This behaviour may also exceed the rate limits on your email provider,
 so it's best to check before cranking it up!
 
 # Troubleshooting

data/docs/development.md

@@ -6,25 +6,26 @@
 
 # Development
 
-A setup for developing under any available Ruby version is
-available in the container directory.
+A Dockerfile is available to allow testing with all available Ruby versions,
+see the `dev/container` directory.
 
 # Testing
 
 ## Feature Specs
 
-Specs under `specs/features` are integration specs run against a local IMAP server
-controlled by Docker Compose.
-Before running the test suite, it needs to be started:
+Specs under `specs/features` are integration specs run against
+two local IMAP servers controlled by Docker Compose.
+
+Start them before running the test suite
 
 ```sh
-$ docker-compose up -d
+$ docker-compose -f dev/docker-compose.yml up -d
 ```
 
 or, with Podman
 
 ```sh
-$ podman-compose -f docker-compose.yml up -d
+$ podman-compose -f dev/docker-compose.yml up -d
 ```
 
 ```sh
@@ -62,24 +63,23 @@ use `last_command_started.output`.
 
 ```ruby
 require "net/imap"
+require_relative "spec/features/support/30_email_server_helpers"
 
-imap = Net::IMAP.new("localhost", {port: 8993, ssl: {verify_mode: 0}})
-username = "address@example.com"
-imap.login(username, "pass")
+include EmailServerHelpers
 
-message = "From: #{username}\nSubject: Some Subject\n\nHello!\n"
-response = imap.append("INBOX", message, nil, nil)
+test_connection = test_server_connection_parameters
 
-imap.examine("INBOX")
-uids = imap.uid_search(["ALL"]).sort
+test_imap = Net::IMAP.new(test_connection[:server], test_connection[:connection_options])
+test_imap.login(test_connection[:username], test_connection[:password])
 
-fetch_data_items = imap.uid_fetch(uids, ["BODY[]"])
-```
+message = "From: #{test_connection[:username]}\nSubject: Some Subject\n\nHello!\n"
+response = test_imap.append("INBOX", message, nil, nil)
 
-# Older Ruby Versions
+test_imap.examine("INBOX")
+uids = test_imap.uid_search(["ALL"]).sort
 
-Dockerfiles are available for all the supported Ruby versions,
-see the `container` directory.
+fetch_data_items = test_imap.uid_fetch(uids, ["BODY[]"])
+```
 
 # Contributing
 

data/docs/migrate-server-keep-address.md

@@ -1,29 +1,47 @@
 # Migrate to a new e-mail server while keeping your existing address
 
-While switching e-mail provider (from provider `A` to `B`), you might want to keep the same address (`mymail@domain.com`), and restrieve all your existing e-mails on your new server `B`. `imap-backup` can do that too!
+While switching e-mail provider (from provider `A` to `B`),
+you might want to keep the same address (`mymail@domain.com`),
+and copy all your existing e-mails to your new server `B`.
+`imap-backup` can do that too!
 
-1. Backup your e-mails: use [`imap-backup setup`](/docs/commands/setup.md) to setup connection to your old provider `A`, then launch [`imap-backup backup`](/docs/commands/backup.md). 
+It is best to use [`imap-backup migrate`](/docs/commands/migrate.md)
+and not [`imap-backup restore`](/docs/commands/restore.md) here because
+`migrate` simply copies emails to folders with the same name as the ones
+they were downloaded from, while `restore` changes the names of restored
+folders if folders with the same name already exist on the destination server.
+
+1. Backup your e-mails: use [`imap-backup setup`](/docs/commands/setup.md)
+  to setup connection to your old provider `A`,
+  then launch [`imap-backup backup`](/docs/commands/backup.md).
 1. Actually switch your e-mail service provider (update your DNS MX and all that...).
-1. It is best to use [`imap-backup migrate`](/docs/commands/migrate.md) and not [`imap-backup restore`](/docs/commands/restore.md) here, but both the source and the destination have the same address... You need to manually rename your old account first:
+1. As both the source and the destination have the same address,
+  you need to manually rename your old account first:
 
-	1. Modify your configuration file manually (i.e. not via `imap-backup setup`) and rename your account to `mymail-old@domain.com`:
+    1. Modify your configuration file manually
+      (i.e. not via `imap-backup setup`) and
+      rename your account to `mymail-old@domain.com`:
 
-		```diff
-		"accounts": [
-		{
-		-	"username": "mymail@domain.com",
-		+	"username": "mymail-old@domain.com",
-			"password": "...",
-		-	"local_path": "/some/path/.imap-backup/mymail_domain.com",
-		+	"local_path": "/some/path/.imap-backup/mymail-old_domain.com",
-			"folders": [...],
-			"server": "..."
-		}
-		```
+        ```diff
+          "accounts": [
+            {
+        -     "username": "mymail@domain.com",
+        +     "username": "mymail-old@domain.com",
+              "password": "...",
+        -     "local_path": "/some/path/.imap-backup/mymail_domain.com",
+        +     "local_path": "/some/path/.imap-backup/mymail-old_domain.com",
+              "folders": [...],
+              "server": "..."
+            }
+        ```
 
-	1. Rename the backup directory from `mymail_domain.com` to `mymail-old_domain.com`.
+    1. Rename the backup directory from `mymail_domain.com`
+      to `mymail-old_domain.com`.
 
-1. Set up a new account giving access to the new provider `B` using `imap-backup setup`.
-1. Now you can use `imap-backup migrate`, optionnally adapting [delimiters and prefixes configuration](/docs/delimiters-and-prefixes.md) if need be:
+1. Set up a new account giving access to the new provider `B`
+  using `imap-backup setup`.
+1. Now you can use `imap-backup migrate`, optionally adapting
+  [delimiters and prefixes configuration](/docs/delimiters-and-prefixes.md)
+  if need be:
 
-		imap-backup migrate mymail-old@domain.com mymail@domain.com [options]
+		imap-backup migrate mymail-old@domain.com mymail@domain.com [options]

data/lib/imap/backup/account/backup.rb

@@ -20,7 +20,7 @@ module Imap::Backup
     def run
       Logger.logger.info "Running backup of account: #{account.username}"
       # start the connection so we get logging messages in the right order
-      account.client
+      account.client.login
 
       Account::FolderEnsurer.new(account: account).run
       Account::LocalOnlyFolderDeleter.new(account: account).run if account.mirror_mode

data/lib/imap/backup/account/client_factory.rb

@@ -2,8 +2,8 @@ require "socket"
 
 require "email/provider"
 require "imap/backup/client/apple_mail"
+require "imap/backup/client/automatic_login_wrapper"
 require "imap/backup/client/default"
-require "retry_on_error"
 
 module Imap; end
 
@@ -11,10 +11,6 @@ module Imap::Backup
   class Account; end
 
   class Account::ClientFactory
-    include RetryOnError
-
-    LOGIN_RETRY_CLASSES = [::EOFError, ::Errno::ECONNRESET, ::SocketError].freeze
-
     attr_reader :account
 
     def initialize(account:)
@@ -24,20 +20,17 @@ module Imap::Backup
     end
 
     def run
-      retry_on_error(errors: LOGIN_RETRY_CLASSES) do
-        options = provider_options
-        Logger.logger.debug(
-          "Creating IMAP instance: #{server}, options: #{options.inspect}"
-        )
-        client =
-          if provider.is_a?(Email::Provider::AppleMail)
-            Client::AppleMail.new(server, account, options)
-          else
-            Client::Default.new(server, account, options)
-          end
-        client.login
-        client
-      end
+      options = provider_options
+      Logger.logger.debug(
+        "Creating IMAP instance: #{server}, options: #{options.inspect}"
+      )
+      client =
+        if provider.is_a?(Email::Provider::AppleMail)
+          Client::AppleMail.new(server, account, options)
+        else
+          Client::Default.new(server, account, options)
+        end
+      Client::AutomaticLoginWrapper.new(client: client)
     end
 
     private

data/lib/imap/backup/cli/folder_enumerator.rb

@@ -30,7 +30,7 @@ module Imap::Backup
       Pathname.glob(glob) do |path|
         name = source_folder_name(path)
         serializer = Serializer.new(source_local_path, name)
-        folder = destination_folder_for(name)
+        folder = destination_folder_for_path(name)
         yield serializer, folder
       end
     end
@@ -55,8 +55,8 @@ module Imap::Backup
         end
     end
 
-    def destination_folder_for(name)
-      parts = name.split(source_delimiter)
+    def destination_folder_for_path(name)
+      parts = name.split("/")
       no_source_prefix =
         if source_prefix_clipped != "" && parts.first == source_prefix_clipped
           parts[1..]

data/lib/imap/backup/cli/stats.rb

@@ -35,6 +35,7 @@ module Imap::Backup
       end
 
       def stats
+        Logger.logger.debug("[Stats] loading configuration")
         config = load_config(**options)
         account = account(config, email)
 
@@ -46,6 +47,7 @@ module Imap::Backup
 
           serializer = Serializer.new(account.local_path, folder.name)
           local_uids = serializer.uids
+          Logger.logger.debug("[Stats] fetching email list for '#{folder.name}'")
           remote_uids = folder.uids
           {
             folder: folder.name,

data/lib/imap/backup/cli/transfer.rb

@@ -0,0 +1,163 @@
+require "imap/backup/cli/backup"
+require "imap/backup/cli/helpers"
+require "imap/backup/cli/folder_enumerator"
+require "imap/backup/migrator"
+require "imap/backup/mirror"
+
+module Imap; end
+
+module Imap::Backup
+  class CLI::Transfer < Thor
+    include Thor::Actions
+    include CLI::Helpers
+
+    ACTIONS = %i(migrate mirror).freeze
+
+    attr_reader :action
+    attr_accessor :automatic_namespaces
+    attr_accessor :config_path
+    attr_accessor :destination_delimiter
+    attr_reader :destination_email
+    attr_accessor :destination_prefix
+    attr_reader :options
+    attr_accessor :reset
+    attr_accessor :source_delimiter
+    attr_reader :source_email
+    attr_accessor :source_prefix
+
+    def initialize(action, source_email, destination_email, options)
+      super([])
+      @action = action
+      @source_email = source_email
+      @destination_email = destination_email
+      @options = options
+      @automatic_namespaces = nil
+      @config_path = nil
+      @destination_delimiter = nil
+      @destination_prefix = nil
+      @reset = nil
+      @source_delimiter = nil
+      @source_prefix = nil
+    end
+
+    no_commands do
+      def run
+        raise "Unknown action '#{action}'" if !ACTIONS.include?(action)
+
+        process_options!
+        prepare_mirror if action == :mirror
+
+        folders.each do |serializer, folder|
+          case action
+          when :migrate
+            Migrator.new(serializer, folder, reset: reset).run
+          when :mirror
+            Mirror.new(serializer, folder).run
+          end
+        end
+      end
+
+      def process_options!
+        self.automatic_namespaces = options[:automatic_namespaces] || false
+        self.config_path = options[:config]
+        self.destination_delimiter = options[:destination_delimiter]
+        self.destination_prefix = options[:destination_prefix]
+        self.source_delimiter = options[:source_delimiter]
+        self.source_prefix = options[:source_prefix]
+        self.reset = options[:reset] || false
+        check_accounts!
+        choose_prefixes_and_delimiters!
+      end
+
+      def check_accounts!
+        if destination_email == source_email
+          raise "Source and destination accounts cannot be the same!"
+        end
+
+        raise "Account '#{destination_email}' does not exist" if !destination_account
+
+        raise "Account '#{source_email}' does not exist" if !source_account
+      end
+
+      def choose_prefixes_and_delimiters!
+        if automatic_namespaces
+          ensure_no_prefix_or_delimiter_parameters!
+          query_servers_for_settings
+        else
+          add_prefix_and_delimiter_defaults
+        end
+      end
+
+      def ensure_no_prefix_or_delimiter_parameters!
+        if destination_delimiter
+          raise "--automatic-namespaces is incompatible with --destination-delimiter"
+        end
+        if destination_prefix
+          raise "--automatic-namespaces is incompatible with --destination-prefix"
+        end
+        raise "--automatic-namespaces is incompatible with --source-delimiter" if source_delimiter
+        raise "--automatic-namespaces is incompatible with --source-prefix" if source_prefix
+      end
+
+      def query_servers_for_settings
+        self.destination_prefix, self.destination_delimiter = account_settings(destination_account)
+        self.source_prefix, self.source_delimiter = account_settings(source_account)
+      end
+
+      def account_settings(account)
+        namespaces = account.client.namespace
+        personal = namespaces.personal.first
+        [personal.prefix, personal.delim]
+      end
+
+      def add_prefix_and_delimiter_defaults
+        self.destination_delimiter ||= "/"
+        self.destination_prefix ||= ""
+        self.source_delimiter ||= "/"
+        self.source_prefix ||= ""
+      end
+
+      def prepare_mirror
+        warn_if_source_account_is_not_in_mirror_mode
+
+        CLI::Backup.new(config: config_path, accounts: source_email).run
+      end
+
+      def warn_if_source_account_is_not_in_mirror_mode
+        return if source_account.mirror_mode
+
+        message =
+          "The account '#{source_account.username}' " \
+          "is not set up to make mirror backups"
+        Logger.logger.warn message
+      end
+
+      def config
+        @config ||= load_config(config: config_path)
+      end
+
+      def enumerator_options
+        {
+          destination: destination_account,
+          destination_delimiter: destination_delimiter,
+          destination_prefix: destination_prefix,
+          source: source_account,
+          source_delimiter: source_delimiter,
+          source_prefix: source_prefix
+        }
+      end
+
+      def folders
+        CLI::FolderEnumerator.new(**enumerator_options)
+      end
+
+      def destination_account
+        config.accounts.find { |a| a.username == destination_email }
+      end
+
+      def source_account
+        config.accounts.find { |a| a.username == source_email }
+      end
+    end
+  end
+end

data/lib/imap/backup/cli.rb

@@ -10,18 +10,38 @@ module Imap::Backup
     autoload :Backup, "imap/backup/cli/backup"
     autoload :Folders, "imap/backup/cli/folders"
     autoload :Local, "imap/backup/cli/local"
-    autoload :Migrate, "imap/backup/cli/migrate"
-    autoload :Mirror, "imap/backup/cli/mirror"
     autoload :Remote, "imap/backup/cli/remote"
     autoload :Restore, "imap/backup/cli/restore"
     autoload :Setup, "imap/backup/cli/setup"
     autoload :Stats, "imap/backup/cli/stats"
+    autoload :Transfer, "imap/backup/cli/transfer"
     autoload :Utils, "imap/backup/cli/utils"
 
     include Helpers
 
     VERSION_ARGUMENTS = %w(-v --version).freeze
 
+    NAMESPACE_CONFIGURATION_DESCRIPTION = <<~DESC.freeze
+      Some IMAP servers use namespaces (i.e. prefixes like "INBOX"),
+      while others, while others concatenate the names of subfolders
+      with a charater ("delimiter") other than "/".
+
+      In these cases there are two choices.
+
+      You can use the `--automatic-namespaces` option.
+      This wil query the source and detination servers for their
+      namespace configuration and will adapt paths accordingly.
+      This option requires that both the source and destination
+      servers are available and work with the provided parameters
+      and authentication.
+
+      If automatic configuration does not work as desired, there are the
+      `--source-prefix=`, `--source-delimiter=`,
+      `--destination-prefix=` and `--destination-delimiter=` parameters.
+      To check what values you should use, check the output of the
+      `imap-backup remote namespaces EMAIL` command.
+    DESC
+
     default_task :backup
 
     def self.start(*args)
@@ -83,19 +103,22 @@ module Imap::Backup
       All emails which have been backed up for the "source account" (SOURCE_EMAIL) are
       uploaded to the "destination account" (DESTINATION_EMAIL).
 
-      When one or other account has namespaces (i.e. prefixes like "INBOX"),
-      use the `--source-prefix=` and/or `--destination-prefix=` options.
+      Some configuration may be necessary, as follows:
 
-      When one or other account uses a delimiter other than `/` (i.e. `.`),
-      use the `--source-delimiter=` and/or `--destination-delimiter=` options.
+      #{NAMESPACE_CONFIGURATION_DESCRIPTION}
 
-      If you you want to delete existing emails in destination folders,
+      Finally, if you want to delete existing emails in destination folders,
       use the `--reset` option. In this case, all existing emails are
       deleted before uploading the migrated emails.
     DESC
     config_option
     quiet_option
     verbose_option
+    method_option(
+      "automatic-namespaces",
+      type: :boolean,
+      desc: "automatically choose delimiters and prefixes"
+    )
     method_option(
       "destination-delimiter",
       type: :string,
@@ -126,7 +149,7 @@ module Imap::Backup
     )
     def migrate(source_email, destination_email)
       non_logging_options = Imap::Backup::Logger.setup_logging(options)
-      Migrate.new(source_email, destination_email, **non_logging_options).run
+      Transfer.new(:migrate, source_email, destination_email, non_logging_options).run
     end
 
     desc(
@@ -140,7 +163,7 @@ module Imap::Backup
       If a folder list is configured for the SOURCE_EMAIL account,
       only the folders indicated by the setting are copied.
 
-      First, runs the download of the SOURCE_EMAIL account.
+      First, it runs the download of the SOURCE_EMAIL account.
       If the SOURCE_EMAIL account is **not** configured to be in 'mirror' mode,
       a warning is printed.
 
@@ -152,6 +175,11 @@ module Imap::Backup
     config_option
     quiet_option
     verbose_option
+    method_option(
+      "automatic-namespaces",
+      type: :boolean,
+      desc: "automatically choose delimiters and prefixes"
+    )
     method_option(
       "destination-delimiter",
       type: :string,
@@ -176,7 +204,7 @@ module Imap::Backup
     )
     def mirror(source_email, destination_email)
       non_logging_options = Imap::Backup::Logger.setup_logging(options)
-      Mirror.new(source_email, destination_email, **non_logging_options).run
+      Transfer.new(:mirror, source_email, destination_email, non_logging_options).run
     end
 
     desc "remote SUBCOMMAND [OPTIONS]", "View info about online accounts"
@@ -211,9 +239,14 @@ module Imap::Backup
 
     desc "stats EMAIL [OPTIONS]", "Print stats for each account folder"
     long_desc <<~DESC
-      For each account folder, lists emails that are yet to be downloaded "server",
-      are downloaded (exist on server and locally) "both" and those which
-      are only present in the backup (as they have been deleted on the server) "local".
+      For each account folder, lists three counts of emails:
+
+      1. "server" - those yet to be downloaded,
+
+      2. "both" - those that exist on server and are backed up,
+
+      3. "local" - those which are only present in the backup (as they have been deleted
+        on the server).
     DESC
     config_option
     format_option

data/lib/imap/backup/client/automatic_login_wrapper.rb

@@ -0,0 +1,43 @@
+require "retry_on_error"
+
+module Imap; end
+
+module Imap::Backup
+  module Client; end
+
+  class Client::AutomaticLoginWrapper
+    include RetryOnError
+
+    LOGIN_RETRY_CLASSES = [::EOFError, ::Errno::ECONNRESET, ::SocketError].freeze
+
+    attr_reader :client
+    attr_reader :login_called
+
+    def initialize(client:)
+      @client = client
+      @login_called = false
+    end
+
+    def method_missing(method_name, *arguments, &block)
+      if login_called
+        client.send(method_name, *arguments, &block)
+      else
+        do_first_login
+        client.send(method_name, *arguments, &block) if method_name != :login
+      end
+    end
+
+    def respond_to_missing?(method_name, _include_private = false)
+      client.respond_to?(method_name)
+    end
+
+    private
+
+    def do_first_login
+      retry_on_error(errors: LOGIN_RETRY_CLASSES) do
+        client.login
+        @login_called = true
+      end
+    end
+  end
+end

data/lib/imap/backup/serializer/appender.rb

@@ -29,7 +29,7 @@ module Imap::Backup
       rollback_on_error do
         serialized = to_serialized(message)
         mbox.append serialized
-        imap.append uid, serialized.length, flags: flags
+        imap.append uid, serialized.bytesize, flags: flags
       rescue StandardError => e
         raise <<-ERROR.gsub(/^\s*/m, "")
           [#{folder}] failed to append message #{uid}: #{message}.

data/lib/imap/backup/serializer/delayed_metadata_serializer.rb

@@ -33,7 +33,7 @@ module Imap::Backup
       tsx.fail_outside_transaction!(:append)
       mboxrd_message = Email::Mboxrd::Message.new(message)
       serialized = mboxrd_message.to_serialized
-      tsx.data[:metadata] << {uid: uid, length: serialized.length, flags: flags}
+      tsx.data[:metadata] << {uid: uid, length: serialized.bytesize, flags: flags}
       mbox.append(serialized)
     end
 

data/lib/imap/backup/serializer/integrity_checker.rb

@@ -15,6 +15,9 @@ module Imap::Backup
     end
 
     def run
+      Logger.logger.debug(
+        "[IntegrityChecker] checking '#{imap.pathname}' against '#{mbox.pathname}'"
+      )
       if !imap.valid?
         message = ".imap file '#{imap.pathname}' is corrupt"
         raise Serializer::FolderIntegrityError, message
@@ -56,14 +59,18 @@ module Imap::Backup
     def check_mbox_length!
       last = imap.messages[-1]
 
-      if mbox.length < last.offset + last.length
+      expected = last.offset + last.length
+      Logger.logger.debug(
+        "[IntegrityChecker] mbox length is #{mbox.length}, expected length is #{expected}"
+      )
+      if mbox.length < expected
         message =
           ".mbox file '#{mbox.pathname}' is shorter than indicated by " \
           ".imap file '#{imap.pathname}'"
         raise Serializer::FolderIntegrityError, message
       end
 
-      if mbox.length > last.offset + last.length
+      if mbox.length > expected
         message =
           ".mbox file '#{mbox.pathname}' is longer than indicated by " \
           ".imap file '#{imap.pathname}'"
@@ -77,6 +84,11 @@ module Imap::Backup
 
         next if text.start_with?("From ")
 
+        Logger.logger.debug(
+          "[IntegrityChecker] looking for message with UID #{m.uid} " \
+          "at offset #{m.offset}, " \
+          "mbox starts with '#{text[0..200]}', expecting 'From '"
+        )
         message =
           "Message #{m.uid} not found at expected offset #{m.offset} " \
           "in file '#{mbox.pathname}'"

data/lib/imap/backup/setup/global_options/download_strategy_chooser.rb

@@ -7,11 +7,9 @@ class Imap::Backup::Setup; end
 class Imap::Backup::Setup::GlobalOptions
   class DownloadStrategyChooser
     attr_reader :config
-    attr_reader :highline
 
-    def initialize(config:, highline:)
+    def initialize(config:)
       @config = config
-      @highline = highline
     end
 
     def run
@@ -81,5 +79,9 @@ class Imap::Backup::Setup::GlobalOptions
         highline.ask "Press a key "
       end
     end
+
+    def highline
+      Imap::Backup::Setup.highline
+    end
   end
 end

data/lib/imap/backup/setup/global_options.rb

@@ -8,11 +8,9 @@ module Imap::Backup
 
   class Setup::GlobalOptions
     attr_reader :config
-    attr_reader :highline
 
-    def initialize(config:, highline:)
+    def initialize(config:)
       @config = config
-      @highline = highline
     end
 
     def run
@@ -46,8 +44,12 @@ module Imap::Backup
       current = strategies.find { |s| s[:key] == config.download_strategy }
       changed = config.download_strategy_modified ? " *" : ""
       menu.choice("change download strategy (currently: '#{current[:description]}')#{changed}") do
-        DownloadStrategyChooser.new(config: config, highline: Setup.highline).run
+        DownloadStrategyChooser.new(config: config).run
       end
     end
+
+    def highline
+      Imap::Backup::Setup.highline
+    end
   end
 end

data/lib/imap/backup/setup.rb

@@ -76,7 +76,7 @@ module Imap::Backup
     def modify_global_options(menu)
       changed = config.modified? ? " *" : ""
       menu.choice("modify global options#{changed}") do
-        GlobalOptions.new(config: config, highline: Setup.highline).run
+        GlobalOptions.new(config: config).run
       end
     end
 

data/lib/imap/backup/version.rb

@@ -1,7 +1,7 @@
 module Imap; end
 
 module Imap::Backup
-  MAJOR    = 11
+  MAJOR    = 12
   MINOR    = 1
   REVISION = 0
   PRE      = nil

data/lib/retry_on_error.rb

@@ -1,11 +1,12 @@
 module RetryOnError
-  def retry_on_error(errors:, limit: 10)
+  def retry_on_error(errors:, limit: 10, on_error: nil)
     tries ||= 1
     yield
   rescue *errors => e
     if tries < limit
       message = "#{e}, attempt #{tries} of #{limit}"
       Imap::Backup::Logger.logger.debug message
+      on_error&.call
       tries += 1
       retry
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: imap-backup
 version: !ruby/object:Gem::Version
-  version: 11.1.0
+  version: 12.1.0
 platform: ruby
 authors:
 - Joe Yates
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-08-31 00:00:00.000000000 Z
+date: 2023-09-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: highline
@@ -247,14 +247,14 @@ files:
 - lib/imap/backup/cli/helpers.rb
 - lib/imap/backup/cli/local.rb
 - lib/imap/backup/cli/local/check.rb
-- lib/imap/backup/cli/migrate.rb
-- lib/imap/backup/cli/mirror.rb
 - lib/imap/backup/cli/remote.rb
 - lib/imap/backup/cli/restore.rb
 - lib/imap/backup/cli/setup.rb
 - lib/imap/backup/cli/stats.rb
+- lib/imap/backup/cli/transfer.rb
 - lib/imap/backup/cli/utils.rb
 - lib/imap/backup/client/apple_mail.rb
+- lib/imap/backup/client/automatic_login_wrapper.rb
 - lib/imap/backup/client/default.rb
 - lib/imap/backup/configuration.rb
 - lib/imap/backup/downloader.rb

data/lib/imap/backup/cli/migrate.rb

@@ -1,87 +0,0 @@
-require "imap/backup/cli/folder_enumerator"
-require "imap/backup/migrator"
-
-module Imap; end
-
-module Imap::Backup
-  class CLI::Migrate < Thor
-    include Thor::Actions
-    include CLI::Helpers
-
-    attr_reader :destination_delimiter
-    attr_reader :destination_email
-    attr_reader :destination_prefix
-    attr_reader :config_path
-    attr_reader :reset
-    attr_reader :source_delimiter
-    attr_reader :source_email
-    attr_reader :source_prefix
-
-    def initialize(
-      source_email,
-      destination_email,
-      config: nil,
-      destination_delimiter: "/",
-      destination_prefix: "",
-      reset: false,
-      source_delimiter: "/",
-      source_prefix: ""
-    )
-      super([])
-      @destination_delimiter = destination_delimiter
-      @destination_email = destination_email
-      @destination_prefix = destination_prefix
-      @config_path = config
-      @reset = reset
-      @source_delimiter = source_delimiter
-      @source_email = source_email
-      @source_prefix = source_prefix
-    end
-
-    no_commands do
-      def run
-        check_accounts!
-        folders.each do |serializer, folder|
-          Migrator.new(serializer, folder, reset: reset).run
-        end
-      end
-
-      def check_accounts!
-        if destination_email == source_email
-          raise "Source and destination accounts cannot be the same!"
-        end
-
-        raise "Account '#{destination_email}' does not exist" if !destination_account
-
-        raise "Account '#{source_email}' does not exist" if !source_account
-      end
-
-      def config
-        @config ||= load_config(config: config_path)
-      end
-
-      def enumerator_options
-        {
-          destination: destination_account,
-          destination_delimiter: destination_delimiter,
-          destination_prefix: destination_prefix,
-          source: source_account,
-          source_delimiter: source_delimiter,
-          source_prefix: source_prefix
-        }
-      end
-
-      def folders
-        CLI::FolderEnumerator.new(**enumerator_options)
-      end
-
-      def destination_account
-        config.accounts.find { |a| a.username == destination_email }
-      end
-
-      def source_account
-        config.accounts.find { |a| a.username == source_email }
-      end
-    end
-  end
-end

data/lib/imap/backup/cli/mirror.rb

@@ -1,97 +0,0 @@
-require "imap/backup/cli/folder_enumerator"
-require "imap/backup/mirror"
-
-module Imap; end
-
-module Imap::Backup
-  class CLI::Mirror < Thor
-    include Thor::Actions
-    include CLI::Helpers
-
-    attr_reader :destination_delimiter
-    attr_reader :destination_email
-    attr_reader :destination_prefix
-    attr_reader :config_path
-    attr_reader :source_delimiter
-    attr_reader :source_email
-    attr_reader :source_prefix
-
-    def initialize(
-      source_email,
-      destination_email,
-      config: nil,
-      destination_delimiter: "/",
-      destination_prefix: "",
-      source_delimiter: "/",
-      source_prefix: ""
-    )
-      super([])
-      @destination_delimiter = destination_delimiter
-      @destination_email = destination_email
-      @destination_prefix = destination_prefix
-      @config_path = config
-      @source_delimiter = source_delimiter
-      @source_email = source_email
-      @source_prefix = source_prefix
-    end
-
-    no_commands do
-      def run
-        check_accounts!
-        warn_if_source_account_is_not_in_mirror_mode
-
-        CLI::Backup.new(config: config_path, accounts: source_email).run
-
-        folders.each do |serializer, folder|
-          Mirror.new(serializer, folder).run
-        end
-      end
-
-      def check_accounts!
-        if destination_email == source_email
-          raise "Source and destination accounts cannot be the same!"
-        end
-
-        raise "Account '#{destination_email}' does not exist" if !destination_account
-
-        raise "Account '#{source_email}' does not exist" if !source_account
-      end
-
-      def warn_if_source_account_is_not_in_mirror_mode
-        return if source_account.mirror_mode
-
-        message =
-          "The account '#{source_account.username}' " \
-          "is not set up to make mirror backups"
-        Logger.logger.warn message
-      end
-
-      def config
-        @config = load_config(config: config_path)
-      end
-
-      def enumerator_options
-        {
-          destination: destination_account,
-          destination_delimiter: destination_delimiter,
-          destination_prefix: destination_prefix,
-          source: source_account,
-          source_delimiter: source_delimiter,
-          source_prefix: source_prefix
-        }
-      end
-
-      def folders
-        CLI::FolderEnumerator.new(**enumerator_options)
-      end
-
-      def destination_account
-        config.accounts.find { |a| a.username == destination_email }
-      end
-
-      def source_account
-        config.accounts.find { |a| a.username == source_email }
-      end
-    end
-  end
-end