litestream 0.4.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3437fcda9be45b8f50a4fcae89e6cc07d3ea590d7e3f0d2f10e50c9f8c6058b6
-  data.tar.gz: f53e83796ed362491b50929d2924d070b6efa166830f7de6195e39300121be81
+  metadata.gz: c7942f70131f11cfcfad19c07442ab78571ad5bb28f67d8e38f1c35573a04413
+  data.tar.gz: 7584c2778dff02a7ded7078656ba57247146a67924c2ef05f4e1fe8407ea04e9
 SHA512:
-  metadata.gz: d8731b80f97dee5bc512e4f4acd58aebb63a435379736d281edc8bd2b42e6cc524c2eb7a55fefc2cf93218b91b2fb12c4fe180685031c9924dacd73a918fe344
-  data.tar.gz: 209fec5c3e355675feeee661ca64194cae370b85cbe90e162aaa2df059b2ccc5806b44e94daf706c29140e106e716eb1d3505a5efeeec65dfecbe4ee763ece09
+  metadata.gz: a48912bbcafb448473ebfd6ff378db5faa0a7f1830b786dff1b6574f8bd9ce210fd1f6b2f6652555544a38923a68fd87562abbca8e497db4282facc91a353ce0
+  data.tar.gz: a2a798e38ccf63d1fd21008bf1ef626a77652f8a8d1ef192fb1eeb11344f933d209f661dc3d24afd905d534ad5c295f4a0afb07b5cc45781ce152b2431746d72

data/README.md

@@ -92,9 +92,135 @@ bin/rails litestream:replicate -- -exec "foreman start"
 
 This example utilizes the `-exec` option available on [the `replicate` command](https://litestream.io/reference/replicate/) which provides basic process management, since Litestream will exit when the child process exits. In this example, we only launch our collection of Rails application processes (like Rails and SolidQueue, for example) after the Litestream replication process is ready.
 
-The rake task is the recommended way to interact with the Litestream replication process in your Rails application or Ruby project. But, you _can_ take full manual control over the replication process and simply run the `litestream replicate --config config/litestream.yml` command to start the Litestream process. Since the gem installs the native executable via Bundler, the `litestream` command will be available in your `PATH`.
+The Litestream `replicate` command supports the following options, which can be passed through the rake task:
+```shell
+-config PATH
+    Specifies the configuration file.
+    Defaults to /etc/litestream.yml
+
+-exec CMD
+    Executes a subcommand. Litestream will exit when the child
+    process exits. Useful for simple process management.
+
+-no-expand-env
+    Disables environment variable expansion in configuration file.
+```
+
+### Restoration
+
+You can restore any replicated database at any point using the gem's provided `litestream:restore` rake task. This rake task requires that you specify which specific database you want to restore. As with the `litestream:replicate` task, you pass arguments to the rake task via argument forwarding. For example, to restore the production database, you would run:
+
+```shell
+bin/rails litestream:restore -- --database=storage/production.sqlite3
+# or
+bundle exec rake litestream:restore -- --database=storage/production.sqlite3
+```
+
+You can restore any of the databases specified in your `config/litestream.yml` file. The `--database` argument should be the path to the database file you want to restore and must match the value for the `path` key of one of your configured databases. The `litestream:restore` rake task will automatically load the configuration file and set the environment variables before calling the Litestream executable.
+
+If you need to pass arguments through the rake task to the underlying `litestream` command, that can be done with additional forwarded arguments:
+
+```shell
+bin/rails litestream:replicate -- --database=storage/production.sqlite3 --if-db-not-exists
+```
+
+You can forward arguments in whatever order you like, you simply need to ensure that the `--database` argument is present. You can also use either a single-dash `-database` or double-dash `--database` argument format. The Litestream `restore` command supports the following options, which can be passed through the rake task:
+
+```shell
+-o PATH
+    Output path of the restored database.
+    Defaults to original DB path.
+
+-if-db-not-exists
+    Returns exit code of 0 if the database already exists.
+
+-if-replica-exists
+    Returns exit code of 0 if no backups found.
+
+-parallelism NUM
+    Determines the number of WAL files downloaded in parallel.
+    Defaults to 8
+
+-replica NAME
+    Restore from a specific replica.
+    Defaults to replica with latest data.
+
+-generation NAME
+    Restore from a specific generation.
+    Defaults to generation with latest data.
+
+-index NUM
+    Restore up to a specific WAL index (inclusive).
+    Defaults to use the highest available index.
+
+-timestamp TIMESTAMP
+    Restore to a specific point-in-time.
+    Defaults to use the latest available backup.
+
+-config PATH
+    Specifies the configuration file.
+    Defaults to /etc/litestream.yml
+
+-no-expand-env
+    Disables environment variable expansion in configuration file.
+```
+
+### Introspection
 
-The full set of commands available to the `litestream` executable are covered in Litestream's [command reference](https://litestream.io/reference/). Currently, only the `replicate` command is provided as a rake task by the gem.
+Litestream offers a handful of commands that allow you to introspect the state of your replication. The gem provides a few rake tasks that wrap these commands for you. For example, you can list the databases that Litestream is configured to replicate:
+
+```shell
+bin/rails litestream:databases
+```
+
+This will return a list of databases and their configured replicas:
+
+```
+path                                                 replicas
+/Users/you/Code/your-app/storage/production.sqlite3  s3
+```
+
+You can also list the generations of a specific database:
+
+```shell
+bin/rails litestream:generations -- --database=storage/production.sqlite3
+```
+
+This will list all generations for the specified database, including stats about their lag behind the primary database and the time range they cover.
+
+```
+name  generation        lag     start                 end
+s3    a295b16a796689f3  -156ms  2024-04-17T00:01:19Z  2024-04-17T00:01:19Z
+```
+
+Finally, you can list the snapshots available for a database:
+
+```shell
+bin/rails litestream:snapshots -- --database=storage/production.sqlite3
+```
+
+This command lists snapshots available for that specified database:
+
+```
+replica  generation        index  size     created
+s3       a295b16a796689f3  1      4645465  2024-04-17T00:01:19Z
+```
+
+### Additional commands
+
+The rake tasks are the recommended way to interact with the Litestream utility in your Rails application or Ruby project. But, you _can_ work directly with the Litestream CLI. Since the gem installs the native executable via Bundler, the `litestream` command will be available in your `PATH`.
+
+The full set of commands available to the `litestream` executable are covered in Litestream's [command reference](https://litestream.io/reference/), but can be summarized as:
+
+```shell
+litestream databases [arguments]
+litestream generations [arguments] DB_PATH|REPLICA_URL
+litestream replicate [arguments]
+litestream restore [arguments] DB_PATH|REPLICA_URL
+litestream snapshots [arguments] DB_PATH|REPLICA_URL
+litestream version
+litestream wal [arguments] DB_PATH|REPLICA_URL
+```
 
 ### Using in development
 

data/exe/litestream

@@ -5,7 +5,6 @@ require "litestream/commands"
 
 begin
   command = [Litestream::Commands.executable, *ARGV]
-  puts command.inspect
   exec(*command)
 rescue Litestream::Commands::UnsupportedPlatformException, Litestream::Commands::ExecutableNotFoundException => e
   warn("ERROR: " + e.message)

data/lib/litestream/commands.rb

@@ -6,90 +6,128 @@ module Litestream
     GEM_NAME = "litestream"
 
     # raised when the host platform is not supported by upstream litestream's binary releases
-    class UnsupportedPlatformException < StandardError
-    end
+    UnsupportedPlatformException = Class.new(StandardError)
 
     # raised when the litestream executable could not be found where we expected it to be
-    class ExecutableNotFoundException < StandardError
-    end
+    ExecutableNotFoundException = Class.new(StandardError)
 
     # raised when LITESTREAM_INSTALL_DIR does not exist
-    class DirectoryNotFoundException < StandardError
-    end
+    DirectoryNotFoundException = Class.new(StandardError)
 
-    def self.platform
-      [:cpu, :os].map { |m| Gem::Platform.local.send(m) }.join("-")
-    end
+    # raised when a litestream command requires a database argument but it isn't provided
+    DatabaseRequiredException = Class.new(StandardError)
+
+    class << self
+      def platform
+        [:cpu, :os].map { |m| Gem::Platform.local.send(m) }.join("-")
+      end
 
-    def self.executable(exe_path: DEFAULT_DIR)
-      litestream_install_dir = ENV["LITESTREAM_INSTALL_DIR"]
-      if litestream_install_dir
-        if File.directory?(litestream_install_dir)
-          warn "NOTE: using LITESTREAM_INSTALL_DIR to find litestream executable: #{litestream_install_dir}"
-          exe_path = litestream_install_dir
-          exe_file = File.expand_path(File.join(litestream_install_dir, "litestream"))
+      def executable(exe_path: DEFAULT_DIR)
+        litestream_install_dir = ENV["LITESTREAM_INSTALL_DIR"]
+        if litestream_install_dir
+          if File.directory?(litestream_install_dir)
+            warn "NOTE: using LITESTREAM_INSTALL_DIR to find litestream executable: #{litestream_install_dir}"
+            exe_path = litestream_install_dir
+            exe_file = File.expand_path(File.join(litestream_install_dir, "litestream"))
+          else
+            raise DirectoryNotFoundException, <<~MESSAGE
+              LITESTREAM_INSTALL_DIR is set to #{litestream_install_dir}, but that directory does not exist.
+            MESSAGE
+          end
         else
-          raise DirectoryNotFoundException, <<~MESSAGE
-            LITESTREAM_INSTALL_DIR is set to #{litestream_install_dir}, but that directory does not exist.
-          MESSAGE
+          if Litestream::Upstream::NATIVE_PLATFORMS.keys.none? { |p| Gem::Platform.match_gem?(Gem::Platform.new(p), GEM_NAME) }
+            raise UnsupportedPlatformException, <<~MESSAGE
+              litestream-ruby does not support the #{platform} platform
+              Please install litestream following instructions at https://litestream.io/install
+            MESSAGE
+          end
+
+          exe_file = Dir.glob(File.expand_path(File.join(exe_path, "*", "litestream"))).find do |f|
+            Gem::Platform.match_gem?(Gem::Platform.new(File.basename(File.dirname(f))), GEM_NAME)
+          end
         end
-      else
-        if Litestream::Upstream::NATIVE_PLATFORMS.keys.none? { |p| Gem::Platform.match_gem?(Gem::Platform.new(p), GEM_NAME) }
-          raise UnsupportedPlatformException, <<~MESSAGE
-            litestream-ruby does not support the #{platform} platform
-            Please install litestream following instructions at https://litestream.io/install
+
+        if exe_file.nil? || !File.exist?(exe_file)
+          raise ExecutableNotFoundException, <<~MESSAGE
+            Cannot find the litestream executable for #{platform} in #{exe_path}
+
+            If you're using bundler, please make sure you're on the latest bundler version:
+
+                gem install bundler
+                bundle update --bundler
+
+            Then make sure your lock file includes this platform by running:
+
+                bundle lock --add-platform #{platform}
+                bundle install
+
+            See `bundle lock --help` output for details.
+
+            If you're still seeing this message after taking those steps, try running
+            `bundle config` and ensure `force_ruby_platform` isn't set to `true`. See
+            https://github.com/fractaledmind/litestream-ruby#check-bundle_force_ruby_platform
+            for more details.
           MESSAGE
         end
 
-        exe_file = Dir.glob(File.expand_path(File.join(exe_path, "*", "litestream"))).find do |f|
-          Gem::Platform.match_gem?(Gem::Platform.new(File.basename(File.dirname(f))), GEM_NAME)
-        end
+        exe_file
       end
 
-      if exe_file.nil? || !File.exist?(exe_file)
-        raise ExecutableNotFoundException, <<~MESSAGE
-          Cannot find the litestream executable for #{platform} in #{exe_path}
+      def replicate(argv = {})
+        execute("replicate", argv)
+      end
 
-          If you're using bundler, please make sure you're on the latest bundler version:
+      def restore(database, argv = {})
+        raise DatabaseRequiredException, "database argument is required for restore command, e.g. litestream:restore -- --database=path/to/database.sqlite" if database.nil?
 
-              gem install bundler
-              bundle update --bundler
+        dir, file = File.split(database)
+        ext = File.extname(file)
+        base = File.basename(file, ext)
+        now = Time.now.utc.strftime("%Y%m%d%H%M%S")
 
-          Then make sure your lock file includes this platform by running:
+        args = {
+          "-o" => File.join(dir, "#{base}-#{now}#{ext}")
+        }.merge(argv)
 
-              bundle lock --add-platform #{platform}
-              bundle install
+        execute("restore", args, database)
+      end
 
-          See `bundle lock --help` output for details.
+      def databases(argv = {})
+        execute("databases", argv)
+      end
 
-          If you're still seeing this message after taking those steps, try running
-          `bundle config` and ensure `force_ruby_platform` isn't set to `true`. See
-          https://github.com/fractaledmind/litestream-ruby#check-bundle_force_ruby_platform
-          for more details.
-        MESSAGE
+      def generations(database, argv = {})
+        raise DatabaseRequiredException, "database argument is required for generations command, e.g. litestream:generations -- --database=path/to/database.sqlite" if database.nil?
+
+        execute("generations", argv, database)
       end
 
-      exe_file
-    end
+      def snapshots(database, argv = {})
+        raise DatabaseRequiredException, "database argument is required for snapshots command, e.g. litestream:snapshots -- --database=path/to/database.sqlite" if database.nil?
 
-    def self.replicate(argv = {})
-      if Litestream.configuration
-        ENV["LITESTREAM_REPLICA_BUCKET"] = Litestream.configuration.replica_bucket
-        ENV["LITESTREAM_ACCESS_KEY_ID"] = Litestream.configuration.replica_key_id
-        ENV["LITESTREAM_SECRET_ACCESS_KEY"] = Litestream.configuration.replica_access_key
+        execute("snapshots", argv, database)
       end
 
-      args = {
-        "--config" => Rails.root.join("config", "litestream.yml").to_s
-      }.merge(argv).to_a.flatten.compact
+      private
 
-      command = [executable, "replicate", *args]
-      puts command.inspect
+      def execute(command, argv = {}, database = nil)
+        if Litestream.configuration
+          ENV["LITESTREAM_REPLICA_BUCKET"] ||= Litestream.configuration.replica_bucket
+          ENV["LITESTREAM_ACCESS_KEY_ID"] ||= Litestream.configuration.replica_key_id
+          ENV["LITESTREAM_SECRET_ACCESS_KEY"] ||= Litestream.configuration.replica_access_key
+        end
 
-      # To release the resources of the Ruby process, just fork and exit.
-      # The forked process executes litestream and replaces itself.
-      if fork.nil?
-        exec(*command)
+        args = {
+          "--config" => Rails.root.join("config", "litestream.yml").to_s
+        }.merge(argv).to_a.flatten.compact
+        cmd = [executable, command, *args, database].compact
+
+        # To release the resources of the Ruby process, just fork and exit.
+        # The forked process executes litestream and replaces itself.
+        if fork.nil?
+          puts cmd.inspect if ENV["DEBUG"]
+          exec(*cmd)
+        end
       end
     end
   end

data/lib/litestream/version.rb

@@ -1,3 +1,3 @@
 module Litestream
-  VERSION = "0.4.0"
+  VERSION = "0.5.1"
 end

data/lib/litestream.rb

@@ -14,7 +14,6 @@ module Litestream
     attr_accessor :database_path, :replica_bucket, :replica_key_id, :replica_access_key
 
     def initialize
-      @mailer_sender = "donotreply@example.com"
     end
   end
 end

data/lib/tasks/litestream_tasks.rake

@@ -13,7 +13,7 @@ namespace :litestream do
     true
   end
 
-  desc "Start a process to monitor and continuously replicate the SQLite databases defined in your configuration file"
+  desc 'Monitor and continuously replicate SQLite databases defined in your config file, e.g. rake litestream:replicate -- -exec "foreman start"'
   task replicate: :environment do
     options = {}
     if (separator_index = ARGV.index("--"))
@@ -24,4 +24,52 @@ namespace :litestream do
 
     Litestream::Commands.replicate(options)
   end
+
+  desc "Restore a SQLite database from a Litestream replica, e.g. rake litestream:restore -- -database=storage/production.sqlite3"
+  task restore: :environment do
+    options = {}
+    if (separator_index = ARGV.index("--"))
+      ARGV.slice(separator_index + 1, ARGV.length)
+        .map { |pair| pair.split("=") }
+        .each { |opt| options[opt[0]] = opt[1] || nil }
+    end
+
+    Litestream::Commands.restore(options.delete("--database") || options.delete("-database"), options)
+  end
+
+  desc "List all databases and associated replicas in the config file, e.g. rake litestream:databases -- -no-expand-env"
+  task databases: :environment do
+    options = {}
+    if (separator_index = ARGV.index("--"))
+      ARGV.slice(separator_index + 1, ARGV.length)
+        .map { |pair| pair.split("=") }
+        .each { |opt| options[opt[0]] = opt[1] || nil }
+    end
+
+    Litestream::Commands.databases(options)
+  end
+
+  desc "List all generations for a database or replica, e.g. rake litestream:generations -- -database=storage/production.sqlite3"
+  task generations: :environment do
+    options = {}
+    if (separator_index = ARGV.index("--"))
+      ARGV.slice(separator_index + 1, ARGV.length)
+        .map { |pair| pair.split("=") }
+        .each { |opt| options[opt[0]] = opt[1] || nil }
+    end
+
+    Litestream::Commands.generations(options.delete("--database") || options.delete("-database"), options)
+  end
+
+  desc "List all snapshots for a database or replica, e.g. rake litestream:snapshots -- -database=storage/production.sqlite3"
+  task snapshots: :environment do
+    options = {}
+    if (separator_index = ARGV.index("--"))
+      ARGV.slice(separator_index + 1, ARGV.length)
+        .map { |pair| pair.split("=") }
+        .each { |opt| options[opt[0]] = opt[1] || nil }
+    end
+
+    Litestream::Commands.snapshots(options.delete("--database") || options.delete("-database"), options)
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: litestream
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.1
 platform: ruby
 authors:
 - Stephen Margheim
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-04-12 00:00:00.000000000 Z
+date: 2024-04-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rubyzip