litestream 0.3.3-x86_64-linux → 0.4.0-x86_64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a811fa93e4a3db5f8ef90904fea642ccb0ea146ace97bf141b716fe6b18e6ce6
-  data.tar.gz: 6b0cae674c1d28cbcefae49793e39df01da662ebcc567458852b819ef87e651b
+  metadata.gz: 296b0fda9f7f9cfce9f44369d76170a5a2437ecd26ea976f668f4e6ac9ccf4da
+  data.tar.gz: 8e18eb8bf7b9cd21e54e98cd00096129146f38d949f4c51c5b7cee34aebbc93f
 SHA512:
-  metadata.gz: 801f7a80f55c91119c41f685b52cf86ca70aa01e74396b85b56fd399e1626a423aa0bee0bea2146813810d7759c72720849ce9fc967b24bea66b8ff2024bf72c
-  data.tar.gz: 5da3badade7b5ab0c282ae208d26aa1ff62e30e29c930bedcf5542747edd7b989fcf0f1252a84c453fef502a685a6e850fb380fe189ba7f99067e7145924351d
+  metadata.gz: 75bab85fa75e5199110b20276b983bdaf404d8696792f83e53d9a06fee1a6cd155c2363ed51b4c8b0c101b45a2d0aaa83a29a18bb06cba31de2428cac4f58336
+  data.tar.gz: f51dbe2cd1c998f834f825cff2749fbbaee824efd9b4339013d753c7d20bb6f4ffca38873f0d7172913afbb9b038a1f049d91fecacde9ee8627ca8bd328d33cf

data/README.md

@@ -49,13 +49,13 @@ LITESTREAM_INSTALL_DIR=.bin
 
 You configure the Litestream executable through the [`config/litestream.yml` file](https://litestream.io/reference/config/), which is a standard Litestream configuration file as if Litestream was running in a traditional installation.
 
-The gem streamlines the configuration process by providing a default configuration file for you. This configuration file will backup one application database to one replication bucket. In order to ensure that no secrets are stored in plain-text in your repository, this configuration file leverages Litestream's support for environment variables. The default configuration file looks like this:
+The gem streamlines the configuration process by providing a default configuration file for you. This configuration file will backup all SQLite databases defined in your `config/database.yml` file to one replication bucket. In order to ensure that no secrets are stored in plain-text in your repository, this configuration file leverages Litestream's support for environment variables. The default configuration file looks like this if you only have one SQLite database:
 
 ```yaml
 dbs:
   - path: $LITESTREAM_DATABASE_PATH
     replicas:
-      - url: $LITESTREAM_REPLICA_URL
+      - url: $LITESTREAM_REPLICA_BUCKET
         access-key-id: $LITESTREAM_ACCESS_KEY_ID
         secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY
 ```
@@ -66,7 +66,7 @@ The gem also provides a default initializer file at `config/initializers/litestr
 litestream_credentials = Rails.application.credentials.litestream
 Litestream.configure do |config|
   config.database_path = ActiveRecord::Base.connection_db_config.database
-  config.replica_url = litestream_credentials.replica_url
+  config.replica_bucket = litestream_credentials.replica_bucket
   config.replica_key_id = litestream_credentials.replica_key_id
   config.replica_access_key = litestream_credentials.replica_access_key
 end
@@ -92,9 +92,94 @@ bin/rails litestream:replicate -- -exec "foreman start"
 
 This example utilizes the `-exec` option available on [the `replicate` command](https://litestream.io/reference/replicate/) which provides basic process management, since Litestream will exit when the child process exits. In this example, we only launch our collection of Rails application processes (like Rails and SolidQueue, for example) after the Litestream replication process is ready.
 
-The rake task is the recommended way to interact with the Litestream replication process in your Rails application or Ruby project. But, you _can_ take full manual control over the replication process and simply run the `litestream replicate --config config/litestream.yml` command to start the Litestream process. Since the gem installs the native executable via Bundler, the `litestream` command will be available in your `PATH`.
+The Litestream `replicate` command supports the following options, which can be passed through the rake task:
+```shell
+-config PATH
+    Specifies the configuration file.
+    Defaults to /etc/litestream.yml
+
+-exec CMD
+    Executes a subcommand. Litestream will exit when the child
+    process exits. Useful for simple process management.
+
+-no-expand-env
+    Disables environment variable expansion in configuration file.
+```
+
+### Restoration
 
-The full set of commands available to the `litestream` executable are covered in Litestream's [command reference](https://litestream.io/reference/). Currently, only the `replicate` command is provided as a rake task by the gem.
+You can restore any replicated database at any point using the gem's provided `litestream:restore` rake task. This rake task requires that you specify which specific database you want to restore. As with the `litestream:replicate` task, you pass arguments to the rake task via argument forwarding. For example, to restore the production database, you would run:
+
+```shell
+bin/rails litestream:restore -- --database=storage/production.sqlite3
+# or
+bundle exec rake litestream:restore -- --database=storage/production.sqlite3
+```
+
+You can restore any of the databases specified in your `config/litestream.yml` file. The `--database` argument should be the path to the database file you want to restore and must match the value for the `path` key of one of your configured databases. The `litestream:restore` rake task will automatically load the configuration file and set the environment variables before calling the Litestream executable.
+
+If you need to pass arguments through the rake task to the underlying `litestream` command, that can be done with additional forwarded arguments:
+
+```shell
+bin/rails litestream:replicate -- --database=storage/production.sqlite3 --if-db-not-exists
+```
+
+You can forward arguments in whatever order you like, you simply need to ensure that the `--database` argument is present. You can also use either a single-dash `-database` or double-dash `--database` argument format. The Litestream `restore` command supports the following options, which can be passed through the rake task:
+
+```shell
+-o PATH
+    Output path of the restored database.
+    Defaults to original DB path.
+
+-if-db-not-exists
+    Returns exit code of 0 if the database already exists.
+
+-if-replica-exists
+    Returns exit code of 0 if no backups found.
+
+-parallelism NUM
+    Determines the number of WAL files downloaded in parallel.
+    Defaults to 8
+
+-replica NAME
+    Restore from a specific replica.
+    Defaults to replica with latest data.
+
+-generation NAME
+    Restore from a specific generation.
+    Defaults to generation with latest data.
+
+-index NUM
+    Restore up to a specific WAL index (inclusive).
+    Defaults to use the highest available index.
+
+-timestamp TIMESTAMP
+    Restore to a specific point-in-time.
+    Defaults to use the latest available backup.
+
+-config PATH
+    Specifies the configuration file.
+    Defaults to /etc/litestream.yml
+
+-no-expand-env
+    Disables environment variable expansion in configuration file.
+```
+
+### Additional commands
+
+The rake tasks are the recommended way to interact with the Litestream utility in your Rails application or Ruby project. But, you _can_ work directly with the Litestream CLI. Since the gem installs the native executable via Bundler, the `litestream` command will be available in your `PATH`.
+
+The full set of commands available to the `litestream` executable are covered in Litestream's [command reference](https://litestream.io/reference/), but can be summarized as:
+
+```shell
+litestream databases [arguments]
+litestream generations [arguments] DB_PATH|REPLICA_URL
+litestream replicate [arguments]
+litestream restore [arguments] DB_PATH|REPLICA_URL
+litestream snapshots [arguments] DB_PATH|REPLICA_URL
+litestream version
+litestream wal [arguments] DB_PATH|REPLICA_URL
+```
 
 ### Using in development
 
@@ -113,7 +198,7 @@ Once you have a MinIO server running, you can create a bucket for Litestream to
 ```ruby
 Litestream.configure do |config|
   config.database_path = ActiveRecord::Base.connection_db_config.database
-  config.replica_url = "s3://mybkt.localhost:9000/"
+  config.replica_bucket = "s3://mybkt.localhost:9000/"
   config.replica_key_id = "minioadmin"
   config.replica_access_key = "minioadmin"
 end

data/exe/litestream

@@ -5,7 +5,6 @@ require "litestream/commands"
 
 begin
   command = [Litestream::Commands.executable, *ARGV]
-  puts command.inspect
   exec(*command)
 rescue Litestream::Commands::UnsupportedPlatformException, Litestream::Commands::ExecutableNotFoundException => e
   warn("ERROR: " + e.message)

data/lib/litestream/commands.rb

@@ -6,16 +6,13 @@ module Litestream
     GEM_NAME = "litestream"
 
     # raised when the host platform is not supported by upstream litestream's binary releases
-    class UnsupportedPlatformException < StandardError
-    end
+    UnsupportedPlatformException = Class.new(StandardError)
 
     # raised when the litestream executable could not be found where we expected it to be
-    class ExecutableNotFoundException < StandardError
-    end
+    ExecutableNotFoundException = Class.new(StandardError)
 
     # raised when LITESTREAM_INSTALL_DIR does not exist
-    class DirectoryNotFoundException < StandardError
-    end
+    DirectoryNotFoundException = Class.new(StandardError)
 
     def self.platform
       [:cpu, :os].map { |m| Gem::Platform.local.send(m) }.join("-")
@@ -74,10 +71,9 @@ module Litestream
 
     def self.replicate(argv = {})
       if Litestream.configuration
-        ENV["LITESTREAM_DATABASE_PATH"] = Litestream.configuration.database_path
-        ENV["LITESTREAM_REPLICA_URL"] = Litestream.configuration.replica_url
-        ENV["LITESTREAM_ACCESS_KEY_ID"] = Litestream.configuration.replica_key_id
-        ENV["LITESTREAM_SECRET_ACCESS_KEY"] = Litestream.configuration.replica_access_key
+        ENV["LITESTREAM_REPLICA_BUCKET"] ||= Litestream.configuration.replica_bucket
+        ENV["LITESTREAM_ACCESS_KEY_ID"] ||= Litestream.configuration.replica_key_id
+        ENV["LITESTREAM_SECRET_ACCESS_KEY"] ||= Litestream.configuration.replica_access_key
       end
 
       args = {
@@ -93,5 +89,32 @@ module Litestream
         exec(*command)
       end
     end
+
+    def self.restore(database, argv = {})
+      if Litestream.configuration
+        ENV["LITESTREAM_REPLICA_BUCKET"] ||= Litestream.configuration.replica_bucket
+        ENV["LITESTREAM_ACCESS_KEY_ID"] ||= Litestream.configuration.replica_key_id
+        ENV["LITESTREAM_SECRET_ACCESS_KEY"] ||= Litestream.configuration.replica_access_key
+      end
+
+      dir, file = File.split(database)
+      ext = File.extname(file)
+      base = File.basename(file, ext)
+      now = Time.now.utc.strftime("%Y%m%d%H%M%S")
+
+      args = {
+        "--config" => Rails.root.join("config", "litestream.yml").to_s,
+        "-o" => File.join(dir, "#{base}-#{now}#{ext}")
+      }.merge(argv).to_a.flatten.compact
+
+      command = [executable, "restore", *args, database]
+      puts command.inspect
+
+      # To release the resources of the Ruby process, just fork and exit.
+      # The forked process executes litestream and replaces itself.
+      if fork.nil?
+        exec(*command)
+      end
+    end
   end
 end

data/lib/litestream/generators/litestream/install_generator.rb

@@ -8,7 +8,7 @@ module Litestream
       source_root File.expand_path("templates", __dir__)
 
       def copy_config_file
-        template "config.yml", "config/litestream.yml"
+        template "config.yml.erb", "config/litestream.yml"
       end
 
       def copy_initializer_file
@@ -27,6 +27,16 @@ module Litestream
           end
         end
       end
+
+      private
+
+      def production_sqlite_databases
+        ActiveRecord::Base
+          .configurations
+          .configs_for(env_name: "production", include_hidden: true)
+          .select { |config| ["sqlite3", "litedb"].include? config.adapter }
+          .map(&:database)
+      end
     end
   end
 end

data/lib/litestream/generators/litestream/templates/{config.yml → config.yml.erb}

@@ -10,8 +10,12 @@
 #
 # For more details, see: https://litestream.io/reference/config/
 dbs:
-  - path: $LITESTREAM_DATABASE_PATH
+  <%- production_sqlite_databases.each do |database| -%>
+  - path: <%= database %>
     replicas:
-      - url: $LITESTREAM_REPLICA_URL
+      - type: s3
+        bucket: $LITESTREAM_REPLICA_BUCKET
+        path: <%= database %>
         access-key-id: $LITESTREAM_ACCESS_KEY_ID
         secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY
+  <%- end -%>

data/lib/litestream/generators/litestream/templates/initializer.rb

@@ -1,30 +1,25 @@
 # Use this hook to configure the litestream-ruby gem.
 # All configuration options will be available as environment variables, e.g.
-# config.database_path becomes LITESTREAM_DATABASE_PATH
+# config.replica_bucket becomes LITESTREAM_REPLICA_BUCKET
 # This allows you to configure Litestream using Rails encrypted credentials,
 # or some other mechanism where the values are only avaialble at runtime.
 
 Litestream.configure do |config|
   # An example of using Rails encrypted credentials to configure Litestream.
   # litestream_credentials = Rails.application.credentials.litestream
-  #
-  # The absolute or relative path to a SQLite database file.
-  # Litestream will monitor this file for changes and replicate them to the
-  # any of the configured replicas specified for this database in the
-  # `litestream.yml` configuration file.
-  # When using SQLite as your database engine for ActiveRecord, you should always
-  # set this to the path of your SQLite database file. You can do so using Rails'
-  # existing knowledge of the database path.
-  # config.database_path = ActiveRecord::Base.connection_db_config.database
 
-  # Short-hand form of specifying a replica location.
-  # When using S3, a value will look like "s3://mybkt.litestream.io/db"
-  # Litestream also supports Azure Blog Storage, Backblaze B2, DigitalOcean Spaces,
+  # Replica-specific bucket location.
+  # This will be your bucket's URL without the `https://` prefix.
+  # For example, if you used DigitalOcean Spaces, your bucket URL could look like:
+  #   https://myapp.fra1.digitaloceanspaces.com
+  # And so you should set your `replica_bucket` to:
+  #   myapp.fra1.digitaloceanspaces.com
+  # Litestream supports Azure Blog Storage, Backblaze B2, DigitalOcean Spaces,
   # Scaleway Object Storage, Google Cloud Storage, Linode Object Storage, and
   # any SFTP server.
   # In this example, we are using Rails encrypted credentials to store the URL to
   # our storage provider bucket.
-  # config.replica_url = litestream_credentials.replica_url
+  # config.replica_bucket = litestream_credentials.replica_bucket
 
   # Replica-specific authentication key.
   # Litestream needs authentication credentials to access your storage provider bucket.

data/lib/litestream/version.rb

@@ -1,3 +1,3 @@
 module Litestream
-  VERSION = "0.3.3"
+  VERSION = "0.4.0"
 end

data/lib/litestream.rb

@@ -11,10 +11,9 @@ module Litestream
   end
 
   class Configuration
-    attr_accessor :database_path, :replica_url, :replica_key_id, :replica_access_key
+    attr_accessor :database_path, :replica_bucket, :replica_key_id, :replica_access_key
 
     def initialize
-      @mailer_sender = "donotreply@example.com"
     end
   end
 end

data/lib/tasks/litestream_tasks.rake

@@ -6,15 +6,14 @@ namespace :litestream do
       next
     end
 
-    puts "LITESTREAM_DATABASE_PATH=#{Litestream.configuration.database_path}"
-    puts "LITESTREAM_REPLICA_URL=#{Litestream.configuration.replica_url}"
+    puts "LITESTREAM_REPLICA_BUCKET=#{Litestream.configuration.replica_bucket}"
     puts "LITESTREAM_ACCESS_KEY_ID=#{Litestream.configuration.replica_key_id}"
     puts "LITESTREAM_SECRET_ACCESS_KEY=#{Litestream.configuration.replica_access_key}"
 
     true
   end
 
-  desc "Start a process to monitor and continuously replicate the SQLite databases defined in your configuration file"
+  desc 'Monitor and continuously replicate SQLite databases defined in your config file, e.g. rake litestream:replicate -- -exec "foreman start"'
   task replicate: :environment do
     options = {}
     if (separator_index = ARGV.index("--"))
@@ -25,4 +24,16 @@ namespace :litestream do
 
     Litestream::Commands.replicate(options)
   end
+
+  desc "Restore a SQLite database from a Litestream replica, e.g. rake litestream:restore -- -database=storage/production.sqlite3"
+  task restore: :environment do
+    options = {}
+    if (separator_index = ARGV.index("--"))
+      ARGV.slice(separator_index + 1, ARGV.length)
+        .map { |pair| pair.split("=") }
+        .each { |opt| options[opt[0]] = opt[1] || nil }
+    end
+
+    Litestream::Commands.restore(options.delete("--database") || options.delete("-database"), options)
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: litestream
 version: !ruby/object:Gem::Version
-  version: 0.3.3
+  version: 0.4.0
 platform: x86_64-linux
 authors:
 - Stephen Margheim
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-02-06 00:00:00.000000000 Z
+date: 2024-04-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rubyzip
@@ -69,7 +69,7 @@ files:
 - lib/litestream.rb
 - lib/litestream/commands.rb
 - lib/litestream/generators/litestream/install_generator.rb
-- lib/litestream/generators/litestream/templates/config.yml
+- lib/litestream/generators/litestream/templates/config.yml.erb
 - lib/litestream/generators/litestream/templates/initializer.rb
 - lib/litestream/railtie.rb
 - lib/litestream/upstream.rb