litestream 0.9.0-x86_64-darwin → 0.10.0-x86_64-darwin

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: bc937c14c8cdbc08db031431903637338b95fed9cee32fcffa945da97bcc5927
4
- data.tar.gz: 8b305efd28e72ef7316fc53e790ca10ef29cc317d74fd72474ed4007a0469936
3
+ metadata.gz: 40caaac876c83fb253b5be358ee51822a27788fd789531f058c6945014716726
4
+ data.tar.gz: 49650381680cb1f359c5945e1b7d9b38dca95cfdd446fad0a0534a357ea7686d
5
5
  SHA512:
6
- metadata.gz: 672231f8716e812ee516d13cfab97386001cd9084f2aa03418d1f17838bbaaa8442e6ccc4ddf3c3edd654743e2f2f11b8f72b5d887f79060d405ac03d2de9832
7
- data.tar.gz: e19b81794bfc2fae07aff2bc1b6c9084a10bd309fda38f2a1f8340791262475fd4d0a5171bd067322a4342c31bb69fcfbf8eea9fd27ec497b76b37dd3bf746ad
6
+ metadata.gz: 53bddbe69e324e5bf376192578b4a34d9e0a7780e8cb08692e06e0419bd7bded4058597f51a1fc8e8d4909d3f57bf11e3fb1a2d25c025823a21520b0522b2196
7
+ data.tar.gz: a0a1392997dd3f2aa69a6a7805b7838f1b87e84b7e23bdc1ccf5c84a689b5da66e327ff7a0126502ec7a3db9fb9746f5c31db8b0a3796472f5ae916039f67b32
data/README.md CHANGED
@@ -6,17 +6,23 @@
6
6
 
7
7
  Install the gem and add to the application's Gemfile by executing:
8
8
 
9
- $ bundle add litestream
9
+ ```sh
10
+ bundle add litestream
11
+ ```
10
12
 
11
13
  If bundler is not being used to manage dependencies, install the gem by executing:
12
14
 
13
- $ gem install litestream
15
+ ```sh
16
+ gem install litestream
17
+ ```
14
18
 
15
19
  After installing the gem, run the installer:
16
20
 
17
- $ rails generate litestream:install
21
+ ```sh
22
+ rails generate litestream:install
23
+ ```
18
24
 
19
- The installer will create a configuration file at `config/litestream.yml`, an initializer file for configuring the gem at `config/initializers/litestream.rb`, as well as a `Procfile` so that you can run the Litestream replication process alongside your Rails application in production.
25
+ The installer will create a configuration file at `config/litestream.yml` and an initializer file for configuring the gem at `config/initializers/litestream.rb`.
20
26
 
21
27
  This gem wraps the standalone executable version of the [Litestream](https://litestream.io/install/source/) utility. These executables are platform specific, so there are actually separate underlying gems per platform, but the correct gem will automatically be picked for your platform. Litestream itself doesn't support Windows, so this gem doesn't either.
22
28
 
@@ -77,14 +83,28 @@ However, if you need manual control over the Litestream configuration, you can m
77
83
 
78
84
  ### Replication
79
85
 
80
- By default, the gem will create or append to a `Procfile` to start the Litestream process via the gem's provided `litestream:replicate` rake task. This rake task will automatically load the configuration file and set the environment variables before starting the Litestream process. You can also execute this rake task yourself:
86
+ In order to stream changes to your configured replicas, you need to start the Litestream replication process.
81
87
 
82
- ```shell
83
- bin/rails litestream:replicate
84
- # or
85
- bundle exec rake litestream:replicate
88
+ The simplest way to run the Litestream replication process is use the Puma plugin provided by the gem. This allows you to run the Litestream replication process together with Puma and have Puma monitor and manage it. You just need to add
89
+
90
+ ```ruby
91
+ plugin :litestream
86
92
  ```
87
93
 
94
+ to your `puma.rb` configuration.
95
+
96
+ If you would prefer to run the Litestream replication process separately from Puma, you can use the provided `litestream:replicate` rake task. This rake task will automatically load the configuration file and set the environment variables before starting the Litestream process.
97
+
98
+ The simplest way to spin up a Litestream process separately from your Rails application is to use a `Procfile`:
99
+
100
+ ```yaml
101
+ # Procfile
102
+ rails: bundle exec rails server --port $PORT
103
+ litestream: bin/rails litestream:replicate
104
+ ```
105
+
106
+ Alternatively, you could setup a `systemd` service to manage the Litestream replication process, but setting this up is outside the scope of this README.
107
+
88
108
  If you need to pass arguments through the rake task to the underlying `litestream` command, that can be done with argument forwarding:
89
109
 
90
110
  ```shell
@@ -94,6 +114,7 @@ bin/rails litestream:replicate -- -exec "foreman start"
94
114
  This example utilizes the `-exec` option available on [the `replicate` command](https://litestream.io/reference/replicate/) which provides basic process management, since Litestream will exit when the child process exits. In this example, we only launch our collection of Rails application processes (like Rails and SolidQueue, for example) after the Litestream replication process is ready.
95
115
 
96
116
  The Litestream `replicate` command supports the following options, which can be passed through the rake task:
117
+
97
118
  ```shell
98
119
  -config PATH
99
120
  Specifies the configuration file.
@@ -168,37 +189,19 @@ You can forward arguments in whatever order you like, you simply need to ensure
168
189
 
169
190
  ### Verification
170
191
 
171
- You can verify the integrity of your backed-up databases using the gem's provided `litestream:verify` rake task. This rake task requires that you specify which specific database you want to verify. As with the `litestream:restore` tasks, you pass arguments to the rake task via argument forwarding. For example, to verify the production database, you would run:
192
+ You can verify the integrity of your backed-up databases using the gem's provided `Litestream.verify!` method. The method takes the path to a database file that you have configured Litestream to backup; that is, it takes one of the `path` values under the `dbs` key in your `litestream.yml` configuration file. For example, to verify the production database, you would run:
172
193
 
173
- ```shell
174
- bin/rails litestream:verify -- --database=storage/production.sqlite3
175
- # or
176
- bundle exec rake litestream:verify -- --database=storage/production.sqlite3
194
+ ```ruby
195
+ Litestream.verify! "storage/production.sqlite3"
177
196
  ```
178
197
 
179
- The `litestream:verify` rake task takes the same options as the `litestream:restore` rake task. After restoring the backup, the rake task will verify the integrity of the restored database by ensuring that the restored database file
198
+ In order to verify that the backup for that database is both restorable and fresh, the method will add a new row to that database under the `_litestream_verification` table, which it will create if needed. It will then wait 10 seconds to give the Litestream utility time to replicate that change to whatever storage providers you have configured. After that, it will download the latest backup from that storage provider and ensure that this verification row is present in the backup. If the verification row is _not_ present, the method will raise a `Litestream::VerificationFailure` exception. This check ensures that the restored database file
180
199
 
181
200
  1. exists,
182
201
  2. can be opened by SQLite, and
183
- 3. sufficiently matches the original database file.
184
-
185
- Since point 3 is subjective, the rake task will output a message providing both the file size and number of tables of both the "original" and "restored" databases. You must manually verify that the restored database is within an acceptable range of the original database.
186
-
187
- The rake task will output a message similar to the following:
188
-
189
- ```
190
- size
191
- original 21688320
192
- restored 21688320
193
- delta 0
194
-
195
- tables
196
- original 9
197
- restored 9
198
- delta 0
199
- ```
202
+ 3. has up-to-date data.
200
203
 
201
- After restoring the backup, the `litestream:verify` rake task will delete the restored database file. If you need the restored database file, use the `litestream:restore` rake task instead.
204
+ After restoring the backup, the `Litestream.verify!` method will delete the restored database file. If you need the restored database file, use the `litestream:restore` rake task or `Litestream::Commands.restore` method instead.
202
205
 
203
206
  ### Introspection
204
207
 
@@ -88,46 +88,6 @@ module Litestream
88
88
  execute("restore", argv, database, async: async, tabled_output: false)
89
89
  end
90
90
 
91
- def verify(database, async: false, **argv)
92
- raise DatabaseRequiredException, "database argument is required for verify command, e.g. litestream:verify -- --database=path/to/database.sqlite" if database.nil? || !File.exist?(database)
93
- argv.stringify_keys!
94
-
95
- dir, file = File.split(database)
96
- ext = File.extname(file)
97
- base = File.basename(file, ext)
98
- now = Time.now.utc.strftime("%Y%m%d%H%M%S")
99
- backup = File.join(dir, "#{base}-#{now}#{ext}")
100
- args = {
101
- "-o" => backup
102
- }.merge(argv)
103
- restore(database, async: false, **args)
104
-
105
- restored_schema = `sqlite3 #{backup} "select name, type from sqlite_schema;"`.chomp.split("\n")
106
- restored_data = restored_schema.map { _1.split("|") }.group_by(&:last)
107
- restored_rows_count = restored_data["table"]&.sum { |tbl, _| `sqlite3 #{backup} "select count(*) from #{tbl};"`.chomp.to_i }
108
-
109
- original_schema = `sqlite3 #{database} "select name, type from sqlite_schema;"`.chomp.split("\n")
110
- original_data = original_schema.map { _1.split("|") }.group_by(&:last)
111
- original_rows_count = original_data["table"]&.sum { |tbl, _| `sqlite3 #{database} "select count(*) from #{tbl};"`.chomp.to_i }
112
-
113
- Dir.glob(backup + "*").each { |file| File.delete(file) }
114
-
115
- {
116
- "original" => {
117
- "path" => database,
118
- "tables" => original_data["table"]&.size,
119
- "indexes" => original_data["index"]&.size,
120
- "rows" => original_rows_count
121
- },
122
- "restored" => {
123
- "path" => backup,
124
- "tables" => restored_data["table"]&.size,
125
- "indexes" => restored_data["index"]&.size,
126
- "rows" => restored_rows_count
127
- }
128
- }
129
- end
130
-
131
91
  def databases(async: false, **argv)
132
92
  execute("databases", argv, async: async, tabled_output: true)
133
93
  end
@@ -1,9 +1,13 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require "rails/railtie"
3
+ require "rails/engine"
4
4
 
5
5
  module Litestream
6
- class Railtie < ::Rails::Railtie
6
+ class Engine < ::Rails::Engine
7
+ isolate_namespace Litestream
8
+
9
+ config.litestream = ActiveSupport::OrderedOptions.new
10
+
7
11
  # Load the `litestream:install` generator into the host Rails app
8
12
  generators do
9
13
  require_relative "generators/litestream/install_generator"
@@ -13,5 +17,11 @@ module Litestream
13
17
  rake_tasks do
14
18
  load "tasks/litestream_tasks.rake"
15
19
  end
20
+
21
+ initializer "litestream.config" do
22
+ config.litestream.each do |name, value|
23
+ Litestream.public_send(:"#{name}=", value)
24
+ end
25
+ end
16
26
  end
17
27
  end
@@ -15,19 +15,6 @@ module Litestream
15
15
  template "initializer.rb", "config/initializers/litestream.rb"
16
16
  end
17
17
 
18
- def create_or_update_procfile
19
- if File.exist?("Procfile")
20
- append_to_file "Procfile", "litestream: bin/rails litestream:replicate"
21
- else
22
- create_file "Procfile" do
23
- <<~PROCFILE
24
- rails: bundle exec rails server --port $PORT
25
- litestream: bin/rails litestream:replicate
26
- PROCFILE
27
- end
28
- end
29
- end
30
-
31
18
  private
32
19
 
33
20
  def production_sqlite_databases
@@ -1,3 +1,3 @@
1
1
  module Litestream
2
- VERSION = "0.9.0"
2
+ VERSION = "0.10.0"
3
3
  end
data/lib/litestream.rb CHANGED
@@ -1,5 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "sqlite3"
4
+
3
5
  module Litestream
4
6
  class << self
5
7
  attr_accessor :configuration
@@ -16,9 +18,118 @@ module Litestream
16
18
  def initialize
17
19
  end
18
20
  end
21
+
22
+ VerificationFailure = Class.new(StandardError)
23
+
24
+ mattr_writer :username
25
+ mattr_writer :password
26
+
27
+ class << self
28
+ def verify!(database_path)
29
+ database = SQLite3::Database.new(database_path)
30
+ database.execute("CREATE TABLE IF NOT EXISTS _litestream_verification (id INTEGER PRIMARY KEY, uuid BLOB)")
31
+ sentinel = SecureRandom.uuid
32
+ database.execute("INSERT INTO _litestream_verification (uuid) VALUES (?)", [sentinel])
33
+ # give the Litestream replication process time to replicate the sentinel value
34
+ sleep 10
35
+
36
+ backup_path = "tmp/#{Time.now.utc.strftime("%Y%m%d%H%M%S")}_#{sentinel}.sqlite3"
37
+ Litestream::Commands.restore(database_path, **{"-o" => backup_path})
38
+
39
+ backup = SQLite3::Database.new(backup_path)
40
+ result = backup.execute("SELECT 1 FROM _litestream_verification WHERE uuid = ? LIMIT 1", sentinel) # => [[1]] || []
41
+
42
+ raise VerificationFailure, "Verification failed, sentinel not found" if result.empty?
43
+ ensure
44
+ database.execute("DELETE FROM _litestream_verification WHERE uuid = ?", sentinel)
45
+ database.close
46
+ Dir.glob(backup_path + "*").each { |file| File.delete(file) }
47
+ end
48
+
49
+ # use method instead of attr_accessor to ensure
50
+ # this works if variable set after Litestream is loaded
51
+ def username
52
+ @username ||= ENV["LITESTREAM_USERNAME"] || @@username
53
+ end
54
+
55
+ # use method instead of attr_accessor to ensure
56
+ # this works if variable set after Litestream is loaded
57
+ def password
58
+ @password ||= ENV["LITESTREAM_PASSWORD"] || @@password
59
+ end
60
+
61
+ def replicate_process
62
+ info = {}
63
+ if !`which systemctl`.empty?
64
+ systemctl_status = `systemctl status litestream`.chomp
65
+ # ["● litestream.service - Litestream",
66
+ # " Loaded: loaded (/lib/systemd/system/litestream.service; enabled; vendor preset: enabled)",
67
+ # " Active: active (running) since Tue 2023-07-25 13:49:43 UTC; 8 months 24 days ago",
68
+ # " Main PID: 1179656 (litestream)",
69
+ # " Tasks: 9 (limit: 1115)",
70
+ # " Memory: 22.9M",
71
+ # " CPU: 10h 49.843s",
72
+ # " CGroup: /system.slice/litestream.service",
73
+ # " └─1179656 /usr/bin/litestream replicate",
74
+ # "",
75
+ # "Warning: some journal files were not opened due to insufficient permissions."]
76
+ systemctl_status.split("\n").each do |line|
77
+ line.strip!
78
+ if line.start_with?("Main PID:")
79
+ _key, value = line.split(":")
80
+ pid, _name = value.strip.split(" ")
81
+ info[:pid] = pid
82
+ elsif line.start_with?("Active:")
83
+ _key, value = line.split(":")
84
+ value, _ago = value.split(";")
85
+ status, timestamp = value.split(" since ")
86
+ info[:started] = DateTime.strptime(timestamp.strip, "%Y-%m-%d %H:%M:%S %Z")
87
+ info[:status] = status.split("(").first.strip
88
+ end
89
+ end
90
+ else
91
+ litestream_replicate_ps = `ps -a | grep litestream | grep replicate`.chomp
92
+ litestream_replicate_ps.split("\n").each do |line|
93
+ next unless line.include?("litestream replicate")
94
+ pid, * = line.split(" ")
95
+ info[:pid] = pid
96
+ state, _, lstart = `ps -o "state,lstart" #{pid}`.chomp.split("\n").last.partition(/\s+/)
97
+
98
+ info[:status] = case state[0]
99
+ when "I" then "idle"
100
+ when "R" then "running"
101
+ when "S" then "sleeping"
102
+ when "T" then "stopped"
103
+ when "U" then "uninterruptible"
104
+ when "Z" then "zombie"
105
+ end
106
+ info[:started] = DateTime.strptime(lstart.strip, "%a %b %d %H:%M:%S %Y")
107
+ end
108
+ end
109
+ info
110
+ end
111
+
112
+ def databases
113
+ databases = Commands.databases
114
+
115
+ databases.each do |db|
116
+ generations = Commands.generations(db["path"])
117
+ snapshots = Commands.snapshots(db["path"])
118
+ db["path"] = db["path"].gsub(Rails.root.to_s, "[ROOT]")
119
+
120
+ db["generations"] = generations.map do |generation|
121
+ id = generation["generation"]
122
+ replica = generation["name"]
123
+ generation["snapshots"] = snapshots.select { |snapshot| snapshot["generation"] == id && snapshot["replica"] == replica }
124
+ .map { |s| s.slice("index", "size", "created") }
125
+ generation.slice("generation", "name", "lag", "start", "end", "snapshots")
126
+ end
127
+ end
128
+ end
129
+ end
19
130
  end
20
131
 
21
132
  require_relative "litestream/version"
22
133
  require_relative "litestream/upstream"
23
134
  require_relative "litestream/commands"
24
- require_relative "litestream/railtie" if defined?(::Rails::Railtie)
135
+ require_relative "litestream/engine" if defined?(::Rails::Engine)
@@ -0,0 +1,69 @@
1
+ require "puma/plugin"
2
+
3
+ # Copied from https://github.com/rails/solid_queue/blob/15408647f1780033dad223d3198761ea2e1e983e/lib/puma/plugin/solid_queue.rb
4
+ Puma::Plugin.create do
5
+ attr_reader :puma_pid, :litestream_pid, :log_writer
6
+
7
+ def start(launcher)
8
+ @log_writer = launcher.log_writer
9
+ @puma_pid = $$
10
+
11
+ launcher.events.on_booted do
12
+ @litestream_pid = fork do
13
+ Thread.new { monitor_puma }
14
+ Litestream::Commands.replicate(async: true)
15
+ end
16
+
17
+ in_background do
18
+ monitor_litestream
19
+ end
20
+ end
21
+
22
+ launcher.events.on_stopped { stop_litestream }
23
+ launcher.events.on_restart { stop_litestream }
24
+ end
25
+
26
+ private
27
+
28
+ def stop_litestream
29
+ Process.waitpid(litestream_pid, Process::WNOHANG)
30
+ log_writer.log "Stopping Litestream..."
31
+ Process.kill(:INT, litestream_pid) if litestream_pid
32
+ Process.wait(litestream_pid)
33
+ rescue Errno::ECHILD, Errno::ESRCH
34
+ end
35
+
36
+ def monitor_puma
37
+ monitor(:puma_dead?, "Detected Puma has gone away, stopping Litestream...")
38
+ end
39
+
40
+ def monitor_litestream
41
+ monitor(:litestream_dead?, "Detected Litestream has gone away, stopping Puma...")
42
+ end
43
+
44
+ def monitor(process_dead, message)
45
+ loop do
46
+ if send(process_dead)
47
+ log message
48
+ Process.kill(:INT, $$)
49
+ break
50
+ end
51
+ sleep 2
52
+ end
53
+ end
54
+
55
+ def litestream_dead?
56
+ Process.waitpid(litestream_pid, Process::WNOHANG)
57
+ false
58
+ rescue Errno::ECHILD, Errno::ESRCH
59
+ true
60
+ end
61
+
62
+ def puma_dead?
63
+ Process.ppid != puma_pid
64
+ end
65
+
66
+ def log(...)
67
+ log_writer.log(...)
68
+ end
69
+ end
@@ -80,66 +80,4 @@ namespace :litestream do
80
80
 
81
81
  Litestream::Commands.snapshots(database, async: true, **options)
82
82
  end
83
-
84
- desc "verify backup of SQLite database from a Litestream replica, e.g. rake litestream:verify -- -database=storage/production.sqlite3"
85
- task verify: :environment do
86
- options = {}
87
- if (separator_index = ARGV.index("--"))
88
- ARGV.slice(separator_index + 1, ARGV.length)
89
- .map { |pair| pair.split("=") }
90
- .each { |opt| options[opt[0]] = opt[1] || nil }
91
- end
92
- database = options.delete("--database") || options.delete("-database")
93
- options.symbolize_keys!
94
-
95
- result = Litestream::Commands.verify(database, async: true, **options)
96
- original_tables = result["original"]["tables"]
97
- restored_tables = result["restored"]["tables"]
98
- original_indexes = result["original"]["indexes"]
99
- restored_indexes = result["restored"]["indexes"]
100
- original_rows = result["original"]["rows"]
101
- restored_rows = result["restored"]["rows"]
102
-
103
- same_number_of_tables = original_tables == restored_tables
104
- same_number_of_indexes = original_indexes == restored_indexes
105
- same_number_of_rows = original_rows == restored_rows
106
-
107
- if same_number_of_tables && same_number_of_indexes && same_number_of_rows
108
- puts "Backup for `#{database}` verified as consistent!\n" + [
109
- " tables #{original_tables}",
110
- " indexes #{original_indexes}",
111
- " rows #{original_rows}"
112
- ].compact.join("\n")
113
- else
114
- abort "Verification failed for #{database}:\n" + [
115
- (unless same_number_of_tables
116
- if original_tables > restored_tables
117
- diff = original_tables - restored_tables
118
- " Backup is missing #{diff} table#{"s" if diff > 1}"
119
- else
120
- diff = restored_tables - original_tables
121
- " Backup has extra #{diff} table#{"s" if diff > 1}"
122
- end
123
- end),
124
- (unless same_number_of_indexes
125
- if original_indexes > restored_indexes
126
- diff = original_indexes - restored_indexes
127
- " Backup is missing #{diff} index#{"es" if diff > 1}"
128
- else
129
- diff = restored_indexes - original_indexes
130
- " Backup has extra #{diff} index#{"es" if diff > 1}"
131
- end
132
- end),
133
- (unless same_number_of_rows
134
- if original_rows > restored_rows
135
- diff = original_rows - restored_rows
136
- " Backup is missing #{diff} row#{"s" if diff > 1}"
137
- else
138
- diff = restored_rows - original_rows
139
- " Backup has extra #{diff} row#{"s" if diff > 1}"
140
- end
141
- end)
142
- ].compact.join("\n")
143
- end
144
- end
145
83
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: litestream
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.9.0
4
+ version: 0.10.0
5
5
  platform: x86_64-darwin
6
6
  authors:
7
7
  - Stephen Margheim
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2024-05-04 00:00:00.000000000 Z
11
+ date: 2024-05-05 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: logfmt
@@ -24,6 +24,20 @@ dependencies:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
26
  version: 0.0.10
27
+ - !ruby/object:Gem::Dependency
28
+ name: sqlite3
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
27
41
  - !ruby/object:Gem::Dependency
28
42
  name: rubyzip
29
43
  requirement: !ruby/object:Gem::Requirement
@@ -82,12 +96,13 @@ files:
82
96
  - exe/x86_64-darwin/litestream
83
97
  - lib/litestream.rb
84
98
  - lib/litestream/commands.rb
99
+ - lib/litestream/engine.rb
85
100
  - lib/litestream/generators/litestream/install_generator.rb
86
101
  - lib/litestream/generators/litestream/templates/config.yml.erb
87
102
  - lib/litestream/generators/litestream/templates/initializer.rb
88
- - lib/litestream/railtie.rb
89
103
  - lib/litestream/upstream.rb
90
104
  - lib/litestream/version.rb
105
+ - lib/puma/plugin/litestream.rb
91
106
  - lib/tasks/litestream_tasks.rake
92
107
  homepage: https://github.com/fractaledmind/litestream-ruby
93
108
  licenses: