pg_easy_replicate 0.1.7 → 0.1.8
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +13 -3
- data/Dockerfile +2 -0
- data/Gemfile.lock +6 -4
- data/README.md +14 -7
- data/lib/pg_easy_replicate/cli.rb +9 -0
- data/lib/pg_easy_replicate/helper.rb +1 -1
- data/lib/pg_easy_replicate/version.rb +1 -1
- data/lib/pg_easy_replicate.rb +60 -8
- data/scripts/e2e-bootstrap.sh +1 -5
- data/scripts/e2e-start.sh +1 -1
- metadata +12 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 73d97a0f8505fef3ac73849e6964d1204c72545eb52fb4059d2c1119ea62228d
|
4
|
+
data.tar.gz: 8ad93a08e70da1945e091110064a9b4be569c2a77328f9960cd7ce52c6becefa
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 895ca04cdabbf15a082b1623890d55e650398f07eb144b2a0c8cf2877557a857926e24bfc336046afdd5097dbd5042d43d344e095338d6ffdb8f259797536c10
|
7
|
+
data.tar.gz: 41dd6428dd0f8c54de4d4b97931d573a0b2ae5fd4f5d61e2908d2e88e7fff0a97ed13c841a8fa2b924bacda0e8ae4c496d3f533d1398be7b15ffe0aa0ba6304a
|
data/CHANGELOG.md
CHANGED
@@ -1,13 +1,23 @@
|
|
1
|
-
## [0.1.
|
1
|
+
## [0.1.8] - 2023-07-23
|
2
|
+
|
3
|
+
- Introduce --copy_schema via pg_dump - #35
|
4
|
+
|
5
|
+
## [0.1.7] - 2023-06-26
|
6
|
+
|
7
|
+
- Perform smoke test with retries in CI - #26
|
8
|
+
- Default schema to `public` #29
|
9
|
+
- Perform vacuum and analyze before and after switchover - #30
|
10
|
+
|
11
|
+
## [0.1.6] - 2023-06-24
|
2
12
|
|
3
13
|
- Bug fix: Support custom schema name
|
4
14
|
- New smoke spec in CI
|
5
15
|
|
6
|
-
## [0.1.5] - 2023-06-
|
16
|
+
## [0.1.5] - 2023-06-24
|
7
17
|
|
8
18
|
- Fix bug in `stop_sync`
|
9
19
|
|
10
|
-
## [0.1.4] - 2023-06-
|
20
|
+
## [0.1.4] - 2023-06-24
|
11
21
|
|
12
22
|
- Drop lockbox dependency
|
13
23
|
- Support password with special chars and test for url encoded URI
|
data/Dockerfile
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,10 +1,10 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
pg_easy_replicate (0.1.
|
4
|
+
pg_easy_replicate (0.1.8)
|
5
5
|
ougai (~> 2.0.0)
|
6
6
|
pg (~> 1.5.3)
|
7
|
-
sequel (
|
7
|
+
sequel (>= 5.69, < 5.71)
|
8
8
|
thor (~> 1.2.2)
|
9
9
|
|
10
10
|
GEM
|
@@ -18,6 +18,7 @@ GEM
|
|
18
18
|
thor
|
19
19
|
tilt
|
20
20
|
json (2.6.3)
|
21
|
+
language_server-protocol (3.17.0.3)
|
21
22
|
method_source (1.0.0)
|
22
23
|
oj (3.14.3)
|
23
24
|
ougai (2.0.0)
|
@@ -50,8 +51,9 @@ GEM
|
|
50
51
|
diff-lcs (>= 1.2.0, < 2.0)
|
51
52
|
rspec-support (~> 3.12.0)
|
52
53
|
rspec-support (3.12.0)
|
53
|
-
rubocop (1.
|
54
|
+
rubocop (1.54.2)
|
54
55
|
json (~> 2.3)
|
56
|
+
language_server-protocol (>= 3.17.0)
|
55
57
|
parallel (~> 1.10)
|
56
58
|
parser (>= 3.2.2.3)
|
57
59
|
rainbow (>= 2.2.2, < 4.0)
|
@@ -78,7 +80,7 @@ GEM
|
|
78
80
|
rubocop-capybara (~> 2.17)
|
79
81
|
rubocop-factory_bot (~> 2.22)
|
80
82
|
ruby-progressbar (1.13.0)
|
81
|
-
sequel (5.
|
83
|
+
sequel (5.70.0)
|
82
84
|
syntax_tree (6.1.1)
|
83
85
|
prettier_print (>= 1.2.0)
|
84
86
|
syntax_tree-haml (4.0.3)
|
data/README.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
[![CI](https://github.com/shayonj/pg_easy_replicate/actions/workflows/ci.yaml/badge.svg?branch=main)](https://github.com/shayonj/pg_easy_replicate/actions/workflows/ci.yaml)
|
4
4
|
[![Smoke spec](https://github.com/shayonj/pg_easy_replicate/actions/workflows/smoke.yaml/badge.svg?branch=main)](https://github.com/shayonj/pg_easy_replicate/actions/workflows/ci.yaml)
|
5
|
-
[![Gem Version](https://badge.fury.io/rb/pg_easy_replicate.svg?
|
5
|
+
[![Gem Version](https://badge.fury.io/rb/pg_easy_replicate.svg?2)](https://badge.fury.io/rb/pg_easy_replicate)
|
6
6
|
|
7
7
|
`pg_easy_replicate` is a CLI orchestrator tool that simplifies the process of setting up [logical replication](https://www.postgresql.org/docs/current/logical-replication.html) between two PostgreSQL databases. `pg_easy_replicate` also supports switchover. After the source (primary database) is fully replicating, `pg_easy_replicate` puts it into read-only mode and via logical replication flushes all data to the new target database. This ensures zero data loss and minimal downtime for the application. This method can be useful for performing minimal downtime (up to <1min, depending) major version upgrades between two PostgreSQL databases, load testing with blue/green database setup and other similar use cases.
|
8
8
|
|
@@ -26,6 +26,8 @@ Battle tested in production at [Tines](https://www.tines.com/) 🚀
|
|
26
26
|
- [Switchover strategies with minimal downtime](#switchover-strategies-with-minimal-downtime)
|
27
27
|
- [Rolling restart strategy](#rolling-restart-strategy)
|
28
28
|
- [DNS Failover strategy](#dns-failover-strategy)
|
29
|
+
- [FAQ](#faq)
|
30
|
+
- [Adding internal user to pgBouncer `userlist`](#adding-internal-user-to-pgbouncer-userlist)
|
29
31
|
- [Contributing](#contributing)
|
30
32
|
|
31
33
|
## Installation
|
@@ -57,7 +59,6 @@ https://hub.docker.com/r/shayonj/pg_easy_replicate
|
|
57
59
|
- PostgreSQL 10 and later
|
58
60
|
- Ruby 2.7 and later
|
59
61
|
- Database user should have permissions for `SUPERUSER` or pass in the special user role that has the privileges to create role, schema, publication and subscription on both databases. More on `--special-user-role` section below.
|
60
|
-
- Both databases should have the same schema
|
61
62
|
|
62
63
|
## Limits
|
63
64
|
|
@@ -115,7 +116,7 @@ $ pg_easy_replicate config_check
|
|
115
116
|
Every sync will need to be bootstrapped before you can set up the sync between two databases. Bootstrap creates a new super user to perform the orchestration required during the rest of the process. It also creates some internal metadata tables for record keeping.
|
116
117
|
|
117
118
|
```bash
|
118
|
-
$ pg_easy_replicate bootstrap --group-name database-cluster-1
|
119
|
+
$ pg_easy_replicate bootstrap --group-name database-cluster-1 --copy-schema
|
119
120
|
|
120
121
|
{"name":"pg_easy_replicate","hostname":"PKHXQVK6DW","pid":21485,"level":30,"time":"2023-06-19T15:51:11.015-04:00","v":0,"msg":"Setting up schema","version":"0.1.0"}
|
121
122
|
...
|
@@ -132,7 +133,7 @@ For AWS the special user role is `rds_superuser`, and for GCP it is `cloudsqlsup
|
|
132
133
|
#### Config Check
|
133
134
|
|
134
135
|
```bash
|
135
|
-
$ pg_easy_replicate config_check --special-user-role="rds_superuser"
|
136
|
+
$ pg_easy_replicate config_check --special-user-role="rds_superuser" --copy-schema
|
136
137
|
|
137
138
|
✅ Config is looking good.
|
138
139
|
```
|
@@ -140,7 +141,7 @@ $ pg_easy_replicate config_check --special-user-role="rds_superuser"
|
|
140
141
|
#### Bootstrap
|
141
142
|
|
142
143
|
```bash
|
143
|
-
$ pg_easy_replicate bootstrap --group-name database-cluster-1 --special-user-role="rds_superuser"
|
144
|
+
$ pg_easy_replicate bootstrap --group-name database-cluster-1 --special-user-role="rds_superuser" --copy-schema
|
144
145
|
|
145
146
|
{"name":"pg_easy_replicate","hostname":"PKHXQVK6DW","pid":21485,"level":30,"time":"2023-06-19T15:51:11.015-04:00","v":0,"msg":"Setting up schema","version":"0.1.0"}
|
146
147
|
...
|
@@ -217,12 +218,12 @@ By default all tables are added for replication but you can create multiple grou
|
|
217
218
|
|
218
219
|
```bash
|
219
220
|
|
220
|
-
$ pg_easy_replicate bootstrap --group-name database-cluster-1
|
221
|
+
$ pg_easy_replicate bootstrap --group-name database-cluster-1 --copy-schema
|
221
222
|
$ pg_easy_replicate start_sync --group-name database-cluster-1 --schema-name public --tables "users, posts, events"
|
222
223
|
|
223
224
|
...
|
224
225
|
|
225
|
-
$ pg_easy_replicate bootstrap --group-name database-cluster-2
|
226
|
+
$ pg_easy_replicate bootstrap --group-name database-cluster-2 --copy-schema
|
226
227
|
$ pg_easy_replicate start_sync --group-name database-cluster-2 --schema-name public --tables "comments, views"
|
227
228
|
|
228
229
|
...
|
@@ -247,6 +248,12 @@ In this strategy, you have a weighted based DNS system (example [AWS Route53 wei
|
|
247
248
|
|
248
249
|
Next, you can set up a program that watches the `stats` and waits until `switchover_completed_at` is reporting as `true`. Once that happens it updates the weight in the DNS weighted group where 100% of the requests now go to the new/target database. Note: Keeping a low `ttl` is recommended.
|
249
250
|
|
251
|
+
## FAQ
|
252
|
+
|
253
|
+
### Adding internal user to pgBouncer `userlist`
|
254
|
+
|
255
|
+
`pg_easy_replicate` creates a special user to orchestrate the replication. If you us pgBouncer, you may need to allow `pger_su_h1a4fb` as a user that can perform login by adding it to the `userlist`.
|
256
|
+
|
250
257
|
## Contributing
|
251
258
|
|
252
259
|
PRs most welcome. You can get started locally by
|
@@ -12,9 +12,14 @@ module PgEasyReplicate
|
|
12
12
|
aliases: "-s",
|
13
13
|
desc:
|
14
14
|
"Name of the role that has superuser permissions. Usually useful for AWS (rds_superuser) or GCP (cloudsqlsuperuser)."
|
15
|
+
method_option :copy_schema,
|
16
|
+
aliases: "-c",
|
17
|
+
boolean: true,
|
18
|
+
desc: "Copy schema to the new database"
|
15
19
|
def config_check
|
16
20
|
PgEasyReplicate.assert_config(
|
17
21
|
special_user_role: options[:special_user_role],
|
22
|
+
copy_schema: options[:copy_schema],
|
18
23
|
)
|
19
24
|
|
20
25
|
puts "✅ Config is looking good."
|
@@ -28,6 +33,10 @@ module PgEasyReplicate
|
|
28
33
|
aliases: "-s",
|
29
34
|
desc:
|
30
35
|
"Name of the role that has superuser permissions. Usually useful with AWS (rds_superuser) or GCP (cloudsqlsuperuser)."
|
36
|
+
method_option :copy_schema,
|
37
|
+
aliases: "-c",
|
38
|
+
boolean: true,
|
39
|
+
desc: "Copy schema to the new database"
|
31
40
|
desc "bootstrap",
|
32
41
|
"Sets up temporary tables for information required during runtime"
|
33
42
|
def bootstrap
|
data/lib/pg_easy_replicate.rb
CHANGED
@@ -4,6 +4,7 @@ require "json"
|
|
4
4
|
require "ougai"
|
5
5
|
require "pg"
|
6
6
|
require "sequel"
|
7
|
+
require "open3"
|
7
8
|
|
8
9
|
require "pg_easy_replicate/helper"
|
9
10
|
require "pg_easy_replicate/version"
|
@@ -15,16 +16,21 @@ require "pg_easy_replicate/cli"
|
|
15
16
|
|
16
17
|
Sequel.default_timezone = :utc
|
17
18
|
module PgEasyReplicate
|
19
|
+
SCHEMA_FILE_LOCATION = "/tmp/pger_schema.sql"
|
20
|
+
|
18
21
|
class Error < StandardError
|
19
22
|
end
|
20
23
|
|
21
24
|
extend Helper
|
22
25
|
|
23
26
|
class << self
|
24
|
-
def config(special_user_role: nil)
|
27
|
+
def config(special_user_role: nil, copy_schema: false)
|
25
28
|
abort_with("SOURCE_DB_URL is missing") if source_db_url.nil?
|
26
29
|
abort_with("TARGET_DB_URL is missing") if target_db_url.nil?
|
27
30
|
|
31
|
+
system("which pg_dump")
|
32
|
+
pg_dump_exists = $CHILD_STATUS.success?
|
33
|
+
|
28
34
|
@config ||=
|
29
35
|
begin
|
30
36
|
q =
|
@@ -47,14 +53,20 @@ module PgEasyReplicate
|
|
47
53
|
connection_url: target_db_url,
|
48
54
|
user: db_user(target_db_url),
|
49
55
|
),
|
56
|
+
pg_dump_exists: pg_dump_exists,
|
50
57
|
}
|
51
58
|
rescue => e
|
52
59
|
abort_with("Unable to check config: #{e.message}")
|
53
60
|
end
|
54
61
|
end
|
55
62
|
|
56
|
-
def assert_config(special_user_role: nil)
|
57
|
-
config_hash =
|
63
|
+
def assert_config(special_user_role: nil, copy_schema: false)
|
64
|
+
config_hash =
|
65
|
+
config(special_user_role: special_user_role, copy_schema: copy_schema)
|
66
|
+
|
67
|
+
if copy_schema && !config_hash.dig(:pg_dump_exists)
|
68
|
+
abort_with("pg_dump must exist if copy_schema (-c) is passed")
|
69
|
+
end
|
58
70
|
|
59
71
|
unless assert_wal_level_logical(config_hash.dig(:source_db))
|
60
72
|
abort_with("WAL_LEVEL should be LOGICAL on source DB")
|
@@ -74,7 +86,15 @@ module PgEasyReplicate
|
|
74
86
|
|
75
87
|
def bootstrap(options)
|
76
88
|
logger.info("Setting up schema")
|
77
|
-
|
89
|
+
setup_internal_schema
|
90
|
+
|
91
|
+
if options[:copy_schema]
|
92
|
+
logger.info("Setting up schema on targer database")
|
93
|
+
copy_schema(
|
94
|
+
source_conn_string: source_db_url,
|
95
|
+
target_conn_string: target_db_url,
|
96
|
+
)
|
97
|
+
end
|
78
98
|
|
79
99
|
logger.info("Setting up replication user on source database")
|
80
100
|
create_user(
|
@@ -103,7 +123,7 @@ module PgEasyReplicate
|
|
103
123
|
|
104
124
|
if options[:everything]
|
105
125
|
logger.info("Dropping schema")
|
106
|
-
|
126
|
+
drop_internal_schema
|
107
127
|
end
|
108
128
|
|
109
129
|
if options[:everything] || options[:sync]
|
@@ -130,7 +150,7 @@ module PgEasyReplicate
|
|
130
150
|
abort_with("Unable to cleanup: #{e.message}")
|
131
151
|
end
|
132
152
|
|
133
|
-
def
|
153
|
+
def drop_internal_schema
|
134
154
|
Query.run(
|
135
155
|
query: "DROP SCHEMA IF EXISTS #{internal_schema_name} CASCADE",
|
136
156
|
connection_url: source_db_url,
|
@@ -141,7 +161,7 @@ module PgEasyReplicate
|
|
141
161
|
raise "Unable to drop schema: #{e.message}"
|
142
162
|
end
|
143
163
|
|
144
|
-
def
|
164
|
+
def setup_internal_schema
|
145
165
|
sql = <<~SQL
|
146
166
|
create schema if not exists #{internal_schema_name};
|
147
167
|
grant usage on schema #{internal_schema_name} to #{db_user(source_db_url)};
|
@@ -169,7 +189,39 @@ module PgEasyReplicate
|
|
169
189
|
end
|
170
190
|
end
|
171
191
|
|
172
|
-
|
192
|
+
def copy_schema(source_conn_string:, target_conn_string:)
|
193
|
+
export_schema(conn_string: source_conn_string)
|
194
|
+
import_schema(conn_string: target_conn_string)
|
195
|
+
end
|
196
|
+
|
197
|
+
def export_schema(conn_string:)
|
198
|
+
logger.info("Exporting schema to #{SCHEMA_FILE_LOCATION}")
|
199
|
+
_, stderr, status =
|
200
|
+
Open3.capture3(
|
201
|
+
"pg_dump",
|
202
|
+
conn_string,
|
203
|
+
"-f",
|
204
|
+
SCHEMA_FILE_LOCATION,
|
205
|
+
"--schema-only",
|
206
|
+
)
|
207
|
+
|
208
|
+
success = status.success?
|
209
|
+
raise stderr unless success
|
210
|
+
rescue => e
|
211
|
+
raise "Unable to export schema: #{e.message}"
|
212
|
+
end
|
213
|
+
|
214
|
+
def import_schema(conn_string:)
|
215
|
+
logger.info("Importing schema from #{SCHEMA_FILE_LOCATION}")
|
216
|
+
|
217
|
+
_, stderr, status =
|
218
|
+
Open3.capture3("psql", "-f", SCHEMA_FILE_LOCATION, conn_string)
|
219
|
+
|
220
|
+
success = status.success?
|
221
|
+
raise stderr unless success
|
222
|
+
rescue => e
|
223
|
+
raise "Unable to import schema: #{e.message}"
|
224
|
+
end
|
173
225
|
|
174
226
|
def assert_wal_level_logical(db_config)
|
175
227
|
db_config&.find do |r|
|
data/scripts/e2e-bootstrap.sh
CHANGED
@@ -12,8 +12,4 @@ export PGPASSWORD='jamesbond123@7!'"'"'3aaR'
|
|
12
12
|
|
13
13
|
pgbench --initialize -s 5 --foreign-keys --host localhost -U jamesbond -d postgres
|
14
14
|
|
15
|
-
|
16
|
-
cat schema.sql | psql --host localhost -U jamesbond -d postgres -p 5433
|
17
|
-
rm schema.sql
|
18
|
-
|
19
|
-
bundle exec bin/pg_easy_replicate config_check
|
15
|
+
bundle exec bin/pg_easy_replicate config_check --copy-schema
|
data/scripts/e2e-start.sh
CHANGED
@@ -12,7 +12,7 @@ export PGPASSWORD='jamesbond123@7!'"'"''"'"'3aaR'
|
|
12
12
|
|
13
13
|
# Bootstrap and cleanup
|
14
14
|
echo "===== Performing Bootstrap and cleanup"
|
15
|
-
bundle exec bin/pg_easy_replicate bootstrap -g cluster-1
|
15
|
+
bundle exec bin/pg_easy_replicate bootstrap -g cluster-1 --copy-schema
|
16
16
|
bundle exec bin/pg_easy_replicate start_sync -g cluster-1 -s public
|
17
17
|
bundle exec bin/pg_easy_replicate stats -g cluster-1
|
18
18
|
bundle exec bin/pg_easy_replicate switchover -g cluster-1
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pg_easy_replicate
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.8
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Shayon Mukherjee
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-07-23 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: ougai
|
@@ -42,16 +42,22 @@ dependencies:
|
|
42
42
|
name: sequel
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
|
-
- - "
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '5.69'
|
48
|
+
- - "<"
|
46
49
|
- !ruby/object:Gem::Version
|
47
|
-
version: 5.
|
50
|
+
version: '5.71'
|
48
51
|
type: :runtime
|
49
52
|
prerelease: false
|
50
53
|
version_requirements: !ruby/object:Gem::Requirement
|
51
54
|
requirements:
|
52
|
-
- - "
|
55
|
+
- - ">="
|
56
|
+
- !ruby/object:Gem::Version
|
57
|
+
version: '5.69'
|
58
|
+
- - "<"
|
53
59
|
- !ruby/object:Gem::Version
|
54
|
-
version: 5.
|
60
|
+
version: '5.71'
|
55
61
|
- !ruby/object:Gem::Dependency
|
56
62
|
name: thor
|
57
63
|
requirement: !ruby/object:Gem::Requirement
|