pg_easy_replicate 0.1.0

data/README.md

@@ -0,0 +1,213 @@
+# pg_easy_replicate
+
+`pg_easy_replicate` is a CLI orchestrator tool that simplifies the process of setting up [logical replication](https://www.postgresql.org/docs/current/logical-replication.html) between two PostgreSQL databases. `pg_easy_replicate` also supports switchover. After the source (primary database) is fully replicating, `pg_easy_replicate` puts it into read-only mode and via logical replication flushes all data to the new target database. This ensures zero data loss and minimal downtime for the application. This method can be useful for performing minimal downtime major version upgrades between two PostgreSQL databases, load testing with blue/green database setup and other similar use cases.
+
+- [Installation](#installation)
+- [Requirements](#requirements)
+- [Limits](#limits)
+- [Usage](#usage)
+- [CLI](#cli)
+- [Replicating all tables with a single group](#replicating-all-tables-with-a-single-group)
+  - [Config check](#config-check)
+  - [Bootstrap](#bootstrap)
+  - [Start sync](#start-sync)
+  - [Stats](#stats)
+  - [Performing switchover](#performing-switchover)
+- [Replicating single database with custom tables](#replicating-single-database-with-custom-tables)
+- [Switchover strategies with minimal downtime](#switchover-strategies-with-minimal-downtime)
+  - [Rolling restart strategy](#rolling-restart-strategy)
+  - [DNS Failover strategy](#dns-failover-strategy)
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "pg_easy_replicate"
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install pg_easy_replicate
+
+This will include all dependencies accordingly as well. Make sure the following requirements are satisfied.
+
+Or via Docker:
+
+    docker pull shayonj/pg_easy_replicate:latest
+
+https://hub.docker.com/r/shayonj/pg_easy_replicate
+
+## Requirements
+
+- PostgreSQL 10 and later
+- Ruby 2.7 and later
+- Database user should have permissions for `SUPERUSER`
+- Both databases should have the same schema
+
+## Limits
+
+All [Logical Replication Restrictions](https://www.postgresql.org/docs/current/logical-replication-restrictions.html) apply.
+
+## Usage
+
+Ensure `SOURCE_DB_URL` and `TARGET_DB_URL` are present as environment variables in the runtime environment. The URL are of the postgres connection string format. Example:
+
+```bash
+$ export SOURCE_DB_URL="postgres://USERNAME:PASSWORD@localhost:5432/DATABASE_NAME"
+$ export TARGET_DB_URL="postgres://USERNAME:PASSWORD@localhost:5433/DATABASE_NAME"
+```
+
+Any `pg_easy_replicate` command can be run the same way with the docker image as well. As long the container is running in an environment where it has access to both the databases. Example
+
+```bash
+docker run -it --rm shayonj/pg_easy_replicate:latest \
+  -e SOURCE_DB_URL="postgres://USERNAME:PASSWORD@localhost:5432/DATABASE_NAME" \
+  -e TARGET_DB_URL="postgres://USERNAME:PASSWORD@localhost:5433/DATABASE_NAME" \
+  pg_easy_replicate config_check
+```
+
+## CLI
+
+```bash
+$  pg_easy_replicate
+pg_easy_replicate commands:
+  pg_easy_replicate bootstrap -g, --group-name=GROUP_NAME    # Sets up temporary tables for information required during runtime
+  pg_easy_replicate cleanup -g, --group-name=GROUP_NAME      # Cleans up all bootstrapped data for the respective group
+  pg_easy_replicate config_check                             # Prints if source and target database have the required config
+  pg_easy_replicate help [COMMAND]                           # Describe available commands or one specific command
+  pg_easy_replicate start_sync -g, --group-name=GROUP_NAME   # Starts the logical replication from source database to target database provisioned in the group
+  pg_easy_replicate stats  -g, --group-name=GROUP_NAME       # Prints the statistics in JSON for the group
+  pg_easy_replicate stop_sync -g, --group-name=GROUP_NAME    # Stop the logical replication from source database to target database provisioned in the group
+  pg_easy_replicate switchover  -g, --group-name=GROUP_NAME  # Puts the source database in read only mode after all the data is flushed and written
+  pg_easy_replicate version                                  # Prints the version
+
+```
+
+## Replicating all tables with a single group
+
+You can create as many groups as you want for a single database. Groups are just a logical isolation of a single replication.
+
+### Config check
+
+```bash
+$ pg_easy_replicate config_check
+
+✅ Config is looking good.
+```
+
+### Bootstrap
+
+Every sync will need to be bootstrapped before you can set up the sync between two databases. Bootstrap creates a new super user to perform the orchestration required during the rest of the process. It also creates some internal metadata tables for record keeping.
+
+```bash
+$ pg_easy_replicate bootstrap --group-name database-cluster-1
+
+{"name":"pg_easy_replicate","hostname":"PKHXQVK6DW","pid":21485,"level":30,"time":"2023-06-19T15:51:11.015-04:00","v":0,"msg":"Setting up schema","version":"0.1.0"}
+...
+```
+
+### Start sync
+
+Once the bootstrap is complete, you can start the sync. Starting the sync sets up the publication, subscription and performs other minor housekeeping things.
+
+```bash
+$ pg_easy_replicate start_sync --group-name database-cluster-1
+
+{"name":"pg_easy_replicate","hostname":"PKHXQVK6DW","pid":22113,"level":30,"time":"2023-06-19T15:54:54.874-04:00","v":0,"msg":"Setting up publication","publication_name":"pger_publication_database_cluster_1","version":"0.1.0"}
+...
+```
+
+### Stats
+
+You can inspect or watch stats any time during the sync process. The stats give you can an idea of when the sync started, current flush/write lag, how many tables are in `replicating`, `copying` or other stages, and more.
+
+You can poll these stats to perform any other after the switchover is done. The stats include a `switchover_completed_at` which is updated once the switch over is complete.
+
+```bash
+$ pg_easy_replicate stats --group-name database-cluster-1
+
+{
+  "lag_stats": [
+    {
+      "pid": 66,
+      "client_addr": "192.168.128.2",
+      "user_name": "jamesbond",
+      "application_name": "pger_subscription_database_cluster_1",
+      "state": "streaming",
+      "sync_state": "async",
+      "write_lag": "0.0",
+      "flush_lag": "0.0",
+      "replay_lag": "0.0"
+    }
+  ],
+  "message_lsn_receipts": [
+    {
+      "received_lsn": "0/1674688",
+      "last_msg_send_time": "2023-06-19 19:56:35 UTC",
+      "last_msg_receipt_time": "2023-06-19 19:56:35 UTC",
+      "latest_end_lsn": "0/1674688",
+      "latest_end_time": "2023-06-19 19:56:35 UTC"
+    }
+  ],
+  "sync_started_at": "2023-06-19 19:54:54 UTC",
+  "sync_failed_at": null,
+  "switchover_completed_at": null
+
+  ....
+```
+
+### Performing switchover
+
+`pg_easy_replicate` doesn't kick off the switchover on its own. When you start the sync via `start_sync`, it starts the replication between the two databases. Once you have had the time to monitor stats and any other key metrics, you can kick off the `switchover`.
+
+`switchover` will wait until all tables in the group are replicating and the delta for lag is <200kb (by calculating the `pg_wal_lsn_diff` between `sent_lsn` and `write_lsn`) and then perform the switch.
+
+The switch is made by putting the user on the source database in `READ ONLY` mode, so that it is not accepting any more writes and waits for the flush lag to be `0`. It is up to user to kick of a rolling restart of your application containers or failover DNS (more on these below in strategies) after the switchover is complete, so that your application isn't sending any read/write requests to the old/source database.
+
+```bash
+$ pg_easy_replicate switchover  --group-name database-cluster-1
+
+{"name":"pg_easy_replicate","hostname":"PKHXQVK6DW","pid":24192,"level":30,"time":"2023-06-19T16:05:23.033-04:00","v":0,"msg":"Watching lag stats","version":"0.1.0"}
+...
+```
+
+## Replicating single database with custom tables
+
+By default all tables are added for replication but you can create multiple groups with custom tables for the same database. Example
+
+```bash
+
+$ pg_easy_replicate bootstrap --group-name database-cluster-1
+$ pg_easy_replicate start_sync --group-name database-cluster-1 --schema-name public --tables "users, posts, events"
+
+...
+
+$ pg_easy_replicate bootstrap --group-name database-cluster-2
+$ pg_easy_replicate start_sync --group-name database-cluster-2 --schema-name public --tables "comments, views"
+
+...
+$ pg_easy_replicate switchover  --group-name database-cluster-1
+$ pg_easy_replicate switchover  --group-name database-cluster-2
+...
+```
+
+## Switchover strategies with minimal downtime
+
+For minimal downtime, it'd be best to watch/tail the stats and wait until `switchover_completed_at` is updated with a timestamp. Once that happens you can perform any of the following strategies. Note: These are just suggestions and `pg_easy_replicate` doesn't provide any functionalities for this.
+
+### Rolling restart strategy
+
+In this strategy, you have a change ready to go which instructs your application to start connecting to the new database. Either using an environment variable or similar. Depending on the application type, it may or may not require a rolling restart.
+
+Next, you can set up a program that watches the `stats` and waits until `switchover_completed_at` is reporting as `true`. Once that happens it kicks off a rolling restart of your application containers so they can start making connections to the DNS of the new database.
+
+### DNS Failover strategy
+
+In this strategy, you have a weighted based DNS system (example [AWS Route53 weighted records](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-values-weighted.html)) where 100% of traffic goes to a primary origin and 0% to a secondary origin. The primary origin here is the DNS host for your source database and secondary origin is the DNS host for your target database. You can set up your application ahead of time to interact with the database using DNS from the weighted group.
+
+Next, you can set up a program that watches the `stats` and waits until `switchover_completed_at` is reporting as `true`. Once that happens it updates the weight in the DNS weighted group where 100% of the requests now go to the new/target database. Note: Keeping a lot `ttl` is recommended.

data/Rakefile

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "standalone_migrations"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: [:spec, :rubocop]

data/bin/console

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "pg_easy_replicate"
+require "pry"
+
+Pry.start

data/bin/pg_easy_replicate

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "pg_easy_replicate"
+
+PgEasyReplicate::CLI.start(ARGV)

data/bin/release.sh

@@ -0,0 +1,28 @@
+export VERSION=$1
+echo "VERSION: ${VERSION}"
+
+echo "=== Pushing tags to github ===="
+git tag v"$VERSION"
+git push origin --tags
+
+echo "=== Building Gem ===="
+gem build pg_easy_replicate.gemspec
+
+echo "=== Pushing gem ===="
+gem push pg_easy_replicate-"$VERSION".gem
+
+echo "=== Sleeping for 5s ===="
+sleep 5
+
+echo "=== Building Image ===="
+docker build . --build-arg VERSION="$VERSION" -t shayonj/pg-osc:"$VERSION"
+
+echo "=== Tagging Image ===="
+docker image tag shayonj/pg-osc:"$VERSION" shayonj/pg-osc:latest
+
+echo "=== Pushing Image ===="
+docker push shayonj/pg-osc:"$VERSION"
+docker push shayonj/pg-osc:latest
+
+echo "=== Cleaning up ===="
+rm pg_easy_replicate-"$VERSION".gem

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/docker-compose.yml

@@ -0,0 +1,34 @@
+version: "3.7"
+services:
+  source_db:
+    image: postgres:12-alpine
+    ports:
+      - "5432:5432"
+    environment:
+      POSTGRES_USER: jamesbond
+      POSTGRES_PASSWORD: jamesbond
+      POSTGRES_DB: postgres
+    command:
+      - "postgres"
+      - "-c"
+      - "wal_level=logical"
+    networks:
+      localnet:
+
+  target_db:
+    image: postgres:12-alpine
+    ports:
+      - "5433:5432"
+    environment:
+      POSTGRES_USER: jamesbond
+      POSTGRES_PASSWORD: jamesbond
+      POSTGRES_DB: postgres
+    command:
+      - "postgres"
+      - "-c"
+      - "wal_level=logical"
+    networks:
+      localnet:
+
+networks:
+  localnet:

data/lib/pg_easy_replicate/cli.rb

@@ -0,0 +1,121 @@
+# frozen_string_literal: true
+
+require "thor"
+
+module PgEasyReplicate
+  class CLI < Thor
+    package_name "pg_easy_replicate"
+
+    desc "config_check",
+         "Prints if source and target database have the required config"
+    def config_check
+      PgEasyReplicate.assert_config
+
+      puts "✅ Config is looking good."
+    end
+
+    method_option :group_name,
+                  aliases: "-g",
+                  required: true,
+                  desc: "Name of the group to provision"
+    desc "bootstrap",
+         "Sets up temporary tables for information required during runtime"
+    def bootstrap
+      PgEasyReplicate.bootstrap(options)
+    end
+
+    desc "cleanup", "Cleans up all bootstrapped data for the respective group"
+    method_option :group_name,
+                  aliases: "-g",
+                  required: true,
+                  desc: "Name of the group previously provisioned"
+    method_option :everything,
+                  aliases: "-e",
+                  desc:
+                    "Cleans up all bootstrap tables, users and any publication/subscription"
+    method_option :sync,
+                  aliases: "-s",
+                  desc:
+                    "Cleans up the publication and subscription for the respective group"
+    def cleanup
+      PgEasyReplicate.cleanup(options)
+    end
+
+    desc "start_sync",
+         "Starts the logical replication from source database to target database provisioned in the group"
+    method_option :group_name,
+                  aliases: "-g",
+                  required: true,
+                  desc: "Name of the group to provision"
+    method_option :group_name,
+                  aliases: "-g",
+                  required: true,
+                  desc:
+                    "Name of the grouping for this collection of source and target DB"
+    method_option :schema_name,
+                  aliases: "-s",
+                  desc:
+                    "Name of the schema tables are in, only required if passsing list of tables"
+    method_option :tables,
+                  aliases: "-t",
+                  desc:
+                    "Comma separated list of table names. Default: All tables"
+    def start_sync
+      PgEasyReplicate::Orchestrate.start_sync(options)
+    end
+
+    desc "stop_sync",
+         "Stop the logical replication from source database to target database provisioned in the group"
+    method_option :group_name,
+                  aliases: "-g",
+                  required: true,
+                  desc: "Name of the group previously provisioned"
+    def stop_sync
+      PgEasyReplicate::Orchestrate.stop_sync(options[:group_name])
+    end
+
+    desc "switchover ",
+         "Puts the source database in read only mode after all the data is flushed and written"
+    method_option :group_name,
+                  aliases: "-g",
+                  required: true,
+                  desc: "Name of the group previously provisioned"
+    method_option :lag_delta_size,
+                  aliases: "-l",
+                  desc:
+                    "The size of the lag to watch for before switchover. Default 200KB."
+    # method_option :bi_directional,
+    #               aliases: "-b",
+    #               desc:
+    #                 "Setup replication from target database to source database"
+    def switchover
+      PgEasyReplicate::Orchestrate.switchover(
+        group_name: options[:group_name],
+        lag_delta_size: options[:lag_delta_size],
+      )
+    end
+
+    desc "stats ", "Prints the statistics in JSON for the group"
+    method_option :group_name,
+                  aliases: "-g",
+                  required: true,
+                  desc: "Name of the group previously provisioned"
+    method_option :watch, aliases: "-w", desc: "Tail the stats"
+    def stats
+      if options[:watch]
+        PgEasyReplicate::Stats.follow(options[:group_name])
+      else
+        PgEasyReplicate::Stats.print(options[:group_name])
+      end
+    end
+
+    desc "version", "Prints the version"
+    def version
+      puts PgEasyReplicate::VERSION
+    end
+
+    def self.exit_on_failure?
+      true
+    end
+  end
+end

data/lib/pg_easy_replicate/group.rb

@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+module PgEasyReplicate
+  class Group
+    extend Helper
+    class << self
+      def setup
+        conn =
+          Query.connect(
+            connection_url: source_db_url,
+            schema: internal_schema_name,
+          )
+        return if conn.table_exists?("groups")
+        conn.create_table("groups") do
+          primary_key(:id)
+          column(:name, String, null: false)
+          column(:table_names, String, text: true)
+          column(:schema_name, String)
+          column(:created_at, Time, default: Sequel::CURRENT_TIMESTAMP)
+          column(:updated_at, Time, default: Sequel::CURRENT_TIMESTAMP)
+          column(:started_at, Time)
+          column(:failed_at, Time)
+          column(:switchover_completed_at, Time)
+        end
+      ensure
+        conn&.disconnect
+      end
+
+      def drop
+        conn =
+          Query.connect(
+            connection_url: source_db_url,
+            schema: internal_schema_name,
+          ).drop_table?("groups")
+      ensure
+        conn&.disconnect
+      end
+
+      def create(options)
+        groups.insert(
+          name: options[:name],
+          table_names: options[:table_names],
+          schema_name: options[:schema_name],
+          started_at: options[:started_at],
+          failed_at: options[:failed_at],
+        )
+      rescue => e
+        abort_with("Adding group entry failed: #{e.message}")
+      end
+
+      def update(
+        group_name:,
+        started_at: nil,
+        switchover_completed_at: nil,
+        failed_at: nil
+      )
+        set = {
+          started_at: started_at&.utc,
+          switchover_completed_at: switchover_completed_at&.utc,
+          failed_at: failed_at&.utc,
+          updated_at: Time.now.utc,
+        }.compact
+        groups.where(name: group_name).update(set)
+      rescue => e
+        abort_with("Updating group entry failed: #{e.message}")
+      end
+
+      def find(group_name)
+        groups.first(name: group_name)
+      rescue => e
+        abort_with("Finding group entry failed: #{e.message}")
+      end
+
+      def delete(group_name)
+        groups.where(name: group_name).delete
+      rescue => e
+        abort_with("Deleting group entry failed: #{e.message}")
+      end
+
+      private
+
+      def groups
+        conn =
+          Query.connect(
+            connection_url: source_db_url,
+            schema: internal_schema_name,
+          )
+        conn[:groups]
+      end
+    end
+  end
+end

data/lib/pg_easy_replicate/helper.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+module PgEasyReplicate
+  module Helper
+    def source_db_url
+      ENV.fetch("SOURCE_DB_URL", nil)
+    end
+
+    def secondary_source_db_url
+      ENV.fetch("SECONDARY_SOURCE_DB_URL", nil)
+    end
+
+    def target_db_url
+      ENV.fetch("TARGET_DB_URL", nil)
+    end
+
+    def logger
+      PgEasyReplicate.logger
+    end
+
+    def internal_schema_name
+      "pger"
+    end
+
+    def internal_user_name
+      "pger_su"
+    end
+
+    def publication_name(group_name)
+      "pger_publication_#{underscore(group_name)}"
+    end
+
+    def subscription_name(group_name)
+      "pger_subscription_#{underscore(group_name)}"
+    end
+
+    def underscore(str)
+      str
+        .gsub(/::/, "/")
+        .gsub(/([A-Z]+)([A-Z][a-z])/, '\1_\2')
+        .gsub(/([a-z\d])([A-Z])/, '\1_\2')
+        .tr("-", "_")
+        .downcase
+    end
+
+    def test_env?
+      ENV.fetch("RACK_ENV", nil) == "test"
+    end
+
+    def connection_info(conn_string)
+      PG::Connection
+        .conninfo_parse(conn_string)
+        .each_with_object({}) do |obj, hash|
+          hash[obj[:keyword].to_sym] = obj[:val]
+        end
+        .compact
+    end
+
+    def db_user(url)
+      connection_info(url)[:user]
+    end
+
+    def abort_with(msg)
+      raise(msg) if test_env?
+      abort(msg)
+    end
+  end
+end