pg_online_schema_change 0.2.0 → 0.5.0

data/docs/load-test.md

@@ -0,0 +1,138 @@
+# Preliminary Load Test
+
+## pg-osc: No downtime schema changes with 7K+ writes/s & 12k+ reads/s
+
+This is a very basic load test performed with `pgbench` against a single instance PostgreSQL DB running on DigitialOcean with the following configuration:
+
+- **128GB RAM**
+- **32vCPU**
+- **695GB Disk**
+- Trasanction based connection pool with **500 pool limit**
+
+Total time taken to run schema change: **<3mins**
+
+## Simulating load with pgbench
+
+**Initialize**
+```
+pgbench -p $PORT --initialize -s 20 -F 20 --foreign-keys --host $HOST -U $USERNAME  -d $DB 
+```
+
+This creates bunch of pgbench tables. The table being used with `pg-osc` is `pgbench_accounts` which has FKs and also references by other tables with FKS, containing 2M rows.
+
+**Begin**
+```
+pgbench -p $PORT -j 72 -c 288 -T 500 -r --host $DB_HOST -U $USERNAME -d $DB  
+```
+
+## Running pg-osc
+
+Simple `ALTER` statement for experimentation purposes.
+
+```sql
+ALTER TABLE pgbench_accounts ADD COLUMN "purchased" BOOLEAN DEFAULT FALSE;
+```
+
+**Execution**
+
+```bash
+bundle exec bin/pg-online-schema-change perform \ 
+-a 'ALTER TABLE pgbench_accounts ADD COLUMN "purchased" BOOLEAN DEFAULT FALSE;' \
+-d "pool" \
+-p 25061
+-h "..." \
+-u "..." \
+--pull-batch-count 2000 \
+--delta-count 200
+```
+
+## Outcome
+
+**pgbench results**
+
+```
+number of transactions actually processed: 1060382
+latency average = 144.874 ms
+tps = 1767.057392 (including connections establishing)
+tps = 1777.971823 (excluding connections establishing)
+statement latencies in milliseconds:
+         0.479  \set aid random(1, 100000 * :scale)
+         0.409  \set bid random(1, 1 * :scale)
+         0.247  \set tid random(1, 10 * :scale)
+         0.208  \set delta random(-5000, 5000)
+         3.136  BEGIN;
+         4.243  UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+         4.488  SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+        71.017  UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+        46.689  UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+         4.035  INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+         4.166  END;
+```
+
+**Metrics**
+![load-test](load-test-1.png)
+
+**New table structure**
+
+Added `purchased` column.
+
+```
+defaultdb=> \d+ pgbench_accounts;
+                                  Table "public.pgbench_accounts"
+  Column   |     Type      | Collation | Nullable | Default | Storage  | Stats target | Description 
+-----------+---------------+-----------+----------+---------+----------+--------------+-------------
+ aid       | integer       |           | not null |         | plain    |              | 
+ bid       | integer       |           |          |         | plain    |              | 
+ abalance  | integer       |           |          |         | plain    |              | 
+ filler    | character(84) |           |          |         | extended |              | 
+ purchased | boolean       |           |          | false   | plain    |              | 
+Indexes:
+    "pgosc_st_pgbench_accounts_815029_pkey" PRIMARY KEY, btree (aid)
+Foreign-key constraints:
+    "pgbench_accounts_bid_fkey" FOREIGN KEY (bid) REFERENCES pgbench_branches(bid)
+Referenced by:
+    TABLE "pgbench_history" CONSTRAINT "pgbench_history_aid_fkey" FOREIGN KEY (aid) REFERENCES pgbench_accounts(aid)
+Options: autovacuum_enabled=false, fillfactor=20
+```
+
+**Logs**
+
+<details>
+<summary>Logs from pg-osc</summary>
+
+```json
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:01.147-05:00","v":0,"msg":"Setting up audit table","audit_table":"pgosc_at_pgbench_accounts_714a8b","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:01.660-05:00","v":0,"msg":"Setting up triggers","version":"0.4.0"}
+NOTICE:  trigger "primary_to_audit_table_trigger" for relation "pgbench_accounts" does not exist, skipping
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:01.814-05:00","v":0,"msg":"Setting up shadow table","shadow_table":"pgosc_st_pgbench_accounts_714a8b","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:02.169-05:00","v":0,"msg":"Running alter statement on shadow table","shadow_table":"pgosc_st_pgbench_accounts_714a8b","parent_table":"pgbench_accounts","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:02.204-05:00","v":0,"msg":"Clearing contents of audit table before copy..","shadow_table":"pgosc_st_pgbench_accounts_714a8b","parent_table":"pgbench_accounts","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:02.240-05:00","v":0,"msg":"Copying contents..","shadow_table":"pgosc_st_pgbench_accounts_714a8b","parent_table":"pgbench_accounts","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:20.481-05:00","v":0,"msg":"Performing ANALYZE!","version":"0.4.0"}
+INFO:  analyzing "public.pgbench_accounts"
+INFO:  "pgbench_accounts": scanned 30000 of 166667 pages, containing 360000 live rows and 200 dead rows; 30000 rows in sample, 2000004 estimated total rows
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:21.078-05:00","v":0,"msg":"Replaying rows, count: 2000","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:21.580-05:00","v":0,"msg":"Replaying rows, count: 2000","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:22.022-05:00","v":0,"msg":"Replaying rows, count: 2000","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:22.490-05:00","v":0,"msg":"Replaying rows, count: 2000","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:22.866-05:00","v":0,"msg":"Replaying rows, count: 661","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:23.212-05:00","v":0,"msg":"Replaying rows, count: 533","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:23.512-05:00","v":0,"msg":"Replaying rows, count: 468","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:23.809-05:00","v":0,"msg":"Remaining rows below delta count, proceeding towards swap","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:23.809-05:00","v":0,"msg":"Performing swap!","version":"0.4.0"}
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:24.259-05:00","v":0,"msg":"Replaying rows, count: 449","version":"0.4.0"}
+NOTICE:  trigger "primary_to_audit_table_trigger" for relation "pgbench_accounts" does not exist, skipping
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:24.650-05:00","v":0,"msg":"Performing ANALYZE!","version":"0.4.0"}
+INFO:  analyzing "public.pgbench_accounts"
+INFO:  "pgbench_accounts": scanned 30000 of 32935 pages, containing 1821834 live rows and 6056 dead rows; 30000 rows in sample, 2000070 estimated total rows
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:24.941-05:00","v":0,"msg":"Validating constraints!","version":"0.4.0"}
+NOTICE:  table "pgosc_st_pgbench_accounts_714a8b" does not exist, skipping
+{"name":"pg-online-schema-change","hostname":"MacBook-Pro.local","pid":13263,"level":30,"time":"2022-02-25T17:22:26.159-05:00","v":0,"msg":"All tasks successfully completed","version":"0.4.0"}
+```
+
+</details>
+
+
+## Conclusion
+
+By tweaking `--pull-batch-count` to `2000` (replay 2k rows at once) and `--delta-count` to `200` (time to swap when remaining rows is <200), `pg-osc` was able to perform the schema change with no impact within very quick time. Depending on the database size and load on the table, you can further tune them to achieve desired impact. At some point this is going to plateau - I can imagine the replay factor not working quite well for say 100k commits/s workloads. So, YMMV.

data/lib/pg_online_schema_change/cli.rb

@@ -1,8 +1,12 @@
+# frozen_string_literal: true
+
 require "thor"
 
 module PgOnlineSchemaChange
+  PULL_BATCH_COUNT = 1000
+  DELTA_COUNT = 20
   class CLI < Thor
-    desc "perform", "Perform the set of operations to safely apply the schema change with minimal locks"
+    desc "perform", "Safely apply schema changes with minimal locks"
     method_option :alter_statement, aliases: "-a", type: :string, required: true,
                                     desc: "The ALTER statement to perform the schema change"
     method_option :schema, aliases: "-s", type: :string, required: true, default: "public",
@@ -11,7 +15,7 @@ module PgOnlineSchemaChange
     method_option :host, aliases: "-h", type: :string, required: true, desc: "Server host where the Database is located"
     method_option :username, aliases: "-u", type: :string, required: true, desc: "Username for the Database"
     method_option :port, aliases: "-p", type: :numeric, required: true, default: 5432, desc: "Port for the Database"
-    method_option :password, aliases: "-w", type: :string, required: true, desc: "Password for the Database"
+    method_option :password, aliases: "-w", type: :string, required: true, desc: "DEPRECATED: Password for the Database. Please pass PGPASSWORD environment variable instead."
     method_option :verbose, aliases: "-v", type: :boolean, default: false, desc: "Emit logs in debug mode"
     method_option :drop, aliases: "-f", type: :boolean, default: false,
                          desc: "Drop the original table in the end after the swap"
@@ -21,11 +25,19 @@ module PgOnlineSchemaChange
                                        desc: "Time to wait before killing backends to acquire lock and/or retrying upto 3 times. It will kill backends if --kill-backends is true, otherwise try upto 3 times and exit if it cannot acquire a lock."
     method_option :copy_statement, aliases: "-c", type: :string, required: false, default: "",
                                    desc: "Takes a .sql file location where you can provide a custom query to be played (ex: backfills) when pgosc copies data from the primary to the shadow table. More examples in README."
+    method_option :pull_batch_count, aliases: "-b", type: :numeric, required: false, default: PULL_BATCH_COUNT,
+                                     desc: "Number of rows to be replayed on each iteration after copy. This can be tuned for faster catch up and swap. Best used with delta-count."
+    method_option :delta_count, aliases: "-e", type: :numeric, required: false, default: DELTA_COUNT,
+                                desc: "Indicates how many rows should be remaining before a swap should be performed. This can be tuned for faster catch up and swap, especially on highly volume tables. Best used with pull-batch-count."
 
     def perform
       client_options = Struct.new(*options.keys.map(&:to_sym)).new(*options.values)
+      PgOnlineSchemaChange.logger(verbose: client_options.verbose)
+
+      PgOnlineSchemaChange.logger.warn("DEPRECATED: -w is deprecated. Please pass PGPASSWORD environment variable instead.") if client_options.password
+
+      client_options.password = ENV["PGPASSWORD"] || client_options.password
 
-      PgOnlineSchemaChange.logger = client_options.verbose
       PgOnlineSchemaChange::Orchestrate.run!(client_options)
     end
 

data/lib/pg_online_schema_change/client.rb

@@ -1,9 +1,11 @@
+# frozen_string_literal: true
+
 require "pg"
 
 module PgOnlineSchemaChange
   class Client
     attr_accessor :alter_statement, :schema, :dbname, :host, :username, :port, :password, :connection, :table, :drop,
-                  :kill_backends, :wait_time_for_lock, :copy_statement
+                  :kill_backends, :wait_time_for_lock, :copy_statement, :pull_batch_count, :delta_count
 
     def initialize(options)
       @alter_statement = options.alter_statement
@@ -16,7 +18,11 @@ module PgOnlineSchemaChange
       @drop = options.drop
       @kill_backends = options.kill_backends
       @wait_time_for_lock = options.wait_time_for_lock
+      @pull_batch_count = options.pull_batch_count
+      @delta_count = options.delta_count
+
       handle_copy_statement(options.copy_statement)
+      handle_validations
 
       @connection = PG.connect(
         dbname: @dbname,
@@ -26,17 +32,19 @@ module PgOnlineSchemaChange
         port: @port,
       )
 
-      raise Error, "Not a valid ALTER statement: #{@alter_statement}" unless Query.alter_statement?(@alter_statement)
-
-      unless Query.same_table?(@alter_statement)
-        raise Error "All statements should belong to the same table: #{@alter_statement}"
-      end
-
       @table = Query.table(@alter_statement)
 
       PgOnlineSchemaChange.logger.debug("Connection established")
     end
 
+    def handle_validations
+      raise Error, "Not a valid ALTER statement: #{@alter_statement}" unless Query.alter_statement?(@alter_statement)
+
+      return if Query.same_table?(@alter_statement)
+
+      raise Error "All statements should belong to the same table: #{@alter_statement}"
+    end
+
     def handle_copy_statement(statement)
       return if statement.nil? || statement == ""
 

data/lib/pg_online_schema_change/functions.rb

@@ -1,4 +1,6 @@
-FUNC_FIX_SERIAL_SEQUENCE = <<~SQL.freeze
+# frozen_string_literal: true
+
+FUNC_FIX_SERIAL_SEQUENCE = <<~SQL
     CREATE OR REPLACE FUNCTION fix_serial_sequence(_table regclass, _newtable text)
     RETURNS void AS
     $func$
@@ -35,7 +37,7 @@ FUNC_FIX_SERIAL_SEQUENCE = <<~SQL.freeze
   $func$  LANGUAGE plpgsql VOLATILE;
 SQL
 
-FUNC_CREATE_TABLE_ALL = <<~SQL.freeze
+FUNC_CREATE_TABLE_ALL = <<~SQL
   CREATE OR REPLACE FUNCTION create_table_all(source_table text, newsource_table text)
     RETURNS void language plpgsql
     as $$

data/lib/pg_online_schema_change/helper.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 module PgOnlineSchemaChange
   module Helper
     def primary_key
@@ -15,7 +17,14 @@ module PgOnlineSchemaChange
       result = Store.send(:get, method)
       return result if result
 
-      raise ArgumentError, "Method `#{method}` doesn't exist."
+      super
+    end
+
+    def respond_to_missing?(method_name, *args)
+      result = Store.send(:get, method)
+      return true if result
+
+      super
     end
   end
 end

data/lib/pg_online_schema_change/orchestrate.rb

@@ -1,8 +1,10 @@
+# frozen_string_literal: true
+
 require "securerandom"
 
 module PgOnlineSchemaChange
   class Orchestrate
-    SWAP_STATEMENT_TIMEOUT = "5s".freeze
+    SWAP_STATEMENT_TIMEOUT = "5s"
 
     extend Helper
 
@@ -21,12 +23,20 @@ module PgOnlineSchemaChange
         Query.run(client.connection, FUNC_FIX_SERIAL_SEQUENCE)
         Query.run(client.connection, FUNC_CREATE_TABLE_ALL)
 
+        setup_store
+      end
+
+      def setup_store
         # Set this early on to ensure their creation and cleanup (unexpected)
         # happens at all times. IOW, the calls from Store.get always return
         # the same value.
         Store.set(:old_primary_table, "pgosc_op_table_#{client.table}")
-        Store.set(:audit_table, "pgosc_at_#{client.table}_#{random_string}")
-        Store.set(:shadow_table, "pgosc_st_#{client.table}_#{random_string}")
+        Store.set(:audit_table, "pgosc_at_#{client.table}_#{pgosc_identifier}")
+        Store.set(:operation_type_column, "operation_type_#{pgosc_identifier}")
+        Store.set(:trigger_time_column, "trigger_time_#{pgosc_identifier}")
+        Store.set(:audit_table_pk, "at_#{pgosc_identifier}_id")
+        Store.set(:audit_table_pk_sequence, "#{audit_table}_#{audit_table_pk}_seq")
+        Store.set(:shadow_table, "pgosc_st_#{client.table}_#{pgosc_identifier}")
       end
 
       def run!(options)
@@ -70,7 +80,7 @@ module PgOnlineSchemaChange
         reader = setup_signals!
         signal = reader.gets.chomp
 
-        while !reader.closed? && IO.select([reader])
+        while !reader.closed? && IO.select([reader]) # rubocop:disable Lint/UnreachableLoop
           logger.info "Signal #{signal} received, cleaning up"
 
           client.connection.cancel
@@ -85,7 +95,7 @@ module PgOnlineSchemaChange
         logger.info("Setting up audit table", { audit_table: audit_table })
 
         sql = <<~SQL
-          CREATE TABLE #{audit_table} (operation_type text, trigger_time timestamp, LIKE #{client.table});
+          CREATE TABLE #{audit_table} (#{audit_table_pk} SERIAL PRIMARY KEY, #{operation_type_column} text, #{trigger_time_column} timestamp, LIKE #{client.table});
         SQL
 
         Query.run(client.connection, sql)
@@ -109,13 +119,13 @@ module PgOnlineSchemaChange
           $$
           BEGIN
             IF ( TG_OP = 'INSERT') THEN
-              INSERT INTO \"#{audit_table}\" select 'INSERT', now(), NEW.* ;
+              INSERT INTO \"#{audit_table}\" select nextval(\'#{audit_table_pk_sequence}\'), 'INSERT', clock_timestamp(), NEW.* ;
               RETURN NEW;
             ELSIF ( TG_OP = 'UPDATE') THEN
-              INSERT INTO \"#{audit_table}\" select 'UPDATE', now(),  NEW.* ;
+              INSERT INTO \"#{audit_table}\" select nextval(\'#{audit_table_pk_sequence}\'), 'UPDATE', clock_timestamp(),  NEW.* ;
               RETURN NEW;
             ELSIF ( TG_OP = 'DELETE') THEN
-              INSERT INTO \"#{audit_table}\" select 'DELETE', now(), OLD.* ;
+              INSERT INTO \"#{audit_table}\" select nextval(\'#{audit_table_pk_sequence}\'), 'DELETE', clock_timestamp(), OLD.* ;
               RETURN NEW;
             END IF;
           END;
@@ -153,7 +163,7 @@ module PgOnlineSchemaChange
         # re-uses transaction with serializable
         # Disabling vacuum to avoid any issues during the process
         result = Query.storage_parameters_for(client, client.table, true) || ""
-        primary_table_storage_parameters = Store.set(:primary_table_storage_parameters, result)
+        Store.set(:primary_table_storage_parameters, result)
 
         logger.debug("Disabling vacuum on shadow and audit table",
                      { shadow_table: shadow_table, audit_table: audit_table })
@@ -185,8 +195,7 @@ module PgOnlineSchemaChange
         # Begin the process to copy data into copy table
         # depending on the size of the table, this can be a time
         # taking operation.
-        logger.info("Clearing contents of audit table before copy..",
-                    { shadow_table: shadow_table, parent_table: client.table })
+        logger.info("Clearing contents of audit table before copy..", { shadow_table: shadow_table, parent_table: client.table })
         Query.run(client.connection, "DELETE FROM #{audit_table}", true)
 
         logger.info("Copying contents..", { shadow_table: shadow_table, parent_table: client.table })
@@ -195,7 +204,7 @@ module PgOnlineSchemaChange
           return Query.run(client.connection, query, true)
         end
 
-        sql = Query.copy_data_statement(client, shadow_table)
+        sql = Query.copy_data_statement(client, shadow_table, true)
         Query.run(client.connection, sql, true)
       ensure
         Query.run(client.connection, "COMMIT;") # commit the serializable transaction
@@ -272,8 +281,10 @@ module PgOnlineSchemaChange
         Query.run(client.connection, sql)
       end
 
-      private def random_string
-        @random_string ||= SecureRandom.hex(3)
+      private
+
+      def pgosc_identifier
+        @pgosc_identifier ||= SecureRandom.hex(3)
       end
     end
   end

data/lib/pg_online_schema_change/query.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require "pg_query"
 require "pg"
 
@@ -5,7 +7,7 @@ module PgOnlineSchemaChange
   class Query
     extend Helper
 
-    INDEX_SUFFIX = "_pgosc".freeze
+    INDEX_SUFFIX = "_pgosc"
     DROPPED_COLUMN_TYPE = :AT_DropColumn
     RENAMED_COLUMN_TYPE = :AT_RenameColumn
     LOCK_ATTEMPT = 4
@@ -15,28 +17,28 @@ module PgOnlineSchemaChange
         PgQuery.parse(query).tree.stmts.all? do |statement|
           statement.stmt.alter_table_stmt.instance_of?(PgQuery::AlterTableStmt) || statement.stmt.rename_stmt.instance_of?(PgQuery::RenameStmt)
         end
-      rescue PgQuery::ParseError => e
+      rescue PgQuery::ParseError
         false
       end
 
       def same_table?(query)
-        tables = PgQuery.parse(query).tree.stmts.map do |statement|
+        tables = PgQuery.parse(query).tree.stmts.filter_map do |statement|
           if statement.stmt.alter_table_stmt.instance_of?(PgQuery::AlterTableStmt)
             statement.stmt.alter_table_stmt.relation.relname
           elsif statement.stmt.rename_stmt.instance_of?(PgQuery::RenameStmt)
             statement.stmt.rename_stmt.relation.relname
           end
-        end.compact
+        end
 
         tables.uniq.count == 1
-      rescue PgQuery::ParseError => e
+      rescue PgQuery::ParseError
         false
       end
 
       def table(query)
-        from_rename_statement = PgQuery.parse(query).tree.stmts.map do |statement|
+        from_rename_statement = PgQuery.parse(query).tree.stmts.filter_map do |statement|
           statement.stmt.rename_stmt&.relation&.relname
-        end.compact[0]
+        end[0]
         PgQuery.parse(query).tables[0] || from_rename_statement
       end
 
@@ -48,7 +50,7 @@ module PgOnlineSchemaChange
         connection.async_exec("BEGIN;")
 
         result = connection.async_exec(query, &block)
-      rescue Exception
+      rescue Exception # rubocop:disable Lint/RescueException
         connection.cancel if connection.transaction_status != PG::PQTRANS_IDLE
         connection.block
         logger.info("Exception raised, rolling back query", { rollback: true, query: query })
@@ -144,11 +146,11 @@ module PgOnlineSchemaChange
         end
 
         references.map do |row|
-          if row["definition"].end_with?("NOT VALID")
-            add_statement = "ALTER TABLE #{row["table_on"]} ADD CONSTRAINT #{row["constraint_name"]} #{row["definition"]};"
-          else
-            add_statement = "ALTER TABLE #{row["table_on"]} ADD CONSTRAINT #{row["constraint_name"]} #{row["definition"]} NOT VALID;"
-          end
+          add_statement = if row["definition"].end_with?("NOT VALID")
+                            "ALTER TABLE #{row["table_on"]} ADD CONSTRAINT #{row["constraint_name"]} #{row["definition"]};"
+                          else
+                            "ALTER TABLE #{row["table_on"]} ADD CONSTRAINT #{row["constraint_name"]} #{row["definition"]} NOT VALID;"
+                          end
 
           drop_statement = "ALTER TABLE #{row["table_on"]} DROP CONSTRAINT #{row["constraint_name"]};"
 
@@ -291,7 +293,7 @@ module PgOnlineSchemaChange
           client.connection.quote_ident(select_column)
         end
 
-        sql = <<~SQL
+        <<~SQL
           INSERT INTO #{shadow_table}(#{insert_into_columns.join(", ")})
           SELECT #{select_columns.join(", ")}
           FROM ONLY #{client.table}

data/lib/pg_online_schema_change/replay.rb

@@ -1,12 +1,12 @@
+# frozen_string_literal: true
+
+# rubocop:disable Metrics/AbcSize, Metrics/MethodLength
+
 module PgOnlineSchemaChange
   class Replay
     extend Helper
 
     class << self
-      PULL_BATCH_COUNT = 1000
-      DELTA_COUNT = 20
-      RESERVED_COLUMNS = %w[operation_type trigger_time].freeze
-
       # This, picks PULL_BATCH_COUNT rows by primary key from audit_table,
       # replays it on the shadow_table. Once the batch is done,
       # it them deletes those PULL_BATCH_COUNT rows from audit_table. Then, pull another batch,
@@ -17,7 +17,7 @@ module PgOnlineSchemaChange
         loop do
           rows = rows_to_play
 
-          raise CountBelowDelta if rows.count <= DELTA_COUNT
+          raise CountBelowDelta if rows.count <= client.delta_count
 
           play!(rows)
         end
@@ -25,7 +25,7 @@ module PgOnlineSchemaChange
 
       def rows_to_play(reuse_trasaction = false)
         select_query = <<~SQL
-          SELECT * FROM #{audit_table} ORDER BY #{primary_key} LIMIT #{PULL_BATCH_COUNT};
+          SELECT * FROM #{audit_table} ORDER BY #{audit_table_pk} LIMIT #{client.pull_batch_count};
         SQL
 
         rows = []
@@ -34,6 +34,10 @@ module PgOnlineSchemaChange
         rows
       end
 
+      def reserved_columns
+        @reserved_columns ||= [trigger_time_column, operation_type_column, audit_table_pk]
+      end
+
       def play!(rows, reuse_trasaction = false)
         logger.info("Replaying rows, count: #{rows.size}")
 
@@ -44,7 +48,7 @@ module PgOnlineSchemaChange
 
           # Remove audit table cols, since we will be
           # re-mapping them for inserts and updates
-          RESERVED_COLUMNS.each do |col|
+          reserved_columns.each do |col|
             new_row.delete(col)
           end
 
@@ -73,7 +77,7 @@ module PgOnlineSchemaChange
             client.connection.escape_string(value)
           end
 
-          case row["operation_type"]
+          case row[operation_type_column]
           when "INSERT"
             values = new_row.map { |_, val| "'#{val}'" }.join(",")
 
@@ -110,13 +114,15 @@ module PgOnlineSchemaChange
         Query.run(client.connection, to_be_replayed.join, reuse_trasaction)
 
         # Delete items from the audit now that are replayed
-        if to_be_deleted_rows.count >= 1
-          delete_query = <<~SQL
-            DELETE FROM #{audit_table} WHERE #{primary_key} IN (#{to_be_deleted_rows.join(",")})
-          SQL
-          Query.run(client.connection, delete_query, reuse_trasaction)
-        end
+        return unless to_be_deleted_rows.count >= 1
+
+        delete_query = <<~SQL
+          DELETE FROM #{audit_table} WHERE #{primary_key} IN (#{to_be_deleted_rows.join(",")})
+        SQL
+        Query.run(client.connection, delete_query, reuse_trasaction)
       end
     end
   end
 end
+
+# rubocop:enable Metrics/AbcSize, Metrics/MethodLength

data/lib/pg_online_schema_change/store.rb

@@ -1,17 +1,21 @@
+# frozen_string_literal: true
+
 require "pg_query"
 require "pg"
 
 module PgOnlineSchemaChange
   class Store
     class << self
-      @@object = {}
+      @object = {}
 
       def get(key)
-        @@object[key.to_s] || @@object[key.to_sym]
+        @object ||= {}
+        @object[key.to_s] || @object[key.to_sym]
       end
 
       def set(key, value)
-        @@object[key.to_sym] = value
+        @object ||= {}
+        @object[key.to_sym] = value
       end
     end
   end

data/lib/pg_online_schema_change/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module PgOnlineSchemaChange
-  VERSION = "0.2.0"
+  VERSION = "0.5.0"
 end

data/lib/pg_online_schema_change.rb

@@ -6,28 +6,24 @@ require "ougai"
 require "pg_online_schema_change/version"
 require "pg_online_schema_change/helper"
 require "pg_online_schema_change/functions"
-require "pg_online_schema_change/cli"
 require "pg_online_schema_change/client"
 require "pg_online_schema_change/query"
 require "pg_online_schema_change/store"
 require "pg_online_schema_change/replay"
 require "pg_online_schema_change/orchestrate"
+require "pg_online_schema_change/cli"
 
 module PgOnlineSchemaChange
   class Error < StandardError; end
   class CountBelowDelta < StandardError; end
   class AccessExclusiveLockNotAcquired < StandardError; end
 
-  def self.logger=(verbose)
-    @@logger ||= begin
-        logger = Ougai::Logger.new($stdout)
-        logger.level = verbose ? Ougai::Logger::TRACE : Ougai::Logger::INFO
-        logger.with_fields = { version: PgOnlineSchemaChange::VERSION }
-        logger
-      end
-  end
-
-  def self.logger
-    @@logger
+  def self.logger(verbose: false)
+    @logger ||= begin
+      logger = Ougai::Logger.new($stdout)
+      logger.level = verbose ? Ougai::Logger::TRACE : Ougai::Logger::INFO
+      logger.with_fields = { version: PgOnlineSchemaChange::VERSION }
+      logger
+    end
   end
 end

data/scripts/release.sh

@@ -0,0 +1,28 @@
+export VERSION=$1
+echo "VERSION: ${VERSION}"
+
+echo "=== Pushing tags to github ===="
+git tag v$VERSION
+git push origin --tags
+
+echo "=== Building Gem ===="
+gem build pg_online_schema_change.gemspec
+
+echo "=== Pushing gem ===="
+gem push pg_online_schema_change-$VERSION.gem
+
+echo "=== Sleeping for 5s ===="
+sleep 5 
+
+echo "=== Building Image ===="
+docker build . --build-arg VERSION=$VERSION -t shayonj/pg-osc:$VERSION
+
+echo "=== Tagging Image ===="
+docker image tag shayonj/pg-osc:$VERSION shayonj/pg-osc:latest
+
+echo "=== Pushing Image ===="
+docker push shayonj/pg-osc:$VERSION
+docker push shayonj/pg-osc:latest
+
+echo "=== Cleaning up ===="
+rm pg_online_schema_change-$VERSION.gem