pgsync 0.5.5 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 32d1e60a133659dde98d3564d21fca4f1866e8f7b58518ef71f6dcaf59cf8c82
-  data.tar.gz: 650b50aee614ed737d2e99278cbb8910f7980a041dcf58d18288f6bc53826635
+  metadata.gz: f6c20cf99ebc0961d8699228883f6c2e9c95c9798cfb6f101cb78803df6b1b43
+  data.tar.gz: 7c719442c07e6db1d4704199aa2b95d32e1ba528797b2d21b15be85f2de71d79
 SHA512:
-  metadata.gz: 8bd3e4373558c8f8519f0c118f51b55ac27a3a82fb1ab9a38e85bfee858e08a80fb2c37cab490878d2dda06121c13f45d81b3df6c87b8ede0e0060a50b94fee6
-  data.tar.gz: 922ecb8f57615025e5b92a93f82010bb667fbc968db12ebddeb49507ce90d6dd4c3c16ea3cd5777817bb321a8fefa54f6483b6ffae872f897e7ada64d68831b0
+  metadata.gz: 1e3d735e6e3002e2fcb74de574f63f534472821aca0154c13d9d7ce6e9fa94426239cef875cd33724fec5a6cf2567b5656a80a9d92f89d9678ee6bee2c2ec70c
+  data.tar.gz: 3ab5d416f4c2364644c015d106631898a3615cc671179f8b1193cda5b1e894512fb39906ec1c27570ca9fbd186c9eb19af3afb88eba093bb21e8bc471c28f18b

data/CHANGELOG.md

@@ -1,9 +1,25 @@
+## 0.6.0 (2020-06-07)
+
+- Added messages for different column types and non-deferrable constraints
+- Added support for wildcards to `--exclude`
+- Improved `--overwrite` and `--preserve` options for foreign keys
+- Improved output for schema sync
+- Fixed `--overwrite` and `--preserve` options for multicolumn primary keys
+- Fixed output for notices
+
+Breaking
+
+- Syncs shared tables instead of raising an error when tables missing in destination
+- Raise an error when `--config` or `--db` option provided and config not found
+- Removed deprecated options
+- Dropped support for Postgres < 9.5
+
 ## 0.5.5 (2020-05-13)
 
 - Added `--jobs` option
-- Added experimental `--defer-constraints` option
-- Added experimental `--disable-user-triggers` option
-- Added experimental `--disable-integrity` option
+- Added `--defer-constraints` option
+- Added `--disable-user-triggers` option
+- Added `--disable-integrity` option
 - Improved error message for older libpq
 
 ## 0.5.4 (2020-05-09)

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2015-2019 Andrew Kane
+Copyright (c) 2015-2020 Andrew Kane
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -33,20 +33,26 @@ This creates `.pgsync.yml` for you to customize. We recommend checking this into
 
 ## How to Use
 
+First, make sure your schema is set up in both databases. We recommend using a schema migration tool for this, but pgsync also provides a few [convenience methods](#schema). Once that’s done, you’re ready to sync data.
+
 Sync all tables
 
 ```sh
 pgsync
 ```
 
-**Note:** pgsync assumes your schema is setup in your `to` database. See the [schema section](#schema) if that’s not the case.
-
 Sync specific tables
 
 ```sh
 pgsync table1,table2
 ```
 
+Works with wildcards as well
+
+```sh
+pgsync "table*"
+```
+
 Sync specific rows (existing rows are overwritten)
 
 ```sh
@@ -65,13 +71,15 @@ Or truncate them
 pgsync products "where store_id = 1" --truncate
 ```
 
-## Exclude Tables
+## Tables
+
+Exclude specific tables
 
 ```sh
-pgsync --exclude users
+pgsync --exclude table1,table2
 ```
 
-To always exclude, add to `.pgsync.yml`.
+Add to `.pgsync.yml` to exclude by default
 
 ```yml
 exclude:
@@ -79,12 +87,14 @@ exclude:
   - table2
 ```
 
-For Rails, you probably want to exclude schema migrations and Active Record metadata.
+Sync tables from all schemas or specific schemas (by default, only the search path is synced)
 
-```yml
-exclude:
-  - schema_migrations
-  - ar_internal_metadata
+```sh
+pgsync --all-schemas
+# or
+pgsync --schemas public,other
+# or
+pgsync public.table1,other.table2
 ```
 
 ## Groups
@@ -104,6 +114,8 @@ And run:
 pgsync group1
 ```
 
+## Variables
+
 You can also use groups to sync a specific record and associated records in other tables.
 
 To get product `123` with its reviews, last 10 coupons, and store, use:
@@ -125,16 +137,12 @@ pgsync product:123
 
 ## Schema
 
-**Note:** pgsync is designed to sync data. You should use a schema migration tool to manage schema changes. The methods in this section are provided for convenience but not recommended.
-
-Sync schema before the data
+Sync schema before the data (this wipes out existing data)
 
 ```sh
 pgsync --schema-first
 ```
 
-**Note:** This wipes out existing data
-
 Specify tables
 
 ```sh
@@ -149,9 +157,9 @@ pgsync --schema-only
 
 pgsync does not try to sync Postgres extensions.
 
-## Sensitive Information
+## Sensitive Data
 
-Prevent sensitive information like email addresses from leaving the remote server.
+Prevent sensitive data like email addresses from leaving the remote server.
 
 Define rules in `.pgsync.yml`:
 
@@ -184,7 +192,7 @@ Options for replacement are:
 - `null`
 - `untouched`
 
-Rules starting with `unique_` require the table to have a primary key. `unique_phone` requires a numeric primary key.
+Rules starting with `unique_` require the table to have a single column primary key. `unique_phone` requires a numeric primary key.
 
 ## Foreign Keys
 
@@ -192,7 +200,7 @@ Foreign keys can make it difficult to sync data. Three options are:
 
 1. Manually specify the order of tables
 2. Use deferrable constraints
-3. Disable triggers, which can silently break referential integrity
+3. Disable foreign key triggers, which can silently break referential integrity
 
 When manually specifying the order, use `--jobs 1` so tables are synced one-at-a-time.
 
@@ -206,16 +214,12 @@ If your tables have [deferrable constraints](https://begriffs.com/posts/2017-08-
 pgsync --defer-constraints
 ```
 
-**Note:** This feature is currently experimental.
-
-To disable triggers and potentially break referential integrity, use:
+To disable foreign key triggers and potentially break referential integrity, use:
 
 ```sh
 pgsync --disable-integrity
 ```
 
-**Note:** This feature is currently experimental.
-
 ## Triggers
 
 Disable user triggers with:
@@ -224,8 +228,6 @@ Disable user triggers with:
 pgsync --disable-user-triggers
 ```
 
-**Note:** This feature is currently experimental.
-
 ## Append-Only Tables
 
 For extremely large, append-only tables, sync in batches.

data/config.yml

@@ -20,6 +20,10 @@ to: postgres://localhost:5432/myapp_development
 #     - table1
 #     - table2
 
+# sync specific schemas
+# schemas:
+#   - public
+
 # protect sensitive information
 data_rules:
   email: unique_email

data/lib/pgsync.rb

@@ -10,15 +10,19 @@ require "shellwords"
 require "tempfile"
 require "uri"
 require "yaml"
+require "open3"
 
 # modules
 require "pgsync/utils"
 require "pgsync/client"
 require "pgsync/data_source"
 require "pgsync/init"
+require "pgsync/schema_sync"
 require "pgsync/sync"
-require "pgsync/table_list"
+require "pgsync/table"
 require "pgsync/table_sync"
+require "pgsync/task"
+require "pgsync/task_resolver"
 require "pgsync/version"
 
 module PgSync

data/lib/pgsync/client.rb

@@ -7,77 +7,72 @@ module PgSync
       output.sync = true
     end
 
-    def perform(testing: true)
-      opts = parse_args
+    def perform
+      result = Slop::Parser.new(slop_options).parse(@args)
+      arguments = result.arguments
+      options = result.to_h
 
-      # TODO throw error in 0.6.0
-      warn "Specify either --db or --config, not both" if opts[:db] && opts[:config]
+      raise Error, "Specify either --db or --config, not both" if options[:db] && options[:config]
+      raise Error, "Cannot use --overwrite with --in-batches" if options[:overwrite] && options[:in_batches]
 
-      if opts.version?
+      if options[:version]
         log VERSION
-      elsif opts.help?
-        log opts
-      # TODO remove deprecated conditions (last two)
-      elsif opts.init? || opts.setup? || opts.arguments[0] == "setup"
-        Init.new.perform(opts)
+      elsif options[:help]
+        log slop_options
+      elsif options[:init]
+        Init.new(arguments, options).perform
       else
-        Sync.new.perform(opts)
+        Sync.new(arguments, options).perform
       end
-    rescue Error, PG::ConnectionBad => e
-      raise e if testing
-      abort colorize(e.message, :red)
+    rescue => e
+      # Error, PG::ConnectionBad, Slop::Error
+      raise e if options && options[:debug]
+      abort colorize(e.message.strip, :red)
     end
 
     def self.start
-      new(ARGV).perform(testing: false)
+      new(ARGV).perform
     end
 
     protected
 
-    def parse_args
-      Slop.parse(@args) do |o|
-        o.banner = %{Usage:
-    pgsync [options]
+    def slop_options
+      o = Slop::Options.new
+      o.banner = %{Usage:
+  pgsync [options]
 
 Options:}
-        o.string "-d", "--db", "database"
-        o.string "-t", "--tables", "tables to sync"
-        o.string "-g", "--groups", "groups to sync"
-        o.integer "-j", "--jobs", "number of tables to sync at a time"
-        o.string "--schemas", "schemas to sync"
-        o.string "--from", "source"
-        o.string "--to", "destination"
-        o.string "--where", "where", help: false
-        o.integer "--limit", "limit", help: false
-        o.string "--exclude", "exclude tables"
-        o.string "--config", "config file"
-        # TODO much better name for this option
-        o.boolean "--to-safe", "accept danger", default: false
-        o.boolean "--debug", "debug", default: false
-        o.boolean "--list", "list", default: false
-        o.boolean "--overwrite", "overwrite existing rows", default: false, help: false
-        o.boolean "--preserve", "preserve existing rows", default: false
-        o.boolean "--truncate", "truncate existing rows", default: false
-        o.boolean "--schema-first", "schema first", default: false
-        o.boolean "--schema-only", "schema only", default: false
-        o.boolean "--all-schemas", "all schemas", default: false
-        o.boolean "--no-rules", "do not apply data rules", default: false
-        o.boolean "--no-sequences", "do not sync sequences", default: false
-        o.boolean "--init", "init", default: false
-        o.boolean "--setup", "setup", default: false, help: false
-        o.boolean "--in-batches", "in batches", default: false, help: false
-        o.integer "--batch-size", "batch size", default: 10000, help: false
-        o.float "--sleep", "sleep", default: 0, help: false
-        o.boolean "--fail-fast", "stop on the first failed table", default: false
-        o.boolean "--defer-constraints", "defer constraints", default: false
-        o.boolean "--disable-user-triggers", "disable non-system triggers", default: false
-        o.boolean "--disable-integrity", "disable foreign key triggers", default: false
-        # o.array "--var", "pass a variable"
-        o.boolean "-v", "--version", "print the version"
-        o.boolean "-h", "--help", "prints help"
-      end
-    rescue Slop::Error => e
-      raise Error, e.message
+      o.string "-d", "--db", "database"
+      o.string "-t", "--tables", "tables to sync"
+      o.string "-g", "--groups", "groups to sync"
+      o.integer "-j", "--jobs", "number of tables to sync at a time"
+      o.string "--schemas", "schemas to sync"
+      o.string "--from", "source"
+      o.string "--to", "destination"
+      o.string "--exclude", "exclude tables"
+      o.string "--config", "config file"
+      o.boolean "--to-safe", "accept danger", default: false
+      o.boolean "--debug", "debug", default: false
+      o.boolean "--list", "list", default: false
+      o.boolean "--overwrite", "overwrite existing rows", default: false, help: false
+      o.boolean "--preserve", "preserve existing rows", default: false
+      o.boolean "--truncate", "truncate existing rows", default: false
+      o.boolean "--schema-first", "schema first", default: false
+      o.boolean "--schema-only", "schema only", default: false
+      o.boolean "--all-schemas", "all schemas", default: false
+      o.boolean "--no-rules", "do not apply data rules", default: false
+      o.boolean "--no-sequences", "do not sync sequences", default: false
+      o.boolean "--init", "init", default: false
+      o.boolean "--in-batches", "in batches", default: false, help: false
+      o.integer "--batch-size", "batch size", default: 10000, help: false
+      o.float "--sleep", "sleep", default: 0, help: false
+      o.boolean "--fail-fast", "stop on the first failed table", default: false
+      o.boolean "--defer-constraints", "defer constraints", default: false
+      o.boolean "--disable-user-triggers", "disable non-system triggers", default: false
+      o.boolean "--disable-integrity", "disable foreign key triggers", default: false
+      o.boolean "-v", "--version", "print the version"
+      o.boolean "-h", "--help", "prints help"
+      o
     end
   end
 end

data/lib/pgsync/data_source.rb

@@ -1,9 +1,11 @@
 module PgSync
   class DataSource
+    include Utils
+
     attr_reader :url
 
-    def initialize(source)
-      @url = resolve_url(source)
+    def initialize(url)
+      @url = url
     end
 
     def exists?
@@ -29,8 +31,18 @@ module PgSync
     # gets visible tables
     def tables
       @tables ||= begin
-        query = "SELECT table_schema, table_name FROM information_schema.tables WHERE table_type = 'BASE TABLE' AND table_schema NOT IN ('information_schema', 'pg_catalog') ORDER BY 1, 2"
-        execute(query).map { |row| "#{row["table_schema"]}.#{row["table_name"]}" }
+        query = <<~SQL
+          SELECT
+            table_schema AS schema,
+            table_name AS table
+          FROM
+            information_schema.tables
+          WHERE
+            table_type = 'BASE TABLE' AND
+            table_schema NOT IN ('information_schema', 'pg_catalog')
+          ORDER BY 1, 2
+        SQL
+        execute(query).map { |row| Table.new(row["schema"], row["table"]) }
       end
     end
 
@@ -38,25 +50,21 @@ module PgSync
       table_set.include?(table)
     end
 
-    def columns(table)
-      query = "SELECT column_name FROM information_schema.columns WHERE table_schema = $1 AND table_name = $2"
-      execute(query, table.split(".", 2)).map { |row| row["column_name"] }
-    end
-
     def sequences(table, columns)
-      execute("SELECT #{columns.map { |f| "pg_get_serial_sequence(#{escape("#{quote_ident_full(table)}")}, #{escape(f)}) AS #{quote_ident(f)}" }.join(", ")}")[0].values.compact
+      execute("SELECT #{columns.map { |f| "pg_get_serial_sequence(#{escape("#{quote_ident_full(table)}")}, #{escape(f)}) AS #{quote_ident(f)}" }.join(", ")}").first.values.compact
     end
 
     def max_id(table, primary_key, sql_clause = nil)
-      execute("SELECT MAX(#{quote_ident(primary_key)}) FROM #{quote_ident_full(table)}#{sql_clause}")[0]["max"].to_i
+      execute("SELECT MAX(#{quote_ident(primary_key)}) FROM #{quote_ident_full(table)}#{sql_clause}").first["max"].to_i
     end
 
     def min_id(table, primary_key, sql_clause = nil)
-      execute("SELECT MIN(#{quote_ident(primary_key)}) FROM #{quote_ident_full(table)}#{sql_clause}")[0]["min"].to_i
+      execute("SELECT MIN(#{quote_ident(primary_key)}) FROM #{quote_ident_full(table)}#{sql_clause}").first["min"].to_i
     end
 
+    # this value comes from pg_get_serial_sequence which is already quoted
     def last_value(seq)
-      execute("select last_value from #{seq}")[0]["last_value"]
+      execute("SELECT last_value FROM #{seq}").first["last_value"]
     end
 
     def truncate(table)
@@ -64,28 +72,31 @@ module PgSync
     end
 
     # https://stackoverflow.com/a/20537829
+    # TODO can simplify with array_position in Postgres 9.5+
     def primary_key(table)
-      query = <<-SQL
+      query = <<~SQL
         SELECT
           pg_attribute.attname,
-          format_type(pg_attribute.atttypid, pg_attribute.atttypmod)
+          format_type(pg_attribute.atttypid, pg_attribute.atttypmod),
+          pg_attribute.attnum,
+          pg_index.indkey
         FROM
           pg_index, pg_class, pg_attribute, pg_namespace
         WHERE
-          pg_class.oid = $2::regclass AND
-          indrelid = pg_class.oid AND
           nspname = $1 AND
+          relname = $2 AND
+          indrelid = pg_class.oid AND
           pg_class.relnamespace = pg_namespace.oid AND
           pg_attribute.attrelid = pg_class.oid AND
           pg_attribute.attnum = any(pg_index.indkey) AND
           indisprimary
       SQL
-      row = execute(query, [table.split(".", 2)[0], quote_ident_full(table)])[0]
-      row && row["attname"]
+      rows = execute(query, [table.schema, table.name])
+      rows.sort_by { |r| r["indkey"].split(" ").index(r["attnum"]) }.map { |r| r["attname"] }
     end
 
     def triggers(table)
-      query = <<-SQL
+      query = <<~SQL
         SELECT
           tgname AS name,
           tgisinternal AS internal,
@@ -108,9 +119,10 @@ module PgSync
           else
             config = {dbname: @url}
           end
+          @concurrent_id = concurrent_id
           PG::Connection.new(config)
         rescue URI::InvalidURIError
-          raise Error, "Invalid connection string"
+          raise Error, "Invalid connection string. Make sure it works with `psql`"
         end
       end
     end
@@ -122,32 +134,17 @@ module PgSync
       end
     end
 
-    def reconnect
-      @conn.reset
-    end
-
-    def dump_command(tables)
-      tables = tables ? tables.keys.map { |t| "-t #{Shellwords.escape(quote_ident_full(t))}" }.join(" ") : ""
-      "pg_dump -Fc --verbose --schema-only --no-owner --no-acl #{tables} -d #{@url}"
+    # reconnect for new thread or process
+    def reconnect_if_needed
+      reconnect if @concurrent_id != concurrent_id
     end
 
-    def restore_command
-      if_exists = Gem::Version.new(pg_restore_version) >= Gem::Version.new("9.4.0")
-      "pg_restore --verbose --no-owner --no-acl --clean #{if_exists ? "--if-exists" : nil} -d #{@url}"
-    end
-
-    def fully_resolve_tables(tables)
-      no_schema_tables = {}
-      search_path_index = Hash[search_path.map.with_index.to_a]
-      self.tables.group_by { |t| t.split(".", 2)[-1] }.each do |group, t2|
-        no_schema_tables[group] = t2.sort_by { |t| [search_path_index[t.split(".", 2)[0]] || 1000000, t] }[0]
-      end
-
-      Hash[tables.map { |k, v| [no_schema_tables[k] || k, v] }]
+    def search_path
+      @search_path ||= execute("SELECT unnest(current_schemas(true)) AS schema").map { |r| r["schema"] }
     end
 
-    def search_path
-      @search_path ||= execute("SELECT current_schemas(true)")[0]["current_schemas"][1..-2].split(",")
+    def server_version_num
+      @server_version_num ||= execute("SHOW server_version_num").first["server_version_num"].to_i
     end
 
     def execute(query, params = [])
@@ -167,10 +164,13 @@ module PgSync
 
     private
 
-    def pg_restore_version
-      `pg_restore --version`.lines[0].chomp.split(" ")[-1].split(/[^\d.]/)[0]
-    rescue Errno::ENOENT
-      raise Error, "pg_restore not found"
+    def concurrent_id
+      [Process.pid, Thread.current.object_id]
+    end
+
+    def reconnect
+      @conn.reset
+      @concurrent_id = concurrent_id
     end
 
     def table_set
@@ -185,41 +185,5 @@ module PgSync
         conn.conninfo_hash
       end
     end
-
-    def quote_ident_full(ident)
-      ident.split(".", 2).map { |v| quote_ident(v) }.join(".")
-    end
-
-    def quote_ident(value)
-      PG::Connection.quote_ident(value)
-    end
-
-    def escape(value)
-      if value.is_a?(String)
-        "'#{quote_string(value)}'"
-      else
-        value
-      end
-    end
-
-    # activerecord
-    def quote_string(s)
-      s.gsub(/\\/, '\&\&').gsub(/'/, "''")
-    end
-
-    def resolve_url(source)
-      if source
-        source = source.dup
-        source.gsub!(/\$\([^)]+\)/) do |m|
-          command = m[2..-2]
-          result = `#{command}`.chomp
-          unless $?.success?
-            raise Error, "Command exited with non-zero status:\n#{command}"
-          end
-          result
-        end
-      end
-      source
-    end
   end
 end