pgsync 0.5.2 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: af75f9e9eb115a889d53b10ed89d8355e3ef111868b69eec31a025a8f22db27e
-  data.tar.gz: 5da0d56267ac89f2ffc282e92275a91d3fb83d80fe969207329ec8ce1876472b
+  metadata.gz: 6c4a582f7fd5b997ba91247f2b57ad68aae7b5fad282c32dca873d75667922af
+  data.tar.gz: a7ae6518b84d28c8ad14786793b8d4257d898214b8071ccc6c91e9d4d144e316
 SHA512:
-  metadata.gz: 9f9d78aebf02f75a5fee9413b56bba68ed97334b3134cef5c28f4b5a8a15cbab45e752cb73dcd88028e2b3b2b0fefb20ef9bd4abe0c92d35998a520b0029e961
-  data.tar.gz: a2fdaee8ba32df378ed2538119817b7649fe134a33c0f6013d55d8e1cb76ae6b0a75d3dd10dff4872d0452bc2e41bb0024c5f276f16ed8f0d1c81b114574f757
+  metadata.gz: f008b6d7114e3f11395479de0e85006c5eb48681b13ab613e1468d20a99acf0121566fe5499e236d1e8b07fdd12acc2c3a99399daac3e3e99269efac5e6bd4e5
+  data.tar.gz: 536af357b17f35c7afb0e8efba5411a0ecbec67adbfe364d465a8831dd0e56d108055b1ccb55bc0203d92d9a3215a11d3a287f52e89b40b6526534ddfbc8371e

data/CHANGELOG.md

@@ -1,3 +1,40 @@
+## 0.6.1 (2020-06-07)
+
+- Added Django and Laravel integrations
+
+## 0.6.0 (2020-06-07)
+
+- Added messages for different column types and non-deferrable constraints
+- Added support for wildcards to `--exclude`
+- Improved `--overwrite` and `--preserve` options for foreign keys
+- Improved output for schema sync
+- Fixed `--overwrite` and `--preserve` options for multicolumn primary keys
+- Fixed output for notices
+
+Breaking
+
+- Syncs shared tables instead of raising an error when tables missing in destination
+- Raise an error when `--config` or `--db` option provided and config not found
+- Removed deprecated options
+- Dropped support for Postgres < 9.5
+
+## 0.5.5 (2020-05-13)
+
+- Added `--jobs` option
+- Added `--defer-constraints` option
+- Added `--disable-user-triggers` option
+- Added `--disable-integrity` option
+- Improved error message for older libpq
+
+## 0.5.4 (2020-05-09)
+
+- Fixed output for `--in-batches`
+
+## 0.5.3 (2020-04-03)
+
+- Improved Postgres error messages
+- Fixed behavior of wildcard without schema
+
 ## 0.5.2 (2020-03-27)
 
 - Added `--fail-fast` option

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2015-2019 Andrew Kane
+Copyright (c) 2015-2020 Andrew Kane
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -33,20 +33,26 @@ This creates `.pgsync.yml` for you to customize. We recommend checking this into
 
 ## How to Use
 
-Sync all tables
+First, make sure your schema is set up in both databases. We recommend using a schema migration tool for this, but pgsync also provides a few [convenience methods](#schema). Once that’s done, you’re ready to sync data.
+
+Sync tables
 
 ```sh
 pgsync
 ```
 
-**Note:** pgsync assumes your schema is setup in your `to` database. See the [schema section](#schema) if that’s not the case.
-
 Sync specific tables
 
 ```sh
 pgsync table1,table2
 ```
 
+Works with wildcards as well
+
+```sh
+pgsync "table*"
+```
+
 Sync specific rows (existing rows are overwritten)
 
 ```sh
@@ -65,13 +71,15 @@ Or truncate them
 pgsync products "where store_id = 1" --truncate
 ```
 
-### Exclude Tables
+## Tables
+
+Exclude specific tables
 
 ```sh
-pgsync --exclude users
+pgsync --exclude table1,table2
 ```
 
-To always exclude, add to `.pgsync.yml`.
+Add to `.pgsync.yml` to exclude by default
 
 ```yml
 exclude:
@@ -79,15 +87,17 @@ exclude:
   - table2
 ```
 
-For Rails, you probably want to exclude schema migrations and ActiveRecord metadata.
+Sync tables from all schemas or specific schemas (by default, only the search path is synced)
 
-```yml
-exclude:
-  - schema_migrations
-  - ar_internal_metadata
+```sh
+pgsync --all-schemas
+# or
+pgsync --schemas public,other
+# or
+pgsync public.table1,other.table2
 ```
 
-### Groups
+## Groups
 
 Define groups in `.pgsync.yml`:
 
@@ -104,6 +114,8 @@ And run:
 pgsync group1
 ```
 
+## Variables
+
 You can also use groups to sync a specific record and associated records in other tables.
 
 To get product `123` with its reviews, last 10 coupons, and store, use:
@@ -123,18 +135,14 @@ And run:
 pgsync product:123
 ```
 
-### Schema
-
-**Note:** pgsync is designed to sync data. You should use a schema migration tool to manage schema changes. The methods in this section are provided for convenience but not recommended.
+## Schema
 
-Sync schema before the data
+Sync schema before the data (this wipes out existing data)
 
 ```sh
 pgsync --schema-first
 ```
 
-**Note:** This wipes out existing data
-
 Specify tables
 
 ```sh
@@ -149,13 +157,9 @@ pgsync --schema-only
 
 pgsync does not try to sync Postgres extensions.
 
-## Data Protection
-
-Always make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust. Your best option is to connect over SSH or a VPN. Another option is to use `sslmode=verify-full`. If you don’t do this, your database credentials can be compromised.
-
-## Sensitive Information
+## Sensitive Data
 
-Prevent sensitive information like email addresses from leaving the remote server.
+Prevent sensitive data like email addresses from leaving the remote server.
 
 Define rules in `.pgsync.yml`:
 
@@ -188,7 +192,61 @@ Options for replacement are:
 - `null`
 - `untouched`
 
-Rules starting with `unique_` require the table to have a primary key. `unique_phone` requires a numeric primary key.
+Rules starting with `unique_` require the table to have a single column primary key. `unique_phone` requires a numeric primary key.
+
+## Foreign Keys
+
+Foreign keys can make it difficult to sync data. Three options are:
+
+1. Manually specify the order of tables
+2. Use deferrable constraints
+3. Disable foreign key triggers, which can silently break referential integrity
+
+When manually specifying the order, use `--jobs 1` so tables are synced one-at-a-time.
+
+```sh
+pgsync table1,table2,table3 --jobs 1
+```
+
+If your tables have [deferrable constraints](https://begriffs.com/posts/2017-08-27-deferrable-sql-constraints.html), use:
+
+```sh
+pgsync --defer-constraints
+```
+
+To disable foreign key triggers and potentially break referential integrity, use:
+
+```sh
+pgsync --disable-integrity
+```
+
+## Triggers
+
+Disable user triggers with:
+
+```sh
+pgsync --disable-user-triggers
+```
+
+## Append-Only Tables
+
+For extremely large, append-only tables, sync in batches.
+
+```sh
+pgsync large_table --in-batches
+```
+
+The script will resume where it left off when run again, making it great for backfills.
+
+## Connection Security
+
+Always make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust. Your best option is to connect over SSH or a VPN. Another option is to use `sslmode=verify-full`. If you don’t do this, your database credentials can be compromised.
+
+## Safety
+
+To keep you from accidentally overwriting production, the destination is limited to `localhost` or `127.0.0.1` by default.
+
+To use another host, add `to_safe: true` to your `.pgsync.yml`.
 
 ## Multiple Databases
 
@@ -204,31 +262,50 @@ This creates `.pgsync-db2.yml` for you to edit. Specify a database in commands w
 pgsync --db db2
 ```
 
-## Safety
+## Integrations
 
-To keep you from accidentally overwriting production, the destination is limited to `localhost` or `127.0.0.1` by default.
+- [Django](#django)
+- [Heroku](#heroku)
+- [Laravel](#laravel)
+- [Rails](#rails)
 
-To use another host, add `to_safe: true` to your `.pgsync.yml`.
+### Django
 
-## Large Tables
+If you run `pgsync --init` in a Django project, migrations will be excluded in `.pgsync.yml`.
 
-For extremely large tables, sync in batches.
+```yml
+exclude:
+  - django_migrations
+```
 
-```sh
-pgsync large_table --in-batches
+### Heroku
+
+If you run `pgsync --init` in a Heroku project, the `from` database will be set in `.pgsync.yml`.
+
+```yml
+from: $(heroku config:get DATABASE_URL)?sslmode=require
 ```
 
-The script will resume where it left off when run again, making it great for backfills.
+### Laravel
 
-## Foreign Keys
+If you run `pgsync --init` in a Laravel project, migrations will be excluded in `.pgsync.yml`.
 
-By default, tables are copied in parallel. If you use foreign keys, this can cause violations. You can specify tables to be copied serially with:
+```yml
+exclude:
+  - migrations
+```
 
-```sh
-pgsync group1 --debug
+### Rails
+
+If you run `pgsync --init` in a Rails project, Active Record metadata and schema migrations will be excluded in `.pgsync.yml`.
+
+```yml
+exclude:
+  - ar_internal_metadata
+  - schema_migrations
 ```
 
-## Reference
+## Other Commands
 
 Help
 
@@ -242,6 +319,12 @@ Version
 pgsync --version
 ```
 
+List tables
+
+```sh
+pgsync --list
+```
+
 ## Scripts
 
 Use groups when possible to take advantage of parallelism.

data/config.yml

@@ -20,6 +20,10 @@ to: postgres://localhost:5432/myapp_development
 #     - table1
 #     - table2
 
+# sync specific schemas
+# schemas:
+#   - public
+
 # protect sensitive information
 data_rules:
   email: unique_email

data/exe/pgsync

@@ -1,9 +1,5 @@
 #!/usr/bin/env ruby
 
-# disable Slop warning with Ruby 2.7
-# TODO remove when Slop > 4.7.0 is released
-$VERBOSE = nil
-
 # handle interrupts
 trap("SIGINT") { abort }
 

data/lib/pgsync.rb

@@ -10,15 +10,19 @@ require "shellwords"
 require "tempfile"
 require "uri"
 require "yaml"
+require "open3"
 
 # modules
 require "pgsync/utils"
 require "pgsync/client"
 require "pgsync/data_source"
 require "pgsync/init"
+require "pgsync/schema_sync"
 require "pgsync/sync"
-require "pgsync/table_list"
+require "pgsync/table"
 require "pgsync/table_sync"
+require "pgsync/task"
+require "pgsync/task_resolver"
 require "pgsync/version"
 
 module PgSync

data/lib/pgsync/client.rb

@@ -4,73 +4,75 @@ module PgSync
 
     def initialize(args)
       @args = args
-      $stderr.sync = true
+      output.sync = true
     end
 
-    def perform(testing: true)
-      opts = parse_args
+    def perform
+      result = Slop::Parser.new(slop_options).parse(@args)
+      arguments = result.arguments
+      options = result.to_h
 
-      if opts.version?
+      raise Error, "Specify either --db or --config, not both" if options[:db] && options[:config]
+      raise Error, "Cannot use --overwrite with --in-batches" if options[:overwrite] && options[:in_batches]
+
+      if options[:version]
         log VERSION
-      elsif opts.help?
-        log opts
-      # TODO remove deprecated conditions (last two)
-      elsif opts.init? || opts.setup? || opts.arguments[0] == "setup"
-        Init.new.perform(opts)
+      elsif options[:help]
+        log slop_options
+      elsif options[:init]
+        Init.new(arguments, options).perform
       else
-        Sync.new.perform(opts)
+        Sync.new(arguments, options).perform
       end
-    rescue Error, PG::ConnectionBad => e
-      raise e if testing
-      abort colorize(e.message, 31) # red
+    rescue => e
+      # Error, PG::ConnectionBad, Slop::Error
+      raise e if options && options[:debug]
+      abort colorize(e.message.strip, :red)
     end
 
     def self.start
-      new(ARGV).perform(testing: false)
+      new(ARGV).perform
     end
 
     protected
 
-    def parse_args
-      Slop.parse(@args) do |o|
-        o.banner = %{Usage:
-    pgsync [options]
+    def slop_options
+      o = Slop::Options.new
+      o.banner = %{Usage:
+  pgsync [options]
 
 Options:}
-        o.string "-d", "--db", "database"
-        o.string "-t", "--tables", "tables to sync"
-        o.string "-g", "--groups", "groups to sync"
-        o.string "--schemas", "schemas to sync"
-        o.string "--from", "source"
-        o.string "--to", "destination"
-        o.string "--where", "where", help: false
-        o.integer "--limit", "limit", help: false
-        o.string "--exclude", "exclude tables"
-        o.string "--config", "config file"
-        # TODO much better name for this option
-        o.boolean "--to-safe", "accept danger", default: false
-        o.boolean "--debug", "debug", default: false
-        o.boolean "--list", "list", default: false
-        o.boolean "--overwrite", "overwrite existing rows", default: false, help: false
-        o.boolean "--preserve", "preserve existing rows", default: false
-        o.boolean "--truncate", "truncate existing rows", default: false
-        o.boolean "--schema-first", "schema first", default: false
-        o.boolean "--schema-only", "schema only", default: false
-        o.boolean "--all-schemas", "all schemas", default: false
-        o.boolean "--no-rules", "do not apply data rules", default: false
-        o.boolean "--no-sequences", "do not sync sequences", default: false
-        o.boolean "--init", "init", default: false
-        o.boolean "--setup", "setup", default: false, help: false
-        o.boolean "--in-batches", "in batches", default: false, help: false
-        o.integer "--batch-size", "batch size", default: 10000, help: false
-        o.float "--sleep", "sleep", default: 0, help: false
-        o.boolean "--fail-fast", "stop on the first failed table", default: false
-        # o.array "--var", "pass a variable"
-        o.boolean "-v", "--version", "print the version"
-        o.boolean "-h", "--help", "prints help"
-      end
-    rescue Slop::Error => e
-      raise Error, e.message
+      o.string "-d", "--db", "database"
+      o.string "-t", "--tables", "tables to sync"
+      o.string "-g", "--groups", "groups to sync"
+      o.integer "-j", "--jobs", "number of tables to sync at a time"
+      o.string "--schemas", "schemas to sync"
+      o.string "--from", "source"
+      o.string "--to", "destination"
+      o.string "--exclude", "exclude tables"
+      o.string "--config", "config file"
+      o.boolean "--to-safe", "accept danger", default: false
+      o.boolean "--debug", "debug", default: false
+      o.boolean "--list", "list", default: false
+      o.boolean "--overwrite", "overwrite existing rows", default: false, help: false
+      o.boolean "--preserve", "preserve existing rows", default: false
+      o.boolean "--truncate", "truncate existing rows", default: false
+      o.boolean "--schema-first", "schema first", default: false
+      o.boolean "--schema-only", "schema only", default: false
+      o.boolean "--all-schemas", "all schemas", default: false
+      o.boolean "--no-rules", "do not apply data rules", default: false
+      o.boolean "--no-sequences", "do not sync sequences", default: false
+      o.boolean "--init", "init", default: false
+      o.boolean "--in-batches", "in batches", default: false, help: false
+      o.integer "--batch-size", "batch size", default: 10000, help: false
+      o.float "--sleep", "sleep", default: 0, help: false
+      o.boolean "--fail-fast", "stop on the first failed table", default: false
+      o.boolean "--defer-constraints", "defer constraints", default: false
+      o.boolean "--disable-user-triggers", "disable non-system triggers", default: false
+      o.boolean "--disable-integrity", "disable foreign key triggers", default: false
+      o.boolean "-v", "--version", "print the version"
+      o.boolean "-h", "--help", "prints help"
+      o
     end
   end
 end