pgslice 0.4.1 → 0.4.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '00404860fae8f92d17c1be36fb44f6b8fea7413644bf3f184a2431589c750abc'
-  data.tar.gz: db1520df2c4bd9097927cf73900785c42bebeb67f747a12a857c48fdc6ca64a9
+  metadata.gz: 30d1c5d3ed842d1250761b64f3da34a6e25e0ea17d7689d1eca9f1f0d4db1b8f
+  data.tar.gz: 82cbeccebcbdc6edeff9808d9013635d880aa8af20cb462722bdfe0fca5732a1
 SHA512:
-  metadata.gz: accd820d05e151745c26cf7138394ded42f0b50f3272f051b8a154cf6dbc51271a8f810f41602138ed003bfba72a3279129638d7eb312cff4a967d6ef05bfe2e
-  data.tar.gz: 1b52fc649ff7f7a6832ad69236c3de3343b00cf762bfe9d935dc2be96552dfcee69e55d2ece4926e1d3b0d8a9123c970db31f706250c3d19501796c0a9a38874
+  metadata.gz: 1db66bb9409d26d39952230c127d5edc900cf6070915d8a58ef4bec94a460d2db4588991fc2ed46e3c9205e904c424c6a379711c0ffc985e1723623f2eed3835
+  data.tar.gz: 60612ea69df38bcc43f27e492f93ca96f17dc9528236f3e107ea1e7990b5df372dd7d799ea1bdebd6ae3064903a63e9d086668208ba6d3e5958b6247d237df43

data/CHANGELOG.md

@@ -1,101 +1,127 @@
-## 0.4.1
+## 0.4.6 (2020-05-29)
+
+- Ensure correct order with multi-column primary keys
+- Ensure fill always uses correct date range (bug introduced in 0.4.5)
+
+## 0.4.5 (2018-10-18)
+
+- Added support for Postgres 11 foreign key improvements
+- Improved versioning
+
+## 0.4.4 (2018-08-18)
+
+- Added partitioning by `year`
+- Fixed `--source-table` and `--dest-table` options
+- Added descriptions to options
+
+## 0.4.3 (2018-08-16)
+
+- Fixed sequence ownership
+- Improved help
+
+## 0.4.2 (2018-07-23)
+
+- Added support for Postgres 11 index improvements
+- Added support for all connection options
+
+## 0.4.1 (2018-04-30)
 
 - Better support for schemas
 - Use latest partition for schema
 - Added support for composite primary keys
 
-## 0.4.0
+## 0.4.0 (2017-10-07)
 
 - Added support for declarative partitioning
 - Added support for foreign keys
 
-## 0.3.6
+## 0.3.6 (2017-07-10)
 
 - Fixed drop trigger on `unprep` for non-lowercase tables
 - Fixed index creation for non-lowercase tables
 
-## 0.3.5
+## 0.3.5 (2017-07-06)
 
 - Added support for non-lowercase tables and columns
 
-## 0.3.4
+## 0.3.4 (2017-07-06)
 
 - Added `analyze` method
 - Fixed `fill` with `--dry-run` option
 - Better error message for tables without primary key
 
-## 0.3.3
+## 0.3.3 (2017-03-22)
 
 - Fixed error when creating partitions
 
-## 0.3.2
+## 0.3.2 (2016-12-15)
 
 - Exit with error code on interrupt
 - Fixed `--start` option with `--swapped`
 
-## 0.3.1
+## 0.3.1 (2016-12-13)
 
 - Fixed exception with `--no-partition` option
 - Use proper cast type in `fill` method for legacy `timestamptz` columns
 
-## 0.3.0
+## 0.3.0 (2016-12-12)
 
 - Better query performance for `timestamptz` columns
 - Added support for schemas other than `public`
 
-## 0.2.3
+## 0.2.3 (2016-10-10)
 
 - Added `--dest-table` option to `fill`
 - Fixed errors with `fill` when no partitions created
 
-## 0.2.2
+## 0.2.2 (2016-10-06)
 
 - Set `lock_timeout` on `swap` to prevent bad things from happening
 - Friendlier error messages
 
-## 0.2.1
+## 0.2.1 (2016-09-28)
 
 - Added `--where` option to `fill`
 - Fixed partition detection with `fill`
 - Fixed error for columns named `user` with `fill`
 
-## 0.2.0
+## 0.2.0 (2016-09-22)
 
 - Switched to new trigger, which is about 20% faster
 
-## 0.1.7
+## 0.1.7 (2016-09-14)
 
 - Added `--source-table` option to `fill`
 
-## 0.1.6
+## 0.1.6 (2016-08-04)
 
 - Added `--no-partition` option to `prep`
 - Added `--url` option
 
-## 0.1.5
+## 0.1.5 (2016-04-26)
 
 - Removed `activesupport` dependency for speed
 - Fixed `fill` for months
 
-## 0.1.4
+## 0.1.4 (2016-04-24)
 
 - Added sequence ownership
 - Default to 0 for `--past` and `--future` options
 - Better `fill` with `--swapped`
 
-## 0.1.3
+## 0.1.3 (2016-04-24)
 
 - Fixed table inheritance
 
-## 0.1.2
+## 0.1.2 (2016-04-24)
 
 - Added `--dry-run` option
 - Print sql to stdout instead of stderr
 
-## 0.1.1
+## 0.1.1 (2016-04-24)
 
 - Added sql commands to output
 
-## 0.1.0
+## 0.1.0 (2016-04-24)
 
 - First release

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2016-2017 Andrew Kane
+Copyright (c) 2016-2020 Andrew Kane
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,6 +1,6 @@
 # pgslice
 
-Postgres partitioning as easy as pie. Works great for both new and existing tables, with zero downtime and minimal app changes. Archive older data on a rolling basis to keep your database size under control.
+Postgres partitioning as easy as pie. Works great for both new and existing tables, with zero downtime and minimal app changes. No need to install anything on your database server. Archive older data on a rolling basis to keep your database size under control.
 
 :tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
 
@@ -14,7 +14,7 @@ pgslice is a command line tool. To install, run:
 gem install pgslice
 ```
 
-This will give you the `pgslice` command.
+This will give you the `pgslice` command. If installation fails, you may need to install [dependencies](#dependencies).
 
 ## Steps
 
@@ -32,18 +32,16 @@ This will give you the `pgslice` command.
   pgslice prep <table> <column> <period>
   ```
 
-  Period can be `day` or `month`.
+  Period can be `day`, `month`, or `year`.
 
-  This creates a table named `<table>_intermediate` with the appropriate trigger for partitioning.
+  This creates a partitioned table named `<table>_intermediate`.
 
-4. Add partitions
+4. Add partitions to the intermediate table
 
   ```sh
   pgslice add_partitions <table> --intermediate --past 3 --future 3
   ```
 
-  This creates child tables that inherit from the intermediate table.
-
   Use the `--past` and `--future` options to control the number of partitions.
 
 5. *Optional, for tables with data* - Fill the partitions in batches with data from the original table
@@ -80,7 +78,7 @@ This will give you the `pgslice` command.
 
   ```sql
   pg_dump -c -Fc -t <table>_retired $PGSLICE_URL > <table>_retired.dump
-  psql -c "DROP <table>_retired" $PGSLICE_URL
+  psql -c "DROP TABLE <table>_retired" $PGSLICE_URL
   ```
 
 ## Sample Output
@@ -92,34 +90,13 @@ pgslice prep visits created_at month
 ```
 
 ```sql
--- Postgres 10
-
-BEGIN;
-
-CREATE TABLE visits_intermediate (LIKE visits INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING STORAGE INCLUDING COMMENTS) PARTITION BY RANGE (created_at);
-
-COMMENT ON TABLE visits_intermediate is 'column:created_at,period:day';
-
-COMMIT;
-
--- Postgres 9.6 and below
-
 BEGIN;
 
-CREATE TABLE visits_intermediate (LIKE visits INCLUDING ALL);
+CREATE TABLE "public"."visits_intermediate" (LIKE "public"."visits" INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING STORAGE INCLUDING COMMENTS) PARTITION BY RANGE ("created_at");
 
-CREATE FUNCTION visits_insert_trigger()
-    RETURNS trigger AS $$
-    BEGIN
-        RAISE EXCEPTION 'Create partitions first.';
-    END;
-    $$ LANGUAGE plpgsql;
+CREATE INDEX ON "public"."visits_intermediate" USING btree ("created_at");
 
-CREATE TRIGGER visits_insert_trigger
-    BEFORE INSERT ON visits_intermediate
-    FOR EACH ROW EXECUTE PROCEDURE visits_insert_trigger();
-
-COMMENT ON TRIGGER visits_insert_trigger ON visits_intermediate is 'column:created_at,period:month';
+COMMENT ON TABLE "public"."visits_intermediate" is 'column:createdAt,period:day,cast:date,version:3';
 
 COMMIT;
 ```
@@ -129,73 +106,19 @@ pgslice add_partitions visits --intermediate --past 1 --future 1
 ```
 
 ```sql
--- Postgres 10
-
 BEGIN;
 
-CREATE TABLE visits_201608 PARTITION OF visits_intermediate FOR VALUES FROM ('2016-08-01') TO ('2016-09-01');
-
-ALTER TABLE visits_201608 ADD PRIMARY KEY (id);
-
-CREATE INDEX ON visits_201608 USING btree (user_id);
-
-CREATE TABLE visits_201609 PARTITION OF visits_intermediate FOR VALUES FROM ('2016-09-01') TO ('2016-10-01');
+CREATE TABLE "public"."visits_201808" PARTITION OF "public"."visits_intermediate" FOR VALUES FROM ('2018-08-01') TO ('2018-09-01');
 
-ALTER TABLE visits_201609 ADD PRIMARY KEY (id);
+ALTER TABLE "public"."visits_201808" ADD PRIMARY KEY ("id");
 
-CREATE INDEX ON visits_201609 USING btree (user_id);
+CREATE TABLE "public"."visits_201809" PARTITION OF "public"."visits_intermediate" FOR VALUES FROM ('2018-09-01') TO ('2018-10-01');
 
-CREATE TABLE visits_201610 PARTITION OF visits_intermediate FOR VALUES FROM ('2016-10-01') TO ('2016-11-01');
+ALTER TABLE "public"."visits_201809" ADD PRIMARY KEY ("id");
 
-ALTER TABLE visits_201610 ADD PRIMARY KEY (id);
+CREATE TABLE "public"."visits_201810" PARTITION OF "public"."visits_intermediate" FOR VALUES FROM ('2018-10-01') TO ('2018-11-01');
 
-CREATE INDEX ON visits_201610 USING btree (user_id);
-
-COMMIT;
-
--- Postgres 9.6 and below
-
-BEGIN;
-
-CREATE TABLE visits_201608
-    (CHECK (created_at >= '2016-08-01'::date AND created_at < '2016-09-01'::date))
-    INHERITS (visits_intermediate);
-
-ALTER TABLE visits_201608 ADD PRIMARY KEY (id);
-
-CREATE INDEX ON visits_201608 USING btree (user_id);
-
-CREATE TABLE visits_201609
-    (CHECK (created_at >= '2016-09-01'::date AND created_at < '2016-10-01'::date))
-    INHERITS (visits_intermediate);
-
-ALTER TABLE visits_201609 ADD PRIMARY KEY (id);
-
-CREATE INDEX ON visits_201609 USING btree (user_id);
-
-CREATE TABLE visits_201610
-    (CHECK (created_at >= '2016-10-01'::date AND created_at < '2016-11-01'::date))
-    INHERITS (visits_intermediate);
-
-ALTER TABLE visits_201610 ADD PRIMARY KEY (id);
-
-CREATE INDEX ON visits_201610 USING btree (user_id);
-
-CREATE OR REPLACE FUNCTION visits_insert_trigger()
-    RETURNS trigger AS $$
-    BEGIN
-        IF (NEW.created_at >= '2016-09-01'::date AND NEW.created_at < '2016-10-01'::date) THEN
-            INSERT INTO visits_201609 VALUES (NEW.*);
-        ELSIF (NEW.created_at >= '2016-10-01'::date AND NEW.created_at < '2016-11-01'::date) THEN
-            INSERT INTO visits_201610 VALUES (NEW.*);
-        ELSIF (NEW.created_at >= '2016-08-01'::date AND NEW.created_at < '2016-09-01'::date) THEN
-            INSERT INTO visits_201608 VALUES (NEW.*);
-        ELSE
-            RAISE EXCEPTION 'Date out of range. Ensure partitions are created.';
-        END IF;
-        RETURN NULL;
-    END;
-    $$ LANGUAGE plpgsql;
+ALTER TABLE "public"."visits_201808" ADD PRIMARY KEY ("id");
 
 COMMIT;
 ```
@@ -206,19 +129,19 @@ pgslice fill visits
 
 ```sql
 /* 1 of 3 */
-INSERT INTO visits_intermediate ("id", "user_id", "ip", "created_at")
-    SELECT "id", "user_id", "ip", "created_at" FROM visits
-    WHERE id > 0 AND id <= 10000 AND created_at >= '2016-08-01'::date AND created_at < '2016-11-01'::date
+INSERT INTO "public"."visits_intermediate" ("id", "user_id", "ip", "created_at")
+    SELECT "id", "user_id", "ip", "created_at" FROM "public"."visits"
+    WHERE "id" > 0 AND "id" <= 10000 AND "created_at" >= '2018-08-01'::date AND "created_at" < '2018-11-01'::date
 
 /* 2 of 3 */
-INSERT INTO visits_intermediate ("id", "user_id", "ip", "created_at")
-    SELECT "id", "user_id", "ip", "created_at" FROM visits
-    WHERE id > 10000 AND id <= 20000 AND created_at >= '2016-08-01'::date AND created_at < '2016-11-01'::date
+INSERT INTO "public"."visits_intermediate" ("id", "user_id", "ip", "created_at")
+    SELECT "id", "user_id", "ip", "created_at" FROM "public"."visits"
+    WHERE "id" > 10000 AND "id" <= 20000 AND "created_at" >= '2018-08-01'::date AND "created_at" < '2018-11-01'::date
 
 /* 3 of 3 */
-INSERT INTO visits_intermediate ("id", "user_id", "ip", "created_at")
-    SELECT "id", "user_id", "ip", "created_at" FROM visits
-    WHERE id > 20000 AND id <= 30000 AND created_at >= '2016-08-01'::date AND created_at < '2016-11-01'::date
+INSERT INTO "public"."visits_intermediate" ("id", "user_id", "ip", "created_at")
+    SELECT "id", "user_id", "ip", "created_at" FROM "public"."visits"
+    WHERE "id" > 20000 AND "id" <= 30000 AND "created_at" >= '2018-08-01'::date AND "created_at" < '2018-11-01'::date
 ```
 
 ```sh
@@ -226,13 +149,13 @@ pgslice analyze visits
 ```
 
 ```sql
-ANALYZE VERBOSE visits_201608;
+ANALYZE VERBOSE "public"."visits_201808";
 
-ANALYZE VERBOSE visits_201609;
+ANALYZE VERBOSE "public"."visits_201809";
 
-ANALYZE VERBOSE visits_201610;
+ANALYZE VERBOSE "public"."visits_201810";
 
-ANALYZE VERBOSE visits_intermediate;
+ANALYZE VERBOSE "public"."visits_intermediate";
 ```
 
 ```sh
@@ -244,11 +167,11 @@ BEGIN;
 
 SET LOCAL lock_timeout = '5s';
 
-ALTER TABLE visits RENAME TO visits_retired;
+ALTER TABLE "public"."visits" RENAME TO "visits_retired";
 
-ALTER TABLE visits_intermediate RENAME TO visits;
+ALTER TABLE "public"."visits_intermediate" RENAME TO "visits";
 
-ALTER SEQUENCE visits_id_seq OWNED BY visits.id;
+ALTER SEQUENCE "visits_id_seq" OWNED BY "public"."visits"."id";
 
 COMMIT;
 ```
@@ -261,7 +184,7 @@ To add partitions, use:
 pgslice add_partitions <table> --future 3
 ```
 
-Add this as a cron job to create a new partition each day or month.
+Add this as a cron job to create a new partition each day, month, or year.
 
 ```sh
 # day
@@ -269,6 +192,9 @@ Add this as a cron job to create a new partition each day or month.
 
 # month
 0 0 1 * * pgslice add_partitions <table> --future 3 --url ...
+
+# year
+0 0 1 1 * pgslice add_partitions <table> --future 3 --url ...
 ```
 
 Add a monitor to ensure partitions are being created.
@@ -283,21 +209,22 @@ WHERE
     n.nspname = 'public' AND
     c.relname = '<table>_' || to_char(NOW() + INTERVAL '3 days', 'YYYYMMDD')
     -- for months, use to_char(NOW() + INTERVAL '3 months', 'YYYYMM')
+    -- for years, use to_char(NOW() + INTERVAL '3 years', 'YYYY')
 ```
 
 ## Archiving Partitions
 
-Back up and drop older partitions each day or month.
+Back up and drop older partitions each day, month, or year.
 
 ```sh
-pg_dump -c -Fc -t <table>_201609 $PGSLICE_URL > <table>_201609.dump
-psql -c "DROP <table>_201609" $PGSLICE_URL
+pg_dump -c -Fc -t <table>_201809 $PGSLICE_URL > <table>_201809.dump
+psql -c "DROP TABLE <table>_201809" $PGSLICE_URL
 ```
 
 If you use [Amazon S3](https://aws.amazon.com/s3/) for backups, [s3cmd](https://github.com/s3tools/s3cmd) is a nice tool.
 
 ```sh
-s3cmd put <table>_201609.dump s3://<s3-bucket>/<table>_201609.dump
+s3cmd put <table>_201809.dump s3://<s3-bucket>/<table>_201809.dump
 ```
 
 ## Additional Commands
+On Mac, you must use [Docker for Mac](https://www.docker.com/docker-mac) for the port mapping to localhost to work.
 
 This set up allows you to read and write with the original table name with no knowledge it’s partitioned. However, there are a few things to be aware of.
 
-### Writes
-
-Before Postgres 10, if you use `INSERT` statements with a `RETURNING` clause (as frameworks like Rails do), you’ll no longer receive the id of the newly inserted record(s) back. If you need this, you can either:
-
-1. Insert directly into the partition
-2. Get value before the insert with `SELECT nextval('sequence_name')` (for multiple rows, append `FROM generate_series(1, n)`)
-
 ### Reads
 
-When possible, queries should include the column you partition on to limit the number of partitions the database needs to check.  For instance, if you partition on `created_at`, try to include it in queries:
+When possible, queries should include the column you partition on to limit the number of partitions the database needs to check. For instance, if you partition on `created_at`, try to include it in queries:
 
 ```sql
 SELECT * FROM
@@ -335,7 +255,7 @@ SELECT * FROM
 WHERE
     user_id = 123 AND
     -- for performance
-    created_at >= '2016-09-01' AND created_at < '2016-09-02'
+    created_at >= '2018-09-01' AND created_at < '2018-09-02'
 ```
 
 For this to be effective, ensure `constraint_exclusion` is set to `partition` (default value) or `on`.
@@ -344,6 +264,33 @@ For this to be effective, ensure `constraint_exclusion` is set to `partition` (d
 SHOW constraint_exclusion;
 ```
 
+### Writes
+
+Before Postgres 10, if you use `INSERT` statements with a `RETURNING` clause (as frameworks like Rails do), you’ll no longer receive the id of the newly inserted record(s) back. If you need this, you can either:
+
+1. Insert directly into the partition
+2. Get value before the insert with `SELECT nextval('sequence_name')` (for multiple rows, append `FROM generate_series(1, n)`)
+
+## Rails
+
+For Postgres 10+, specify the primary key for partitioned models to ensure it’s returned.
+
+```ruby
+class Visit < ApplicationRecord
+  self.primary_key = "id"
+end
+```
+
+Before Postgres 10, preload the value.
+
+```ruby
+class Visit < ApplicationRecord
+  before_create do
+    self.id ||= self.class.connection.select_all("SELECT nextval('#{self.class.sequence_name}')").first["nextval"]
+  end
+end
+```
+
 ## One Off Tasks
 
 You can also use pgslice to reduce the size of a table without partitioning by creating a new table, filling it with a subset of records, and swapping it in.
+On Mac, you must use [Docker for Mac](https://www.docker.com/docker-mac) for the port mapping to localhost to work.
 
 Once a table is partitioned, here’s how to change the schema:
 
-- To add, remove, or modify a column, make the update on the master table only
-- To add or remove an index, make the update on the master table and all partitions
+To add, remove, or modify a column, make the update on the master table only.
+
+To add or remove an index or foreign key:
+
+- For Postgres 11+, make the update on the master table only.
+- For Postgres 10, make the update on partitions only.
+- For Postgres < 10, make the update on the master table and all partitions.
 
 ## Declarative Partitioning
 
 Postgres 10 introduces [declarative partitioning](https://www.postgresql.org/docs/10/static/ddl-partitioning.html#ddl-partitioning-declarative). A major benefit is `INSERT` statements with a `RETURNING` clause work as expected. If you prefer to use trigger-based partitioning instead (not recommended), pass the `--trigger-based` option to the `prep` command.
 
+## Data Protection
+
+Always make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust. Your best option is to connect over SSH or a VPN. Another option is to use `sslmode=verify-full`. If you don’t do this, your database credentials can be compromised.
+
+## Dependencies
+
+If installation fails, your system may be missing Ruby or libpq.
+
+On Mac, run:
+
+```sh
+brew install postgresql
+```
+
+On Ubuntu, run:
+
+```sh
+sudo apt-get install ruby-dev libpq-dev build-essential
+```
+
 ## Upgrading
 
 Run:
@@ -394,16 +366,11 @@ This will give you the `pgslice` command.
 - [PostgreSQL Manual](https://www.postgresql.org/docs/current/static/ddl-partitioning.html)
 - [PostgreSQL Wiki](https://wiki.postgresql.org/wiki/Table_partitioning)
 
-## TODO
-
-- Command to sync index changes with partitions
-- Disable indexing for faster `fill`
-- ETA for `fill`
-
 ## Related Projects
 
 Also check out:
 
+- [Dexter](https://github.com/ankane/dexter) - The automatic indexer for Postgres
 - [PgHero](https://github.com/ankane/pghero) - A performance dashboard for Postgres
 - [pgsync](https://github.com/ankane/pgsync) - Sync Postgres data to your local machine
 
@@ -416,7 +383,7 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 - Write, clarify, or fix documentation
 - Suggest or add new features
 
-To run tests, do:
+To get started with development and testing:
 
 ```sh
 git clone https://github.com/ankane/pgslice.git
@@ -425,3 +392,12 @@ bundle install
 createdb pgslice_test
 bundle exec rake
 ```
+
+To test against different versions of Postgres with Docker, use:
+
+```sh
+docker run -p=8000:5432 postgres:10
+TZ=Etc/UTC PGSLICE_URL=postgres://postgres@localhost:8000/postgres bundle exec rake
+```
+
+On Mac, you must use [Docker for Mac](https://www.docker.com/docker-mac) for the port mapping to localhost to work.