strong_migrations 0.6.5 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bad0ba9c075a7c3eae98fc8835b9e56863b3573a381371d3b1affde769da0ad6
-  data.tar.gz: 7d67dbdaf674414b43781dd45b60f1f0d11a6d811ffe9ce8235ffed2889d8383
+  metadata.gz: 947dda0ab2fa336611335b258b4eae34e7b60f38083e162fb6bf80d2237ce819
+  data.tar.gz: 6744411d6b2a33d5ee54b910ef60652fcb992cbb700a59a31da3eb48c2cb5322
 SHA512:
-  metadata.gz: dc889f4d62bba08ae0d628e629dc37d7ecd53300ae92691118ca3db0707a416e7f043bbde41f2a0311acb9bd0f0b973175413f17a7c8aed2e134a6f29fc59b61
-  data.tar.gz: b08c078f6591b4458fadf64c9d1b8e04c3323c8dc9036e88e4e5a7ddc4f8a38bfa39d8e3e0cce2dcda7b9b9fce2d9676140d0cf89b2d668f6e5f2389e825b225
+  metadata.gz: 3e1b51f910a02835ab0612f8be9714dce1dd1024acc3745a4465588447e6e12d8a4bdb2903c66f9c5f8ccc67bb6877ec231ad5979b304aff0ee34e5e4bd1f0a2
+  data.tar.gz: e739903401b7787ad66a929ff1f67b0e0b4ff9366571d25ad0349f680bf013adad95be38a1fc71c4f4039174b4ef7ee30764a9786e6b8677597a0d2bd6434749

data/CHANGELOG.md

@@ -1,3 +1,31 @@
+## 0.7.1 (2020-07-27)
+
+- Added `target_version` option to replace database-specific options
+
+## 0.7.0 (2020-07-22)
+
+- Added `check_down` option
+- Added check for `change_column` with `null: false`
+- Added check for `validate_foreign_key`
+- Improved error messages
+- Made auto analyze less verbose in Postgres
+- Decreasing the length limit of a `varchar` column or adding a limit is not safe in Postgres
+- Removed safety checks for `db` rake tasks (Rails 5+ handles this)
+
+## 0.6.8 (2020-05-13)
+
+- `change_column_null` on a column with a `NOT NULL` constraint is safe in Postgres 12+
+
+## 0.6.7 (2020-05-13)
+
+- Improved comments in initializer
+- Fixed string timeouts for Postgres
+
+## 0.6.6 (2020-05-08)
+
+- Added warnings for missing and long lock timeouts
+- Added install generator
+
 ## 0.6.5 (2020-05-06)
 
 - Fixed deprecation warnings with Ruby 2.7

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2013 Bob Remeika and David Waller, 2015-2019 Andrew Kane
+Copyright (c) 2013 Bob Remeika and David Waller, 2015-2020 Andrew Kane
 
 MIT License
 

data/README.md

@@ -18,7 +18,42 @@ Add this line to your application’s Gemfile:
 gem 'strong_migrations'
 ```
 
-We highly recommend [setting timeouts](#timeouts). You can [mark existing migrations as safe](#existing-migrations) as well.
+And run:
+
+```sh
+bundle install
+rails generate strong_migrations:install
+```
+
+Strong Migrations sets a long statement timeout for migrations so you can set a [short statement timeout](#app-timeouts) for your application.
+
+## How It Works
+
+When you run a migration that’s potentially dangerous, you’ll see an error message like:
+
+```txt
+=== Dangerous operation detected #strong_migrations ===
+
+Active Record caches attributes, which causes problems
+when removing columns. Be sure to ignore the column:
+
+class User < ApplicationRecord
+  self.ignored_columns = ["name"]
+end
+
+Deploy the code, then wrap this step in a safety_assured { ... } block.
+
+class RemoveColumn < ActiveRecord::Migration[6.0]
+  def change
+    safety_assured { remove_column :users, :name }
+  end
+end
+```
+
+An operation is classified as dangerous if it either:
+
+- Blocks reads or writes for more than a few seconds (after a lock is acquired)
+- Has a good chance of causing application errors
 
 ## Checks
 
@@ -27,21 +62,19 @@ Potentially dangerous operations:
 - [removing a column](#removing-a-column)
 - [adding a column with a default value](#adding-a-column-with-a-default-value)
 - [backfilling data](#backfilling-data)
-- [changing the type of a column](#renaming-or-changing-the-type-of-a-column)
-- [renaming a column](#renaming-or-changing-the-type-of-a-column)
+- [changing the type of a column](#changing-the-type-of-a-column)
+- [renaming a column](#renaming-a-column)
 - [renaming a table](#renaming-a-table)
 - [creating a table with the force option](#creating-a-table-with-the-force-option)
-- [using change_column_null with a default value](#using-change_column_null-with-a-default-value)
+- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
 - [executing SQL directly](#executing-SQL-directly)
 
 Postgres-specific checks:
 
-- [adding an index non-concurrently](#adding-an-index)
-- [removing an index non-concurrently](#removing-an-index)
+- [adding an index non-concurrently](#adding-an-index-non-concurrently)
 - [adding a reference](#adding-a-reference)
 - [adding a foreign key](#adding-a-foreign-key)
 - [adding a json column](#adding-a-json-column)
-- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
 
 Best practices:
 
@@ -53,7 +86,7 @@ You can also add [custom checks](#custom-checks) or [disable specific checks](#d
 
 #### Bad
 
-ActiveRecord caches database columns at runtime, so if you drop a column, it can cause exceptions until your app reboots.
+Active Record caches database columns at runtime, so if you drop a column, it can cause exceptions until your app reboots.
 
 ```ruby
 class RemoveSomeColumnFromUsers < ActiveRecord::Migration[6.0]
@@ -65,7 +98,7 @@ end
 
 #### Good
 
-1. Tell ActiveRecord to ignore the column from its cache
+1. Tell Active Record to ignore the column from its cache
 
   ```ruby
   class User < ApplicationRecord
@@ -88,11 +121,9 @@ end
 
 ### Adding a column with a default value
 
-Note: This operation is safe in Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+.
-
 #### Bad
 
-Adding a column with a default value to an existing table causes the entire table to be rewritten.
+In earlier versions of Postgres, MySQL, and MariaDB, adding a column with a default value to an existing table causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
 
 ```ruby
 class AddSomeColumnToUsers < ActiveRecord::Migration[6.0]
@@ -102,6 +133,8 @@ class AddSomeColumnToUsers < ActiveRecord::Migration[6.0]
 end
 ```
 
+In Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+, this no longer requires a table rewrite and is safe.
+
 #### Good
 
 Instead, add the column without a default value, then change the default.
@@ -125,7 +158,7 @@ See the next section for how to backfill.
 
 #### Bad
 
-Backfilling in the same transaction that alters a table locks the table for the [duration of the backfill](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
+Active Record creates a transaction around each migration, and backfilling in the same transaction that alters a table keeps the table locked for the [duration of the backfill](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
 
 ```ruby
 class AddSomeColumnToUsers < ActiveRecord::Migration[6.0]
@@ -155,19 +188,11 @@ class BackfillSomeColumn < ActiveRecord::Migration[6.0]
 end
 ```
 
-### Renaming or changing the type of a column
+### Changing the type of a column
 
 #### Bad
 
-```ruby
-class RenameSomeColumn < ActiveRecord::Migration[6.0]
-  def change
-    rename_column :users, :some_column, :new_name
-  end
-end
-```
-
-or
+Changing the type of a column causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
 
 ```ruby
 class ChangeSomeColumnType < ActiveRecord::Migration[6.0]
@@ -177,17 +202,44 @@ class ChangeSomeColumnType < ActiveRecord::Migration[6.0]
 end
 ```
 
-A few changes are safe in Postgres:
+A few changes don’t require a table rewrite (and are safe) in Postgres:
 
-- Changing between `varchar` and `text` columns
+- Increasing the length limit of a `varchar` column (or removing the limit)
+- Changing a `varchar` column to a `text` column
+- Changing a `text` column to a `varchar` column with no length limit
 - Increasing the precision of a `decimal` or `numeric` column
 - Making a `decimal` or `numeric` column unconstrained
 - Changing between `timestamp` and `timestamptz` columns when session time zone is UTC in Postgres 12+
 
 And a few in MySQL and MariaDB:
 
-- Increasing the length of a `varchar` column from under 255 up to 255
-- Increasing the length of a `varchar` column over 255
+- Increasing the length limit of a `varchar` column from under 255 up to 255
+- Increasing the length limit of a `varchar` column from over 255 to the max limit
+
+#### Good
+
+A safer approach is to:
+
+1. Create a new column
+2. Write to both columns
+3. Backfill data from the old column to the new column
+4. Move reads from the old column to the new column
+5. Stop writing to the old column
+6. Drop the old column
+
+### Renaming a column
+
+#### Bad
+
+Renaming a column that’s in use will cause errors in your application.
+
+```ruby
+class RenameSomeColumn < ActiveRecord::Migration[6.0]
+  def change
+    rename_column :users, :some_column, :new_name
+  end
+end
+```
 
 #### Good
 
@@ -204,6 +256,8 @@ A safer approach is to:
 
 #### Bad
 
+Renaming a table that’s in use will cause errors in your application.
+
 ```ruby
 class RenameUsersToCustomers < ActiveRecord::Migration[6.0]
   def change
@@ -253,33 +307,57 @@ class CreateUsers < ActiveRecord::Migration[6.0]
 end
 ```
 
-### Using change_column_null with a default value
+If you intend to drop an existing table, run `drop_table` first.
+
+### Setting NOT NULL on an existing column
 
 #### Bad
 
-This generates a single `UPDATE` statement to set the default value.
+Setting `NOT NULL` on an existing column blocks reads and writes while the every row is checked.
 
 ```ruby
-class ChangeSomeColumnNull < ActiveRecord::Migration[6.0]
+class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
   def change
-    change_column_null :users, :some_column, false, "default_value"
+    change_column_null :users, :some_column, false
   end
 end
 ```
 
-#### Good
+#### Good - Postgres
+
+Instead, add a check constraint:
+
+```ruby
+class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
+  def change
+    safety_assured do
+      execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
+    end
+  end
+end
+```
 
-Backfill the column [safely](#backfilling-data). Then use:
+Then validate it in a separate migration. A `NOT NULL` check constraint is [functionally equivalent](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c) to setting `NOT NULL` on the column, but it won’t show up in `schema.rb`. In Postgres 12+, once the check constraint is validated, you can safely set `NOT NULL` on the column and drop the check constraint.
 
 ```ruby
-class ChangeSomeColumnNull < ActiveRecord::Migration[6.0]
+class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
   def change
+    safety_assured do
+      execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
+    end
+
+    # in Postgres 12+, you can then safely set NOT NULL on the column
     change_column_null :users, :some_column, false
+    safety_assured do
+      execute 'ALTER TABLE "users" DROP CONSTRAINT "users_some_column_null"'
+    end
   end
 end
 ```
 
-Note: In Postgres, `change_column_null` is still [not safe](#setting-not-null-on-an-existing-column) with this method.
+#### Good - MySQL and MariaDB
+
+[Let us know](https://github.com/ankane/strong_migrations/issues/new) if you have a safe way to do this.
 
 ### Executing SQL directly
 
@@ -293,11 +371,11 @@ class ExecuteSQL < ActiveRecord::Migration[6.0]
 end
 ```
 
-### Adding an index
+### Adding an index non-concurrently
 
 #### Bad
 
-In Postgres, adding an index non-concurrently locks the table.
+In Postgres, adding an index non-concurrently blocks writes.
 
 ```ruby
 class AddSomeIndexToUsers < ActiveRecord::Migration[6.0]
@@ -329,41 +407,11 @@ With [gindex](https://github.com/ankane/gindex), you can generate an index migra
 rails g index table column
 ```
 
-### Removing an index
-
-Note: This check is [opt-in](#opt-in-checks).
-
-#### Bad
-
-In Postgres, removing an index non-concurrently locks the table for a brief period.
-
-```ruby
-class RemoveSomeIndexFromUsers < ActiveRecord::Migration[6.0]
-  def change
-    remove_index :users, :some_column
-  end
-end
-```
-
-#### Good
-
-Remove indexes concurrently.
-
-```ruby
-class RemoveSomeIndexFromUsers < ActiveRecord::Migration[6.0]
-  disable_ddl_transaction!
-
-  def change
-    remove_index :users, column: :some_column, algorithm: :concurrently
-  end
-end
-```
-
 ### Adding a reference
 
 #### Bad
 
-Rails adds an index non-concurrently to references by default, which is problematic for Postgres.
+Rails adds an index non-concurrently to references by default, which blocks writes in Postgres.
 
 ```ruby
 class AddReferenceToUsers < ActiveRecord::Migration[6.0]
@@ -391,7 +439,7 @@ end
 
 #### Bad
 
-In Postgres, new foreign keys are validated by default, which acquires a `ShareRowExclusiveLock` that can be [expensive on large tables](https://travisofthenorth.com/blog/2017/2/2/postgres-adding-foreign-keys-with-zero-downtime).
+In Postgres, adding a foreign key blocks writes on both tables.
 
 ```ruby
 class AddForeignKeyOnUsers < ActiveRecord::Migration[6.0]
@@ -413,7 +461,7 @@ end
 
 #### Good
 
-Instead, validate it in a separate migration with a more agreeable `RowShareLock`. This approach is documented by Postgres to have “[the least impact on other work](https://www.postgresql.org/docs/current/sql-altertable.html).”
+Add the foreign key without validating existing rows, then validate them in a separate migration.
 
 For Rails 5.2+, use:
 
@@ -425,7 +473,7 @@ class AddForeignKeyOnUsers < ActiveRecord::Migration[6.0]
 end
 ```
 
-Then validate it in a separate migration.
+Then:
 
 ```ruby
 class ValidateForeignKeyOnUsers < ActiveRecord::Migration[6.0]
@@ -447,7 +495,7 @@ class AddForeignKeyOnUsers < ActiveRecord::Migration[5.1]
 end
 ```
 
-Then validate it in a separate migration.
+Then:
 
 ```ruby
 class ValidateForeignKeyOnUsers < ActiveRecord::Migration[5.1]
@@ -463,7 +511,7 @@ end
 
 #### Bad
 
-In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries.
+In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries in your application.
 
 ```ruby
 class AddPropertiesToUsers < ActiveRecord::Migration[6.0]
@@ -485,48 +533,6 @@ class AddPropertiesToUsers < ActiveRecord::Migration[6.0]
 end
 ```
 
-### Setting NOT NULL on an existing column
-
-#### Bad
-
-In Postgres, setting `NOT NULL` on an existing column requires an `AccessExclusiveLock`, which is expensive on large tables.
-
-```ruby
-class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
-  def change
-    change_column_null :users, :some_column, false
-  end
-end
-```
-
-#### Good
-
-Instead, add a constraint:
-
-```ruby
-class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
-  def change
-    safety_assured do
-      execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
-    end
-  end
-end
-```
-
-Then validate it in a separate migration.
-
-```ruby
-class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
-  def change
-    safety_assured do
-      execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
-    end
-  end
-end
-```
-
-Note: This is not 100% the same as `NOT NULL` column constraint. Here’s a [good explanation](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
-
 ### Keeping non-unique indexes to three columns or less
 
 #### Bad
@@ -587,16 +593,12 @@ Note: Since `remove_column` always requires a `safety_assured` block, it’s not
 
 ## Opt-in Checks
 
-Some operations rarely cause issues in practice, but can be checked if desired. Enable checks with:
-
-```ruby
-StrongMigrations.enable_check(:remove_index)
-```
+### Removing an index non-concurrently
 
-To start a check only after a specific migration, use:
+Postgres supports removing indexes concurrently, but removing them non-concurrently shouldn’t be an issue for most applications. You can enable this check with:
 
 ```ruby
-StrongMigrations.enable_check(:remove_index, start_after: 20170101000000)
+StrongMigrations.enable_check(:remove_index)
 ```
 
 ## Disable Checks
@@ -609,6 +611,14 @@ StrongMigrations.disable_check(:add_index)
 
 Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
 
+## Down Migrations / Rollbacks
+
+By default, checks are disabled when migrating down. Enable them with:
+
+```ruby
+StrongMigrations.check_down = true
+```
+
 ## Custom Messages
 
 To customize specific messages, create an initializer with:
@@ -619,26 +629,69 @@ StrongMigrations.error_messages[:add_column_default] = "Your custom instructions
 
 Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
 
-## Timeouts
+## Migration Timeouts
 
-It’s a good idea to set a long statement timeout and a short lock timeout for migrations. This way, migrations can run for a while, and if a migration can’t acquire a lock in a timely manner, other statements won’t be stuck behind it.
+It’s extremely important to set a short lock timeout for migrations. This way, if a migration can’t acquire a lock in a timely manner, other statements won’t be stuck behind it. We also recommend setting a long statement timeout so migrations can run for a while.
 
 Create `config/initializers/strong_migrations.rb` with:
 
 ```ruby
-StrongMigrations.statement_timeout = 1.hour
 StrongMigrations.lock_timeout = 10.seconds
+StrongMigrations.statement_timeout = 1.hour
 ```
 
 Or set the timeouts directly on the database user that runs migrations. For Postgres, use:
 
 ```sql
-ALTER ROLE myuser SET statement_timeout = '1h';
 ALTER ROLE myuser SET lock_timeout = '10s';
+ALTER ROLE myuser SET statement_timeout = '1h';
 ```
 
 Note: If you use PgBouncer in transaction mode, you must set timeouts on the database user.
 
+## App Timeouts
+
+We recommend adding timeouts to `config/database.yml` to prevent connections from hanging and individual queries from taking up too many resources in controllers, jobs, the Rails console, and other places.
+
+For Postgres:
+
+```yml
+production:
+  connect_timeout: 5
+  variables:
+    statement_timeout: 15s
+    lock_timeout: 10s
+```
+
+Note: If you use PgBouncer in transaction mode, you must set the statement and lock timeouts on the database user as shown above.
+
+For MySQL:
+
+```yml
+production:
+  connect_timeout: 5
+  read_timeout: 5
+  write_timeout: 5
+  variables:
+    max_execution_time: 15000 # ms
+    lock_wait_timeout: 10 # sec
+
+```
+
+For MariaDB:
+
+```yml
+production:
+  connect_timeout: 5
+  read_timeout: 5
+  write_timeout: 5
+  variables:
+    max_statement_time: 15 # sec
+    lock_wait_timeout: 10 # sec
+```
+
+For HTTP connections, Redis, and other services, check out [this guide](https://github.com/ankane/the-ultimate-guide-to-ruby-timeouts).
+
 ## Existing Migrations
 
 To mark migrations as safe that were created before installing this gem, create an initializer with:
@@ -651,14 +704,14 @@ Use the version from your latest migration.
 
 ## Target Version
 
-If your development database version is different from production, you can specify the production version so the right checks are run in development.
+If your development database version is different from production, you can specify the production version so the right checks run in development.
 
 ```ruby
-StrongMigrations.target_postgresql_version = "10"
-StrongMigrations.target_mysql_version = "8.0.12"
-StrongMigrations.target_mariadb_version = "10.3.2"
+StrongMigrations.target_version = 10 # or "8.0.12", "10.3.2", etc
 ```
 
+The major version works well for Postgres, while the full version is recommended for MySQL and MariaDB.
+
 For safety, this option only affects development and test environments. In other environments, the actual server version is always used.
 
 ## Analyze Tables
@@ -686,22 +739,20 @@ Columns can flip order in `db/schema.rb` when you have multiple developers. One
 task "db:schema:dump": "strong_migrations:alphabetize_columns"
 ```
 
-## Dangerous Tasks
-
-For safety, dangerous database tasks are disabled in production - `db:drop`, `db:reset`, `db:schema:load`, and `db:structure:load`. To get around this, use:
-
-```sh
-SAFETY_ASSURED=1 rails db:drop
-```
-
 ## Permissions
 
 We recommend using a [separate database user](https://ankane.org/postgres-users) for migrations when possible so you don’t need to grant your app user permission to alter tables.
 
+## Smaller Projects
+
+You probably don’t need this gem for smaller projects, as operations that are unsafe at scale can be perfectly safe on smaller, low-traffic tables.
+
 ## Additional Reading
 
 - [Rails Migrations with No Downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
 - [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/braintree-product-technology/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
+- [An Overview of DDL Algorithms in MySQL](https://mydbops.wordpress.com/2020/03/04/an-overview-of-ddl-algorithms-in-mysql-covers-mysql-8/)
+- [MariaDB InnoDB Online DDL Overview](https://mariadb.com/kb/en/innodb-online-ddl-overview/)
 
 ## Credits