strong_migrations 0.8.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1f885cc0a8b0fb30e3bc4d000e82584fdf3850943563b4b426d3cfc39d70015c
-  data.tar.gz: d0dd29ea45ccc3b7c571e8bf03fa98c401d5489d1f5459488a0eff411cf6880e
+  metadata.gz: a22e260ec0e3e09954c65535725a3ddc4439b9cf94182a21cf5963e12d8485fe
+  data.tar.gz: 865633da561d77e615df16a9a00f8524a95bf73edc5590e3932855aaf544d4f7
 SHA512:
-  metadata.gz: c03a023d4c593d2868e524ca0f3e7ebd93e07e087e02f0d7090210b00fdb1fdc50d2abb74ab7df4131bddc6052fc5b9c10ce902ede8c708927476e40ad49d07f
-  data.tar.gz: 479e34db0101a15e1db1a593f01773da5665baa5cda3403ad32ea58dca00a0d0faee0ef813cf1975b7e9f99f1591e7f4f2be5c456c96fd6ae4633b48b6a0b3b4
+  metadata.gz: f9b4df7a67aae2c8e1f7c58327b0e7e298261a33d9810dff4b5634bee3e000f1b26e64ad7b3997c8fc6c1cfef3a5cbeabe35ecbe508ed902679e3c9a720ecfb8
+  data.tar.gz: f4a7ecb6e64c40d1be1025a978387b94b9c2e47897cc25568c32dc32043292bc8fe14c9e6471108a4d84a1b1704459707f4fcf8a25227b7a86d8bcf2237dc19d

data/CHANGELOG.md

@@ -1,3 +1,32 @@
+## 1.0.0 (2022-03-21)
+
+New safe operations with MySQL and MariaDB
+
+- Setting `NOT NULL` on an existing column with strict mode enabled
+
+New safe operations with Postgres
+
+- Changing between `text` and `citext` when not indexed
+- Changing a `string` column to a `citext` column when not indexed
+- Changing a `citext` column to a `string` column with no `:limit` when not indexed
+- Changing a `cidr` column to an `inet` column
+- Increasing `:precision` of an `interval` or `time` column
+
+New unsafe operations with Postgres
+
+- Adding a column with a callable default value
+- Decreasing `:precision` of a `datetime` column
+- Decreasing `:limit` of a `timestamptz` column
+- Passing a default value to `change_column_null`
+
+Other
+
+- Added experimental support for lock timeout retries
+- Added `target_sql_mode` option
+- Added error for `change_column_null` with default value with `safe_by_default` option
+- Fixed instructions for `remove_columns` with options
+- Dropped support for Postgres < 10, MySQL < 5.7, and MariaDB < 10.2
+
 ## 0.8.0 (2022-02-09)
 
 - Fixed error with versioned schema with Active Record 7.0.2+

data/README.md

@@ -4,7 +4,7 @@ Catch unsafe migrations in development
 
 &nbsp;&nbsp;✓&nbsp;&nbsp;Detects potentially dangerous operations<br />&nbsp;&nbsp;✓&nbsp;&nbsp;Prevents them from running by default<br />&nbsp;&nbsp;✓&nbsp;&nbsp;Provides instructions on safer ways to do what you want
 
-Supports for PostgreSQL, MySQL, and MariaDB
+Supports PostgreSQL, MySQL, and MariaDB
 
 :tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
 
@@ -67,7 +67,6 @@ Potentially dangerous operations:
 - [renaming a table](#renaming-a-table)
 - [creating a table with the force option](#creating-a-table-with-the-force-option)
 - [adding a check constraint](#adding-a-check-constraint)
-- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
 - [executing SQL directly](#executing-SQL-directly)
 
 Postgres-specific checks:
@@ -76,6 +75,7 @@ Postgres-specific checks:
 - [adding a reference](#adding-a-reference)
 - [adding a foreign key](#adding-a-foreign-key)
 - [adding a json column](#adding-a-json-column)
+- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
 
 Best practices:
 
@@ -204,19 +204,26 @@ class ChangeSomeColumnType < ActiveRecord::Migration[7.0]
 end
 ```
 
-A few changes don’t require a table rewrite (and are safe) in Postgres:
+Some changes don’t require a table rewrite and are safe in Postgres:
 
-- Increasing the length limit of a `varchar` column (or removing the limit)
-- Changing a `varchar` column to a `text` column
-- Changing a `text` column to a `varchar` column with no length limit
-- Increasing the precision of a `decimal` or `numeric` column
-- Making a `decimal` or `numeric` column unconstrained
-- Changing between `timestamp` and `timestamptz` columns when session time zone is UTC in Postgres 12+
+Type | Safe Changes
+--- | ---
+`cidr` | Changing to `inet`
+`citext` | Changing to `text` if not indexed, changing to `string` with no `:limit` if not indexed
+`datetime` | Increasing or removing `:precision`, changing to `timestamptz` when session time zone is UTC in Postgres 12+
+`decimal` | Increasing `:precision` at same `:scale`, removing `:precision` and `:scale`
+`interval` | Increasing or removing `:precision`
+`numeric` | Increasing `:precision` at same `:scale`, removing `:precision` and `:scale`
+`string` | Increasing or removing `:limit`, changing to `text`, changing `citext` if not indexed
+`text` | Changing to `string` with no `:limit`, changing to `citext` if not indexed
+`time` | Increasing or removing `:precision`
+`timestamptz` | Increasing or removing `:limit`, changing to `datetime` when session time zone is UTC in Postgres 12+
 
-And a few in MySQL and MariaDB:
+And some in MySQL and MariaDB:
 
-- Increasing the length limit of a `varchar` column from under 255 up to 255
-- Increasing the length limit of a `varchar` column from over 255 to the max limit
+Type | Safe Changes
+--- | ---
+`string` | Increasing `:limit` from under 255 up to 255, increasing `:limit` from over 255 to the max
 
 #### Good
 
@@ -353,86 +360,6 @@ end
 
 [Let us know](https://github.com/ankane/strong_migrations/issues/new) if you have a safe way to do this (check constraints can be added with `NOT ENFORCED`, but enforcing blocks writes).
 
-### Setting NOT NULL on an existing column
-
-:turtle: Safe by default available
-
-#### Bad
-
-Setting `NOT NULL` on an existing column blocks reads and writes while every row is checked.
-
-```ruby
-class SetSomeColumnNotNull < ActiveRecord::Migration[7.0]
-  def change
-    change_column_null :users, :some_column, false
-  end
-end
-```
-
-#### Good - Postgres
-
-Instead, add a check constraint.
-
-For Rails 6.1, use:
-
-```ruby
-class SetSomeColumnNotNull < ActiveRecord::Migration[7.0]
-  def change
-    add_check_constraint :users, "some_column IS NOT NULL", name: "users_some_column_null", validate: false
-  end
-end
-```
-
-For Rails < 6.1, use:
-
-```ruby
-class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
-  def change
-    safety_assured do
-      execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
-    end
-  end
-end
-```
-
-Then validate it in a separate migration. A `NOT NULL` check constraint is [functionally equivalent](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c) to setting `NOT NULL` on the column (but it won’t show up in `schema.rb` in Rails < 6.1). In Postgres 12+, once the check constraint is validated, you can safely set `NOT NULL` on the column and drop the check constraint.
-
-For Rails 6.1, use:
-
-```ruby
-class ValidateSomeColumnNotNull < ActiveRecord::Migration[7.0]
-  def change
-    validate_check_constraint :users, name: "users_some_column_null"
-
-    # in Postgres 12+, you can then safely set NOT NULL on the column
-    change_column_null :users, :some_column, false
-    remove_check_constraint :users, name: "users_some_column_null"
-  end
-end
-```
-
-For Rails < 6.1, use:
-
-```ruby
-class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
-  def change
-    safety_assured do
-      execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
-    end
-
-    # in Postgres 12+, you can then safely set NOT NULL on the column
-    change_column_null :users, :some_column, false
-    safety_assured do
-      execute 'ALTER TABLE "users" DROP CONSTRAINT "users_some_column_null"'
-    end
-  end
-end
-```
-
-#### Good - MySQL and MariaDB
-
-[Let us know](https://github.com/ankane/strong_migrations/issues/new) if you have a safe way to do this.
-
 ### Executing SQL directly
 
 Strong Migrations can’t ensure safety for raw SQL statements. Make really sure that what you’re doing is safe, then use:
@@ -587,6 +514,82 @@ class AddPropertiesToUsers < ActiveRecord::Migration[7.0]
 end
 ```
 
+### Setting NOT NULL on an existing column
+
+:turtle: Safe by default available
+
+#### Bad
+
+In Postgres, setting `NOT NULL` on an existing column blocks reads and writes while every row is checked.
+
+```ruby
+class SetSomeColumnNotNull < ActiveRecord::Migration[7.0]
+  def change
+    change_column_null :users, :some_column, false
+  end
+end
+```
+
+#### Good
+
+Instead, add a check constraint.
+
+For Rails 6.1, use:
+
+```ruby
+class SetSomeColumnNotNull < ActiveRecord::Migration[7.0]
+  def change
+    add_check_constraint :users, "some_column IS NOT NULL", name: "users_some_column_null", validate: false
+  end
+end
+```
+
+For Rails < 6.1, use:
+
+```ruby
+class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
+  def change
+    safety_assured do
+      execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
+    end
+  end
+end
+```
+
+Then validate it in a separate migration. A `NOT NULL` check constraint is [functionally equivalent](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c) to setting `NOT NULL` on the column (but it won’t show up in `schema.rb` in Rails < 6.1). In Postgres 12+, once the check constraint is validated, you can safely set `NOT NULL` on the column and drop the check constraint.
+
+For Rails 6.1, use:
+
+```ruby
+class ValidateSomeColumnNotNull < ActiveRecord::Migration[7.0]
+  def change
+    validate_check_constraint :users, name: "users_some_column_null"
+
+    # in Postgres 12+, you can then safely set NOT NULL on the column
+    change_column_null :users, :some_column, false
+    remove_check_constraint :users, name: "users_some_column_null"
+  end
+end
+```
+
+For Rails < 6.1, use:
+
+```ruby
+class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
+  def change
+    safety_assured do
+      execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
+    end
+
+    # in Postgres 12+, you can then safely set NOT NULL on the column
+    change_column_null :users, :some_column, false
+    safety_assured do
+      execute 'ALTER TABLE "users" DROP CONSTRAINT "users_some_column_null"'
+    end
+  end
+end
+```
+
 ### Keeping non-unique indexes to three columns or less
 
 #### Bad
@@ -696,7 +699,7 @@ To customize specific messages, create an initializer with:
 StrongMigrations.error_messages[:add_column_default] = "Your custom instructions"
 ```
 
-Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
+Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations/error_messages.rb) for the list of keys.
 
 ## Migration Timeouts
 
@@ -718,6 +721,25 @@ ALTER ROLE myuser SET statement_timeout = '1h';
 
 Note: If you use PgBouncer in transaction mode, you must set timeouts on the database user.
 
+## Lock Timeout Retries [experimental]
+
+There’s the option to automatically retry statements when the lock timeout is reached. Here’s how it works:
+
+- If a lock timeout happens outside a transaction, the statement is retried
+- If it happens inside the DDL transaction, the entire migration is retried (only applicable to Postgres)
+
+Add to `config/initializers/strong_migrations.rb`:
+
+```ruby
+StrongMigrations.lock_timeout_retries = 3
+```
+
+Set the delay between retries with:
+
+```ruby
+StrongMigrations.lock_timeout_retry_delay = 10.seconds
+```
+
 ## App Timeouts
 
 We recommend adding timeouts to `config/database.yml` to prevent connections from hanging and individual queries from taking up too many resources in controllers, jobs, the Rails console, and other places.

data/lib/strong_migrations/adapters/abstract_adapter.rb

@@ -0,0 +1,61 @@
+module StrongMigrations
+  module Adapters
+    class AbstractAdapter
+      def initialize(checker)
+        @checker = checker
+      end
+
+      def name
+        "Unknown"
+      end
+
+      def min_version
+      end
+
+      def set_statement_timeout(timeout)
+        raise StrongMigrations::Error, "Statement timeout not supported for this database"
+      end
+
+      def set_lock_timeout(timeout)
+        raise StrongMigrations::Error, "Lock timeout not supported for this database"
+      end
+
+      def check_lock_timeout(limit)
+        # do nothing
+      end
+
+      def add_column_default_safe?
+        false
+      end
+
+      def change_type_safe?(table, column, type, options, existing_column, existing_type)
+        false
+      end
+
+      def rewrite_blocks
+        "reads and writes"
+      end
+
+      private
+
+      def connection
+        @checker.send(:connection)
+      end
+
+      def select_all(statement)
+        connection.select_all(statement)
+      end
+
+      def target_version(target_version)
+        target_version ||= StrongMigrations.target_version
+        version =
+          if target_version && StrongMigrations.developer_env?
+            target_version.to_s
+          else
+            yield
+          end
+        Gem::Version.new(version)
+      end
+    end
+  end
+end

data/lib/strong_migrations/adapters/mariadb_adapter.rb

@@ -0,0 +1,29 @@
+module StrongMigrations
+  module Adapters
+    class MariaDBAdapter < MySQLAdapter
+      def name
+        "MariaDB"
+      end
+
+      def min_version
+        "10.2"
+      end
+
+      def server_version
+        @server_version ||= begin
+          target_version(StrongMigrations.target_mariadb_version) do
+            select_all("SELECT VERSION()").first["VERSION()"].split("-").first
+          end
+        end
+      end
+
+      def set_statement_timeout(timeout)
+        select_all("SET max_statement_time = #{connection.quote(timeout)}")
+      end
+
+      def add_column_default_safe?
+        server_version >= Gem::Version.new("10.3.2")
+      end
+    end
+  end
+end

data/lib/strong_migrations/adapters/mysql_adapter.rb

@@ -0,0 +1,87 @@
+# note: MariaDB inherits from this adapter
+# when making changes, be sure to see how it affects it
+module StrongMigrations
+  module Adapters
+    class MySQLAdapter < AbstractAdapter
+      def name
+        "MySQL"
+      end
+
+      def min_version
+        "5.7"
+      end
+
+      def server_version
+        @server_version ||= begin
+          target_version(StrongMigrations.target_mysql_version) do
+            select_all("SELECT VERSION()").first["VERSION()"].split("-").first
+          end
+        end
+      end
+
+      def set_statement_timeout(timeout)
+        # use ceil to prevent no timeout for values under 1 ms
+        select_all("SET max_execution_time = #{connection.quote((timeout.to_f * 1000).ceil)}")
+      end
+
+      def set_lock_timeout(timeout)
+        select_all("SET lock_wait_timeout = #{connection.quote(timeout)}")
+      end
+
+      def check_lock_timeout(limit)
+        lock_timeout = connection.select_all("SHOW VARIABLES LIKE 'lock_wait_timeout'").first["Value"]
+        # lock timeout is an integer
+        if lock_timeout.to_i > limit
+          warn "[strong_migrations] DANGER: Lock timeout is longer than #{limit} seconds: #{lock_timeout}"
+        end
+      end
+
+      def analyze_table(table)
+        connection.execute "ANALYZE TABLE #{connection.quote_table_name(table.to_s)}"
+      end
+
+      def add_column_default_safe?
+        server_version >= Gem::Version.new("8.0.12")
+      end
+
+      def change_type_safe?(table, column, type, options, existing_column, existing_type)
+        safe = false
+
+        case type.to_s
+        when "string"
+          # https://dev.mysql.com/doc/refman/5.7/en/innodb-online-ddl-operations.html
+          # https://mariadb.com/kb/en/innodb-online-ddl-operations-with-the-instant-alter-algorithm/#changing-the-data-type-of-a-column
+          # increased limit, but doesn't change number of length bytes
+          # 1-255 = 1 byte, 256-65532 = 2 bytes, 65533+ = too big for varchar
+          limit = options[:limit] || 255
+          safe = ["varchar"].include?(existing_type) &&
+            limit >= existing_column.limit &&
+            (limit <= 255 || existing_column.limit > 255)
+        end
+
+        safe
+      end
+
+      def strict_mode?
+        sql_modes = sql_modes()
+        sql_modes.include?("STRICT_ALL_TABLES") || sql_modes.include?("STRICT_TRANS_TABLES")
+      end
+
+      def rewrite_blocks
+        "writes"
+      end
+
+      private
+
+      # do not memoize
+      # want latest value
+      def sql_modes
+        if StrongMigrations.target_sql_mode && StrongMigrations.developer_env?
+          StrongMigrations.target_sql_mode.split(",")
+        else
+          select_all("SELECT @@SESSION.sql_mode").first["@@SESSION.sql_mode"].split(",")
+        end
+      end
+    end
+  end
+end