schema_ferry 0.1.2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a34cb64f18cd33a93ebbc3636bd1e76e170279d98ec5d9c59b8ef14e8f141fd9
-  data.tar.gz: f3ae1fbea45f7b7f5c4d5c79e5704fcc680955a77a9123a73975618c32752b8f
+  metadata.gz: e954e39c0309126d404ec1a7b95cfcaf062cbdf18525e6d179e8747c96d95aaf
+  data.tar.gz: 29c7495b19fb18d7cb9ea1a17fb64b8778dc7deeefeeee50858b0584adc57fe8
 SHA512:
-  metadata.gz: 9fa8d644af36d128f1922258299fc96ba682a068f1195fced8e8446e64067c63e5d5ee81b3541f07e162d0e2588495946e111ca5076da608ab9054491f9a8cdd
-  data.tar.gz: 18dd78144c3e9ccbba232da685b0248eb22be16a73ac55090edfe6638a7d93d8ef794bcf1babad248a95f324b9bf8a3e4bac4e3db19a2b4d735d09dd45a9cffe
+  metadata.gz: 939a4335e6ad17b7eabbde3d4f3979e09009493cef5b474ec033cfbf853f4def56fb505d82aa9ab4366ab6ff6158196aed1f9a4ea762470cc5e3938a148f181e
+  data.tar.gz: d7e983858d24f45938fbda3a8c76d4a0de06d496c949a91320903b0c49b1ff440450570440e7726860e6c2158bbddec080d3a92c78918f62182f84321ad21b42

data/CHANGELOG.md

@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.0] - 2026-07-06
+
+### Removed
+
+- `add_index` — declaring a PostgreSQL-only index existed solely as a replacement path for skipped FULLTEXT indexes, but that undermined the tool's own "faithful mirror of MySQL" design: it was the only way to declare something with no MySQL counterpart. It also had its own bug — an index with a `where:` clause was dropped and re-created on every single run, because PostgreSQL rewrites predicates (operators, casts, parenthesization) when it stores them, so a declared clause can never reliably match what's on the target. A `pg_trgm` GIN index (or similar) is still the way to replace a FULLTEXT index, but only by hand, once you're fully cut over to PostgreSQL.
+
+### Changed
+
+- `FULLTEXT`/`SPATIAL` indexes and spatial columns (`POINT`, `GEOMETRY`, `POLYGON`, `LINESTRING`, …) now raise `ConversionError` instead of printing a warning and silently excluding themselves. A warning is easy to miss in an unattended cron run's output; raising fails the same way any other error does — a non-zero exit status a monitor can catch. Exclude them explicitly with `ignore_index` / `ignore_column`.
+
+### Fixed
+
+- `POINT` columns are now caught and raise `ConversionError`, like other unsupported spatial types (`GEOMETRY`/`POLYGON`/`LINESTRING`/etc.), instead of silently becoming a meaningless `integer NOT NULL` column. ActiveRecord's mysql2 adapter misreports `POINT` as plain `:integer` — the string "point" happens to match an unanchored `/int/i` pattern deep in ActiveRecord's generic type map — unlike the others, which already came through as an unrecognized type and were rejected.
+
+## [0.1.3] - 2026-07-06
+
+### Fixed
+
+- The 0.1.2 fix only covered the two reported cases, not the underlying pattern: any type/option combination that PostgreSQL can't actually reproduce back to ActiveRecord causes `apply!` to re-run `change_column` forever, because ridgepole compares the declaration against a state it can never match.
+  - `DOUBLE` columns (`:float`) are read with `limit: 53` (the type's internal bit width), but a PostgreSQL column never reports a limit back. `limit` is no longer emitted for `:float` columns.
+  - Plain `DECIMAL` columns with a default (not just ones bumped from `BIGINT UNSIGNED`) now render their default as a string, matching the form ActiveRecord's schema dumper uses for decimal defaults — the same fix from 0.1.2, generalized to `TypeMapper` so it applies to every decimal column instead of only the unsigned-bigint conversion path.
+
 ## [0.1.2] - 2026-07-06
 
 ### Fixed

data/README.md

@@ -6,7 +6,7 @@ You're migrating a production MySQL database to PostgreSQL. Moving the data take
 - **Sensible defaults, fully customizable** — built-in type mappings handle most cases; override anything with a few DSL rules
 - **Safe to iterate** — `dry_run` shows the exact changes that would be applied, before touching anything
 
-schema_ferry is designed to run repeatedly (e.g. via cron). Data migration is out of scope — pair it with [pgloader](https://pgloader.io/) (one-shot bulk copy) or CDC replication (AWS DMS, [Debezium](https://debezium.io/), …), which load rows into the tables schema_ferry has created.
+schema_ferry is designed to run repeatedly (e.g. via cron). Data migration is out of scope — pair it with [pgloader](https://github.com/dimitri/pgloader) (one-shot bulk copy) or CDC replication (AWS DMS, [Debezium](https://github.com/debezium/debezium), …), which load rows into the tables schema_ferry keeps in sync.
 
 ## Requirements
 
@@ -43,7 +43,7 @@ pipeline.apply!   # applies the schema to PostgreSQL
 
 There is also `pipeline.schemafile`, which returns the generated schema as a string without connecting to the target.
 
-`apply!` makes the target match the generated schema — including **dropping** columns and indexes that are not part of it. Before running against a target that holds data, read [Destructive changes](#destructive-changes) below.
+`apply!` makes the target match the generated schema — including **dropping** columns and indexes from the target that are not part of it. Before running against a target that holds data, read [Coverage](#coverage) below.
 
 ### CLI
 
@@ -86,12 +86,24 @@ end
 
 The same rules work in a CLI definition file.
 
-### Destructive changes
+### Coverage
 
-`apply!` delegates to [ridgepole](https://github.com/ridgepole/ridgepole), which makes the target match the generated schema. For tables it manages, **columns and indexes that are missing from the generated schema are dropped from the target** (e.g. a column excluded via `ignore_column`, or an index created by hand on the target — declare those with `add_index` instead). Tables absent from the generated schema are themselves left untouched.
+schema_ferry syncs what can be done automatically — exactly where possible, or as an approximation with a warning where it isn't — and leaves the rest to add by hand, later. Where there's no reasonable equivalent at all, it raises instead of guessing. Add the column or index by hand once you're fully cut over to PostgreSQL — not before, since `apply!` delegates to [ridgepole](https://github.com/ridgepole/ridgepole), which drops anything missing from the generated schema on a managed table, including something added by hand as a stand-in — but never drops a table that's absent from it entirely.
 
 Review `dry_run` output before your first `apply!` and whenever you change the conversion rules — those are the moments that introduce drops. Unattended runs in between only mirror changes made to the MySQL schema; if even those need review, schedule `dry-run` instead and apply by hand.
 
+Normalized automatically, with a warning to stderr:
+
+- **Index prefix lengths** (`KEY (col(10))`) are dropped silently — PostgreSQL indexes the full column.
+- **Identifiers over 63 bytes** (MySQL allows 64): index and foreign key names are shortened deterministically (`first 54 bytes + _ + 8-char digest`), so repeated runs stay stable. Overlong table names are only warned about — rename those yourself.
+- **Zero-date defaults** (`'0000-00-00 00:00:00'`) are invalid in PostgreSQL and are dropped.
+- **BIGINT UNSIGNED columns on a foreign key** (either side) become signed `bigint` instead of `numeric(20)` — a numeric column cannot reference a bigint primary key. Values above 2⁶³−1 will not fit, the same trade-off as for `BIGINT UNSIGNED` primary keys.
+
+Raises instead:
+
+- **FULLTEXT indexes** — PostgreSQL has no equivalent construct (a `pg_trgm` GIN index is a common approximation, but it's not the same search semantics, so schema_ferry doesn't create one for you). Because of the drop behavior above, you can't pre-create a replacement during the sync period — add one once you're fully cut over to PostgreSQL. `ignore_index` them.
+- **Spatial columns** (`POINT`, `GEOMETRY`, `POLYGON`, `LINESTRING`, …) — PostgreSQL has no built-in equivalent without PostGIS, which schema_ferry does not manage. `ignore_column` them.
+
 ## DSL reference
 
 ### Top-level
@@ -113,7 +125,6 @@ Review `dry_run` output before your first `apply!` and whenever you change the c
 | `map_column :col, type: :type, default: value` | …and give it an explicit default |
 | `ignore_column :col` | Exclude a column |
 | `ignore_index :index_name` | Exclude an index |
-| `add_index :col, ...options` | Declare a PostgreSQL-only index (options: `name`, `unique`, `using`, `opclass`, `where`, `order`) |
 
 Ignoring a column also drops indexes and foreign keys that reference it. Renaming tables or columns is out of scope — clean up names after the cutover with a regular migration.
 
@@ -128,7 +139,7 @@ Ignoring a column also drops indexes and foreign keys that reference it. Renamin
 | `TINYINT(1)` | `boolean` | see the caveat above if a column holds more than 0/1 |
 | `TINYINT`…`BIGINT` (signed) | `smallint` / `integer` / `bigint` | widths normalized to PostgreSQL's three integer sizes |
 | `TINYINT`…`INT` `UNSIGNED` | one size larger | e.g. `INT UNSIGNED` → `bigint` |
-| `BIGINT UNSIGNED` | `numeric(20)` | PostgreSQL has no unsigned 8-byte integer; emitted with a warning. Columns on a foreign key become signed `bigint` instead — see below |
+| `BIGINT UNSIGNED` | `numeric(20)` | PostgreSQL has no unsigned 8-byte integer; emitted with a warning. Columns on a foreign key become signed `bigint` instead — see above |
 | `FLOAT` / `DOUBLE` | `double precision` | |
 | `DECIMAL(p,s)` | `numeric(p,s)` | |
 | `DATETIME` / `TIMESTAMP` | `timestamp` | use `map_type :datetime, to: :timestamptz` for `timestamptz` |
@@ -139,16 +150,6 @@ Ignoring a column also drops indexes and foreign keys that reference it. Renamin
 
 `map_type` / `map_column` take Rails-style abstract type symbols (`:string`, `:integer`, `:jsonb`, …), not raw SQL type names.
 
-### Automatic adjustments (with warnings)
-
-Some MySQL constructs have no PostgreSQL equivalent. schema_ferry handles them and prints a `[schema_ferry]` warning to stderr:
-
-- **FULLTEXT / SPATIAL indexes** are skipped. Declare a replacement with `add_index` (e.g. `add_index :body, using: :gin, opclass: :gin_trgm_ops` — requires `CREATE EXTENSION pg_trgm` on the target, done once by hand) and silence the warning with `ignore_index`. Don't create replacement indexes by hand: anything not in the generated schema is dropped on the next run.
-- **Index prefix lengths** (`KEY (col(10))`) are dropped silently — PostgreSQL indexes the full column.
-- **Identifiers over 63 bytes** (MySQL allows 64): index and foreign key names are shortened deterministically (`first 54 bytes + _ + 8-char digest`), so repeated runs stay stable. Overlong table names are only warned about — rename those yourself.
-- **Zero-date defaults** (`'0000-00-00 00:00:00'`) are invalid in PostgreSQL and are dropped.
-- **BIGINT UNSIGNED columns on a foreign key** (either side) become signed `bigint` instead of `numeric(20)` — a numeric column cannot reference a bigint primary key. Values above 2⁶³−1 will not fit, the same trade-off as for `BIGINT UNSIGNED` primary keys.
-
 ## How it works
 
 Each run executes a three-stage pipeline:

data/lib/schema_ferry/converter/column_converter.rb

@@ -45,11 +45,7 @@ module SchemaFerry
         else
           emit_warning "column #{raw[:name].inspect}: BIGINT UNSIGNED has no PostgreSQL " \
                        "integer equivalent; mapped to decimal(20, 0)."
-          # AR's schema dumper renders decimal defaults as a string (e.g.
-          # `default: "0"`), not a numeric literal — match that representation
-          # or ridgepole re-applies the default on every run.
-          new_default = raw[:default]&.to_s
-          raw.merge(type: :decimal, limit: nil, precision: 20, scale: 0, default: new_default)
+          raw.merge(type: :decimal, limit: nil, precision: 20, scale: 0)
         end
       end
 

data/lib/schema_ferry/converter/schema_converter.rb

@@ -8,6 +8,21 @@ module SchemaFerry
       # PostgreSQL has no FULLTEXT/SPATIAL equivalent that ridgepole can express.
       UNSUPPORTED_INDEX_TYPES = %i[fulltext spatial].freeze
 
+      # MySQL spatial column types have no PostgreSQL equivalent (that would
+      # require PostGIS, which schema_ferry does not manage). Most of these
+      # already raise TypeMapper's ConversionError, since ActiveRecord's
+      # mysql2 adapter reports them as an unrecognized type (nil). POINT is
+      # the sole exception: AR matches sql_type against an unanchored /int/i
+      # regex (see register_class_with_limit in
+      # ActiveRecord::ConnectionAdapters::AbstractAdapter#initialize_type_map),
+      # which matches "point" as a substring — so it's misreported as plain
+      # :integer instead. Without this check it would sail through as a
+      # meaningless integer column instead of raising. It must be caught
+      # here, by sql_type, and raised explicitly — the same failure mode as
+      # any other unsupported type, not a warning that's easy to miss in a
+      # cron log.
+      MISDETECTED_SPATIAL_SQL_TYPE = /\Apoint\b/i
+
       def initialize(config)
         @column_converter = ColumnConverter.new(TypeMapper.new(config.global_type_overrides))
         @table_rules      = config.table_rules
@@ -82,47 +97,46 @@ module SchemaFerry
       def convert_columns(raw_columns, table_name, rule, ignored, fk_columns)
         raw_columns
           .reject { |c| ignored.include?(c[:name]) }
-          .map    { |c| @column_converter.call(c, table_name, rule, fk_columns) }
+          .map do |c|
+            check_misdetected_spatial_type!(c, table_name, rule)
+            @column_converter.call(c, table_name, rule, fk_columns)
+          end
       end
 
-      def convert_indexes(raw_indexes, table_name, rule, ignored)
-        converted = raw_indexes
-                    .reject { |idx| rule&.ignored_indexes&.include?(idx[:name]) }
-                    .reject { |idx| skip_unsupported_index?(idx, table_name) }
-                    .reject { |idx| idx[:columns].intersect?(ignored) }
-                    .map    { |idx| build_index_schema(idx, table_name) }
-        converted + extra_indexes(rule, table_name)
+      # map_column already lets a rule override any column's type before it
+      # ever reaches this check (ColumnConverter#call skips TypeMapper
+      # entirely when an override is present), same as for any other type.
+      def check_misdetected_spatial_type!(raw, table_name, rule)
+        return unless raw[:sql_type].to_s.match?(MISDETECTED_SPATIAL_SQL_TYPE)
+        return if rule&.column_type_overrides&.key?(raw[:name])
+
+        raise ConversionError, "#{table_name}.#{raw[:name]}: MySQL #{raw[:sql_type]} columns have no " \
+                               "PostgreSQL equivalent without PostGIS, which schema_ferry does not " \
+                               "manage. Exclude it with ignore_column :#{raw[:name]}."
       end
 
-      def extra_indexes(rule, table_name)
-        (rule&.extra_indexes || []).map do |extra|
-          opts = extra[:options]
-          name = opts[:name] || "index_#{table_name}_on_#{extra[:columns].join("_")}"
-          IndexSchema.new(
-            name:    IdentifierShortener.shorten(name.to_s, kind: "index", table: table_name),
-            columns: extra[:columns],
-            unique:  opts[:unique],
-            using:   opts[:using],
-            opclass: opts[:opclass],
-            where:   opts[:where],
-            orders:  opts[:order],
-            lengths: nil
-          )
-        end
+      def convert_indexes(raw_indexes, table_name, rule, ignored)
+        raw_indexes
+          .reject { |idx| rule&.ignored_indexes&.include?(idx[:name]) }
+          .reject { |idx| idx[:columns].intersect?(ignored) }
+          .map do |idx|
+            check_unsupported_index_type!(idx, table_name)
+            build_index_schema(idx, table_name)
+          end
       end
 
       def convert_foreign_keys(raw)
         surviving_foreign_keys(raw).map { |fk| build_fk_schema(fk) }
       end
 
-      def skip_unsupported_index?(idx, table_name)
-        return false unless UNSUPPORTED_INDEX_TYPES.include?(idx[:type])
+      # A warning here would be easy to miss in an unattended cron run.
+      # Raising makes this failure show up the same way as any other error:
+      # a non-zero exit status a monitor can catch.
+      def check_unsupported_index_type!(idx, table_name)
+        return unless UNSUPPORTED_INDEX_TYPES.include?(idx[:type])
 
-        emit_warning "#{table_name}: skipping #{idx[:type].to_s.upcase} index #{idx[:name].inspect} " \
-                     "(no PostgreSQL equivalent). Declare a replacement with add_index " \
-                     "(e.g. add_index :col, using: :gin, opclass: :gin_trgm_ops) and " \
-                     "silence this warning with ignore_index :#{idx[:name]}."
-        true
+        raise ConversionError, "#{table_name}: #{idx[:type].to_s.upcase} index #{idx[:name].inspect} " \
+                               "has no PostgreSQL equivalent. Exclude it with ignore_index :#{idx[:name]}."
       end
 
       def build_index_schema(raw, table_name)
@@ -131,8 +145,6 @@ module SchemaFerry
           columns: raw[:columns],
           unique:  raw[:unique],
           using:   raw[:using],
-          opclass: nil, # MySQL has no operator classes
-          where:   raw[:where],
           lengths: raw[:lengths],
           orders:  raw[:orders]
         )

data/lib/schema_ferry/converter/type_mapper.rb

@@ -3,9 +3,10 @@
 module SchemaFerry
   module Converter
     class TypeMapper
-      # PG has no limit concept for text/binary/bigint; drop the MySQL-derived
-      # limits.
-      LIMIT_STRIPPED_TYPES = %i[text binary bigint].freeze
+      # PG has no limit concept for text/binary/bigint/float; drop the
+      # MySQL-derived limits (float's limit: 53 is DOUBLE's internal bit
+      # width, not something AR ever reads back from a PG column).
+      LIMIT_STRIPPED_TYPES = %i[text binary bigint float].freeze
 
       # PG's default timestamp precision is 6. Spelling it out makes ridgepole
       # see a diff against the PG export (which omits it) on every run.
@@ -43,8 +44,15 @@ module SchemaFerry
         pg_type, adjusted = normalize_integer(adjusted) if pg_type == :integer
         adjusted.delete(:limit) if LIMIT_STRIPPED_TYPES.include?(pg_type)
         strip_default_precision(pg_type, adjusted)
-        # PG numeric(20) equals numeric(20,0) and is exported without scale.
-        adjusted[:scale] = nil if pg_type == :decimal && adjusted[:scale]&.zero?
+        if pg_type == :decimal
+          # PG numeric(20) equals numeric(20,0) and is exported without scale.
+          adjusted[:scale] = nil if adjusted[:scale]&.zero?
+          # AR's schema dumper renders decimal defaults as a string (e.g.
+          # `default: "0"`), not a numeric literal. ridgepole compares against
+          # that dumped form, so a BigDecimal/Integer default never matches
+          # and gets re-applied on every run.
+          adjusted[:default] = adjusted[:default]&.to_s
+        end
 
         [pg_type, adjusted]
       end

data/lib/schema_ferry/dsl/table_rule.rb

@@ -6,11 +6,8 @@ module SchemaFerry
       UNSET = Object.new.freeze
       private_constant :UNSET
 
-      EXTRA_INDEX_OPTIONS = %i[name unique using where opclass order].freeze
-
       attr_reader :table_name, :column_type_overrides,
-                  :column_default_overrides, :ignored_columns, :ignored_indexes,
-                  :extra_indexes
+                  :column_default_overrides, :ignored_columns, :ignored_indexes
 
       def initialize(table_name)
         @table_name               = table_name.to_s
@@ -18,7 +15,6 @@ module SchemaFerry
         @column_default_overrides = {}
         @ignored_columns          = []
         @ignored_indexes          = []
-        @extra_indexes            = []
       end
 
       def map_column(column_name, type:, default: UNSET)
@@ -33,19 +29,6 @@ module SchemaFerry
       def ignore_index(index_name)
         @ignored_indexes << index_name.to_s
       end
-
-      # Declares a PostgreSQL-side index that does not exist in MySQL (e.g. a
-      # pg_trgm GIN index replacing a skipped FULLTEXT index). Declared indexes
-      # are part of the generated schema, so sync keeps them.
-      def add_index(*columns, **options)
-        unknown = options.keys - EXTRA_INDEX_OPTIONS
-        unless unknown.empty?
-          raise ConfigError, "add_index: unknown option(s) #{unknown.map(&:inspect).join(", ")} " \
-                             "(allowed: #{EXTRA_INDEX_OPTIONS.map(&:inspect).join(", ")})"
-        end
-
-        @extra_indexes << { columns: columns.map(&:to_s), options: options }
-      end
     end
   end
 end

data/lib/schema_ferry/schema_model.rb

@@ -32,8 +32,6 @@ module SchemaFerry
     :columns, # Array<String>
     :unique,  # Boolean
     :using,   # Symbol | nil
-    :opclass, # Symbol | Hash | nil (PostgreSQL operator class, e.g. :gin_trgm_ops)
-    :where,   # String | nil
     :lengths, # Integer | Hash | nil
     :orders,  # Symbol | Hash | nil
     keyword_init: true

data/lib/schema_ferry/source/mysql_reader.rb

@@ -64,7 +64,6 @@ module SchemaFerry
             unique:  idx.unique,
             using:   idx.using,
             type:    idx.type, # :fulltext | :spatial | nil
-            where:   idx.where,
             lengths: idx.lengths.presence,
             orders:  idx.orders.presence
           }

data/lib/schema_ferry/target/schemafile_renderer.rb

@@ -113,8 +113,6 @@ module SchemaFerry
         parts = ["name: #{idx.name.inspect}"]
         parts << "unique: true" if idx.unique
         parts << "using: #{idx.using.inspect}" if idx.using
-        parts << "opclass: #{idx.opclass.inspect}" if idx.opclass
-        parts << "where: #{idx.where.inspect}" if idx.where
         parts << "order: #{idx.orders.inspect}" if idx.orders
         "  t.index #{cols}, #{parts.join(", ")}"
       end

data/lib/schema_ferry/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SchemaFerry
-  VERSION = "0.1.2"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: schema_ferry
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.2.0
 platform: ruby
 authors:
 - kyuuri1791