schema_ferry 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +20 -1
- data/README.md +34 -32
- data/lib/schema_ferry/cli.rb +19 -10
- data/lib/schema_ferry/config.rb +109 -0
- data/lib/schema_ferry/{converter → core}/column_converter.rb +4 -4
- data/lib/schema_ferry/{converter → core}/enum_check_builder.rb +8 -6
- data/lib/schema_ferry/{converter/identifier_shortener.rb → core/identifier_shortenable.rb} +4 -14
- data/lib/schema_ferry/core/schema_model.rb +60 -0
- data/lib/schema_ferry/core/schemafile_renderer.rb +109 -0
- data/lib/schema_ferry/core/table_converter.rb +159 -0
- data/lib/schema_ferry/core/translator.rb +53 -0
- data/lib/schema_ferry/{converter → core}/type_mapper.rb +2 -2
- data/lib/schema_ferry/errors.rb +2 -0
- data/lib/schema_ferry/io/mysql_reader.rb +80 -0
- data/lib/schema_ferry/{target/ridgepole_runner.rb → io/postgres_writer.rb} +11 -6
- data/lib/schema_ferry/pipeline.rb +24 -7
- data/lib/schema_ferry/support/drop_detectable.rb +17 -0
- data/lib/schema_ferry/support/warnings.rb +13 -0
- data/lib/schema_ferry/version.rb +1 -1
- data/lib/schema_ferry.rb +14 -14
- metadata +20 -14
- data/lib/schema_ferry/converter/schema_converter.rb +0 -172
- data/lib/schema_ferry/dsl/config.rb +0 -77
- data/lib/schema_ferry/dsl/table_rule.rb +0 -34
- data/lib/schema_ferry/schema_model.rb +0 -56
- data/lib/schema_ferry/source/connection_registry.rb +0 -29
- data/lib/schema_ferry/source/mysql_reader.rb +0 -88
- data/lib/schema_ferry/target/schemafile_renderer.rb +0 -143
- data/lib/schema_ferry/warnings.rb +0 -13
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 689fd7901be3470c89447bcf83a61fb4185880e53f5723afa3385d2c23850a29
|
|
4
|
+
data.tar.gz: e74cd92c6c2b0c446b4b57e5460311f9885044f3f1f1ad2b86b23fc926cb0671
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c918782073cfb88339251517b2985a022b7035eb813c21045afe9ba66b252adaf32350820ac756c035e795411963af678747a5aeb00ec82404ab1bc10b006ee7
|
|
7
|
+
data.tar.gz: 8a8d656502ee2ad147212f3920b19136ec31ca88f7b99d2391ade83046f9a0db476bbc163fdc496034e2d3048ff58219674aee03f8315a07b15b5ab4b576f9ac
|
data/CHANGELOG.md
CHANGED
|
@@ -5,7 +5,26 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
-
## [
|
|
8
|
+
## [0.3.0] - 2026-07-11
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- `apply!(allow_drops: false)` — refuses to apply anything (raising `SchemaFerry::DropNotAllowedError` instead) whenever the diff contains a column, index, foreign key, or constraint drop. The default remains `allow_drops: true`, matching the gem's "MySQL is the source of truth" philosophy, but this gives an escape hatch for handoff periods where a target-only addition might otherwise get swept up in a legitimate MySQL-driven drop. The CLI exposes the same behavior via `schema_ferry apply --disable-drops`.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- `map_column :col, type: :type` is now `column :col, map_type_to: :type`. `map_column` implied a mapping between two columns, which isn't what it does — it overrides one column's own type in place. `column` pairs with the existing `table` block, and `map_type_to:` mirrors `map_type`'s `to:` keyword for the same kind of type override.
|
|
17
|
+
- `apply!` now wraps the whole migration in a single transaction (via ridgepole's `--pre-query`/`--post-query`), so a failure partway through no longer leaves a half-applied schema on the target.
|
|
18
|
+
- A table name over 63 bytes now raises `SchemaFerry::ConversionError` instead of just warning. `apply!` was never actually going to succeed for it anyway — ActiveRecord's own PostgreSQL schema statements reject an over-length `create_table` outright — so the warning was misleading about how far the pipeline would actually get. `ignore_table` it, or rename it in MySQL.
|
|
19
|
+
|
|
20
|
+
### Removed
|
|
21
|
+
|
|
22
|
+
- `Pipeline#schemafile` — it exposed the generated ridgepole Schemafile as a string without connecting to the target. That text is an internal hand-off format between schema_ferry and ridgepole, and keeping it public meant committing to a stable, human-friendly rendering of it forever. Pipeline's API is now just `dry_run` and `apply!`; to preview what a run would do, use `dry_run` — it shows the actual diff against the target rather than the full declared schema.
|
|
23
|
+
|
|
24
|
+
### Fixed
|
|
25
|
+
|
|
26
|
+
- A MySQL `MULTIPOINT` column silently passed through as a plain PostgreSQL integer instead of raising. ActiveRecord misdetects `MULTIPOINT` as `:integer`, the same bug that already affected `POINT` — the check only matched `POINT` by name, so `MULTIPOINT` slipped past it. All eight MySQL spatial types (`GEOMETRY`, `POINT`, `LINESTRING`, `POLYGON`, `MULTIPOINT`, `MULTILINESTRING`, `MULTIPOLYGON`, `GEOMETRYCOLLECTION`) are now matched directly by `sql_type`, so they all raise the same explicit "no PostgreSQL equivalent without PostGIS" error instead of some going through a generic "unknown type" message.
|
|
27
|
+
- `Pipeline#apply!(allow_drops: false)` read the MySQL schema twice — once for the drop pre-check and once for the actual apply — so a schema change landing on MySQL between the two reads could apply a drop the pre-check never saw. The schema is now read and converted once, and the same Schemafile is used for both the pre-check and the apply, matching what the CLI already did.
|
|
9
28
|
|
|
10
29
|
## [0.2.0] - 2026-07-06
|
|
11
30
|
|
data/README.md
CHANGED
|
@@ -6,7 +6,7 @@ You're migrating a production MySQL database to PostgreSQL. Moving the data take
|
|
|
6
6
|
- **Sensible defaults, fully customizable** — built-in type mappings handle most cases; override anything with a few DSL rules
|
|
7
7
|
- **Safe to iterate** — `dry_run` shows the exact changes that would be applied, before touching anything
|
|
8
8
|
|
|
9
|
-
schema_ferry is designed to run repeatedly (
|
|
9
|
+
schema_ferry is designed to run repeatedly — as a step in whatever CI/CD pipeline you already have (Jenkins, Step Functions, GitHub Actions, …). Data migration is out of scope — pair it with [pgloader](https://github.com/dimitri/pgloader) (one-shot bulk copy) or CDC replication (AWS DMS, [Debezium](https://github.com/debezium/debezium), …), which load rows into the tables schema_ferry keeps in sync.
|
|
10
10
|
|
|
11
11
|
## Requirements
|
|
12
12
|
|
|
@@ -41,13 +41,11 @@ pipeline.dry_run # returns the changes that would be applied, without applying
|
|
|
41
41
|
pipeline.apply! # applies the schema to PostgreSQL
|
|
42
42
|
```
|
|
43
43
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
`apply!` makes the target match the generated schema — including **dropping** columns and indexes from the target that are not part of it. Before running against a target that holds data, read [Coverage](#coverage) below.
|
|
44
|
+
`apply!` makes the target match the generated schema — including **dropping** columns, indexes, foreign keys, and constraints from the target that are not part of it. Before running against a target that holds data, read [Handoff](#handoff) below.
|
|
47
45
|
|
|
48
46
|
### CLI
|
|
49
47
|
|
|
50
|
-
|
|
48
|
+
As a step in an existing pipeline, or whenever you'd rather not write a runner script, there is a small CLI. Put the same DSL (without the `SchemaFerry.define` wrapper) in a `Ferryfile`:
|
|
51
49
|
|
|
52
50
|
```ruby
|
|
53
51
|
source "mysql2://user:password@host:3306/source_db"
|
|
@@ -60,9 +58,10 @@ Then:
|
|
|
60
58
|
schema_ferry dry-run # show what would change (reads ./Ferryfile)
|
|
61
59
|
schema_ferry apply # apply to PostgreSQL
|
|
62
60
|
schema_ferry apply -c path/to/Ferryfile # explicit definition file path
|
|
61
|
+
schema_ferry apply --disable-drops # refuse instead of applying if the diff contains a drop
|
|
63
62
|
```
|
|
64
63
|
|
|
65
|
-
Each command prints the changes it applied (or would apply) followed by a one-line summary (`118 tables synced, 3 changes applied`). The exit status is 0 on success and 1 on any error, so
|
|
64
|
+
Each command prints the changes it applied (or would apply) followed by a one-line summary (`118 tables synced, 3 changes applied`). The exit status is 0 on success and 1 on any error, so your monitoring can rely on it.
|
|
66
65
|
|
|
67
66
|
### Custom conversion rules
|
|
68
67
|
|
|
@@ -75,7 +74,7 @@ pipeline = SchemaFerry.define do
|
|
|
75
74
|
map_type :json, to: :json # e.g. opt out of the default json → jsonb conversion
|
|
76
75
|
|
|
77
76
|
table :users do
|
|
78
|
-
|
|
77
|
+
column :is_admin, map_type_to: :boolean # override a specific column's type
|
|
79
78
|
ignore_column :legacy_field # exclude a column
|
|
80
79
|
ignore_index :idx_old_legacy # exclude an index
|
|
81
80
|
end
|
|
@@ -84,25 +83,7 @@ pipeline = SchemaFerry.define do
|
|
|
84
83
|
end
|
|
85
84
|
```
|
|
86
85
|
|
|
87
|
-
The same rules work in a
|
|
88
|
-
|
|
89
|
-
### Coverage
|
|
90
|
-
|
|
91
|
-
schema_ferry syncs what can be done automatically — exactly where possible, or as an approximation with a warning where it isn't — and leaves the rest to add by hand, later. Where there's no reasonable equivalent at all, it raises instead of guessing. Add the column or index by hand once you're fully cut over to PostgreSQL — not before, since `apply!` delegates to [ridgepole](https://github.com/ridgepole/ridgepole), which drops anything missing from the generated schema on a managed table, including something added by hand as a stand-in — but never drops a table that's absent from it entirely.
|
|
92
|
-
|
|
93
|
-
Review `dry_run` output before your first `apply!` and whenever you change the conversion rules — those are the moments that introduce drops. Unattended runs in between only mirror changes made to the MySQL schema; if even those need review, schedule `dry-run` instead and apply by hand.
|
|
94
|
-
|
|
95
|
-
Normalized automatically, with a warning to stderr:
|
|
96
|
-
|
|
97
|
-
- **Index prefix lengths** (`KEY (col(10))`) are dropped silently — PostgreSQL indexes the full column.
|
|
98
|
-
- **Identifiers over 63 bytes** (MySQL allows 64): index and foreign key names are shortened deterministically (`first 54 bytes + _ + 8-char digest`), so repeated runs stay stable. Overlong table names are only warned about — rename those yourself.
|
|
99
|
-
- **Zero-date defaults** (`'0000-00-00 00:00:00'`) are invalid in PostgreSQL and are dropped.
|
|
100
|
-
- **BIGINT UNSIGNED columns on a foreign key** (either side) become signed `bigint` instead of `numeric(20)` — a numeric column cannot reference a bigint primary key. Values above 2⁶³−1 will not fit, the same trade-off as for `BIGINT UNSIGNED` primary keys.
|
|
101
|
-
|
|
102
|
-
Raises instead:
|
|
103
|
-
|
|
104
|
-
- **FULLTEXT indexes** — PostgreSQL has no equivalent construct (a `pg_trgm` GIN index is a common approximation, but it's not the same search semantics, so schema_ferry doesn't create one for you). Because of the drop behavior above, you can't pre-create a replacement during the sync period — add one once you're fully cut over to PostgreSQL. `ignore_index` them.
|
|
105
|
-
- **Spatial columns** (`POINT`, `GEOMETRY`, `POLYGON`, `LINESTRING`, …) — PostgreSQL has no built-in equivalent without PostGIS, which schema_ferry does not manage. `ignore_column` them.
|
|
86
|
+
The same rules work in a `Ferryfile`.
|
|
106
87
|
|
|
107
88
|
## DSL reference
|
|
108
89
|
|
|
@@ -121,14 +102,14 @@ Raises instead:
|
|
|
121
102
|
|
|
122
103
|
| Method | Description |
|
|
123
104
|
|---|---|
|
|
124
|
-
| `
|
|
125
|
-
| `
|
|
105
|
+
| `column :col, map_type_to: :type` | Override a column's type |
|
|
106
|
+
| `column :col, map_type_to: :type, default: value` | …and give it an explicit default |
|
|
126
107
|
| `ignore_column :col` | Exclude a column |
|
|
127
108
|
| `ignore_index :index_name` | Exclude an index |
|
|
128
109
|
|
|
129
|
-
Ignoring a column also drops indexes and foreign keys that reference it.
|
|
110
|
+
Ignoring a column also drops indexes and foreign keys that reference it.
|
|
130
111
|
|
|
131
|
-
**tinyint(1) caveat:** ActiveRecord reads `tinyint(1)` as boolean, including its default (`DEFAULT 2` is read as `true`). If a `tinyint(1)` column actually holds 0/1/2-style values, override both the type and the default: `
|
|
112
|
+
**tinyint(1) caveat:** ActiveRecord reads `tinyint(1)` as boolean, including its default (`DEFAULT 2` is read as `true`). If a `tinyint(1)` column actually holds 0/1/2-style values, override both the type and the default: `column :flags, map_type_to: :integer, default: 2`. Without an explicit default, schema_ferry drops the unreliable boolean default and warns.
|
|
132
113
|
|
|
133
114
|
## Default type mapping
|
|
134
115
|
|
|
@@ -139,7 +120,7 @@ Ignoring a column also drops indexes and foreign keys that reference it. Renamin
|
|
|
139
120
|
| `TINYINT(1)` | `boolean` | see the caveat above if a column holds more than 0/1 |
|
|
140
121
|
| `TINYINT`…`BIGINT` (signed) | `smallint` / `integer` / `bigint` | widths normalized to PostgreSQL's three integer sizes |
|
|
141
122
|
| `TINYINT`…`INT` `UNSIGNED` | one size larger | e.g. `INT UNSIGNED` → `bigint` |
|
|
142
|
-
| `BIGINT UNSIGNED` | `numeric(20)` | PostgreSQL has no unsigned 8-byte integer; emitted with a warning. Columns on a foreign key become signed `bigint` instead — see
|
|
123
|
+
| `BIGINT UNSIGNED` | `numeric(20)` | PostgreSQL has no unsigned 8-byte integer; emitted with a warning. Columns on a foreign key become signed `bigint` instead — see [Handoff](#handoff) below |
|
|
143
124
|
| `FLOAT` / `DOUBLE` | `double precision` | |
|
|
144
125
|
| `DECIMAL(p,s)` | `numeric(p,s)` | |
|
|
145
126
|
| `DATETIME` / `TIMESTAMP` | `timestamp` | use `map_type :datetime, to: :timestamptz` for `timestamptz` |
|
|
@@ -148,7 +129,28 @@ Ignoring a column also drops indexes and foreign keys that reference it. Renamin
|
|
|
148
129
|
| `JSON` | `jsonb` | opt out with `map_type :json, to: :json` |
|
|
149
130
|
| `ENUM(...)` | `varchar` | add `enum_as :check` to enforce the values with a CHECK constraint |
|
|
150
131
|
|
|
151
|
-
`map_type` / `
|
|
132
|
+
`map_type` / `column`'s `map_type_to:` take Rails-style abstract type symbols (`:string`, `:integer`, `:jsonb`, …), not raw SQL type names.
|
|
133
|
+
|
|
134
|
+
## Handoff
|
|
135
|
+
|
|
136
|
+
MySQL is the source of truth: `apply!` makes PostgreSQL match the generated schema exactly, so anything else on the target — including a column or index added by hand as an early stand-in — gets dropped. That's intentional. Add the real thing by hand once you're fully cut over to PostgreSQL, not before. The one exception is a table absent from the generated schema entirely — that's left alone.
|
|
137
|
+
|
|
138
|
+
Though that's the philosophy, if you don't want it, you can switch it off with `apply!(allow_drops: false)`: `apply!` then only goes through when the diff is drop-free; otherwise it raises `SchemaFerry::DropNotAllowedError` instead of applying anything to the target.
|
|
139
|
+
|
|
140
|
+
Within that generated schema, schema_ferry syncs what can be done automatically — exactly where possible, or as an approximation with a warning where it isn't — and leaves the rest to add by hand, later. Where there's no reasonable equivalent at all, it raises instead of guessing.
|
|
141
|
+
|
|
142
|
+
Normalized automatically, with a warning to stderr:
|
|
143
|
+
|
|
144
|
+
- **Index prefix lengths** (`KEY (col(10))`) are dropped silently — PostgreSQL indexes the full column.
|
|
145
|
+
- **Index and foreign key names over 63 bytes** (MySQL allows 64) are shortened deterministically (`first 54 bytes + _ + 8-char digest`), so repeated runs stay stable.
|
|
146
|
+
- **Zero-date defaults** (`'0000-00-00 00:00:00'`) are invalid in PostgreSQL and are dropped.
|
|
147
|
+
- **BIGINT UNSIGNED columns on a foreign key** (either side) become signed `bigint` instead of `numeric(20)` — a numeric column cannot reference a bigint primary key. Values above 2⁶³−1 will not fit, the same trade-off as for `BIGINT UNSIGNED` primary keys.
|
|
148
|
+
|
|
149
|
+
Raises instead:
|
|
150
|
+
|
|
151
|
+
- **FULLTEXT indexes** — PostgreSQL has no equivalent construct (a `pg_trgm` GIN index is a common approximation, but it's not the same search semantics, so schema_ferry doesn't create one for you). Because of the drop behavior above, you can't pre-create a replacement during the sync period — add one once you're fully cut over to PostgreSQL. `ignore_index` them.
|
|
152
|
+
- **Spatial columns** (`POINT`, `GEOMETRY`, `POLYGON`, `LINESTRING`, …) — PostgreSQL has no built-in equivalent without PostGIS, which schema_ferry does not manage. `ignore_column` them.
|
|
153
|
+
- **Table names over 63 bytes** (MySQL allows 64) — unlike index and foreign key names, schema_ferry won't rename a table for you (too invasive to do silently). `ignore_table` it, or rename it in MySQL.
|
|
152
154
|
|
|
153
155
|
## How it works
|
|
154
156
|
|
data/lib/schema_ferry/cli.rb
CHANGED
|
@@ -3,16 +3,16 @@
|
|
|
3
3
|
require "optparse"
|
|
4
4
|
|
|
5
5
|
module SchemaFerry
|
|
6
|
-
# Minimal command-line interface: `schema_ferry <apply|dry-run> [-c FILE]`.
|
|
7
|
-
# #run returns a process exit code instead of calling Kernel#exit so the
|
|
8
|
-
# class stays testable.
|
|
9
6
|
class CLI
|
|
7
|
+
include Support::DropDetectable
|
|
8
|
+
|
|
10
9
|
DEFAULT_CONFIG_PATH = "Ferryfile"
|
|
11
10
|
COMMANDS = %w[apply dry-run].freeze
|
|
12
11
|
|
|
13
12
|
def initialize(stdout: $stdout, stderr: $stderr)
|
|
14
|
-
@stdout
|
|
15
|
-
@stderr
|
|
13
|
+
@stdout = stdout
|
|
14
|
+
@stderr = stderr
|
|
15
|
+
@allow_drops = true
|
|
16
16
|
end
|
|
17
17
|
|
|
18
18
|
def run(argv)
|
|
@@ -33,10 +33,18 @@ module SchemaFerry
|
|
|
33
33
|
private
|
|
34
34
|
|
|
35
35
|
def run_command(command)
|
|
36
|
-
config
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
36
|
+
config = load_config
|
|
37
|
+
mysql_tables = IO::MysqlReader.new(config.source_url).read_all
|
|
38
|
+
schemafile = Core::Translator.new(config).translate(mysql_tables)
|
|
39
|
+
dry_run = command == "dry-run"
|
|
40
|
+
writer = IO::PostgresWriter.new(config.target_url)
|
|
41
|
+
|
|
42
|
+
if command == "apply" && !@allow_drops
|
|
43
|
+
drops = detect_drops(writer.run(schemafile, dry_run: true))
|
|
44
|
+
raise DropNotAllowedError, "refused: the diff contains drop(s):\n#{drops.join("\n")}" unless drops.empty?
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
output = writer.run(schemafile, dry_run: dry_run)
|
|
40
48
|
|
|
41
49
|
@stdout.puts output
|
|
42
50
|
@stdout.puts summary(schemafile, output, dry_run: dry_run)
|
|
@@ -72,13 +80,14 @@ module SchemaFerry
|
|
|
72
80
|
path = @config_path || DEFAULT_CONFIG_PATH
|
|
73
81
|
raise ConfigError, "definition file not found: #{path}" unless File.exist?(path)
|
|
74
82
|
|
|
75
|
-
|
|
83
|
+
Config.load_file(path)
|
|
76
84
|
end
|
|
77
85
|
|
|
78
86
|
def parser
|
|
79
87
|
@parser ||= OptionParser.new do |o|
|
|
80
88
|
o.banner = "Usage: schema_ferry <apply|dry-run> [options]"
|
|
81
89
|
o.on("-c", "--config FILE", "Definition file (default: #{DEFAULT_CONFIG_PATH})") { |v| @config_path = v }
|
|
90
|
+
o.on("--disable-drops", "apply: refuse if the diff contains any drop") { @allow_drops = false }
|
|
82
91
|
o.on("-h", "--help", "Show this help") { @mode = :help }
|
|
83
92
|
o.on("--version", "Show version") { @mode = :version }
|
|
84
93
|
end
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SchemaFerry
|
|
4
|
+
class Config
|
|
5
|
+
attr_reader :source_url, :target_url,
|
|
6
|
+
:global_type_overrides,
|
|
7
|
+
:table_rules,
|
|
8
|
+
:ignored_tables,
|
|
9
|
+
:enum_mode
|
|
10
|
+
|
|
11
|
+
class << self
|
|
12
|
+
# Ruby entry point
|
|
13
|
+
def build(&block)
|
|
14
|
+
evaluate { |config| config.instance_eval(&block) }
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
# CLI entry point
|
|
18
|
+
def load_file(path)
|
|
19
|
+
evaluate { |config| config.instance_eval(File.read(path), path.to_s, 1) }
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
private
|
|
23
|
+
|
|
24
|
+
def evaluate
|
|
25
|
+
new.tap do |config|
|
|
26
|
+
yield config
|
|
27
|
+
config.validate!
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def initialize
|
|
33
|
+
@global_type_overrides = {}
|
|
34
|
+
@table_rules = Hash.new(TableRule::EMPTY)
|
|
35
|
+
@ignored_tables = []
|
|
36
|
+
@enum_mode = :string
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def source(url)
|
|
40
|
+
@source_url = url
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def target(url)
|
|
44
|
+
@target_url = url
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def map_type(mysql_type, to:)
|
|
48
|
+
@global_type_overrides[mysql_type.to_sym] = to.to_sym
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
def table(table_name, &)
|
|
52
|
+
rule = TableRule.new(table_name)
|
|
53
|
+
rule.instance_eval(&)
|
|
54
|
+
@table_rules[table_name.to_s] = rule
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def ignore_table(table_name)
|
|
58
|
+
@ignored_tables << table_name.to_s
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
# How to convert MySQL enum columns:
|
|
62
|
+
# :string — plain varchar, values not enforced (default)
|
|
63
|
+
# :check — varchar plus a CHECK constraint restricting the values
|
|
64
|
+
def enum_as(mode)
|
|
65
|
+
modes = %i[string check]
|
|
66
|
+
unless modes.include?(mode.to_sym)
|
|
67
|
+
raise ConfigError, "enum_as accepts #{modes.map(&:inspect).join(" or ")}, got #{mode.inspect}"
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
@enum_mode = mode.to_sym
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def validate!
|
|
74
|
+
raise ConfigError, 'source is not configured. Add: source "mysql2://..."' unless @source_url
|
|
75
|
+
raise ConfigError, 'target is not configured. Add: target "postgresql://..."' unless @target_url
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# Vocabulary available inside a `table :name do ... end` block.
|
|
79
|
+
class TableRule
|
|
80
|
+
attr_reader :table_name, :column_type_overrides,
|
|
81
|
+
:column_default_overrides, :ignored_columns, :ignored_indexes
|
|
82
|
+
|
|
83
|
+
def initialize(table_name)
|
|
84
|
+
@table_name = table_name.to_s
|
|
85
|
+
@column_type_overrides = {}
|
|
86
|
+
@column_default_overrides = {}
|
|
87
|
+
@ignored_columns = []
|
|
88
|
+
@ignored_indexes = []
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
EMPTY = new("").freeze # Null object for tables without a `table ... do` block.
|
|
92
|
+
DEFAULT_OMITTED = Object.new.freeze
|
|
93
|
+
private_constant :DEFAULT_OMITTED
|
|
94
|
+
|
|
95
|
+
def column(column_name, map_type_to:, default: DEFAULT_OMITTED)
|
|
96
|
+
@column_type_overrides[column_name.to_s] = map_type_to.to_sym
|
|
97
|
+
@column_default_overrides[column_name.to_s] = default unless default.equal?(DEFAULT_OMITTED)
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def ignore_column(column_name)
|
|
101
|
+
@ignored_columns << column_name.to_s
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
def ignore_index(index_name)
|
|
105
|
+
@ignored_indexes << index_name.to_s
|
|
106
|
+
end
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
end
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module SchemaFerry
|
|
4
|
-
module
|
|
4
|
+
module Core
|
|
5
5
|
class ColumnConverter
|
|
6
|
-
include Warnings
|
|
6
|
+
include Support::Warnings
|
|
7
7
|
|
|
8
8
|
# PG has no unsigned integers: bump to the next size that holds the full
|
|
9
9
|
# unsigned range. Keys/values are AR byte limits.
|
|
@@ -16,7 +16,7 @@ module SchemaFerry
|
|
|
16
16
|
def call(raw, table_name, rule, fk_columns = [])
|
|
17
17
|
raw = bump_unsigned_integer(raw, table_name, fk_columns.include?(raw[:name]))
|
|
18
18
|
raw = drop_zero_date_default(raw, table_name)
|
|
19
|
-
override = rule
|
|
19
|
+
override = rule.column_type_overrides[raw[:name]]
|
|
20
20
|
col_opts = raw.slice(:limit, :precision, :scale, :null, :default, :default_function, :comment)
|
|
21
21
|
|
|
22
22
|
if override
|
|
@@ -71,7 +71,7 @@ module SchemaFerry
|
|
|
71
71
|
emit_warning "#{rule.table_name}.#{raw[:name]}: dropping default #{default.inspect} — " \
|
|
72
72
|
"MySQL reported a tinyint(1) default as boolean, which is unreliable under " \
|
|
73
73
|
"a type override. Restore it explicitly: " \
|
|
74
|
-
"
|
|
74
|
+
"column :#{raw[:name]}, map_type_to: :#{override}, default: <value>"
|
|
75
75
|
return nil
|
|
76
76
|
end
|
|
77
77
|
default
|
|
@@ -1,16 +1,18 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module SchemaFerry
|
|
4
|
-
module
|
|
4
|
+
module Core
|
|
5
5
|
# Builds CHECK constraints enforcing MySQL enum values on varchar columns
|
|
6
6
|
# (enum_as :check).
|
|
7
7
|
class EnumCheckBuilder
|
|
8
|
-
|
|
8
|
+
include IdentifierShortenable
|
|
9
|
+
|
|
10
|
+
def call(raw_table, rule)
|
|
9
11
|
raw_table[:columns].filter_map do |col|
|
|
10
|
-
next if
|
|
12
|
+
next if rule.ignored_columns.include?(col[:name])
|
|
11
13
|
# A type override takes the column away from varchar; the caller owns
|
|
12
14
|
# any constraint then.
|
|
13
|
-
next if rule
|
|
15
|
+
next if rule.column_type_overrides.key?(col[:name])
|
|
14
16
|
|
|
15
17
|
values = enum_values(col[:sql_type])
|
|
16
18
|
next if values.nil?
|
|
@@ -24,8 +26,8 @@ module SchemaFerry
|
|
|
24
26
|
def build_constraint(table_name, col_name, values)
|
|
25
27
|
CheckConstraintSchema.new(
|
|
26
28
|
expression: expression(col_name, values),
|
|
27
|
-
name:
|
|
28
|
-
|
|
29
|
+
name: shorten_identifier("chk_#{table_name}_#{col_name}",
|
|
30
|
+
kind: "check constraint", table: table_name)
|
|
29
31
|
)
|
|
30
32
|
end
|
|
31
33
|
|
|
@@ -3,19 +3,17 @@
|
|
|
3
3
|
require "digest"
|
|
4
4
|
|
|
5
5
|
module SchemaFerry
|
|
6
|
-
module
|
|
6
|
+
module Core
|
|
7
7
|
# PostgreSQL truncates identifiers to 63 bytes (MySQL allows 64). A silently
|
|
8
8
|
# truncated index name makes ridgepole see a diff on every run, so names that
|
|
9
9
|
# would overflow are shortened deterministically instead.
|
|
10
|
-
module
|
|
11
|
-
|
|
10
|
+
module IdentifierShortenable
|
|
11
|
+
include Support::Warnings
|
|
12
12
|
|
|
13
13
|
MAX_BYTES = 63
|
|
14
14
|
HASH_LENGTH = 8
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
def shorten(name, kind:, table:)
|
|
16
|
+
def shorten_identifier(name, kind:, table:)
|
|
19
17
|
return name if name.nil? || name.bytesize <= MAX_BYTES
|
|
20
18
|
|
|
21
19
|
prefix = name.byteslice(0, MAX_BYTES - HASH_LENGTH - 1)
|
|
@@ -24,14 +22,6 @@ module SchemaFerry
|
|
|
24
22
|
"#{MAX_BYTES}-byte identifier limit; renamed to #{short.inspect}."
|
|
25
23
|
short
|
|
26
24
|
end
|
|
27
|
-
|
|
28
|
-
def warn_long_table_name(name)
|
|
29
|
-
return if name.bytesize <= MAX_BYTES
|
|
30
|
-
|
|
31
|
-
emit_warning "table name #{name.inspect} exceeds PostgreSQL's #{MAX_BYTES}-byte " \
|
|
32
|
-
"identifier limit and will be truncated by PostgreSQL. Rename the " \
|
|
33
|
-
"table before applying to avoid ridgepole re-creating it on every run."
|
|
34
|
-
end
|
|
35
25
|
end
|
|
36
26
|
end
|
|
37
27
|
end
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Plain data structs for the converted (PostgreSQL-side) schema: TableConverter
|
|
4
|
+
# builds these, SchemafileRenderer reads them.
|
|
5
|
+
module SchemaFerry
|
|
6
|
+
module Core
|
|
7
|
+
TableSchema = Struct.new(
|
|
8
|
+
:name, # String
|
|
9
|
+
:primary_key, # String | Array<String> | nil
|
|
10
|
+
:pk_type, # Symbol: :bigint, :integer, :string, etc. (single-column PK only)
|
|
11
|
+
:pk_limit, # Integer | nil (limit of a string PK column)
|
|
12
|
+
:comment, # String | nil
|
|
13
|
+
:columns, # Array<ColumnSchema>
|
|
14
|
+
:indexes, # Array<IndexSchema>
|
|
15
|
+
:foreign_keys, # Array<ForeignKeySchema>
|
|
16
|
+
:check_constraints, # Array<CheckConstraintSchema>
|
|
17
|
+
keyword_init: true
|
|
18
|
+
)
|
|
19
|
+
|
|
20
|
+
ColumnSchema = Struct.new(
|
|
21
|
+
:name, # String
|
|
22
|
+
:type, # Symbol (:string, :integer, :jsonb, …)
|
|
23
|
+
:limit, # Integer | nil
|
|
24
|
+
:precision, # Integer | nil
|
|
25
|
+
:scale, # Integer | nil
|
|
26
|
+
:null, # Boolean
|
|
27
|
+
:default, # Object | nil
|
|
28
|
+
:default_function, # String | nil (e.g. "CURRENT_TIMESTAMP")
|
|
29
|
+
:comment, # String | nil
|
|
30
|
+
keyword_init: true
|
|
31
|
+
)
|
|
32
|
+
|
|
33
|
+
IndexSchema = Struct.new(
|
|
34
|
+
:name, # String
|
|
35
|
+
:columns, # Array<String>
|
|
36
|
+
:unique, # Boolean
|
|
37
|
+
:using, # Symbol | nil
|
|
38
|
+
:lengths, # Integer | Hash | nil
|
|
39
|
+
:orders, # Symbol | Hash | nil
|
|
40
|
+
keyword_init: true
|
|
41
|
+
)
|
|
42
|
+
|
|
43
|
+
CheckConstraintSchema = Struct.new(
|
|
44
|
+
:expression, # String (e.g. "kind IN ('a', 'b')")
|
|
45
|
+
:name, # String
|
|
46
|
+
keyword_init: true
|
|
47
|
+
)
|
|
48
|
+
|
|
49
|
+
ForeignKeySchema = Struct.new(
|
|
50
|
+
:from_table, # String
|
|
51
|
+
:to_table, # String
|
|
52
|
+
:column, # String
|
|
53
|
+
:primary_key, # String
|
|
54
|
+
:on_update, # Symbol | nil
|
|
55
|
+
:on_delete, # Symbol | nil
|
|
56
|
+
:name, # String | nil
|
|
57
|
+
keyword_init: true
|
|
58
|
+
)
|
|
59
|
+
end
|
|
60
|
+
end
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SchemaFerry
|
|
4
|
+
module Core
|
|
5
|
+
class SchemafileRenderer
|
|
6
|
+
Call = Struct.new(:name, :args, :opts, :children)
|
|
7
|
+
Raw = Struct.new(:source)
|
|
8
|
+
|
|
9
|
+
def render(tables)
|
|
10
|
+
calls = tables.map { |table| table_call(table) }
|
|
11
|
+
calls += tables.flat_map { |table| table.foreign_keys.map { |fk| foreign_key_call(fk) } }
|
|
12
|
+
calls.map { |call| serialize(call) }.join("\n\n")
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
private
|
|
16
|
+
|
|
17
|
+
def table_call(table)
|
|
18
|
+
children = table.columns.map { |col| column_call(col) }
|
|
19
|
+
children += table.indexes.map { |idx| index_call(idx) }
|
|
20
|
+
children += Array(table.check_constraints).map { |chk| check_constraint_call(chk) }
|
|
21
|
+
Call.new("create_table", [table.name], table_options(table), children)
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
def table_options(table)
|
|
25
|
+
opts = { force: :cascade }
|
|
26
|
+
case table.primary_key
|
|
27
|
+
when nil
|
|
28
|
+
opts[:id] = false
|
|
29
|
+
when String
|
|
30
|
+
opts[:primary_key] = table.primary_key if table.primary_key != "id"
|
|
31
|
+
if table.pk_type && table.pk_type != :bigint
|
|
32
|
+
opts[:id] = table.pk_type
|
|
33
|
+
opts[:limit] = table.pk_limit
|
|
34
|
+
end
|
|
35
|
+
when Array
|
|
36
|
+
opts[:primary_key] = table.primary_key
|
|
37
|
+
end
|
|
38
|
+
opts[:comment] = table.comment
|
|
39
|
+
opts
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def column_call(col)
|
|
43
|
+
Call.new("t.#{col.type}", [col.name], {
|
|
44
|
+
limit: col.limit,
|
|
45
|
+
precision: col.precision,
|
|
46
|
+
scale: col.scale,
|
|
47
|
+
default: default_option(col),
|
|
48
|
+
null: (false if col.null == false),
|
|
49
|
+
comment: col.comment
|
|
50
|
+
})
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def default_option(col)
|
|
54
|
+
return col.default unless col.default.nil?
|
|
55
|
+
|
|
56
|
+
Raw.new("-> { #{col.default_function.inspect} }") if col.default_function
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def index_call(idx)
|
|
60
|
+
Call.new("t.index", [idx.columns], {
|
|
61
|
+
name: idx.name,
|
|
62
|
+
unique: (true if idx.unique),
|
|
63
|
+
using: idx.using,
|
|
64
|
+
order: idx.orders
|
|
65
|
+
})
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def check_constraint_call(chk)
|
|
69
|
+
Call.new("t.check_constraint", [chk.expression], { name: chk.name })
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
def foreign_key_call(foreign_key)
|
|
73
|
+
Call.new("add_foreign_key", [foreign_key.from_table, foreign_key.to_table], {
|
|
74
|
+
column: (foreign_key.column if custom_fk_column?(foreign_key)),
|
|
75
|
+
primary_key: (foreign_key.primary_key if foreign_key.primary_key && foreign_key.primary_key != "id"),
|
|
76
|
+
name: foreign_key.name,
|
|
77
|
+
on_update: foreign_key.on_update,
|
|
78
|
+
on_delete: foreign_key.on_delete
|
|
79
|
+
})
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# ridgepole's PostgreSQL export omits column: when it follows the Rails
|
|
83
|
+
# convention, and it compares foreign keys by their literal options —
|
|
84
|
+
# emitting column: for conventional names re-creates the FK every run.
|
|
85
|
+
def custom_fk_column?(foreign_key)
|
|
86
|
+
foreign_key.column && foreign_key.column != "#{foreign_key.to_table.singularize}_id"
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def serialize(call, indent = "")
|
|
90
|
+
head = "#{indent}#{[call.name, arguments(call)].reject(&:empty?).join(" ")}"
|
|
91
|
+
return head unless call.children
|
|
92
|
+
|
|
93
|
+
["#{head} do |t|",
|
|
94
|
+
*call.children.map { |child| serialize(child, "#{indent} ") },
|
|
95
|
+
"#{indent}end"].join("\n")
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
def arguments(call)
|
|
99
|
+
positional = call.args.map { |arg| literal(arg) }
|
|
100
|
+
keyword = call.opts.filter_map { |key, value| "#{key}: #{literal(value)}" unless value.nil? }
|
|
101
|
+
(positional + keyword).join(", ")
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
def literal(value)
|
|
105
|
+
value.is_a?(Raw) ? value.source : value.inspect
|
|
106
|
+
end
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
end
|