percona_migrator 0.1.0.rc.5 → 0.1.0.rc.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: e7301e478b66a555e4abe723cd9429914fba7820
4
- data.tar.gz: fbf40c010ac9272f913b1e79285cb5f61fb11839
3
+ metadata.gz: 7c92cf9bc3f2fd48fbe1277d9196037bb9d92fec
4
+ data.tar.gz: 8df8aceabfb8822fe30c3c1cae0d9055b58e2834
5
5
  SHA512:
6
- metadata.gz: 39543206254ddc6f383446adca091ead90f3ac4a879dad0f7b246b8443d3bbeadc5eb95500fa26040fb8c28ef6182ad94e0620a1a4e8dd593f885f057555ed41
7
- data.tar.gz: 011386f3f3e8866fb9db4c19c98f9e050c4899168bb5c0eb9d78ddd56451ad5eeb89caaa3973f7574f21038b89dbcda4f511353103c760f577007043e8f82af0
6
+ metadata.gz: a1ee2cb57ce1d7062f34d6a3c604081f2c7c7863c76e2acccf7cdadb010239e4eb94350e0c28a17eb44a90f61c7c940239682dbec5e5d2bddcd18239386b1259
7
+ data.tar.gz: e9fb460fb9eaaecee4ef5a6e5d424a23da11b9717522fd4e4914fa01d27230c5c813dac1160c460f161414077e41f64f4348a94619f0c7e43f8071abc6a57100
data/CHANGELOG.md CHANGED
@@ -6,6 +6,18 @@ Please follow the format in [Keep a Changelog](http://keepachangelog.com/)
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.1.0.rc.6] - 2016-04-07
10
+
11
+ ### Added
12
+
13
+ - Support non-ddl migrations by implementing the methods for the ActiveRecord
14
+ quering to work.
15
+
16
+ ### Changed
17
+
18
+ - Refactor the PerconaAdapter to use the Runner as connection client, as all the
19
+ other adapters.
20
+
9
21
  ## [0.1.0.rc.5] - 2016-03-29
10
22
 
11
23
  ### Changed
@@ -22,7 +34,7 @@ Please follow the format in [Keep a Changelog](http://keepachangelog.com/)
22
34
  - Support for foreing keys in db/schema.rb when using [Foreigner
23
35
  gem](https://github.com/matthuhiggins/foreigner) in Rails 3 apps. This allows to
24
36
  define foreign keys with #execute, but does not provide support for
25
- #add_foreign_key yet.
37
+ add_foreign_key yet.
26
38
 
27
39
  ## [0.1.0.rc.3] - 2016-03-10
28
40
 
data/README.md CHANGED
@@ -1,26 +1,36 @@
1
- # PerconaMigrator [![Build Status](https://travis-ci.org/redbooth/percona_migrator.svg?branch=master)](https://travis-ci.org/redbooth/percona_migrator) [![Code Climate](https://codeclimate.com/github/redbooth/percona_migrator/badges/gpa.svg)](https://codeclimate.com/github/redbooth/percona_migrator)
1
+ # Percona Migrator [![Build Status](https://travis-ci.org/redbooth/percona_migrator.svg?branch=master)](https://travis-ci.org/redbooth/percona_migrator) [![Code Climate](https://codeclimate.com/github/redbooth/percona_migrator/badges/gpa.svg)](https://codeclimate.com/github/redbooth/percona_migrator)
2
2
 
3
- Percona Migrator is a tool for running online and non-blocking
4
- DDL `ActiveRecord::Migrations` using `pt-online-schema-change` command-line tool of
3
+ Percona Migrator is an **ActiveRecord connection adapter** that allows running
4
+ **MySQL online and non-blocking DDL** `ActiveRecord::Migration` without needing
5
+ to use a different DSL other than Rails' migrations DSL.
6
+
7
+ It uses `pt-online-schema-change` command-line tool of
5
8
  [Percona
6
9
  Toolkit](https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html)
7
- which supports foreign key constraints.
8
-
9
- It adds a `db:percona_migrate:up` runs your migration using the
10
- `pt-online-schema-change` command. It will apply exactly the same changes as
11
- if you run it with `db:migrate:up` avoiding deadlocks and without the need to
12
- change how you write regular rails migrations.
13
-
14
- It also disables `rake db:migrate:up` for the ddl migrations on envs with
15
- PERCONA_TOOLKIT var set to ensure all these migrations use Percona in production.
10
+ which runs MySQL alter table statements without downtime.
16
11
 
17
12
  ## Installation
18
13
 
19
- Percona Migrator relies on `pt-online-schema-change` from [Percona
14
+ Percona Migrator relies on `pt-online-schema-change` from [Percona
20
15
  Toolkit](https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html)
21
16
 
22
- For mac, you can install it with homebrew typing `brew install percona-toolkit`. For
23
- linux machines check out the [Percona Toolkit download
17
+ ### Mac
18
+
19
+ `brew install percona-toolkit`
20
+
21
+ ### Linux
22
+
23
+ #### Ubuntu/Debian based
24
+
25
+ `apt-get install percona-toolkit`
26
+
27
+ #### Arch Linux
28
+
29
+ `pacman -S percona-toolkit perl-dbd-mysql`
30
+
31
+ #### Other distros
32
+
33
+ For other Linux distributions check out the [Percona Toolkit download
24
34
  page](https://www.percona.com/downloads/percona-toolkit/) to find the package
25
35
  that fits your distribution.
26
36
 
@@ -42,20 +52,46 @@ Or install it yourself as:
42
52
 
43
53
  ## Usage
44
54
 
45
- Percona Migrator is meant to be used only on production or production-like
46
- environments. To that end, it will only run if the `PERCONA_TOOLKIT`
47
- environment variable is present.
55
+ Once you added it to your app's Gemfile, you can create and run Rails migrations
56
+ as usual.
57
+
58
+ All the `ALTER TABLE` statements will be executed with
59
+ `pt-online-schema-change`, which will provide additional output to the
60
+ migration.
61
+
62
+ ### LHM support
63
+
64
+ If you moved to Soundcloud's [Lhm](https://github.com/soundcloud/lhm) already,
65
+ we got you covered. Percona Migrator overrides Lhm's DSL so that all the alter
66
+ statements also go through `pt-online-schema-change` as well.
67
+
68
+ You can keep your Lhm migrations and start using Rails migration's DSL back
69
+ again in your next migration.
70
+
71
+ ## How it works
72
+
73
+ When booting your Rails app, Percona Migrator extends the
74
+ `ActiveRecord::Migration#migrate` method to reset the connection and reestablish
75
+ it using the `PerconaAdapter` instead of the one you defined in your
76
+ `config/database.yml`.
48
77
 
49
- From that same environment where you added the variable, execute the following:
78
+ Then, when any migration DSL methods such as `add_column` or `create_table` are
79
+ executed, they all go to the
80
+ [PerconaAdapter](https://github.com/redbooth/percona_migrator/blob/master/lib/active_record/connection_adapters/percona_adapter.rb).
81
+ There, the methods that require `ALTER TABLE` SQL statements, like `add_column`,
82
+ are overriden to get executed with
83
+ [PerconaMigrator::Runner](https://github.com/redbooth/percona_migrator/blob/master/lib/percona_migrator/runner.rb),
84
+ which deals with the `pt-online-schema-change` binary. All the others, like
85
+ `create_table`, are delegated to the ActiveRecord's built in Mysql2Adapter and
86
+ so they follow the regular path.
50
87
 
51
- 1. `bundle exec rake db:migrate:status` to find out your migration's version
52
- number
53
- 2. `rake db:percona_migrate:up VERSION=<version>`. This will run the migration
54
- and mark it as up. Otherwise, if the migration fails, it will still be listed as down
88
+ [PerconaMigrator::Runner](https://github.com/redbooth/percona_migrator/blob/master/lib/percona_migrator/runner.rb)
89
+ spawns a new process that runs the `pt-online-schema-change` binary present in
90
+ the system, with the apropriate arguments for the generated SQL.
55
91
 
56
- You can also mark the migration as run manually, by executing `bundle exec rake
57
- db:migrate:mark_as_up VERSION=<version>`. Likewise, there's a `bundle exec rake
58
- db:migrate:mark_as_down VERSION=<version>` that may be of help.
92
+ When an any error occurs, an `ActiveRecord::StatementInvalid` exception is
93
+ raised and the migration is aborted, as all other ActiveRecord connection
94
+ adapters.
59
95
 
60
96
  ## Development
61
97
 
@@ -9,19 +9,19 @@ module ActiveRecord
9
9
  # Establishes a connection to the database that's used by all Active
10
10
  # Record objects.
11
11
  def self.percona_connection(config)
12
- connection = mysql2_connection(config)
13
- client = connection.raw_connection
12
+ mysql2_connection = mysql2_connection(config)
14
13
 
15
- config.merge!(
16
- logger: logger,
17
- runner: PerconaMigrator::Runner.new(logger),
18
- cli_generator: PerconaMigrator::CliGenerator.new(config)
14
+ cli_generator = PerconaMigrator::CliGenerator.new(config)
15
+ runner = PerconaMigrator::Runner.new(
16
+ logger,
17
+ cli_generator,
18
+ mysql2_connection
19
19
  )
20
20
 
21
- connection_options = { mysql_adapter: connection }
21
+ connection_options = { mysql_adapter: mysql2_connection }
22
22
 
23
23
  ConnectionAdapters::PerconaMigratorAdapter.new(
24
- client,
24
+ runner,
25
25
  logger,
26
26
  connection_options,
27
27
  config
@@ -42,58 +42,48 @@ module ActiveRecord
42
42
 
43
43
  ADAPTER_NAME = 'Percona'.freeze
44
44
 
45
- def_delegators :mysql_adapter, :tables, :select_values, :exec_delete,
46
- :exec_insert, :exec_query, :last_inserted_id, :select, :create_table,
47
- :drop_table
45
+ def_delegators :mysql_adapter, :last_inserted_id, :each_hash
48
46
 
49
- def initialize(connection, logger, connection_options, config)
47
+ def initialize(connection, _logger, connection_options, _config)
50
48
  super
49
+ @visitor = BindSubstitution.new(self)
51
50
  @mysql_adapter = connection_options[:mysql_adapter]
52
- @logger = logger
53
- @runner = config[:runner]
54
- @cli_generator = config[:cli_generator]
55
51
  end
56
52
 
57
- # Returns true, as this adapter supports migrations
58
- def supports_migrations?
59
- true
53
+ def exec_delete(sql, name, binds)
54
+ execute(to_sql(sql, binds), name)
55
+ @connection.affected_rows
60
56
  end
57
+ alias :exec_update :exec_delete
61
58
 
62
- # Delegates #each_hash to the mysql adapter
63
- #
64
- # @param result [Mysql2::Result]
65
- def each_hash(result)
66
- if block_given?
67
- mysql_adapter.each_hash(result, &Proc.new)
68
- else
69
- mysql_adapter.each_hash(result)
70
- end
59
+ def exec_insert(sql, name, binds)
60
+ execute(to_sql(sql, binds), name)
71
61
  end
72
62
 
73
- def new_column(field, default, type, null, collation)
74
- Column.new(field, default, type, null, collation)
63
+ def exec_query(sql, name = 'SQL', _binds = [])
64
+ result = execute(sql, name)
65
+ ActiveRecord::Result.new(result.fields, result.to_a)
75
66
  end
76
67
 
77
- # Adds a new column to the named table
78
- #
79
- # @param table_name [String, Symbol]
80
- # @param column_name [String, Symbol]
81
- # @param type [Symbol]
82
- # @param options [Hash] optional
83
- def add_column(table_name, column_name, type, options = {})
84
- super
85
- command = cli_generator.generate(table_name, @sql)
86
- log(@sql, nil) { runner.execute(command) }
68
+ # Executes a SELECT query and returns an array of rows. Each row is an
69
+ # array of field values.
70
+ def select_rows(sql, name = nil)
71
+ execute(sql, name).to_a
87
72
  end
88
73
 
89
- # Removes the column(s) from the table definition
90
- #
91
- # @param table_name [String, Symbol]
92
- # @param column_names [String, Symbol, Array<String>, Array<Symbol>]
93
- def remove_column(table_name, *column_names)
94
- super
95
- command = cli_generator.generate(table_name, @sql)
96
- log(@sql, nil) { runner.execute(command) }
74
+ # Executes a SELECT query and returns an array of record hashes with the
75
+ # column names as keys and column values as values.
76
+ def select(sql, name = nil, binds = [])
77
+ exec_query(sql, name, binds).to_a
78
+ end
79
+
80
+ # Returns true, as this adapter supports migrations
81
+ def supports_migrations?
82
+ true
83
+ end
84
+
85
+ def new_column(field, default, type, null, collation)
86
+ Column.new(field, default, type, null, collation)
97
87
  end
98
88
 
99
89
  # Adds a new index to the table
@@ -103,10 +93,7 @@ module ActiveRecord
103
93
  # @param options [Hash] optional
104
94
  def add_index(table_name, column_name, options = {})
105
95
  index_name, index_type, index_columns, index_options = add_index_options(table_name, column_name, options)
106
- execute "ADD #{index_type} INDEX #{quote_column_name(index_name)} (#{index_columns})#{index_options}"
107
-
108
- command = cli_generator.generate(table_name, @sql)
109
- log(@sql, nil) { runner.execute(command) }
96
+ execute "ALTER TABLE #{quote_table_name(table_name)} ADD #{index_type} INDEX #{quote_column_name(index_name)} (#{index_columns})#{index_options}"
110
97
  end
111
98
 
112
99
  # Remove the given index from the table.
@@ -115,59 +102,17 @@ module ActiveRecord
115
102
  # @param options [Hash] optional
116
103
  def remove_index(table_name, options = {})
117
104
  index_name = index_name_for_remove(table_name, options)
118
- execute "DROP INDEX #{quote_column_name(index_name)}"
119
-
120
- command = cli_generator.generate(table_name, @sql)
121
- log(@sql, nil) { runner.execute(command) }
122
- end
123
-
124
- # Records the SQL statement to be executed. This is used to then delegate
125
- # the execution to Percona's pt-online-schema-change.
126
- #
127
- # @param sql [String]
128
- # @param _name [String] optional
129
- def execute(sql, _name = nil)
130
- @sql = sql
131
- true
105
+ execute "ALTER TABLE #{quote_table_name(table_name)} DROP INDEX #{quote_column_name(index_name)}"
132
106
  end
133
107
 
134
- # Executes the passed statement through pt-online-schema-change if it's
135
- # an alter statement, or through the mysql adapter otherwise
136
- #
137
- # @param sql [String]
138
- # @param name [String]
139
- def percona_execute(sql, name)
140
- if alter_statement?(sql)
141
- command = cli_generator.parse_statement(sql)
142
- log(sql, nil) { runner.execute(command) }
143
- else
144
- mysql_adapter.execute(sql, name)
145
- end
146
- end
147
-
148
- # This abstract method leaves up to the connection adapter freeing the
149
- # result, if it needs to. Check out: https://github.com/rails/rails/blob/330c6af05c8b188eb072afa56c07d5fe15767c3c/activerecord/lib/active_record/connection_adapters/abstract_mysql_adapter.rb#L247
150
- #
151
- # @param sql [String]
152
- # @param name [String] optional
153
- def execute_and_free(sql, name = nil)
154
- yield mysql_adapter.execute(sql, name)
155
- end
156
-
157
- def error_number(exception)
108
+ # Returns the MySQL error number from the exception. The
109
+ # AbstractMysqlAdapter requires it to be implemented
110
+ def error_number(_exception)
158
111
  end
159
112
 
160
113
  private
161
114
 
162
- attr_reader :mysql_adapter, :logger, :runner, :cli_generator
163
-
164
- # Checks whether the sql statement is an ALTER TABLE
165
- #
166
- # @param sql [String]
167
- # @return [Boolean]
168
- def alter_statement?(sql)
169
- sql =~ /alter table/i
170
- end
115
+ attr_reader :mysql_adapter
171
116
  end
172
117
  end
173
118
  end
@@ -1,30 +1,32 @@
1
1
  module PerconaMigrator
2
+ class InvalidAlterStatement < StandardError; end
2
3
 
3
4
  # Represents the '--alter' argument of Percona's pt-online-schema-change
4
5
  # See https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html
5
6
  class AlterArgument
6
- ALTER_TABLE_REGEX = /ALTER TABLE `(\w+)` /
7
+ ALTER_TABLE_REGEX = /\AALTER TABLE `(\w+)` /
8
+
9
+ attr_reader :table_name
7
10
 
8
11
  # Constructor
9
12
  #
10
13
  # @param statement [String]
14
+ # @raise [InvalidAlterStatement] if the statement is not an ALTER TABLE
11
15
  def initialize(statement)
12
16
  @statement = statement
17
+
18
+ match = statement.match(ALTER_TABLE_REGEX)
19
+ raise InvalidAlterStatement unless match
20
+
21
+ @table_name = match.captures[0]
13
22
  end
14
23
 
15
- # Returns the '--alter' pt-online-schema-change argumment as a string. See
24
+ # Returns the '--alter' pt-online-schema-change argument as a string. See
16
25
  # https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html
17
26
  def to_s
18
27
  "--alter \"#{parsed_statement}\""
19
28
  end
20
29
 
21
- # Returns the name of the table the alter statement refers to
22
- #
23
- # @return [String]
24
- def table_name
25
- statement.match(ALTER_TABLE_REGEX).captures[0]
26
- end
27
-
28
30
  private
29
31
 
30
32
  attr_reader :statement
@@ -48,18 +48,6 @@ module PerconaMigrator
48
48
  connection_config.merge(adapter: 'percona')
49
49
  )
50
50
  end
51
-
52
- # It executes the passed statement through the PerconaMigratorAdapter
53
- # if it's an alter statement. It uses the mysql adapter otherwise.
54
- #
55
- # This is because +pt-online-schema-change+ is intended for alter
56
- # statements only.
57
- #
58
- # @param sql [String]
59
- # @param name [String] optional
60
- def execute(sql, name = nil)
61
- percona_execute(sql, name)
62
- end
63
51
  end
64
52
  end
65
53
  end
@@ -47,11 +47,35 @@ module PerconaMigrator
47
47
  # Constructor
48
48
  #
49
49
  # @param logger [IO]
50
- def initialize(logger)
50
+ def initialize(logger, cli_generator, mysql_adapter)
51
51
  @logger = logger
52
+ @cli_generator = cli_generator
53
+ @mysql_adapter = mysql_adapter
52
54
  @status = nil
53
55
  end
54
56
 
57
+ # Executes the passed sql statement using pt-online-schema-change for ALTER
58
+ # TABLE statements, or the specified mysql adapter otherwise.
59
+ #
60
+ # @param sql [String]
61
+ def query(sql)
62
+ if alter_statement?(sql)
63
+ command = cli_generator.parse_statement(sql)
64
+ execute(command)
65
+ else
66
+ mysql_adapter.execute(sql)
67
+ end
68
+ end
69
+
70
+ # Returns the number of rows affected by the last UPDATE, DELETE or INSERT
71
+ # statements
72
+ #
73
+ # @return [Integer]
74
+ def affected_rows
75
+ mysql_adapter.raw_connection.affected_rows
76
+ end
77
+
78
+ # TODO: rename it so we don't confuse it with AR's #execute
55
79
  # Runs and logs the given command
56
80
  #
57
81
  # @param command [String]
@@ -64,7 +88,15 @@ module PerconaMigrator
64
88
 
65
89
  private
66
90
 
67
- attr_reader :command, :logger, :status
91
+ attr_reader :command, :logger, :status, :cli_generator, :mysql_adapter
92
+
93
+ # Checks whether the sql statement is an ALTER TABLE
94
+ #
95
+ # @param sql [String]
96
+ # @return [Boolean]
97
+ def alter_statement?(sql)
98
+ sql =~ /\Aalter table/i
99
+ end
68
100
 
69
101
  # Logs the start and end of the execution
70
102
  #
@@ -84,7 +116,9 @@ module PerconaMigrator
84
116
 
85
117
  # Executes the command outputing any errors
86
118
  #
87
- # @raise [Errno::ENOENT] if pt-online-schema-change can't be found
119
+ # @raise [NoStatusError] if the spawned process' status can't be retrieved
120
+ # @raise [SignalError] if the spawned process receives a signal
121
+ # @raise [CommandNotFoundError] if pt-online-schema-change can't be found
88
122
  def run_command
89
123
  message = nil
90
124
  Open3.popen3(command) do |_stdin, stdout, stderr, waith_thr|
@@ -1,3 +1,3 @@
1
1
  module PerconaMigrator
2
- VERSION = '0.1.0.rc.5'.freeze
2
+ VERSION = '0.1.0.rc.6'.freeze
3
3
  end
@@ -7,8 +7,8 @@ require 'percona_migrator/version'
7
7
  Gem::Specification.new do |spec|
8
8
  spec.name = 'percona_migrator'
9
9
  spec.version = PerconaMigrator::VERSION
10
- spec.authors = ['Ilya Zayats', 'Pau Pérez', 'Fran Casas']
11
- spec.email = ['ilya.zayats@redbooth.com', 'pau.perez@redbooth.com', 'fran.casas@redbooth.com']
10
+ spec.authors = ['Ilya Zayats', 'Pau Pérez', 'Fran Casas', 'Jorge Morante']
11
+ spec.email = ['ilya.zayats@redbooth.com', 'pau.perez@redbooth.com', 'fran.casas@redbooth.com', 'jorge.morante@redbooth.com']
12
12
 
13
13
  spec.summary = %q{pt-online-schema-change runner for ActiveRecord migrations}
14
14
  spec.description = %q{Execute your ActiveRecord migrations with Percona's pt-online-schema-change}
metadata CHANGED
@@ -1,16 +1,17 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: percona_migrator
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0.rc.5
4
+ version: 0.1.0.rc.6
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ilya Zayats
8
8
  - Pau Pérez
9
9
  - Fran Casas
10
+ - Jorge Morante
10
11
  autorequire:
11
12
  bindir: bin
12
13
  cert_chain: []
13
- date: 2016-03-29 00:00:00.000000000 Z
14
+ date: 2016-04-07 00:00:00.000000000 Z
14
15
  dependencies:
15
16
  - !ruby/object:Gem::Dependency
16
17
  name: rails
@@ -127,6 +128,7 @@ email:
127
128
  - ilya.zayats@redbooth.com
128
129
  - pau.perez@redbooth.com
129
130
  - fran.casas@redbooth.com
131
+ - jorge.morante@redbooth.com
130
132
  executables: []
131
133
  extensions: []
132
134
  extra_rdoc_files: []