departure 1.0.1 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 04cbb41217f09b9e06bb015359b9189f8342d5b1
4
- data.tar.gz: 280d997d365765664307e0851a71c86a631e473f
3
+ metadata.gz: ef3cc8883b3b4dc294f1b94b7d93744e98812849
4
+ data.tar.gz: 242eac0339dd6f9aedf020d321cd9e6cc0bc24fe
5
5
  SHA512:
6
- metadata.gz: 145620cb84bcb3bf28d3504e3a35713bcfcb265e3627da6faa2c9a01b77bf1bd4e2b8ffa34d4ecd2bff6fe39df68f454374694a4198286a475029fa90353ef3e
7
- data.tar.gz: 9eac5663f4e2fa00d1d2f47a3222f63f65f16144c49f9bcb4438d49d3e4c996abb236074616972848b971c0e2992db586bdd2574f05048079afe002b9b90d518
6
+ metadata.gz: 515ab293664312a15f3482f1b0cc0156c325b59df452fa82a9a700adbff81197d1cfffcb4b1296a1afdad47f828a6aeffe0e5b831b61ff37dd89fab0127abdd4
7
+ data.tar.gz: 35b2746785ba419acddf8e2629e564f9382e7107fd4e57dcd75750e7440f347b3d1e4ce4c96a22a330096bec2bb383ef039070244a522dfd27df87018f850dd1
data/.gitignore CHANGED
@@ -9,4 +9,4 @@
9
9
  /tmp/
10
10
  .byebug_history
11
11
  tags
12
- percona_migrator_error.log
12
+ departure_error.log
@@ -11,45 +11,50 @@ Please follow the format in [Keep a Changelog](http://keepachangelog.com/)
11
11
  ### Removed
12
12
  ### Fixed
13
13
 
14
- ## [1.0.1] - 2017-04-19
14
+ ## [2.0.0 - 2017-04-20]
15
15
 
16
16
  ### Added
17
17
  ### Changed
18
+
19
+ - Rename the gem from percona_migrator to departure
20
+
18
21
  ### Removed
19
22
 
20
23
  - Percona_migrator's deprecation warnings when installing and running the gem.
21
24
 
22
25
  ### Fixed
23
26
 
24
- ## [1.0.0] - 2017-04-19
27
+ ## [1.1.0] - 2017-04-07
25
28
 
26
29
  ### Added
27
- ### Changed
28
30
 
29
- - Renamed gem to Departure
31
+ - Allow passing any `pt-online-schema-change`'s arguments through the
32
+ `PERCONA_ARGS` env var when executing a migration with `rake db:migrate:up`
33
+ or `db:migrate:down`.
34
+ - Allow setting global percona arguments via gem configuration
35
+ - Filter MySQL's password from logs
30
36
 
31
- ### Removed
32
- ### Fixed
37
+ ### Changed
38
+
39
+ - Enable default pt-online-schema-change replicas discovering mechanism.
40
+ So far, this was purposely set to `none`. To keep this same behaviour
41
+ provide the `PERCONA_ARGS=--recursion-method=none` env var when running the
42
+ migration.
33
43
 
34
- ## [0.1.1] - 2017-03-10
44
+ ## [1.0.0] - 2016-11-30
35
45
 
36
46
  ### Added
37
47
 
48
+ - Show pt-online-schema-change's stdout while the migration is running instead
49
+ of at then and all at once.
38
50
  - Store pt-online-schema-change's stderr to percona_migrator_error.log in the
39
51
  default Rails tmp folder.
40
- - Allow configuring the tmp directory where the error log gets written into.
41
- - Output a deprecation warning when installing and running the gem. It will be
42
- replaced by Departure soon.
43
-
44
- ### Changed
45
-
46
- - No longer a hard dependency on mysql2 0.3.20. Will accept 0.3.20 or any
47
- higher patch revisions.
48
-
49
- ### Fixed
50
-
51
- - Output pt-online-schema-change's stdout while the migration is running instead
52
- of all of it at the end.
52
+ - Allow configuring the tmp directory where the error log gets written into,
53
+ with the `tmp_path` configuration setting.
54
+ - Support for ActiveRecord 4.0. Adds the following migration methods:
55
+ - #rename_index, #change_column_null, #add_reference, #remove_reference,
56
+ #set_field_encoding, #add_timestamps, #remove_timestamps, #rename_table,
57
+ #rename_column
53
58
 
54
59
  ## [0.1.0.rc.7] - 2016-09-15
55
60
 
data/README.md CHANGED
@@ -26,6 +26,21 @@ Toolkit](https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change
26
26
 
27
27
  `brew install percona-toolkit`
28
28
 
29
+ If when running a migration you see an error like:
30
+
31
+ ```
32
+ PerconaMigrator::Error: Cannot connect to MySQL: Cannot connect to MySQL because
33
+ the Perl DBI module is not installed or not found.
34
+ ```
35
+
36
+ You also need to install the DBI and DBD::MySQL modules from `cpan`.
37
+
38
+ ```
39
+ $ sudo cpan
40
+ cpan> install DBI
41
+ cpan> install DBD::mysql
42
+ ```
43
+
29
44
  ### Linux
30
45
 
31
46
  #### Ubuntu/Debian based
@@ -67,6 +82,48 @@ All the `ALTER TABLE` statements will be executed with
67
82
  `pt-online-schema-change`, which will provide additional output to the
68
83
  migration.
69
84
 
85
+ ### pt-online-schema-change arguments
86
+
87
+ #### with environment variable
88
+
89
+ You can specify any `pt-online-schema-change` arguments when running the
90
+ migration. All what you pass in the PERCONA_ARGS env var, will be bypassed to the
91
+ binary, overwriting any default values. Note the format is the same as in
92
+ `pt-online-schema-change`. Check the full list in [Percona Toolkit
93
+ documentation](https://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html#options)
94
+
95
+ ```ruby
96
+ $ PERCONA_ARGS='--chunk-time=1' bundle exec rake db:migrate:up VERSION=xxx
97
+ ```
98
+
99
+ or even mulitple arguments
100
+
101
+ ```ruby
102
+ $ PERCONA_ARGS='--chunk-time=1 --critical-load=55' bundle exec rake db:migrate:up VERSION=xxx
103
+ ```
104
+
105
+ This however, only works for `db:migrate:up` or `db:migrate:down` rake tasks and
106
+ not with `db:migrate`. The settings you provide can't be generalized as these
107
+ vary depending on the database table and the kind of changes you apply.
108
+
109
+ #### with global configuration
110
+
111
+ You can specify any `pt-online-schema-change` arguments in global gem configuration
112
+ using `global_percona_args` option.
113
+
114
+ ```ruby
115
+ PerconaMigrator.configure do |config|
116
+ config.global_percona_args = '--chunk-time=1 --critical-load=55'
117
+ end
118
+ ```
119
+
120
+ Unlike using `PERCONA_ARGS`, options provided with global configuration will be applied
121
+ every time sql command is executed via `pt-online-schema-change`.
122
+
123
+ Arguments provided in global configuration can be overwritten with `PERCONA_ARGS` env variable.
124
+
125
+ We recommend using this option with caution and only when you understand the consequences.
126
+
70
127
  ### LHM support
71
128
 
72
129
  If you moved to Soundcloud's [Lhm](https://github.com/soundcloud/lhm) already,
@@ -129,7 +186,8 @@ git commits and tags, and push the `.gem` file to
129
186
  ## Contributing
130
187
 
131
188
  Bug reports and pull requests are welcome on GitHub at
132
- https://github.com/redbooth/percona_migrator.
189
+ https://github.com/redbooth/percona_migrator. They need to be opened against
190
+ `master` or `v3.2` only if the changes fix a bug in Rails 3.2 apps.
133
191
 
134
192
  Please note that this project is released with a Contributor Code of Conduct. By
135
193
  participating in this project you agree to abide by its terms.
@@ -0,0 +1,17 @@
1
+ # Releasing Percona Migrator
2
+
3
+ All releases come from the master branch. All other branches won't be maintained
4
+ and will receive bug fix releases only.
5
+
6
+ In order to give support to a new major Rails version, we'll branch off of
7
+ master, name it following the Rails repo convention, such as `v4.2`, and
8
+ we'll keep it open for bug fixes.
9
+
10
+ 1. Update `lib/percona_migrator/version.rb` accordingly
11
+ 2. Review the `CHANGELOG.md` and add a new section following the format
12
+ `[version] - YYYY-MM-DD`. We conform to the guidelines of
13
+ http://keepachangelog.com/
14
+ 3. Commit the changes with the message `Prepare release VERSION`
15
+ 4. Execute the release rake task as `bundle exec rake release`. It creates the
16
+ tag, builds and pushes the gem to Rubygems.
17
+ 5. Announce it! :tada:
@@ -0,0 +1,17 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+ #
4
+ # This file was generated by Bundler.
5
+ #
6
+ # The application 'rspec' is installed as part of a gem, and
7
+ # this file is here to facilitate running it.
8
+ #
9
+
10
+ require "pathname"
11
+ ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile",
12
+ Pathname.new(__FILE__).realpath)
13
+
14
+ require "rubygems"
15
+ require "bundler/setup"
16
+
17
+ load Gem.bin_path("rspec-core", "rspec")
data/config.yml CHANGED
@@ -1,3 +1,3 @@
1
1
  username: 'root'
2
2
  password: ''
3
- database: 'percona_migrator_test'
3
+ database: 'departure_test'
@@ -7,8 +7,8 @@ require 'departure/version'
7
7
  Gem::Specification.new do |spec|
8
8
  spec.name = 'departure'
9
9
  spec.version = Departure::VERSION
10
- spec.authors = ['Ilya Zayats', 'Pau Pérez', 'Fran Casas', 'Jorge Morante']
11
- spec.email = ['ilya.zayats@redbooth.com', 'pau.perez@redbooth.com', 'fran.casas@redbooth.com', 'jorge.morante@redbooth.com']
10
+ spec.authors = ['Ilya Zayats', 'Pau Pérez', 'Fran Casas', 'Jorge Morante', 'Adrian Serafin']
11
+ spec.email = ['ilya.zayats@redbooth.com', 'pau.perez@redbooth.com', 'fran.casas@redbooth.com', 'jorge.morante@redbooth.com', 'adrian@softmad.pl']
12
12
 
13
13
  spec.summary = %q{pt-online-schema-change runner for ActiveRecord migrations}
14
14
  spec.description = %q{Execute your ActiveRecord migrations with Percona's pt-online-schema-change. Formerly known as Percona Migrator.}
@@ -18,12 +18,13 @@ Gem::Specification.new do |spec|
18
18
  spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
19
19
  spec.require_paths = ['lib']
20
20
 
21
- spec.add_runtime_dependency 'rails', '~>3.2.22'
22
- spec.add_runtime_dependency 'mysql2', '~>0.3.20'
21
+ spec.add_runtime_dependency 'rails', '~> 4.0.0'
22
+ spec.add_runtime_dependency 'mysql2', '0.3.20'
23
23
 
24
24
  spec.add_development_dependency 'bundler', '~> 1.10'
25
25
  spec.add_development_dependency 'rake', '~> 10.0'
26
26
  spec.add_development_dependency 'rspec', '~> 3.4', '>= 3.4.0'
27
27
  spec.add_development_dependency 'rspec-its', '~> 1.2'
28
28
  spec.add_development_dependency 'byebug', '~> 8.2', '>= 8.2.1'
29
+ spec.add_development_dependency 'climate_control', '~> 0.0.3'
29
30
  end
@@ -5,15 +5,21 @@ require 'departure'
5
5
  require 'forwardable'
6
6
 
7
7
  module ActiveRecord
8
- class Base
8
+ module ConnectionHandling
9
9
  # Establishes a connection to the database that's used by all Active
10
10
  # Record objects.
11
- def self.percona_connection(config)
11
+ def percona_connection(config)
12
12
  mysql2_connection = mysql2_connection(config)
13
13
 
14
+ config[:username] = 'root' if config[:username].nil?
15
+
16
+ connection_details = Departure::ConnectionDetails.new(config)
14
17
  verbose = ActiveRecord::Migration.verbose
15
- percona_logger = Departure::LoggerFactory.build(verbose: verbose)
16
- cli_generator = Departure::CliGenerator.new(config)
18
+ sanitizers = [
19
+ Departure::LogSanitizers::PasswordSanitizer.new(connection_details)
20
+ ]
21
+ percona_logger = Departure::LoggerFactory.build(sanitizers: sanitizers, verbose: verbose)
22
+ cli_generator = Departure::CliGenerator.new(connection_details)
17
23
 
18
24
  runner = Departure::Runner.new(
19
25
  percona_logger,
@@ -45,7 +51,7 @@ module ActiveRecord
45
51
 
46
52
  ADAPTER_NAME = 'Percona'.freeze
47
53
 
48
- def_delegators :mysql_adapter, :last_inserted_id, :each_hash
54
+ def_delegators :mysql_adapter, :last_inserted_id, :each_hash, :set_field_encoding
49
55
 
50
56
  def initialize(connection, _logger, connection_options, _config)
51
57
  super
@@ -59,7 +65,7 @@ module ActiveRecord
59
65
  end
60
66
  alias :exec_update :exec_delete
61
67
 
62
- def exec_insert(sql, name, binds)
68
+ def exec_insert(sql, name, binds, pk = nil, sequence_name = nil)
63
69
  execute(to_sql(sql, binds), name)
64
70
  end
65
71
 
@@ -77,7 +83,7 @@ module ActiveRecord
77
83
  # Executes a SELECT query and returns an array of record hashes with the
78
84
  # column names as keys and column values as values.
79
85
  def select(sql, name = nil, binds = [])
80
- exec_query(sql, name, binds).to_a
86
+ exec_query(sql, name, binds)
81
87
  end
82
88
 
83
89
  # Returns true, as this adapter supports migrations
@@ -85,8 +91,8 @@ module ActiveRecord
85
91
  true
86
92
  end
87
93
 
88
- def new_column(field, default, type, null, collation)
89
- Column.new(field, default, type, null, collation)
94
+ def new_column(field, default, type, null, collation, extra = "")
95
+ Column.new(field, default, type, null, collation, extra)
90
96
  end
91
97
 
92
98
  # Adds a new index to the table
@@ -2,6 +2,7 @@ require 'active_record'
2
2
  require 'active_support/all'
3
3
 
4
4
  require 'departure/version'
5
+ require 'departure/log_sanitizers/password_sanitizer'
5
6
  require 'departure/runner'
6
7
  require 'departure/cli_generator'
7
8
  require 'departure/logger'
@@ -9,6 +10,7 @@ require 'departure/null_logger'
9
10
  require 'departure/logger_factory'
10
11
  require 'departure/configuration'
11
12
  require 'departure/errors'
13
+ require 'departure/command'
12
14
 
13
15
  require 'departure/railtie' if defined?(Rails)
14
16
 
@@ -28,6 +30,21 @@ module Departure
28
30
  # Hooks Percona Migrator into Rails migrations by replacing the configured
29
31
  # database adapter
30
32
  def self.load
33
+ ActiveRecord::Migrator.instance_eval do
34
+ class << self
35
+ alias_method(:original_migrate, :migrate)
36
+ end
37
+
38
+ # Checks whether arguments are being passed through PERCONA_ARGS when running
39
+ # the db:migrate rake task
40
+ #
41
+ # @raise [ArgumentsNotSupported] if PERCONA_ARGS has any value
42
+ def migrate(migrations_paths, target_version = nil, &block)
43
+ raise ArgumentsNotSupported if ENV['PERCONA_ARGS'].present?
44
+ original_migrate(migrations_paths, target_version, &block)
45
+ end
46
+ end
47
+
31
48
  ActiveRecord::Migration.class_eval do
32
49
  alias_method :original_migrate, :migrate
33
50
 
@@ -55,10 +72,9 @@ module Departure
55
72
  # Make all connections in the connection pool to use PerconaAdapter
56
73
  # instead of the current adapter.
57
74
  def reconnect_with_percona
58
- connection_config = ActiveRecord::Base.connection_config
59
- ActiveRecord::Base.establish_connection(
60
- connection_config.merge(adapter: 'percona')
61
- )
75
+ connection_config = ActiveRecord::Base
76
+ .connection_config.merge(adapter: 'percona')
77
+ ActiveRecord::Base.establish_connection(connection_config)
62
78
  end
63
79
  end
64
80
  end
@@ -1,49 +1,36 @@
1
+ require 'departure/dsn'
2
+ require 'departure/option'
1
3
  require 'departure/alter_argument'
4
+ require 'departure/connection_details'
5
+ require 'departure/user_options'
2
6
 
3
7
  module Departure
4
8
 
5
- # Represents the 'DSN' argument of Percona's pt-online-schema-change
6
- # See https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html#dsn-options
7
- class DSN
8
-
9
- # Constructor
10
- #
11
- # @param database [String, Symbol]
12
- # @param table_name [String, Symbol]
13
- def initialize(database, table_name)
14
- @database = database
15
- @table_name = table_name
16
- end
17
-
18
- # Returns the pt-online-schema-change DSN string. See
19
- # https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html#dsn-options
20
- def to_s
21
- "D=#{database},t=#{table_name}"
22
- end
23
-
24
- private
25
-
26
- attr_reader :table_name, :database
27
- end
28
-
29
9
  # Generates the equivalent Percona's pt-online-schema-change command to the
30
10
  # given SQL statement
31
- class CliGenerator # Command
32
- BASE_COMMAND = 'pt-online-schema-change'
33
- BASE_OPTIONS = %w(
34
- --execute
35
- --statistics
36
- --recursion-method=none
37
- --alter-foreign-keys-method=auto
38
- )
39
-
40
- # Constructor
11
+ #
12
+ # --no-check-alter is used to allow running CHANGE COLUMN statements. For
13
+ # more details, check: www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html#cmdoption-pt-online-schema-change--[no]check-alter
14
+ #
15
+ class CliGenerator
16
+ COMMAND_NAME = 'pt-online-schema-change'.freeze
17
+ DEFAULT_OPTIONS = Set.new(
18
+ [
19
+ Option.new('execute'),
20
+ Option.new('statistics'),
21
+ Option.new('alter-foreign-keys-method', 'auto'),
22
+ Option.new('no-check-alter')
23
+ ]
24
+ ).freeze
25
+
26
+ # TODO: Better doc.
27
+ #
28
+ # Constructor. Specify any arguments to pass to pt-online-schema-change
29
+ # passing the PERCONA_ARGS env var when executing the migration
41
30
  #
42
31
  # @param connection_data [Hash]
43
- def initialize(connection_data)
44
- @connection_data = connection_data
45
- init_base_command
46
- add_connection_details
32
+ def initialize(connection_details)
33
+ @connection_details = connection_details
47
34
  end
48
35
 
49
36
  # Generates the percona command. Fills all the connection credentials from
@@ -57,9 +44,9 @@ module Departure
57
44
  # @return [String]
58
45
  def generate(table_name, statement)
59
46
  alter_argument = AlterArgument.new(statement)
60
- dsn = DSN.new(database, table_name)
47
+ dsn = DSN.new(connection_details.database, table_name)
61
48
 
62
- "#{self} #{dsn} #{alter_argument}"
49
+ "#{command} #{all_options} #{dsn} #{alter_argument}"
63
50
  end
64
51
 
65
52
  # Generates the percona command for a raw MySQL statement. Fills all the
@@ -72,63 +59,27 @@ module Departure
72
59
  # @return [String]
73
60
  def parse_statement(statement)
74
61
  alter_argument = AlterArgument.new(statement)
75
- dsn = DSN.new(database, alter_argument.table_name)
62
+ dsn = DSN.new(connection_details.database, alter_argument.table_name)
76
63
 
77
- "#{self} #{dsn} #{alter_argument}"
64
+ "#{command} #{all_options} #{dsn} #{alter_argument}"
78
65
  end
79
66
 
80
67
  private
81
68
 
82
- attr_reader :connection_data
83
-
84
- # Sets up the command with its options
85
- def init_base_command
86
- @command = [BASE_COMMAND, BASE_OPTIONS.join(' ')]
87
- end
88
-
89
- # Adds the host, user and password, if present, to the command
90
- def add_connection_details
91
- @command.push("-h #{host}")
92
- @command.push("-u #{user}")
93
- @command.push("-p #{password}") if password.present?
94
- end
95
-
96
- # Returns the command as a string that can be executed in a shell
97
- #
98
- # @return [String]
99
- def to_s
100
- @command.join(' ')
101
- end
102
-
103
- # Returns the database host name, defaulting to localhost. If PERCONA_DB_HOST
104
- # is passed its value will be used instead
105
- #
106
- # @return [String]
107
- def host
108
- ENV['PERCONA_DB_HOST'] || connection_data[:host] || 'localhost'
109
- end
69
+ attr_reader :connection_details
110
70
 
111
- # Returns the database user. If PERCONA_DB_USER is passed its value will be
112
- # used instead
113
- #
114
- # @return [String]
115
- def user
116
- ENV['PERCONA_DB_USER'] || connection_data[:username]
71
+ def command
72
+ "#{COMMAND_NAME} #{connection_details}"
117
73
  end
118
74
 
119
- # Returns the database user's password. If PERCONA_DB_PASSWORD is passed its
120
- # value will be used instead
75
+ # Returns all the arguments to execute pt-online-schema-change with
121
76
  #
122
77
  # @return [String]
123
- def password
124
- ENV['PERCONA_DB_PASSWORD'] || connection_data[:password]
125
- end
126
-
127
- # TODO: Doesn't the abstract adapter already handle this somehow?
128
- # Returns the database name. If PERCONA_DB_NAME is passed its value will be
129
- # used instead
130
- def database
131
- ENV['PERCONA_DB_NAME'] || connection_data[:database]
78
+ def all_options
79
+ env_variable_options = UserOptions.new
80
+ global_configuration_options = UserOptions.new(Departure.configuration.global_percona_args)
81
+ options = env_variable_options.merge(global_configuration_options).merge(DEFAULT_OPTIONS)
82
+ options.to_a.join(' ')
132
83
  end
133
84
  end
134
85
  end
@@ -0,0 +1,96 @@
1
+ module Departure
2
+ # Executes the given command returning it's status and errors
3
+ class Command
4
+ COMMAND_NOT_FOUND = 127
5
+
6
+ # Constructor
7
+ #
8
+ # @param command_line [String]
9
+ # @param error_log_path [String]
10
+ # @param logger [#write_no_newline]
11
+ def initialize(command_line, error_log_path, logger)
12
+ @command_line = command_line
13
+ @error_log_path = error_log_path
14
+ @logger = logger
15
+ end
16
+
17
+ # Executes the command returning its status. It also prints its stdout to
18
+ # the logger and its stderr to the file specified in error_log_path.
19
+ #
20
+ # @raise [NoStatusError] if the spawned process' status can't be retrieved
21
+ # @raise [SignalError] if the spawned process received a signal
22
+ # @raise [CommandNotFoundError] if pt-online-schema-change can't be found
23
+ #
24
+ # @return [Process::Status]
25
+ def run
26
+ log_started
27
+
28
+ run_in_process
29
+
30
+ log_finished
31
+
32
+ validate_status!
33
+ status
34
+ end
35
+
36
+ private
37
+
38
+ attr_reader :command_line, :error_log_path, :logger, :status
39
+
40
+ # Runs the command in a separate process, capturing its stdout and
41
+ # execution status
42
+ def run_in_process
43
+ Open3.popen3(full_command) do |_stdin, stdout, _stderr, waith_thr|
44
+ begin
45
+ loop do
46
+ IO.select([stdout])
47
+ data = stdout.read_nonblock(8)
48
+ logger.write_no_newline(data)
49
+ end
50
+ rescue EOFError
51
+ # noop
52
+ ensure
53
+ @status = waith_thr.value
54
+ end
55
+ end
56
+ end
57
+
58
+ # Builds the actual command including stderr redirection to the specified
59
+ # log file
60
+ #
61
+ # @return [String]
62
+ def full_command
63
+ "#{command_line} 2> #{error_log_path}"
64
+ end
65
+
66
+ # Validates the status of the execution
67
+ #
68
+ # @raise [NoStatusError] if the spawned process' status can't be retrieved
69
+ # @raise [SignalError] if the spawned process received a signal
70
+ # @raise [CommandNotFoundError] if pt-online-schema-change can't be found
71
+ def validate_status!
72
+ raise SignalError.new(status) if status.signaled?
73
+ raise CommandNotFoundError if status.exitstatus == COMMAND_NOT_FOUND
74
+ raise Error, error_message unless status.success?
75
+ end
76
+
77
+ # Returns the error message that appeared in the process' stderr
78
+ #
79
+ # @return [String]
80
+ def error_message
81
+ File.read(error_log_path)
82
+ end
83
+
84
+ # Logs when the execution started
85
+ def log_started
86
+ logger.write("\n")
87
+ logger.say("Running #{command_line}\n\n", true)
88
+ end
89
+
90
+ # Prints a line break to keep the logs separate from the execution time
91
+ # print by the migration
92
+ def log_finished
93
+ logger.write("\n")
94
+ end
95
+ end
96
+ end
@@ -1,10 +1,11 @@
1
1
  module Departure
2
2
  class Configuration
3
- attr_accessor :tmp_path
3
+ attr_accessor :tmp_path, :global_percona_args
4
4
 
5
5
  def initialize
6
6
  @tmp_path = '.'.freeze
7
7
  @error_log_filename = 'departure_error.log'.freeze
8
+ @global_percona_args = nil
8
9
  end
9
10
 
10
11
  def error_log_path
@@ -0,0 +1,70 @@
1
+ module Departure
2
+ # Holds the parameters of the DB connection and formats them to string
3
+ class ConnectionDetails
4
+
5
+ # Constructor
6
+ #
7
+ # @param [Hash] connection parametes as used in #establish_conneciton
8
+ def initialize(connection_data)
9
+ @connection_data = connection_data
10
+ end
11
+
12
+ # Returns the details formatted as an string to be used with
13
+ # pt-online-schema-change. It follows the mysql client's format.
14
+ #
15
+ # @return [String]
16
+ def to_s
17
+ @to_s ||= "-h #{host} -u #{user} #{password_argument}"
18
+ end
19
+
20
+ # TODO: Doesn't the abstract adapter already handle this somehow?
21
+ # Returns the database name. If PERCONA_DB_NAME is passed its value will be
22
+ # used instead
23
+ #
24
+ # Returns the database name
25
+ #
26
+ # @return [String]
27
+ def database
28
+ ENV.fetch('PERCONA_DB_NAME', connection_data[:database])
29
+ end
30
+
31
+ # Returns the password fragment of the details string if a password is passed
32
+ #
33
+ # @return [String]
34
+ def password_argument
35
+ if password.present?
36
+ "-p #{password}"
37
+ else
38
+ ''
39
+ end
40
+ end
41
+
42
+ private
43
+
44
+ attr_reader :connection_data
45
+
46
+ # Returns the database host name, defaulting to localhost. If PERCONA_DB_HOST
47
+ # is passed its value will be used instead
48
+ #
49
+ # @return [String]
50
+ def host
51
+ ENV.fetch('PERCONA_DB_HOST', connection_data[:host]) || 'localhost'
52
+ end
53
+
54
+ # Returns the database user. If PERCONA_DB_USER is passed its value will be
55
+ # used instead
56
+ #
57
+ # @return [String]
58
+ def user
59
+ ENV.fetch('PERCONA_DB_USER', connection_data[:username])
60
+ end
61
+
62
+ # Returns the database user's password. If PERCONA_DB_PASSWORD is passed its
63
+ # value will be used instead
64
+ #
65
+ # @return [String]
66
+ def password
67
+ ENV.fetch('PERCONA_DB_PASSWORD', connection_data[:password])
68
+ end
69
+ end
70
+ end
@@ -0,0 +1,26 @@
1
+ module Departure
2
+
3
+ # Represents the 'DSN' argument of Percona's pt-online-schema-change
4
+ # See https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html#dsn-options
5
+ class DSN
6
+
7
+ # Constructor
8
+ #
9
+ # @param database [String, Symbol]
10
+ # @param table_name [String, Symbol]
11
+ def initialize(database, table_name)
12
+ @database = database
13
+ @table_name = table_name
14
+ end
15
+
16
+ # Returns the pt-online-schema-change DSN string. See
17
+ # https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html#dsn-options
18
+ def to_s
19
+ "D=#{database},t=#{table_name}"
20
+ end
21
+
22
+ private
23
+
24
+ attr_reader :table_name, :database
25
+ end
26
+ end
@@ -32,4 +32,8 @@ module Departure
32
32
  'Please install pt-online-schema-change. Check: https://www.percona.com/doc/percona-toolkit for further details'
33
33
  end
34
34
  end
35
+
36
+ # Used to prevent running the db:migrate rake task when providing arguments
37
+ # through PERCONA_ARGS env var
38
+ class ArgumentsNotSupported < Error; end
35
39
  end
@@ -0,0 +1,21 @@
1
+ module Departure
2
+ module LogSanitizers
3
+ class PasswordSanitizer
4
+ PASSWORD_REPLACEMENT = '[filtered_password]'
5
+
6
+ delegate :password_argument, to: :connection_details
7
+
8
+ def initialize(connection_details)
9
+ @connection_details = connection_details
10
+ end
11
+
12
+ def execute(log_statement)
13
+ return log_statement if password_argument.blank?
14
+ log_statement.gsub(password_argument, PASSWORD_REPLACEMENT)
15
+ end
16
+
17
+ private
18
+ attr_accessor :connection_details
19
+ end
20
+ end
21
+ end
@@ -5,6 +5,10 @@ module Departure
5
5
  # seen from the connection adapter.
6
6
  class Logger
7
7
 
8
+ def initialize(sanitizers)
9
+ @sanitizers = sanitizers
10
+ end
11
+
8
12
  # Outputs the message through the stdout, following the
9
13
  # ActiveRecord::Migration log format
10
14
  #
@@ -18,14 +22,22 @@ module Departure
18
22
  #
19
23
  # @param text [String]
20
24
  def write(text = '')
21
- puts(text)
25
+ puts(sanitize(text))
22
26
  end
23
27
 
24
28
  # Outputs the text through the stdout without adding a new line at the end
25
29
  #
26
30
  # @param text [String]
27
31
  def write_no_newline(text)
28
- print(text)
32
+ print(sanitize(text))
33
+ end
34
+
35
+ private
36
+
37
+ attr_accessor :sanitizers
38
+
39
+ def sanitize(text)
40
+ sanitizers.inject(text) { |memo, sanitizer| sanitizer.execute(memo) }
29
41
  end
30
42
  end
31
43
  end
@@ -6,9 +6,9 @@ module Departure
6
6
  #
7
7
  # @param verbose [Boolean]
8
8
  # @return [#say, #write]
9
- def self.build(verbose: true)
9
+ def self.build(sanitizers: [], verbose: true)
10
10
  if verbose
11
- Departure::Logger.new
11
+ Departure::Logger.new(sanitizers)
12
12
  else
13
13
  Departure::NullLogger.new
14
14
  end
@@ -0,0 +1,63 @@
1
+ module Departure
2
+ class Option
3
+ attr_reader :name, :value
4
+
5
+ # Builds an instance by parsing its name and value out of the given string.
6
+ # Note the string must conform to "--<arg>=<value>" format.
7
+ #
8
+ # @param string [String]
9
+ # @return [Option]
10
+ def self.from_string(string)
11
+ pair = string.split('=')
12
+ name = pair[0][2..-1]
13
+ value = pair[1]
14
+
15
+ new(name, value)
16
+ end
17
+
18
+ # Constructor
19
+ #
20
+ # @param name [String]
21
+ # @param optional value [String]
22
+ def initialize(name, value = nil)
23
+ @name = name
24
+ @value = value
25
+ end
26
+
27
+ # Compares two options
28
+ #
29
+ # @param [Option]
30
+ # @return [Boolean]
31
+ def ==(another_option)
32
+ name == another_option.name
33
+ end
34
+ alias :eql? :==
35
+
36
+ # Returns the option's hash
37
+ #
38
+ # @return [Fixnum]
39
+ def hash
40
+ name.hash
41
+ end
42
+
43
+ # Returns the option as string following the "--<name>=<value>" format
44
+ #
45
+ # @return [String]
46
+ def to_s
47
+ "--#{name}#{value_as_string}"
48
+ end
49
+
50
+ private
51
+
52
+ # Returns the value fragment of the option string if any value is specified
53
+ #
54
+ # @return [String]
55
+ def value_as_string
56
+ if value.nil?
57
+ ''
58
+ else
59
+ "=#{value}"
60
+ end
61
+ end
62
+ end
63
+ end
@@ -5,7 +5,6 @@ module Departure
5
5
  # It executes pt-online-schema-change commands in a new process and gets its
6
6
  # output and status
7
7
  class Runner
8
- COMMAND_NOT_FOUND = 127
9
8
 
10
9
  # Constructor
11
10
  #
@@ -17,8 +16,7 @@ module Departure
17
16
  @logger = logger
18
17
  @cli_generator = cli_generator
19
18
  @mysql_adapter = mysql_adapter
20
- @status = nil
21
- @config = config
19
+ @error_log_path = config.error_log_path
22
20
  end
23
21
 
24
22
  # Executes the passed sql statement using pt-online-schema-change for ALTER
@@ -27,8 +25,8 @@ module Departure
27
25
  # @param sql [String]
28
26
  def query(sql)
29
27
  if alter_statement?(sql)
30
- command = cli_generator.parse_statement(sql)
31
- execute(command)
28
+ command_line = cli_generator.parse_statement(sql)
29
+ execute(command_line)
32
30
  else
33
31
  mysql_adapter.execute(sql)
34
32
  end
@@ -45,18 +43,15 @@ module Departure
45
43
  # TODO: rename it so we don't confuse it with AR's #execute
46
44
  # Runs and logs the given command
47
45
  #
48
- # @param command [String]
46
+ # @param command_line [String]
49
47
  # @return [Boolean]
50
- def execute(command)
51
- @command = command
52
- logging { run_command }
53
- validate_status
54
- status
48
+ def execute(command_line)
49
+ Command.new(command_line, error_log_path, logger).run
55
50
  end
56
51
 
57
52
  private
58
53
 
59
- attr_reader :command, :logger, :status, :cli_generator, :mysql_adapter, :config
54
+ attr_reader :logger, :cli_generator, :mysql_adapter, :error_log_path
60
55
 
61
56
  # Checks whether the sql statement is an ALTER TABLE
62
57
  #
@@ -65,66 +60,5 @@ module Departure
65
60
  def alter_statement?(sql)
66
61
  sql =~ /\Aalter table/i
67
62
  end
68
-
69
- # Logs the start and end of the execution
70
- #
71
- # @yield
72
- def logging
73
- log_started
74
- yield
75
- log_finished
76
- end
77
-
78
- # Logs when the execution started
79
- def log_started
80
- logger.write("\n")
81
- logger.say("Running #{command}\n\n", true)
82
- end
83
-
84
- # Executes the command and prints its output to the stdout
85
- def run_command
86
- Open3.popen3("#{command} 2> #{error_log_path}") do |_stdin, stdout, _stderr, waith_thr|
87
- begin
88
- loop do
89
- IO.select([stdout])
90
- data = stdout.read_nonblock(8)
91
- logger.write_no_newline(data)
92
- end
93
- rescue EOFError
94
- # noop
95
- ensure
96
- @status = waith_thr.value
97
- end
98
- end
99
- end
100
-
101
- # Validates the status of the execution
102
- #
103
- # @raise [NoStatusError] if the spawned process' status can't be retrieved
104
- # @raise [SignalError] if the spawned process received a signal
105
- # @raise [CommandNotFoundError] if pt-online-schema-change can't be found
106
- def validate_status
107
- raise SignalError.new(status) if status.signaled?
108
- raise CommandNotFoundError if status.exitstatus == COMMAND_NOT_FOUND
109
- raise Error, error_message unless status.success?
110
- end
111
-
112
- # Prints a line break to keep the logs separate from the execution time
113
- # print by the migration
114
- def log_finished
115
- logger.write("\n")
116
- end
117
-
118
- # The path where the percona toolkit stderr will be written
119
- #
120
- # @return [String]
121
- def error_log_path
122
- config.error_log_path
123
- end
124
-
125
- # @return [String]
126
- def error_message
127
- File.read(error_log_path)
128
- end
129
63
  end
130
64
  end
@@ -0,0 +1,44 @@
1
+ module Departure
2
+ # Encapsulates the pt-online-schema-change options defined by the user
3
+ class UserOptions
4
+ delegate :each, :merge, to: :to_set
5
+
6
+ # Constructor
7
+ #
8
+ # @param arguments [String]
9
+ def initialize(arguments = ENV['PERCONA_ARGS'])
10
+ @arguments = arguments
11
+ end
12
+
13
+ private
14
+
15
+ attr_reader :arguments
16
+
17
+ # Returns the arguments the user defined but without duplicates
18
+ #
19
+ # @return [Set]
20
+ def to_set
21
+ Set.new(user_options)
22
+ end
23
+
24
+ # Returns Option instances from the arguments the user specified, if any
25
+ #
26
+ # @return [Array]
27
+ def user_options
28
+ if arguments
29
+ build_options
30
+ else
31
+ []
32
+ end
33
+ end
34
+
35
+ # Builds Option instances from the user arguments
36
+ #
37
+ # @return [Array<Option>]
38
+ def build_options
39
+ arguments.split(' ').map do |argument|
40
+ Option.from_string(argument)
41
+ end
42
+ end
43
+ end
44
+ end
@@ -1,3 +1,3 @@
1
1
  module Departure
2
- VERSION = '1.0.1'.freeze
2
+ VERSION = '2.0.0'.freeze
3
3
  end
@@ -19,7 +19,7 @@ class TestDatabase
19
19
  drop_and_create_schema_migrations_table
20
20
  end
21
21
 
22
- # Creates the test database and the comments table in it.
22
+ # Creates the #{database} database and the comments table in it.
23
23
  # Before, it drops both if they already exist
24
24
  def setup_test_database
25
25
  drop_and_create_test_database
metadata CHANGED
@@ -1,17 +1,18 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: departure
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.1
4
+ version: 2.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ilya Zayats
8
8
  - Pau Pérez
9
9
  - Fran Casas
10
10
  - Jorge Morante
11
+ - Adrian Serafin
11
12
  autorequire:
12
13
  bindir: bin
13
14
  cert_chain: []
14
- date: 2017-04-19 00:00:00.000000000 Z
15
+ date: 2017-04-20 00:00:00.000000000 Z
15
16
  dependencies:
16
17
  - !ruby/object:Gem::Dependency
17
18
  name: rails
@@ -19,26 +20,26 @@ dependencies:
19
20
  requirements:
20
21
  - - "~>"
21
22
  - !ruby/object:Gem::Version
22
- version: 3.2.22
23
+ version: 4.0.0
23
24
  type: :runtime
24
25
  prerelease: false
25
26
  version_requirements: !ruby/object:Gem::Requirement
26
27
  requirements:
27
28
  - - "~>"
28
29
  - !ruby/object:Gem::Version
29
- version: 3.2.22
30
+ version: 4.0.0
30
31
  - !ruby/object:Gem::Dependency
31
32
  name: mysql2
32
33
  requirement: !ruby/object:Gem::Requirement
33
34
  requirements:
34
- - - "~>"
35
+ - - '='
35
36
  - !ruby/object:Gem::Version
36
37
  version: 0.3.20
37
38
  type: :runtime
38
39
  prerelease: false
39
40
  version_requirements: !ruby/object:Gem::Requirement
40
41
  requirements:
41
- - - "~>"
42
+ - - '='
42
43
  - !ruby/object:Gem::Version
43
44
  version: 0.3.20
44
45
  - !ruby/object:Gem::Dependency
@@ -123,6 +124,20 @@ dependencies:
123
124
  - - ">="
124
125
  - !ruby/object:Gem::Version
125
126
  version: 8.2.1
127
+ - !ruby/object:Gem::Dependency
128
+ name: climate_control
129
+ requirement: !ruby/object:Gem::Requirement
130
+ requirements:
131
+ - - "~>"
132
+ - !ruby/object:Gem::Version
133
+ version: 0.0.3
134
+ type: :development
135
+ prerelease: false
136
+ version_requirements: !ruby/object:Gem::Requirement
137
+ requirements:
138
+ - - "~>"
139
+ - !ruby/object:Gem::Version
140
+ version: 0.0.3
126
141
  description: Execute your ActiveRecord migrations with Percona's pt-online-schema-change.
127
142
  Formerly known as Percona Migrator.
128
143
  email:
@@ -130,6 +145,7 @@ email:
130
145
  - pau.perez@redbooth.com
131
146
  - fran.casas@redbooth.com
132
147
  - jorge.morante@redbooth.com
148
+ - adrian@softmad.pl
133
149
  executables: []
134
150
  extensions: []
135
151
  extra_rdoc_files: []
@@ -142,23 +158,32 @@ files:
142
158
  - Gemfile
143
159
  - LICENSE.txt
144
160
  - README.md
161
+ - RELEASING.md
145
162
  - Rakefile
146
163
  - bin/console
164
+ - bin/rspec
147
165
  - bin/setup
148
166
  - config.yml
149
167
  - configuration.rb
150
168
  - departure.gemspec
169
+ - departure_error.log
151
170
  - lib/active_record/connection_adapters/percona_adapter.rb
152
171
  - lib/departure.rb
153
172
  - lib/departure/alter_argument.rb
154
173
  - lib/departure/cli_generator.rb
174
+ - lib/departure/command.rb
155
175
  - lib/departure/configuration.rb
176
+ - lib/departure/connection_details.rb
177
+ - lib/departure/dsn.rb
156
178
  - lib/departure/errors.rb
179
+ - lib/departure/log_sanitizers/password_sanitizer.rb
157
180
  - lib/departure/logger.rb
158
181
  - lib/departure/logger_factory.rb
159
182
  - lib/departure/null_logger.rb
183
+ - lib/departure/option.rb
160
184
  - lib/departure/railtie.rb
161
185
  - lib/departure/runner.rb
186
+ - lib/departure/user_options.rb
162
187
  - lib/departure/version.rb
163
188
  - lib/lhm.rb
164
189
  - lib/lhm/adapter.rb