jetpants 0.7.0

Sign up to get free protection for your applications and to get access to all the features.
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source :rubygems
2
+
3
+ gemspec
@@ -0,0 +1,88 @@
1
+ = Jetpants
2
+
3
+ == OVERVIEW:
4
+
5
+ \Jetpants is an automation toolkit for handling monstrously large MySQL database topologies. It is geared towards common operational tasks like cloning slaves, rebalancing shards, and performing master promotions. It features a command suite for easy use by operations staff, though it's also a full Ruby library for use in developing custom migration scripts and database automation.
6
+
7
+ \Jetpants supports a <b>range-based sharding scheme</b> for MySQL by providing a fast way to split shards that are approaching capacity or I/O limitations. \Jetpants is able to accomplish this without any locking, downtime, data inconsistency, or query failures. Dynamically resizable range-based sharding allows you to scale MySQL horizontally in a sane manner, without any need for a central lookup service or massive pre-allocation of tiny shards.
8
+
9
+ == MOTIVATION:
10
+
11
+ \Jetpants was created by {Tumblr}[http://www.tumblr.com/] to help manage our database infrastructure. It handles automation tasks for our entire database topology, which as of June 2012 consists of approximately:
12
+ * 200 dedicated database servers
13
+ * 12 global (unsharded) functional pools
14
+ * 45 shard pools
15
+ * 21 terabytes total of unique relational data on masters
16
+ * 60 billion total unique relational rows on masters
17
+
18
+ One of the primary requirements for \Jetpants was speed. On our hardware, <b>\Jetpants can divide a 750GB, billion-row shard in half in about six hours</b> -- or even faster if you're diving into thirds or fourths. It can also <b>clone slaves at line speed on gigabit ethernet</b>, including to multiple destinations at once, using a novel "chained copy" approach.
19
+
20
+ For more background on the initial motivations behind \Jetpants, please see {Evan Elias's presentation at Velocity Europe 2011}[https://github.com/tumblr/jetpants/blob/master/doc/VelocityEurope2011Presentation.pdf?raw=true].
21
+
22
+ == COMMAND SUITE FEATURES:
23
+
24
+ The \Jetpants command suite offers easy command-line interaction with complex MySQL automation tasks.
25
+
26
+ * Clone slaves efficiently, including to multiple targets simultaneously
27
+ * Split a range-based shard into N new shards with zero downtime and no failed queries
28
+ * Perform master promotions and other pool topology changes
29
+ * Defragment tables quickly in parallelized chunks
30
+ * Interact with your database topology in a REPL environment via <tt>jetpants console</tt> mode
31
+
32
+ For more information on the command suite, please see doc/commands.rdoc ({view on GitHub}[https://github.com/tumblr/jetpants/blob/master/doc/commands.rdoc]).
33
+
34
+ == LIBRARY FEATURES:
35
+
36
+ \Jetpants is also a Ruby module which you can use to build complex database migration scripts and other customized automation. It provides object modeling for databases, hosts, global/functional pools, sharded pools, and your database topology as a whole.
37
+
38
+ * Utilize scriptable versions of all command suite functionality
39
+ * Crawl replication topology programmatically
40
+ * Import or export arbitrary portions of a data set
41
+ * Copy large files quickly and efficiently, including to multiple simultaneous destinations
42
+ * Manipulate server settings or concurrently execute arbitrary UNIX commands / administrative MySQL queries on multiple servers
43
+
44
+
45
+ == ASSUMPTIONS AND REQUIREMENTS:
46
+
47
+ The base classes of \Jetpants currently make a number of assumptions about your environment and database topology. Please see doc/requirements.rdoc ({view on GitHub}[https://github.com/tumblr/jetpants/blob/master/doc/requirements.rdoc]).
48
+
49
+
50
+ == CONFIGURATION:
51
+
52
+ \Jetpants supports a global configuration file at <tt>/etc/jetpants.yaml</tt>, as well as per-user configuration files at <tt>~/.jetpants.yaml</tt>.
53
+
54
+ At least one of these files must exist for \Jetpants to function properly, since certain options (database schema name, database credentials, etc) are mandatory and cannot be inferred.
55
+
56
+ Please see doc/configuration.rdoc ({view on GitHub}[https://github.com/tumblr/jetpants/blob/master/doc/configuration.rdoc]) for information on configuring \Jetpants.
57
+
58
+
59
+ == PLUGINS:
60
+
61
+ \Jetpants offers an extensible plugin system. Plugins are Ruby code (such as stand-alone gems) that add to \Jetpants by supplying callback methods, and/or overriding core methods.
62
+
63
+ It is highly recommended that you tie \Jetpants into your site's asset tracker / hardware management system by writing a custom plugin. This will allow \Jetpants to automatically know what database pools and shards are present, and to make topological changes immediately be reflected in your site's configuration. <b>Several complex \Jetpants features (including shard splits) actually require an asset tracker plugin in order to function, since these processes involve obtaining spare nodes and manipulating multiple pools in your database topology.</b>
64
+
65
+ Other recommended uses of plugins include integration with your site's monitoring system, trending system, query killers, and environment-specific overrides to various core methods.
66
+
67
+ For more information on how to write plugins and use the Jetpants::CallbackHandler system, please see doc/plugins.rdoc ({view on GitHub}[https://github.com/tumblr/jetpants/blob/master/doc/plugins.rdoc])
68
+
69
+ == CREDITS:
70
+
71
+ * <b>Evan Elias</b>: Lead developer. Core class implementations, shard split logic, plugin system
72
+ * <b>Dallas Marlow</b>: Master promotion logic, command suite and console structure, MySQL internals expertise
73
+
74
+ == LICENSE:
75
+
76
+ Copyright 2012 Tumblr, Inc.
77
+
78
+ Licensed under the Apache License, Version 2.0 (the "License");
79
+ you may not use this file except in compliance with the License.
80
+ You may obtain a copy of the License at
81
+
82
+ [http://www.apache.org/licenses/LICENSE-2.0]
83
+
84
+ Unless required by applicable law or agreed to in writing, software
85
+ distributed under the License is distributed on an "AS IS" BASIS,
86
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
87
+ See the License for the specific language governing permissions and
88
+ limitations under the License.
@@ -0,0 +1,442 @@
1
+ #!/usr/bin/env ruby
2
+ jetpants_base_dir = File.expand_path(File.dirname(__FILE__) + '/..')
3
+ $:.unshift File.join(jetpants_base_dir, 'lib')
4
+ %w[thor pry state_machine highline/import terminal-table colored].each {|g| require g}
5
+ # load tasks
6
+ Dir[File.join jetpants_base_dir, 'tasks', '**'].each {|f| require f}
7
+
8
+ module Jetpants
9
+
10
+ class CommandSuite < Thor
11
+
12
+ def initialize *args
13
+ super
14
+ # setup pry
15
+ Pry.config.prompt = proc {|object| "# #{object} > "}
16
+ end
17
+
18
+ # Override Thor.dispatch to allow simple callbacks, which must be before_foo /
19
+ # after_foo *class* methods of Jetpants::CommandSuite.
20
+ # These aren't as full-featured as normal Jetpants::Callback: you can only have
21
+ # ONE before_foo or after_foo method (they override instead of stacking); no arg
22
+ # passing; no callback abort exception type. Mostly useful for plugins overriding
23
+ # reminder text before or after a task.
24
+ def self.dispatch(task, given_args, given_ops, config)
25
+ task_name = task || given_args[0]
26
+ self.send "before_#{task_name}" if self.respond_to? "before_#{task_name}"
27
+ super
28
+ self.send "after_#{task_name}" if self.respond_to? "after_#{task_name}"
29
+ end
30
+
31
+ desc 'console', 'Jetpants interactive console'
32
+ def console
33
+ Jetpants.pry
34
+ end
35
+ def self.before_console
36
+ message = [ 'Welcome to the Jetpants Console. A few notes:',
37
+ ' - Jetpants interacts with databases via ssh and the root user (BE CAREFUL).',
38
+ ' - Jetpants relies on having access to a root ssh key in order to perform these operations.',
39
+ ' - This console operates from inside the Jetpants module namespace by default (Jetpants::DB.new == DB.new, etc).',
40
+ ' - Jetpants uses a global config file at /etc/jetpants.yaml, or a user override config file at ~/.jetpants.yaml.',
41
+ ].join "\n"
42
+ print "\n#{message}\n\n"
43
+ end
44
+
45
+ desc 'promotion', 'perform a master promotion'
46
+ method_option :demote, :desc => 'node to demote'
47
+ method_option :promote, :desc => 'node to promote'
48
+ def promotion
49
+ Tasks::Promotion.new(options)
50
+ end
51
+
52
+ desc 'show_slaves', 'show the current slaves of a master'
53
+ method_option :node, :desc => 'node to query for slaves'
54
+ def show_slaves
55
+ node = options[:node] || ask('Please enter IP of node to query for slaves: ')
56
+ error "node (#{node}) does not appear to be an IP." unless is_ip? node
57
+ node = Jetpants::DB.new node
58
+ slaves = node.slaves rescue error("unable to connect to node #{node}")
59
+
60
+ inform "node (#{node}) is a current slave of #{node.master}" if node.is_slave?
61
+
62
+ if node.has_slaves?
63
+ current_slaves = Terminal::Table.new :title => "slaves of master (#{node})", :headings => ["slave", "seconds behind master"] do |rows|
64
+ slaves.each do |slave|
65
+ rows << [slave, slave.seconds_behind_master]
66
+ end
67
+ end
68
+ puts current_slaves
69
+ else
70
+ inform "node (#{node}) currently has no slaves."
71
+ end
72
+ end
73
+
74
+ desc 'show_master', 'show the current master of a node'
75
+ method_option :node, :desc => 'node to query for master'
76
+ method_option :siblings, :desc => 'show nodes current slave siblings'
77
+ def show_master
78
+ node = options[:node] || ask('Please enter the IP address of a node to query for master: ')
79
+ error "node (#{node}) does not appear to be an IP." unless is_ip? node
80
+ node = Jetpants::DB.new node
81
+
82
+ if node.has_slaves?
83
+ inform "node (#{node}) is a master to the following nodes: #{node.slaves.join(', ')}"
84
+ end rescue error("unable to connect to node #{node}")
85
+
86
+ if node.is_slave?
87
+ inform "node (#{node}) is a slave of master (#{node.master})"
88
+ if options[:siblings]
89
+ current_siblings = Terminal::Table.new :title => "siblings of slave (#{node})", :headings => ["sibling slave", "seconds behind master (#{node.master})"] do |rows|
90
+ node.master.slaves.reject {|slave| slave == node}.each do |slave|
91
+ rows << [slave, slave.seconds_behind_master]
92
+ end
93
+ end
94
+ puts current_siblings
95
+ end
96
+ else
97
+ inform "node (#{node}) does not appear to be a slave"
98
+ end
99
+ end
100
+
101
+ desc 'node_info', 'show information about a given node'
102
+ method_option :node, :desc => 'node to query for information'
103
+ def node_info
104
+ node = options[:node] || ask('Please enter node: ')
105
+ error "node address (#{node}) does not appear to be an ip" unless is_ip? node
106
+
107
+ node = Jetpants::DB.new node
108
+ role = case
109
+ when node.has_slaves?
110
+ :master
111
+ when node.is_slave?
112
+ :slave
113
+ else
114
+ :node
115
+ end rescue error("unable to connect to node #{node}")
116
+
117
+ node_info = Terminal::Table.new :title => "info on #{role}: #{node}"
118
+
119
+ binary_log, binary_log_position = node.binlog_coordinates
120
+ node_info << [:binary_log, binary_log]
121
+ node_info << [:binary_log_position, binary_log_position]
122
+ node_info << [:read_only, node.read_only? ? 'true' : 'false']
123
+
124
+ if node.is_slave?
125
+ slave_status = node.slave_status
126
+ slave_siblings = node.master.slaves.reject {|slave| slave == node}
127
+
128
+ node_info << [:master, node.master]
129
+ node_info << [:sibling_slaves, slave_siblings.join(', ')]
130
+ node_info << [:replicating, node.replicating? ? 'true' : 'false']
131
+ if node.replicating?
132
+ node_info << [:seconds_behind_master, node.seconds_behind_master]
133
+ node_info << [:master_log, slave_status[:master_log_file]]
134
+ node_info << [:master_position, slave_status[:exec_master_log_pos]]
135
+ end
136
+ end
137
+ puts node_info
138
+
139
+ if node.has_slaves?
140
+ current_slaves = Terminal::Table.new :title => "slaves of #{node}", :headings => ["slave", "seconds behind master"] do |rows|
141
+ node.slaves.each do |slave|
142
+ rows << [slave, slave.seconds_behind_master]
143
+ end
144
+ end
145
+ puts current_slaves
146
+ end
147
+ end
148
+
149
+ desc 'regen_config', 'regenerate the application configuration'
150
+ def regen_config
151
+ Jetpants.topology.write_config
152
+ end
153
+
154
+ desc 'clone_slave', 'clone a standby slave'
155
+ method_option :source, :desc => 'IP of node to clone from'
156
+ method_option :target, :desc => 'IP of node to clone to'
157
+ def clone_slave
158
+ puts "This task clones the data set of a standby slave."
159
+ source = options[:source] || ask('Please enter IP of node to clone from: ')
160
+ error "source (#{source}) does not appear to be an IP." unless is_ip? source
161
+ source = Jetpants::DB.new source
162
+ target = options[:target] || ask('Please enter comma-separated list of IP addresses to clone to: ')
163
+ targets = target.split(',').map do |ip|
164
+ ip.strip!
165
+ error "target (#{ip}) does not appear to be an IP." unless is_ip? ip
166
+ ip.to_db
167
+ end
168
+
169
+ source.start_mysql if ! source.running?
170
+ error "source (#{source}) is not a standby slave" unless source.is_standby?
171
+ source.enslave_siblings!(targets)
172
+ targets.concurrent_each {|t| t.resume_replication; t.catch_up_to_master}
173
+ source.pool.sync_configuration
174
+ puts "Cloning complete."
175
+ Jetpants.topology.write_config
176
+ end
177
+ def self.after_clone_slave
178
+ reminders(
179
+ 'Add the new host(s) to trending and monitoring.'
180
+ )
181
+ end
182
+
183
+ desc 'activate_slave', 'turn a standby slave into an active slave'
184
+ method_option :node, :desc => 'IP of standby slave to activate'
185
+ def activate_slave
186
+ puts "This task turns a standby slave into an active slave, OR alters an active slave's weight."
187
+ node = options[:node] || ask('Please enter node IP: ')
188
+ weight = options[:weight] || ask('Please enter weight, or ENTER for default of 100: ')
189
+ weight = 100 if weight == ''
190
+ weight = weight.to_i
191
+ error "node address (#{node}) does not appear to be an IP" unless is_ip? node
192
+ error "Adding a slave of weight 0 makes no sense, use pull_slave instead" if weight == 0
193
+ node = node.to_db
194
+ node.pool.mark_slave_active(node, weight)
195
+ Jetpants.topology.write_config
196
+ end
197
+
198
+ desc 'weigh_slave', 'change the weight of an active slave'
199
+ alias :weigh_slave :activate_slave
200
+
201
+ desc 'pull_slave', 'turn an active slave into a standby slave'
202
+ method_option :node, :desc => 'IP of active slave to pull'
203
+ def pull_slave
204
+ puts "This task turns an active slave into a standby slave."
205
+ node = options[:node] || ask('Please enter node IP: ')
206
+ error "node address (#{node}) does not appear to be an ip" unless is_ip? node
207
+ node = node.to_db
208
+ node.pool.mark_slave_standby(node)
209
+ Jetpants.topology.write_config
210
+ end
211
+
212
+ desc 'destroy_slave', 'remove a standby slave from its pool'
213
+ method_option :node, :desc => 'IP of standby slave to remove'
214
+ def destroy_slave
215
+ puts "This task removes a standby slave from its pool entirely. THIS IS PERMANENT, ie, it does a RESET SLAVE on the target."
216
+ node = options[:node] || ask('Please enter node IP: ')
217
+ error "node address (#{node}) does not appear to be an IP" unless is_ip? node
218
+ node = node.to_db
219
+ raise "Node is not a standby slave" unless node.is_standby?
220
+ raise "Aborting" unless ask('Please type YES in all capital letters to confirm: ') == 'YES'
221
+ node.pool.remove_slave!(node)
222
+ end
223
+
224
+ desc 'rebuild_slave', 'export and re-import data set on a standby slave'
225
+ method_option :node, :desc => 'IP of standby slave to rebuild'
226
+ def rebuild_slave
227
+ puts "This task exports all data on a standby/backup slave and then re-imports it."
228
+ node = options[:node] || ask('Please enter node IP: ')
229
+ error "node address (#{node}) does not appear to be an ip" unless is_ip? node
230
+ node = node.to_db
231
+ raise "Node is not a standby or backup slave" unless node.is_standby? || node.for_backups?
232
+ raise "Cannot rebuild non-shard slaves from command suite; use jetpants console instead" unless node.pool.is_a?(Shard)
233
+ node.rebuild!
234
+ end
235
+
236
+ desc 'shard_read_only', 'mark a shard as read-only'
237
+ method_option :min_id, :desc => 'Minimum ID of shard to mark as read-only'
238
+ def shard_read_only
239
+ shard_min = options[:min_id] || ask('Please enter min ID of the shard: ')
240
+ s = Jetpants.topology.shard shard_min
241
+ raise "Shard not found" unless s
242
+ s.state = :read_only
243
+ s.sync_configuration
244
+ Jetpants.topology.write_config
245
+ end
246
+
247
+ desc 'shard_offline', 'mark a shard as offline (not readable or writable)'
248
+ method_option :min_id, :desc => 'Minimum ID of shard to mark as offline'
249
+ def shard_offline
250
+ shard_min = options[:min_id] || ask('Please enter min ID of the shard: ')
251
+ s = Jetpants.topology.shard shard_min
252
+ raise "Shard not found" unless s
253
+ s.state = :offline
254
+ s.sync_configuration
255
+ Jetpants.topology.write_config
256
+ end
257
+
258
+ desc 'shard_online', 'mark a shard as fully online (readable and writable)'
259
+ method_option :min_id, :desc => 'Minimum ID of shard to mark as fully online'
260
+ def shard_online
261
+ shard_min = options[:min_id] || ask('Please enter min ID of the shard: ')
262
+ s = Jetpants.topology.shard shard_min
263
+ raise "Shard not found" unless s
264
+ s.state = :ready
265
+ s.sync_configuration
266
+ Jetpants.topology.write_config
267
+ end
268
+
269
+ desc 'shard_split', 'shard split step 1 of 4: spin up child pools with different portions of data set'
270
+ method_option :min_id, :desc => 'Minimum ID of parent shard to split'
271
+ method_option :max_id, :desc => 'Maximum ID of parent shard to split'
272
+ method_option :ranges, :desc => 'Optional comma-separated list of ranges per child ie "1000-1999,2000-2499" (default if omitted: split evenly)'
273
+ method_option :count, :desc => 'How many child shards to split the parent into (only necessary if the ranges option is omitted)'
274
+ def shard_split
275
+ shard_min = options[:min_id] || ask('Please enter min ID of the parent shard: ')
276
+ shard_max = options[:max_id] || ask('Please enter max ID of the parent shard: ')
277
+ s = Jetpants.topology.shard shard_min, shard_max
278
+
279
+ raise "Shard not found" unless s
280
+ raise "Shard isn't in ready state" unless s.state == :ready
281
+
282
+ ranges = options[:ranges] || ask('Optionally enter comma-separated list of ranges per child ie "1000-1999,2000-2499" [default=even split]: ')
283
+ ranges = false if ranges.strip == ''
284
+
285
+ if ranges
286
+ supply_ranges = []
287
+ ranges.split(',').each do |my_range|
288
+ my_range.strip!
289
+ my_min, my_max = my_range.split('-', 2)
290
+ my_min = my_min.strip.to_i
291
+ my_max = my_max.strip.to_i
292
+ raise "Supplied range list has gaps!" if supply_ranges.last && supply_ranges.last[1] + 1 != my_min
293
+ supply_ranges << [my_min, my_max]
294
+ end
295
+ children = supply_ranges.count
296
+ raise "Supplied range does not cover parent completely!" if supply_ranges.first[0] != shard_min.to_i || supply_ranges.last[1] != shard_max.to_i
297
+ s.init_children(children, supply_ranges)
298
+ s.split!
299
+ else
300
+ children = options[:count] || ask('Optionally enter how many children to split into [default=2]: ')
301
+ children = 2 if children == ''
302
+ s.split!(children.to_i)
303
+ end
304
+ end
305
+ def self.before_shard_split
306
+ reminders(
307
+ 'This process may take several hours. You probably want to run this from a screen session.',
308
+ 'Be especially careful if you are relying on SSH Agent Forwarding for your root key, since this is not screen-friendly.'
309
+ )
310
+ end
311
+ def self.after_shard_split
312
+ reminders(
313
+ 'Trending and monitoring setup, as needed.',
314
+ 'Proceed to next step: jetpants shard_split_child_reads'
315
+ )
316
+ end
317
+
318
+ # This step is only really necessary if asset-tracker changes don't immediately reflect in application configuration.
319
+ # (ie, if app configuration is a static file that needs to be deployed to webs.)
320
+ desc 'shard_split_child_reads', 'shard split step 2 of 4: move reads to child shards'
321
+ def shard_split_child_reads
322
+ Jetpants.topology.write_config
323
+ end
324
+ def self.after_shard_split_child_reads
325
+ reminders(
326
+ 'Commit/push the configuration in version control.',
327
+ 'Deploy the configuration to all machines.',
328
+ 'Wait for reads to stop on the old parent master.',
329
+ 'Proceed to next step: jetpants shard_split_child_writes'
330
+ )
331
+ end
332
+
333
+ desc 'shard_split_child_writes', 'shard split step 3 of 4: move writes to child shards'
334
+ method_option :min_id, :desc => 'Minimum ID of parent shard being split'
335
+ method_option :max_id, :desc => 'Maximum ID of parent shard being split'
336
+ def shard_split_child_writes
337
+ shard_min = options[:min_id] || ask('Please enter min ID of the parent shard: ')
338
+ shard_max = options[:max_id] || ask('Please enter max ID of the parent shard: ')
339
+ s = Jetpants.topology.shard shard_min, shard_max
340
+ raise "Shard not found" unless s
341
+ raise "Shard isn't in expected state" unless s.state == :deprecated && s.children.count > 1
342
+ s.move_writes_to_children
343
+ Jetpants.topology.write_config
344
+ end
345
+ def self.after_shard_split_child_writes
346
+ reminders(
347
+ 'Commit/push the configuration in version control.',
348
+ 'Deploy the configuration to all machines.',
349
+ 'Wait for writes to stop on the old parent master.',
350
+ 'Proceed to next step: jetpants shard_split_cleanup',
351
+ )
352
+ end
353
+
354
+ desc 'shard_split_cleanup', 'shard split step 4 of 4: clean up data that replicated to wrong shard'
355
+ method_option :min_id, :desc => 'Minimum ID of parent shard being split'
356
+ method_option :max_id, :desc => 'Maximum ID of parent shard being split'
357
+ def shard_split_cleanup
358
+ shard_min = options[:min_id] || ask('Please enter min ID of the parent shard: ')
359
+ shard_max = options[:max_id] || ask('Please enter max ID of the parent shard: ')
360
+ s = Jetpants.topology.shard shard_min, shard_max
361
+ raise "Shard not found" unless s
362
+ raise "Shard isn't in expected state" unless s.state == :deprecated && s.children.count > 1
363
+ s.cleanup!
364
+ end
365
+ def self.after_shard_split_cleanup
366
+ reminders(
367
+ 'Review old nodes for hardware issues before re-using, or simply cancel them.',
368
+ )
369
+ end
370
+
371
+ desc 'shard_cutover', 'truncate the current last shard range, and add a new shard after it'
372
+ method_option :cutover_id, :desc => 'Minimum ID of new last shard being created'
373
+ def shard_cutover
374
+ cutover_id = options[:cutover_id] || ask('Please enter min ID of the new shard to be created: ')
375
+ cutover_id = cutover_id.to_i
376
+ last_shard = Jetpants.topology.shards.select {|s| s.max_id == 'INFINITY' && s.in_config?}.first
377
+ last_shard_master = last_shard.master
378
+
379
+ # In asset tracker, remove the last shard pool and replace it with a new pool. The new pool
380
+ # has the same master/slaves but now has a non-infinity max ID.
381
+ last_shard.state = :recycle
382
+ last_shard.sync_configuration
383
+ last_shard_replace = Shard.new(last_shard.min_id, cutover_id - 1, last_shard_master)
384
+ last_shard_replace.sync_configuration
385
+ Jetpants.topology.pools << last_shard_replace
386
+
387
+ # Now put another new shard after that one
388
+ new_last_shard_master = Jetpants.topology.claim_spare(role: 'master')
389
+ new_last_shard_slaves = Jetpants.topology.claim_spares(Jetpants.standby_slaves_per_pool, role: 'standby_slave')
390
+ new_last_shard_slaves.each do |x|
391
+ x.change_master_to new_last_shard_master
392
+ x.resume_replication
393
+ end
394
+ new_last_shard = Shard.new(cutover_id, 'INFINITY', new_last_shard_master)
395
+ new_last_shard.sync_configuration
396
+ Jetpants.topology.pools << new_last_shard
397
+
398
+ # Create tables on the new shard's master, obtaining schema from previous last shard
399
+ tables = Table.from_config 'sharded_tables'
400
+ last_shard_master.export_schemata tables
401
+ last_shard_master.host.fast_copy_chain(Jetpants.export_location, new_last_shard_master.host, files: ["create_tables_#{last_shard_master.port}.sql"])
402
+ new_last_shard_master.import_schemata!
403
+
404
+ # regen config file
405
+ Jetpants.topology.write_config
406
+ end
407
+ def self.after_shard_cutover
408
+ reminders(
409
+ 'Trending and monitoring setup, as needed.',
410
+ 'Commit/push the configuration in version control.',
411
+ 'Deploy the configuration to all machines.',
412
+ )
413
+ end
414
+
415
+ no_tasks do
416
+ def is_ip? address
417
+ address =~ /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/
418
+ end
419
+
420
+ def error message
421
+ abort ['ERROR:'.red, message].join ' '
422
+ end
423
+
424
+ def inform message
425
+ puts message.blue
426
+ end
427
+ end
428
+
429
+ def self.reminders(*strings)
430
+ strings.map! {|s| " - #{s}"}
431
+ strings.flatten!
432
+ reminder_text = strings.join "\n"
433
+ noun = (strings.count == 1 ? 'REMINDER' : 'REMINDERS')
434
+ print "\n#{noun}:\n#{reminder_text}\n\n" if strings.count > 0
435
+ end
436
+ end
437
+ end
438
+
439
+ # We load jetpants last so that plugins can monkeypatch Jetpants::CommandSuite if desired.
440
+ require 'jetpants'
441
+
442
+ Jetpants::CommandSuite.start