jetpants 0.7.8 → 0.7.10

data/README.rdoc

@@ -8,16 +8,16 @@
 
 == MOTIVATION:
 
-\Jetpants was created by {Tumblr}[http://www.tumblr.com/] to help manage our database infrastructure. It handles automation tasks for our entire database topology, which as of June 2012 consists of approximately:
+\Jetpants was created by {Tumblr}[http://www.tumblr.com/] to help manage our database infrastructure. It handles automation tasks for our entire database topology, which as of October 2012 consists of approximately:
 * 200 dedicated database servers
-* 12 global (unsharded) functional pools 
-* 45 shard pools
-* 21 terabytes total of unique relational data on masters
-* 60 billion total unique relational rows on masters
+* 5 global (unsharded) functional pools 
+* 58 shard pools
+* 28 terabytes total of unique relational data on masters
+* 100 billion total unique relational rows on masters
 
 One of the primary requirements for \Jetpants was speed. On our hardware, <b>\Jetpants can divide a 750GB, billion-row shard in half in about six hours</b> -- or even faster if you're diving into thirds or fourths. It can also <b>clone slaves at line speed on gigabit ethernet</b>, including to multiple destinations at once, using a novel "chained copy" approach.
 
-For more background on the initial motivations behind \Jetpants, please see {Evan Elias's presentation at Velocity Europe 2011}[https://github.com/tumblr/jetpants/blob/master/doc/VelocityEurope2011Presentation.pdf?raw=true].
+For more background on the initial motivations behind \Jetpants, please see {Evan Elias's presentation at Percona Live NYC 2012}[https://github.com/tumblr/jetpants/blob/master/doc/PerconaLiveNYC2012Presentation.pdf?raw=true].
 
 == COMMAND SUITE FEATURES:
 
@@ -64,6 +64,8 @@ It is highly recommended that you tie \Jetpants into your site's asset tracker /
 
 Other recommended uses of plugins include integration with your site's monitoring system, trending system, query killers, and environment-specific overrides to various core methods.
 
+If you are using \Collins for asset management, \Jetpants now ships with a plugin that offers integration. Please see doc/jetpants_collins.rdoc ({view on GitHub}[https://github.com/tumblr/jetpants/blob/master/doc/jetpants_collins.rdoc]) for usage.
+
 For more information on how to write plugins and use the Jetpants::CallbackHandler system, please see doc/plugins.rdoc ({view on GitHub}[https://github.com/tumblr/jetpants/blob/master/doc/plugins.rdoc])
 
 == FREQUENTLY ASKED QUESTIONS:

data/bin/jetpants

@@ -170,10 +170,10 @@ module Jetpants
     def pools_compact
       puts
       Jetpants.shards.each do |s| 
-        puts "[%-12s] %8s to %-11s = %3s GB" % [s.ip, s.min_id, s.max_id, s.data_set_size(true)]
+        puts "[%-15s] %8s to %-11s = %3s GB" % [s.ip, s.min_id, s.max_id, s.data_set_size(true)]
       end
       Jetpants.functional_partitions.each do |p| 
-        puts "[%-12s] %-23s = %3s GB" % [p.ip, p.name, p.data_set_size(true)]
+        puts "[%-15s] %-23s = %3s GB" % [p.ip, p.name, p.data_set_size(true)]
       end
       puts
     end

data/doc/commands.rdoc

@@ -12,9 +12,9 @@ Here's a more thorough description of the commands, grouped by function:
 
 \Jetpants copies data sets by shutting down the MySQL daemon on the source node and then copying all MySQL files. This is the fastest way to clone MySQL data sets, and is part of the reason why we recommend having 2 standby slaves per pool for high availability.
 
-The copy method in \Jetpants uses a combination of tar, netcat (nc), and pigz. It does not use encryption; we assume you are transferring over a secure local network.  When copying to multiple destinations, \Jetpants creates a "copy chain" using tee and a fifo. For more information on this technique, please see {our post on the Tumblr engineering blog}[http://engineering.tumblr.com/post/7658008285/efficiently-copying-files-to-multiple-destinations].
+The copy method in \Jetpants uses a combination of tar, netcat (nc), and whichever compression binary you have specified in your Jetpants configuration file (if any). It does not use encryption; we assume you are transferring over a secure local network.  When copying to multiple destinations, \Jetpants creates a "copy chain" using tee and a fifo. For more information on this technique, please see {our post on the Tumblr engineering blog}[http://engineering.tumblr.com/post/7658008285/efficiently-copying-files-to-multiple-destinations].
 
-This command does not require an asset tracker plugin, but DOES require that all your database nodes have pigz (a parallel gzip tool) installed. This may become configurable in a future release.
+This command does not require an asset tracker plugin, but DOES require that all your database nodes have installed whichever compression binary you specified in the Jetpants config file.
 
 
 == Master/slave state changes
@@ -73,7 +73,7 @@ Or if you're deploying a brand new pool in an existing topology:
 5. Stop replicating writes from the parent shard, and then take the parent pool offline entirely.
 6. Remove rows that replicated to the wrong child shard. This data will be sparse, since it's only the writes that were made since the shard split process started.
 
-For more information, including diagrams of each step, please see {our presentation at Velocity Europe 2011}[https://github.com/tumblr/jetpants/blob/master/doc/VelocityEurope2011Presentation.pdf?raw=true].
+For more information, including diagrams of each step, please see {our presentation at Percona Live NYC 2012}[https://github.com/tumblr/jetpants/blob/master/doc/PerconaLiveNYC2012Presentation.pdf?raw=true].
 
 Separately, \Jetpants also allows you to alter the range of the last shard in your topology. In a range-based sharding scheme, the last shard has a range of X to infinity; eventually this will be too large of a range, so you need to truncate that shard range and create a "new" last shard after it.  We call this process "shard cutover".
 

data/doc/configuration.rdoc

@@ -20,8 +20,48 @@ mysql_repl_password::       mysql password for replication (mandatory)
 mysql_root_password::       mysql root password (default: false, indicating that \Jetpants should use /root/.my.cnf instead)
 mysql_grant_ips::           mysql user manipulations are applied to these IPs (array; mandatory)
 mysql_grant_privs::         mysql user manipulations grant this set of privileges by default (array; default: \['ALL'])
+compress_with::             command line to perform compression during large file copy operations; see below (default: false)
+decompress_with::           command line to perform decompression during large file copy operations; see below (default: false)
 export_location::           directory to use for data dumping (default: '/tmp')
 verify_replication::        raise exception if the actual replication topology differs from Jetpants' understanding of it (ie, disagreement between asset tracker and probed state), or if MySQL's two replication threads are in different states (one running and the other stopped) on a DB node. (default: true. master promotion tool ignores this, since the demoted master may legitimately be dead/offline)
 plugins::                   hash of plugin name => arbitrary plugin data, usually a nested hash of settings (default: \{})
 ssh_keys::                  array of SSH private key file locations, if not using standard id_dsa or id_rsa. Passed directly to Net::SSH.start's :keys parameter (default: nil)
 sharded_tables::            array of name => \{sharding_key=>X, chunks=>Y} hashes, describing all tables on shards. Required by shard split/rebuild processes (default: \[])
+
+== Compression
+
+\Jetpants has the ability to use compression during large file copy operations, which are performed by commands "jetpants clone_slave" and "jetpants shard_split". Compression is disabled by default in \Jetpants unless you specify a compression program to use via the <tt>compress_with</tt> and <tt>decompress_with</tt> config options. It is highly recommended that you do so, in order to speed up these operations when working with large data sets.
+
+The command lines that you specify should accept input from STDIN and supply output to STDOUT, because they will be used in the middle of a series of piped commands. The binary specified should be in root's PATH on all database nodes. We recommend use of a parallel compression tool, to take advantage of multiple cores.
+
+You will need to do some profiling to determine the best tool to use for your hardware and data set; there's no universal best choice of compression algorithm or settings.
+
+Some example values of these parameters are as follows:
+
+=== Disable compression (default)
+
+  compress_with: false
+  decompress_with: false
+
+=== pigz
+pigz is an open-source parallel gzip tool by Mark Adler. It is available as a package in several Linux distros. It performs well, but is very CPU intensive. More information: http://zlib.net/pigz/
+
+  compress_with: pigz
+  decompress_with: pigz -d
+
+=== qpress
+qpress is a multi-threaded portable file archiver using QuickLZ. A prebuilt package is not available for most Linux distros due to licensing reasons, but a binary can be downloaded from http://www.quicklz.com/. It performs extremely well, especially once tuned.
+
+In order to read from STDIN and write to STDOUT, use <tt>qpress -io</tt>. In this case qpress still requires a filename during compression, even though it is unused. Decompression does not have the same requirement.
+
+The example below uses 4 threads and a block size of 32768KB.
+
+  compress_with: qpress -ioT4K32768 dummyfilename
+  decompress_with: qpress -dioT4
+
+=== lzop
+lzop is a less CPU-intensive compressor. lzop is still single-threaded in v1.x, so its performance may not be ideal for the \Jetpants use-case. Multithreading is planned for v2.x. More information: http://www.lzop.org/
+
+  compress_with: lzop
+  decompress_with: lzop -d
+

data/doc/faq.rdoc

@@ -72,7 +72,11 @@ The main downside to the range-based approach is lack of even distribution of "h
 
 \Jetpants clones slaves by stopping replication, shutting down the MySQL daemon, and then copying the raw files to the destination(s). This is the fastest way to get a consistent clone of a data set in MySQL. After the copy operation is complete, we start MySQL back up on the source and destinations, and then make the destination instances start slaving at the appropriate binlog coordinates.
 
-We perform the copy operation using a combination of tar (for archiving), pigz (for fast compression), and nc (for transferring the data over the network). If there are multiple destinations, we create a serial "copy chain" using tee and a fifo.
+We perform the copy operation using a combination of:
+* <tt>tar</tt>, for archiving
+* a compression tool, if specified in your \Jetpants config file; we recommend <tt>qpress</tt> or <tt>pigz</tt>
+* <tt>nc</tt> (netcat), for transferring the data over the network
+* If there are multiple destinations, we create a serial "copy chain" using <tt>tee</tt> and a FIFO.
 
 Please note that we don't encrypt the data in this process, so we assume you are using it on a private LAN or over a VPN tunnel.
 
@@ -115,7 +119,7 @@ For any given operation that requires an asset tracker, there's one of two reaso
 
 * The operation inherently involves generating a new configuration for your application -- for example, setting a shard to read-only or promoting a standby slave to an active slave. These operations are meaningless outside of your application, since MySQL has no notion of "standby slave" or "degraded shard". \Jetpants has a notion of these things, but needs to persist the information somewhere, and it makes more sense to have \Jetpants relay this information to an external hardware management tool rather than maintain a separate (and potentially conflicting) source of truth.
 
-If you have enough servers to be using a sharded architecture, you hopefully already have some sort of hardware management / asset tracker system in place. \Jetpants is designed to be integrated with this system, but since every site runs something different, this requires that you write some custom plugin code to achieve.
+If you have enough servers to be using a sharded architecture, you hopefully already have some sort of hardware management / asset tracker system in place. \Jetpants is designed to be integrated with this system, but since every site runs something different, this requires that you write some custom plugin code to achieve. (Unless you use \Collins for tracking hardware, in which case you can use the bundled jetpants_collins plugin.)
 
 
 == Can I use \Jetpants with PostgreSQL?

data/doc/jetpants_collins.rdoc

@@ -0,0 +1,95 @@
+= jetpants_collins
+
+== OVERVIEW:
+
+This \Jetpants plugin offers integration with the \Collins hardware asset tracking system. This allows \Jetpants to automatically query the list of pools, shards, hosts, and databases in your topology at start-up time. Furthermore, every change you make to your topology using \Jetpants (master promotions, shard splits, new slaves cloned, etc) will automatically be reflected in \Collins immediately.
+
+== CONFIGURATION:
+
+This plugin has a number of configuration options, some of which are mandatory.
+
+user::          \Collins account username for \Jetpants to use (required)
+password::      \Collins account password (required)
+url::           \Collins URL (required)
+timeout::       \Collins client timeout, in seconds (default: 30)
+datacenter::    \Collins data center name that we're running \Jetpants in the context of (required if multi-datacenter, omit otherwise)
+remote_lookup:: Supply "remoteLookup" parameter for \Collins requests, to search multiple datacenters (default: false)
+
+To enable this plugin, add it to your \Jetpants configuration file (either <tt>/etc/jetpants.yaml</tt> or <tt>~/.jetpants.yaml</tt>). For example, in a single-datacenter environment, you configuration might look like this:
+
+    # ... rest of Jetpants config here
+    
+    plugins:
+        jetpants_collins:
+            user: jetpants
+            password: xxx
+            url: http://collins.yourdomain.com:8080
+        
+        # ... other plugins configured here
+
+== ASSUMPTIONS AND REQUIREMENTS:
+
+Use of this plugin assumes that you already have \Collins set up, and have performed hardware intake for all your servers already.
+
+This plugin also makes some assumptions about the way in which you use \Collins, namely:
+
+* All Linux servers have a TYPE of SERVER_NODE.
+* All MySQL database server hosts will have a PRIMARY_ROLE of DATABASE.
+* All MySQL database server hosts that are in-use will have a STATUS of either ALLOCATED or MAINTENANCE.
+* All MySQL database server hosts that are in-use will have a POOL set matching the name of their pool/shard, and a SECONDARY_ROLE set matching their \Jetpants role within the pool (MASTER, ACTIVE_SLAVE, STANDBY_SLAVE, or BACKUP_SLAVE).
+* You can initially assign PRIMARY_ROLE, STATUS, POOL, and SECONDARY_ROLE to database servers somewhat automatically; see GETTING STARTED, below.
+* All database server hosts that are "spares" (not yet in use, but ready for use in shard splits, shard cutover, or slave cloning) need to have a STATUS of PROVISIONED. These nodes must meet the requirements of spares as defined by the REQUIREMENTS doc that comes with \Jetpants. They should NOT have a POOL set, but they may have a ROLE set to either MASTER or STANDBY_SLAVE. The role will be used to select spare nodes for shard splits and shard cutover.
+* Database server hosts may optionally have an attribute called SLAVE_WEIGHT. The default weight, if omitted, is 100. This field has no effect in \Jetpants, but can be used by your custom configuration generator as needed, if your application supports a notion of different weights for slave selection.
+* Arbitrary metadata regarding pools and shards will be stored in assets with a TYPE of CONFIGURATION. These assets will have a POOL matching the pool's name, a TAG matching the pool's name but prefixed with 'mysql-', a STATUS reflecting the pool's state, and a PRIMARY_ROLE of either MYSQL_POOL or MYSQL_SHARD depending on the type of pool. You can make jetpants_collins create these automatically; see GETTING STARTED, below.
+
+Please note that jetpants_collins does not generate application configuration files, because every web app/framework uses a different format. You will need to write a custom plugin to generate a configuration file for your application as needed, by overriding the Topology#write_config method.
+
+== GETTING STARTED:
+
+Once you've met all of the requirements listed in the previous section, the next step is to tell \Jetpants about your existing pools/shards via <tt>jetpants console</tt>. You only need to do this process once.
+
+Adding functional partitions (global / unsharded pools):
+
+  # Create the pool object, specifying pool name and IP of current master
+  p = Pool.new('my-pool-name', '10.42.3.4')
+  
+  # Tell Jetpants about IPs of any existing active slaves (read slaves), if any.
+  # For example, say this pool has 2 active slaves and 2 standby slaves. \Jetpants
+  # can automatically figure out which slaves exist, but won't automatically know
+  # which ones are active for reads, so you need to tell it.
+  p.has_active_slave('10.42.3.30')
+  p.has_active_slave('10.42.3.32')
+  
+  # Sync the information to Collins
+  p.sync_configuration
+  
+Repeat this process for each functional partition, if you have more than one.
+
+Adding shard pools:
+
+  # Create and sync each shard object, specifying ID range and IP of current master
+  Shard.new(      1,    1000000, '10.42.4.10' ).sync_configuration
+  Shard.new(1000001,    2000000, '10.42.3.112').sync_configuration
+  Shard.new(2000001,    4000000, '10.42.3.45' ).sync_configuration
+  Shard.new(4000001, 'INFINITY', '10.42.3.26' ).sync_configuration
+
+The max ID of the last shard must be 'INFINITY' in order for <tt>jetpants shard_cutover</tt> to work.
+
+
+== MULTI-DATACENTER SUPPORT:
+
+This plugin offers preliminary support for multi-datacenter \Collins deployments. The assumed topology is:
+* Each datacenter has its own copy of \Collins, and they're configured to talk to each other
+* Each datacenter has a node to run \Jetpants from, with the jetpants_collins configuration options differing between datacenters
+* Every database pool has only one true, writable master. This is located in any datacenter.
+* The true master may have several slaves in its own datacenter.
+* The true master may have slaves in other datacenters, but should only have <i>one direct slave per remote datacenter</i>. These remote slaves should have a SECONDARY_ROLE of MASTER in their datacenter's copy of \Collins, and they may have additional slaves of their own (tiered replication).
+* In other words, each datacenter -- and hence each copy of \Collins -- still has at most one MASTER per database pool. However only one of these nodes is the true, writable master; the others are actually slaves of master, and are read-only.
+
+Also, jetpants_collins currently enforces several restrictions on interacting with databases in remote datacenters, to simplify handling of tiered replication:
+
+* jetpants_collins won't change Collins attributes on remote server node assets. If you need to manipulate those assets, do it from the copy of \Jetpants and copy of \Collins in that datacenter.
+* If a local slave node has a master in a remote datacenter, it is ignored/hidden by jetpants_collins. In other words, each datacenter's master is viewed as a "real" master, even if it's actually slaving from another remote master.
+* If a local master node has a slave in a remote datacenter, it's treated as a backup_slave, in order to prevent cross-datacenter master promotions. If any of these remote slaves have slaves of their own, they're ignored/hidden by jetpants_collins.
+
+Due to the nature of this implementation, it works best for setups with 1 active datacenter and 1 or more passive datacenters. This support will be expanded in future releases to better capture the tiered replication roles and support active/active topologies. At that time, these restrictions/simplifications will be lifted wherever possible.

data/doc/plugins.rdoc

@@ -8,9 +8,9 @@ It is highly recommended that you tie \Jetpants into your site's asset tracker (
 
 Other recommended uses of plugins include integration with your site's monitoring system, trending system, query killers, and environment-specific overrides to various core methods.
 
-== Asset tracker
+== Asset tracker examples
 
-=== Example
+=== simple_tracker
 
 We supply a sample plugin called simple_tracker, demonstrating how to go about writing a very basic asset-tracking plugin. This plugin just uses an internal JSON file to keep track of database topology/state, and separately writes app configuration to a YAML file. This isn't actually suitable for production use, but should provide a reasonable starting point for learning the plugin system and building a real asset tracker.
 
@@ -24,8 +24,11 @@ When you first start using simple_tracker, there will be no pools, shards, or sp
 * <tt>jetpants add_shard</tt>
 * <tt>jetpants add_spare</tt>
 
+=== jetpants_collins
 
-=== Methods to override
+We also supply a plugin called jetpants_collins, offering integration with the Collins asset management software. This is fully intended for production use, and powers our automation at Tumblr. For more information on jetpants_collins, please see jetpants_collins.rdoc ({view on GitHub}[https://github.com/tumblr/jetpants/blob/master/doc/jetpants_collins.rdoc]).
+
+=== Rolling your own
 
 If you're writing your own asset-tracker plugin, you will need to override the following methods:
 
@@ -58,7 +61,7 @@ You may also want to override or implement these, though it's not strictly manda
 
 === Name and location
 
-If you define a plugin named "foo" in your \Jetpants config file, on startup \Jetpants will first attempt to require 'foo/foo', and failing that simply 'foo'.
+If you define a plugin named "foo" in your \Jetpants config file, on startup \Jetpants will first attempt to require 'foo/foo' (for loading bundled plugins in <tt>jetpants/plugins/<>tt>), if that fails then simply 'foo' (for loading a stand-alone gem).
 
 Plugins may be located anywhere on your Ruby load path, so packing them as standard gems should work perfectly. You may want to prefix the gem name with "\jetpants_" to avoid conflicts with other gems. \Jetpants also adds its plugins directory to its load path automatically, so that any pre-bundled plugins (like simple_tracker) can be loaded easily.
 

data/doc/requirements.rdoc

@@ -16,7 +16,7 @@ Plugins may freely override these assumptions, and upstream patches are very wel
 * Required Linux binaries that must be installed and in root's PATH on all of your database machines:
   * <tt>service</tt>, a wrapper around init scripts, supporting syntax <tt>service mysql start</tt>, <tt>service mysql status</tt>, etc. Some distros include this by default (typically as /sbin/service or /usr/sbin/service) while others offer it as a package. Implementation varies slightly between distros; currently \Jetpants expects <tt>service mysql status</tt> output to include either "not running" (RHEL/Centos) or "stop/waiting" (Ubuntu) in the output if the MySQL server is not running.
   * <tt>nc</tt>, also known as netcat, a tool for piping data to or from a socket.
-  * <tt>pigz</tt>, an open-source single-binary parallel gzip tool by Mark Adler. A future version of \Jetpants will allow pluggable compression tools, but at present we strictly use <tt>pigz</tt> for compression in all file copy operations.
+  * Whichever compression tool you've specified in the \Jetpants config file for the <tt>compress_with</tt> and <tt>decompress_with</tt> options, if any. (if omitted, compression will not be used for file copy operations.)
 * InnoDB / Percona XtraDB for storage engine. \Jetpants has not been tested with MyISAM, since \Jetpants is geared towards huge tables, and MyISAM is generally a bad fit.
 * All MySQL instances run on port 3306, with only one instance per logical machine.
   * A plugin could override this easily, but would require you to use the --report-host option on all slaves running MySQL 5.1, so that crawling the replication topology is possible. It would also have to override various methods that specify the MySQL init script location, config file location, data directory, etc.
@@ -64,4 +64,4 @@ By default, the <tt>standby_slaves_per_pool</tt> config option is set to 2. This
 
 A "spare" machine in \Jetpants should be in a clean-slate state: MySQL should be installed and have the proper grants and root password, but there should be no data on these machines, and they should not be slaving. \Jetpants will set up replication appropriately when it assigns the nodes to their appropriate pools.
 
-For more information on the shard split process implemented by \Jetpants, including diagrams of each stage of the process, please see {Evan Elias's presentation at Velocity Europe 2011}[https://github.com/tumblr/jetpants/blob/master/doc/VelocityEurope2011Presentation.pdf?raw=true], starting at slide 19.
+For more information on the shard split process implemented by \Jetpants, including diagrams of each stage of the process, please see {Evan Elias's presentation at Percona Live NYC 2012}[https://github.com/tumblr/jetpants/blob/master/doc/PerconaLiveNYC2012Presentation.pdf?raw=true], starting at slide 20.

data/etc/jetpants.yaml.sample

@@ -27,6 +27,11 @@ mysql_grant_privs:
 # has higher capacity anyway.
 export_location:        /some/path/on/root/partition
 
+# If you want to speed up large copy operations in Jetpants, supply relevant
+# command lines here. See configuration.rdoc for more information on formatting.
+compress_with:          false
+decompress_with:        false
+
 # List all tables defined on the shards, along with what column name corresponds
 # to your app's sharding key (to determine which shard a given row lives on),
 # and how many "chunks" to split the data set into when doing an import or

data/lib/jetpants.rb

@@ -36,6 +36,8 @@ module Jetpants
     'plugins'                 =>  {},         # hash of plugin name => arbitrary plugin data (usually a nested hash of settings)
     'ssh_keys'                =>  nil,        # array of SSH key file locations
     'sharded_tables'          =>  [],         # array of name => {sharding_key=>X, chunks=>Y} hashes
+    'compress_with'           =>  false,      # command line to use for compression in large file transfers
+    'decompress_with'         =>  false,      # command line to use for decompression in large file transfers
   }
   %w(/etc/jetpants.yaml ~/.jetpants.yml ~/.jetpants.yaml).each do |path|
     overrides = YAML.load_file(File.expand_path path) rescue {}

data/lib/jetpants/db/state.rb

@@ -105,7 +105,15 @@ module Jetpants
       sleep(interval)
       global_status[:Connections].to_i - conn_counter > threshold
     end
-
+    
+    # Confirms the binlog of this node has not moved during a duration
+    # of [interval] seconds.
+    def taking_writes?(interval=5.0)
+      coords = binlog_coordinates
+      sleep(interval)
+      coords != binlog_coordinates
+    end
+    
     # Returns true if this instance appears to be a standby slave,
     # false otherwise. Note that "standby" in this case is based
     # on whether the slave is actively receiving connections, not
@@ -249,9 +257,17 @@ module Jetpants
       @slaves = []
       slaves_mutex = Mutex.new
       processes = mysql_root_cmd("SHOW PROCESSLIST", :terminator => ';').split("\n")
-      processes.grep(/Binlog Dump/).concurrent_each do |p|
+      
+      # We have to de-dupe the output, since it's possible in weird edge cases for
+      # the same slave to be listed twice
+      ips = {}
+      processes.grep(/Binlog Dump/).each do |p|
         tokens = p.split
         ip, dummy = tokens[2].split ':'
+        ips[ip] = true
+      end
+      
+      ips.keys.concurrent_each do |ip|
         db = ip.to_db
         db.probe
         slaves_mutex.synchronize {@slaves << db if db.master == self}

data/lib/jetpants/host.rb

@@ -192,7 +192,6 @@ module Jetpants
     ###### Directory Copying / Listing / Comparison methods ####################
     
     # Quickly and efficiently recursively copies a directory to one or more target hosts.
-    # Requires that pigz is installed on source (self) and all targets.
     # base_dir::  is base directory to copy from the source (self). Also the default destination base
     #             directory on the targets, if not supplied via next param.
     # targets::   is one of the following:
@@ -225,6 +224,14 @@ module Jetpants
       file_list = filenames.join ' '
       port = (options[:port] || 7000).to_i
       
+      if Jetpants.compress_with || Jetpants.decompress_with
+        comp_bin = Jetpants.compress_with.split(' ')[0]
+        confirm_installed comp_bin
+        output "Using #{comp_bin} for compression"
+      else
+        output "Compression disabled -- no compression method specified in Jetpants config file"
+      end
+      
       # On each destination host, do any initial setup (and optional validation/erasing),
       # and then listen for new files.  If there are multiple destination hosts, all of them
       # except the last will use tee to "chain" the copy along to the next machine.
@@ -233,7 +240,10 @@ module Jetpants
         dir = destinations[t]
         raise "Directory #{t}:#{dir} looks suspicious" if dir.include?('..') || dir.include?('./') || dir == '/' || dir == ''
         
-        t.confirm_installed 'pigz'
+        if Jetpants.compress_with || Jetpants.decompress_with
+          decomp_bin = Jetpants.decompress_with.split(' ')[0]
+          t.confirm_installed decomp_bin
+        end
         t.ssh_cmd "mkdir -p #{dir}"
         
         # Check if contents already exist / non-empty.
@@ -244,8 +254,9 @@ module Jetpants
           dirlist.each {|name, size| raise "File #{name} exists on destination and has nonzero size!" if size.to_i > 0}
         end
         
+        decompression_pipe = Jetpants.decompress_with ? "| #{Jetpants.decompress_with}" : ''
         if i == 0
-          workers << Thread.new { t.ssh_cmd "cd #{dir} && nc -l #{port} | pigz -d | tar xvf -" }
+          workers << Thread.new { t.ssh_cmd "cd #{dir} && nc -l #{port} #{decompression_pipe} | tar xv" }
           t.confirm_listening_on_port port
           t.output "Listening with netcat."
         else
@@ -254,16 +265,16 @@ module Jetpants
           workers << Thread.new { t.ssh_cmd "cd #{dir} && mkfifo #{fifo} && nc #{tt.ip} #{port} <#{fifo} && rm #{fifo}" }
           checker_th = Thread.new { t.ssh_cmd "while [ ! -p #{dir}/#{fifo} ] ; do sleep 1; done" }
           raise "FIFO not found on #{t} after 10 tries" unless checker_th.join(10)
-          workers << Thread.new { t.ssh_cmd "cd #{dir} && nc -l #{port} | tee #{fifo} | pigz -d | tar xvf -" }
+          workers << Thread.new { t.ssh_cmd "cd #{dir} && nc -l #{port} | tee #{fifo} #{decompression_pipe} | tar xv" }
           t.confirm_listening_on_port port
           t.output "Listening with netcat, and chaining to #{tt}."
         end
       end
       
       # Start the copy chain.
-      confirm_installed 'pigz'
       output "Sending files over to #{targets[0]}: #{file_list}"
-      ssh_cmd "cd #{base_dir} && tar vc #{file_list} | pigz | nc #{targets[0].ip} #{port}"
+      compression_pipe = Jetpants.compress_with ? "| #{Jetpants.compress_with}" : ''
+      ssh_cmd "cd #{base_dir} && tar vc #{file_list} #{compression_pipe} | nc #{targets[0].ip} #{port}"
       workers.each {|th| th.join}
       output "File copy complete."
       

data/lib/jetpants/monkeypatch.rb

@@ -17,6 +17,32 @@ module Enumerable
   def concurrent_each_with_index(&block)
     each_with_index.concurrent_each(&block)
   end
+  
+  # Alternative for concurrent_map which also has the ability to limit how
+  # many threads are used. Much less elegant :(
+  def limited_concurrent_map(thread_limit=40)
+    lock = Mutex.new
+    group = ThreadGroup.new
+    items = to_a
+    results = []
+    pos = 0
+    
+    # Number of concurrent threads is the lowest of: self length, supplied thread limit, global concurrency limit
+    [items.length, thread_limit, Jetpants.max_concurrency].min.times do
+      th = Thread.new do
+        while true do
+          my_pos = nil
+          lock.synchronize { my_pos = pos; pos += 1}
+          break unless my_pos < items.length
+          my_result = yield items[my_pos]
+          lock.synchronize { results[my_pos] = my_result }
+        end
+      end
+      group.add th
+    end
+    group.list.each {|th| th.join}
+    results
+  end
 end
 
 # Add Jetpants-specific conversion methods to Object.

data/lib/jetpants/pool.rb

@@ -177,14 +177,14 @@ module Jetpants
       end
       
       binlog_pos = extended_info ? details[@master][:coordinates].join(':') : ''
-      print "\tmaster          = %-13s %-30s %s\n" % [@master.ip, @master.hostname, binlog_pos]
+      print "\tmaster          = %-15s %-30s %s\n" % [@master.ip, @master.hostname, binlog_pos]
       
       [:active, :standby, :backup].each do |type|
         slave_list = slaves(type)
         slave_list.sort.each_with_index do |s, i|
           binlog_pos = extended_info ? details[s][:coordinates].join(':') : ''
           slave_lag = extended_info ? "lag=#{details[s][:lag]}" : ''
-          print "\t%-7s slave #{i + 1} = %-13s %-30s %-26s %s\n" % [type, s.ip, s.hostname, binlog_pos, slave_lag]
+          print "\t%-7s slave #{i + 1} = %-15s %-30s %-26s %s\n" % [type, s.ip, s.hostname, binlog_pos, slave_lag]
         end
       end
       true