jetpants 0.7.0

data/doc/commands.rdoc

@@ -0,0 +1,119 @@
+= Jetpants command suite
+
+Run <tt>jetpants</tt> without any parameters to see a list of all commands available in the suite.
+
+Please note that all \Jetpants commands that take a database node require you to specify it <b>by IP address</b>, rather than hostname. It's usually a good practice to list databases in your application configuration by IP anyway, to remove DNS as a potential bottleneck and single point of failure. By also keeping all commands in \Jetpants by IP, it's easier to confirm that your commands had the expected results by looking at the diff of your regenerated application config file, for example.
+
+Here's a more thorough description of the commands, grouped by function:
+
+== Slave cloning
+
+<b><tt>jetpants clone_slave</tt></b> copies data from a standby slave onto one or more fresh nodes, turning them into additional standby slaves of the same master.
+
+\Jetpants copies data sets by shutting down the MySQL daemon on the source node and then copying all MySQL files. This is the fastest way to clone MySQL data sets, and is part of the reason why we recommend having 2 standby slaves per pool for high availability.
+
+The copy method in \Jetpants uses a combination of tar, netcat (nc), and pigz. It does not use encryption; we assume you are transferring over a secure local network.  When copying to multiple destinations, \Jetpants creates a "copy chain" using tee and a fifo. For more information on this technique, please see {our post on the Tumblr engineering blog}[http://engineering.tumblr.com/post/7658008285/efficiently-copying-files-to-multiple-destinations].
+
+This command does not require an asset tracker plugin, but DOES require that all your database nodes have pigz (a parallel gzip tool) installed. This may become configurable in a future release.
+
+
+== Master/slave state changes
+
+These commands change the type of a slave, or promote a slave to be a master.  <tt>jetpants promotion</tt> does not require an asset tracker, but the other commands all do.
+
+<b><tt>jetpants promotion</tt></b> changes which node in a pool is the master by performing a full MySQL master promotion. This is usable even if the old master is offline or unavailable. All nodes in the pool will now slave off of the new master. If the old master is online/available, it will become a standby slave of the new master.
+
+Please note that the master promotion process enables global READ_ONLY mode on the old master. This is a required step of the standard MySQL master promotion technique. After doing a promotion in \Jetpants, you'll need to update/deploy your application's configuration as quickly as possible, if a plugin doesn't do it automatically for you.
+
+<b><tt>jetpants activate_slave</tt></b> turns a standby slave into an active slave. Use this if you want to generate an app configuration that now sends read queries to a slave that formerly did not receive them.
+
+<b><tt>jetpants weigh_slave</tt></b> alters the weight of an active slave. This is only useful if application supports having read slaves with different selection weights.
+
+<b><tt>jetpants pull_slave</tt></b> turns an active slave into a standby slave. Use this if you want to generate an app configuration file that stops sending read queries to a slave.
+
+<b><tt>jetpants destroy_slave</tt></b> removes a standby slave from its pool entirely, stopping and then resetting replication on the node. Use this prior to recycling or canceling a node that is no longer needed.
+
+
+== Interactive REPL
+
+<b><tt>jetpants console</tt></b> starts an <tt>irb</tt> session that is already in the Jetpants module namespace.
+
+If you're using an asset tracker, this is an easy way to programmatically interact with your database topology.
+This is especially true because the Jetpants module delegates missing methods to its Jetpants::Topology singleton. For example:
+
+  # print summary data on all pools
+  > pools.each &:summary
+
+  # print condensed info on shard pools only
+  > shards.each {|s| print "[%-12s] %8s to %-11s = %-4s\n" % [s.ip, s.min_id, s.max_id, s.data_set_size(true)]}
+
+The first time you install \Jetpants and an asset tracker, you can enter your pool information through the console:
+
+  > p = Pool.new('my-pool-name', '1.2.3.4')   # supply name and master IP
+  > p.sync_configuration                      # update the asset tracker data
+
+Or if you're deploying a brand new pool in an existing topology:
+
+  > master = claim_spare
+  > slaves = claim_spares(2)
+  > p = Pool.new('my-new-pool', master)
+  > slaves.each {|s| s.change_master_to(master); s.start_replication}
+  > p.sync_configuration
+
+== Shard rebalancing
+
+\Jetpants allows you to split a shard into N pieces.  This is a fairly complex process; the steps look roughly like this:
+
+1. Create N new slaves of the shard being split (the "parent" shard). These new slaves will become masters of their own "child" shards later in the process.
+2. Reduce the data set on those slaves so that each contains a different subset of the data. This is done by exporting the data subset, dropping the tables, re-creating the tables, and importing the data subset back in.
+3. Move app reads from the parent shard to the appropriate child shards. You must move reads before moving writes in order to maintain consistency in your application. Writes will continue to go to the parent shard, which then replicates to the child shards.
+4. Move app writes from the parent shard to the appropriate child shards.
+5. Stop replicating writes from the parent shard, and then take the parent pool offline entirely.
+6. Remove rows that replicated to the wrong child shard. This data will be sparse, since it's only the writes that were made since the shard split process started.
+
+For more information, including diagrams of each step, please see {our presentation at Velocity Europe 2011}[https://github.com/tumblr/jetpants/blob/master/doc/VelocityEurope2011Presentation.pdf?raw=true].
+
+Separately, \Jetpants also allows you to alter the range of the last shard in your topology. In a range-based sharding scheme, the last shard has a range of X to infinity; eventually this will be too large of a range, so you need to truncate that shard range and create a "new" last shard after it.  We call this process "shard cutover".
+
+These commands all require an asset tracker in order to function.
+
+<b><tt>jetpants shard_split</tt></b> performs the first steps of the split: spins up the new shard pools with the appropriate portions of the data set. The new shards will still be slaving from their parent though. This step potentially takes several hours, so you may want to use a tool like screen to prevent SSH timeouts.
+
+<b><tt>jetpants shard_split_child_reads</tt></b> regenerates your application config file to move reads to the new child shards.
+
+<b><tt>jetpants shard_split_child_writes</tt></b> regenerates your application config file to move writes to the new child shards. Be sure to move reads first.
+
+<b><tt>jetpants shard_split_cleanup</tt></b> tears down replication between the parent and child shards, makes the parent shard non-readable and non-writable, and then cleans up data that replicated to the wrong child. This may take a little while to run, although typically much less than the initial <tt>jetpants shard_split</tt> takes since data will be sparse.
+
+<b><tt>jetpants shard_cutover</tt></b> truncates the current last shard range, and adds a new shard after it. You need to pick what ID is being used for the cutover; this ID <b>MUST</b> be one "in the future", ie, one your application hasn't hit yet. The cutover process does NOT move any data, since it assumes the new shard it's creating has no data yet. What the cutover process will do is allocate the new shard's hardware, set up replication, create your sharded tables, and update your app configuration file.
+
+
+== Shard degradation
+
+With an asset tracker, \Jetpants allows you to mark a shard as read-only or completely offline, assuming your application supports these notions. (Which, of course, it should -- this is one of the major benefits of a sharded infrastructure!)
+
+<b><tt>jetpants shard_read_only</tt></b> marks a shard as not writable.
+
+<b><tt>jetpants shard_offline</tt></b> marks a shard as completely unavailable.
+
+<b><tt>jetpants shard_online</tt></b> marks a shard as being fully online again, after a prior call to <tt>jetpants shard_read_only</tt> or <tt>jetpants shard_offline</tt>.
+
+
+== Informational commands
+
+These commands do not require an asset tracker. They may be fleshed out further in a future release of \Jetpants.
+
+<b><tt>jetpants node_info</tt></b> displays information on a MySQL instance.
+
+<b><tt>jetpants show_master</tt></b> tells you what node is the master of a given MySQL instance.
+
+<b><tt>jetpants show_slaves</tt></b> displays a list of slaves (along with current slave lag) of a MySQL instance.
+
+
+== Miscellaneous
+
+<b><tt>jetpants rebuild_slave</tt></b> exports all data on a standby slave, drops tables, recreates tables, and then re-imports the data. This is useful for defragmenting a node. Currently it only works on shard pool slaves; this may change in a future release.
+
+<b><tt>jetpants regen_config</tt></b> regenerates your application's configuration file, assuming you are using an asset tracker. This isn't terribly useful in most cases, because any commands that make configuration changes will do this automatically. However, you may want to run this after manually making topology changes via <tt>jetpants console</tt>.
+
+

data/doc/configuration.rdoc

@@ -0,0 +1,27 @@
+= Jetpants Configuration
+
+\Jetpants supports a global configuration file at <tt>/etc/jetpants.yaml</tt>, as well as per-user configuration files at <tt>~/.jetpants.yaml</tt>.
+
+At least one of these files must exist for \Jetpants to function properly.
+
+If both exist, the user configuration will be merged on top of the global configuration, but please note that this is not a "deep" merge. So if the user configuration defines a "plugins" section, this will used as-is and the global "plugins" section will be ignored. This may change in a future release.
+
+For an example global configuration file, please see the included <tt>etc/jetpants.yaml.sample</tt> file.
+
+== Configuration settings
+
+max_concurrency::           Maximum threads to use in import/export operations, or equivalently the maximum connection pool size per database host (default: 40)
+standby_slaves_per_pool::   Minimum number of standby slaves to keep in every pool (default: 2)
+mysql_schema::              database name (mandatory)
+mysql_app_user::            mysql user that your application uses (mandatory)
+mysql_app_password::        mysql password that your application uses (mandatory)
+mysql_repl_user::           mysql user for replication (mandatory)
+mysql_repl_password::       mysql password for replication (mandatory)
+mysql_root_password::       mysql root password (default: false, indicating that \Jetpants should use /root/.my.cnf instead)
+mysql_grant_ips::           mysql user manipulations are applied to these IPs (array; mandatory)
+mysql_grant_privs::         mysql user manipulations grant this set of privileges by default (array; default: \['ALL'])
+export_location::           directory to use for data dumping (default: '/tmp')
+verify_replication::        raise exception if the actual replication topology differs from Jetpants' understanding of it (ie, disagreement between asset tracker and probed state), or if MySQL's two replication threads are in different states (one running and the other stopped) on a DB node. (default: true. master promotion tool ignores this, since the demoted master may legitimately be dead/offline)
+plugins::                   hash of plugin name => arbitrary plugin data, usually a nested hash of settings (default: \{})
+ssh_keys::                  array of SSH private key file locations, if not using standard id_dsa or id_rsa. Passed directly to Net::SSH.start's :keys parameter (default: nil)
+sharded_tables::            array of name => \{sharding_key=>X, chunks=>Y} hashes, describing all tables on shards. Required by shard split/rebuild processes (default: \[])

data/doc/plugins.rdoc

@@ -0,0 +1,120 @@
+= Jetpants Plugins
+
+\Jetpants offers an extensible plugin system. Plugins are Ruby code (such as stand-alone gems) that add to \Jetpants by supplying callback methods, and/or overriding core methods.
+
+== Recommended Uses
+
+It is highly recommended that you tie \Jetpants into your site's asset tracker (ie, hardware management system) by writing a custom plugin. This will allow Jetpants to automatically know what database pools and shards are present, and to make topological changes immediately be reflected in your site's configuration.
+
+Other recommended uses of plugins include integration with your site's monitoring system, trending system, query killers, and environment-specific overrides to various core methods.
+
+== Asset tracker
+
+=== Example
+
+We supply a sample plugin called simple_tracker, demonstrating how to go about writing a very basic asset-tracking plugin. This plugin just uses an internal JSON file to keep track of database topology/state, and separately writes app configuration to a YAML file. This isn't actually suitable for production use, but should provide a reasonable starting point for learning the plugin system and building a real asset tracker.
+
+To use simple_tracker, be sure to define its <tt>tracker_data_file_path</tt> (where it saves its internal asset-tracking data) and <tt>app_config_file_path</tt> (where it exports web app config files).  If you're wondering why these are kept separate, it's to allow you to support pool topology changes that don't immediately impact your application, in case you're doing a complex multi-step migration.
+
+For an example of how to configure the plugin, please see the bottom of etc/jetpants.yaml.sample.
+
+When you first start using simple_tracker, there will be no pools, shards, or spare machines. You can start to add some via <tt>jetpants console</tt>, or you can write a custom script to import data from another source or config file. Either way, some examples of how to add things include:
+
+  # Create a global pool (functional partition)
+  p = Pool.new('some-name', '10.45.67.220')
+  
+  # The slaves of the master will be detected automatically,
+  # but they will all be assumed to be standby slaves by
+  # default. You have to tell Jetpants which ones are active
+  # slaves:
+  p.has_active_slave('10.45.67.226')
+  
+  # save it
+  p.sync_configuration
+  
+  # Create a shard + save it
+  s = Shard.new(1, 100000, '10.45.67.160', :ready)
+  s.sync_configuration
+  
+  # add some spare machines
+  Jetpants.topology.tracker.spares << '10.45.67.130'
+  Jetpants.topology.tracker.spares << '10.45.67.132'
+  
+  # If you're using jetpants console, this works too, since console runs in Jetpants
+  # module namespace, which then delegates missing methods to its topology object:
+  tracker.spares << '10.45.67.134'
+  
+
+=== Methods to override
+
+If you're writing your own asset-tracker plugin, you will need to override the following methods:
+
+* Jetpants::Topology#load_pools
+  * Reads data from an asset tracker to initialize all pools and shards.
+* Jetpants::Topology#write_config
+  * Writes a configuration file for your webapp, or commits configuration changes to a service, depending on how your site handles configuration knowledge
+* Jetpants::Topology#claim_spares
+  * Obtains spare database nodes and marks them as no longer being spare
+* Jetpants::Topology#count_spares
+  * Returns a count of spare database nodes
+* Jetpants::Pool#sync_configuration
+  * Updates the asset tracker with the current status of a pool. 
+  * This should update the asset tracker's internal knowledge of the database topology immediately, but not necessarily cause the application's config file to be regenerated immediately. 
+
+You may also want to override or implement these, though it's not strictly mandatory:
+
+* Jetpants::DB#for_backups?
+  * Defines how to tell the different between standby slaves and backup slaves
+  * Skip this if your architecture doesn't include backup slaves, or if the default implementation (hostnames starting with "backup") is sufficient
+* Jetpants::DB#after_probe_slaves
+  * In order to support master promotions when a master has failed, you will need to implement this to set a slave list from your asset tracker if a MySQL instance isn't available
+* Jetpants::DB#after_probe_master
+  * In order to support operations on a slave that has failed, you will need to implement this to set @master based on information in your asset tracker if a MySQL instance isn't available
+* Jetpants::Pool#after_remove_slave!
+  * If your implementation of Jetpants::Pool#sync_configuration works by iterating over the current slaves of the pool's master, it won't correctly handle the <tt>jetpants destroy_slave</tt> command,
+    or anything else that calls Jetpants::DB#disable_replication! which does a RESET SLAVE. You can instead handle that special case here.
+
+== How to write a plugin
+
+=== Name and location
+
+If you define a plugin named "foo" in your \Jetpants config file, on startup \Jetpants will first attempt to require 'foo/foo', and failing that simply 'foo'.
+
+Plugins may be located anywhere on your Ruby load path, so packing them as standard gems should work perfectly. You may want to prefix the gem name with "\jetpants_" to avoid conflicts with other gems. \Jetpants also adds its plugins directory to its load path automatically, so that any pre-bundled plugins (like simple_tracker) can be loaded easily.
+
+=== Recommended format
+
+The \Jetpants plugin system is designed to be as light and simple as possible. Plugins are just normal Ruby code that gets required after the rest of \Jetpants has been initialized. This allows you to override any method in any \Jetpants class. Simply re-open the class as normal in Ruby.
+
+The base of your plugin may use the Jetpants::Plugin module as an outer namespace, but doing so is not required.
+
+=== Callbacks
+
+To make it easier to chain behavior before or after existing \Jetpants methods, all core \Jetpants classes include a mix-in (Jetpants::CallbackHandler) that adds automatic callback functionality. For example, any time you call method Jetpants::DB#foo, it will magically first call method Jetpants::DB#before_foo if it exists. Similarly, after the foo method finishes, it will magically call method Jetpants::DB#after_foo if it exists.
+
+A particularly nice aspect of callbacks is that they "stack".  For example, if two plugins both implement Jetpants::DB#before_foo, they will <b>both</b> be called before Jetpants::DB#foo. You can even specify the priority of a callback (higher = called first, default is 100) by preceding it with the Jetpants::CallbackHandler.callback_priority decorator. For example:
+
+    module Jetpants
+        class DB
+            callback_priority 150
+            def after_query(*args)
+                puts "After calling DB#query, I am called next due to my high priority"
+            end
+        end
+    end
+    module Jetpants
+        class DB
+            callback_priority 85
+            def after_query(*args)
+                puts "After calling DB#query, other plugins' callbacks with higher priorities get called, then I get called"
+            end
+        end
+    end
+
+Callbacks receive the same parameter list as the base method was called with. Their return values are ignored.
+
+Callbacks may interrupt the control flow by raising a Jetpants::CallbackAbortError exception:
+
+* If this is raised in a before_foo method, all lower-priority before_foo stacked callbacks will be skipped, as will the call to foo and any after_foo callbacks. The exception is automatically caught and is not fatal. The return value of foo will be nil.
+
+* If this is raised in an after_foo method, all subsequent lower-priority after_foo stacked callbacks will be skipped. The return value of foo will be unaffected.

data/doc/requirements.rdoc

@@ -0,0 +1,54 @@
+= Jetpants Requirements and Assumptions
+
+The base classes of \Jetpants currently make a number of assumptions about your environment and database topology. 
+
+Plugins may freely override these assumptions, and upstream patches are very welcome to incorporate support for alternative configurations. We're especially interested in plugins or pull requests that add support for: Postgres and other relational databases; Redis and other non-relational data stores; non-Redhat Linux distributions or *BSD operating systems; master-master topologies; multi-instance-per-host setups; etc. We have attempted to design \Jetpants in a way that is sufficiently flexible to eventually support a wide range of environments.
+
+== Environment
+
+* Using MySQL (or Percona Server), specifically version 5.1 or higher.
+* Using a RHEL/CentOS distribution of Linux.
+  * It should be easy to write a plugin supporting another distribution. The main change might be overriding Jetpants::Host#service, if your distribution doesn't have <tt>/sbin/service</tt>.
+* Using InnoDB / Percona XtraDB for storage engine. \Jetpants has not been tested with MyISAM, since \Jetpants is geared towards huge tables, and MyISAM is generally a bad fit.
+* All MySQL instances run on port 3306, with only one instance per logical machine.
+  * A plugin could override this easily, but would require you to use the --report-host option on all slaves, so that crawling the replication topology is possible. It would also have to override various methods that specify the MySQL init script location, config file location, data directory, etc.
+  * Since there's no "standard" layout for multi-instance MySQL, this won't ever be part of the \Jetpants core, but we may include one implementation as a bundled plugin in a future release.
+* Master-Master replication is not in use. It presents a large number of additional failure states which make automation quite difficult, since transaction IDs are not global in MySQL.
+* Unusual replication topologies -- special-purpose slaves, ring topologies, hierarchical replication -- are not in use. Plugins could certainly override this by adding additional types of slaves, but there are too many options for this to be part of the \Jetpants core.
+* You must be running \Jetpants as a UNIX user with access to an SSH private key, capable of connecting as root to any host in your database topology. If this key has a nonstandard name or location, you must specify this using the <tt>ssh_keys</tt> option in the Jetpants configuration file. SSH agent forwarding should work fine automatically, but be careful of its interaction with UNIX tools like <tt>screen</tt>.
+* The root MySQL user must be able to access the database on localhost via MySQL's UNIX socket. You may specify the root password with Jetpants' <tt>mysql_root_password</tt> option in the configuration file. Or if all of your database hosts has a <tt>/root/.my.cnf</tt> file specifying the root password, you can omit <tt>mysql_root_password</tt> in the configuration file.
+* You must have <tt>pigz</tt> (an open-source parallel gzip tool by Mark Adler) installed on all database hosts. A future version of \Jetpants will allow pluggable compression tools, but at present we strictly use <tt>pigz</tt> for compression in all file copy operations.
+
+
+== Database topology
+
+We define a database <b>pool</b> as 1 master and 0 or more slaves. Each slave falls into one of three classifications:
+* Active slaves: replicas that actively receive reads from your application(s)
+* Standby slaves: replicas that exist for high availability, to replace failed nodes or to quickly spin up new standby replacements.
+* Backup slaves: replicas that never receive application queries and will never be promoted to another class. For example, might be a different hardware spec.
+
+At Tumblr we ensure that all pools have exactly 2 standby slaves. This establishes a minimum level of redundancy at 3 copies of each row. If a master or active slave fails, we use \Jetpants to promote one standby slave in its place, and use the other standby slave to quickly clone a replacement. (\Jetpants clones nodes by stopping the MySQL daemon and then copying the data set; this is substantially faster than using a hot-copy solution, especially on very large and/or very actively written data sets.)
+
+We no longer use dedicated backup slaves at Tumblr, although we still offer support in \Jetpants for them.
+
+== Sharding scheme
+
+We define a database <b>shard</b> as a specialized type of pool, always consisting of 1 master and 2 standby slaves. All shards contain the same set of tables, but each stores a different partition of the data set, divided by ID ranges of a global sharding key.
+
+For example, say your application's natural sharding key is <tt>user_id</tt>. The <tt>users</tt> table itself might not be shardable (since it may require global lookup patterns like find-by-email), but any other table containing a <tt>user_id</tt> column could be sharded. One shard might contain all sharded table data for users 1 through 1000000; another shard would contain data for users 1000001 through 2000000; etc. These ranges may be of uneven sizes. When disk usage approaches capacity or I/O utilization gets too high, \Jetpants can be used to split the shard into N new "child" shards of equal or unequal ranges.
+
+We prefer this range-based scheme due to its simplicity. The ranges can be expressed in a language-neutral JSON or YAML format, which can be shared between application stacks easily. \Jetpants solves all data rebalancing problems for you. There's no need to "pre-allocate" thousands of tiny shards, nor do you need an external lookup service which would be a bottleneck and a single point of failure.
+
+Under this sharding scheme, there will always be a "last" shard, with a range of X to infinity. All new users (or whatever your sharding key is) will have their data placed on this shard. Once there are too many users on this shard, it becomes necessary to truncate its range such that it handles IDs X through Y, where Y is an ID "in the future" (ie, sufficiently higher than your current max ID such that you won't hit ID Y for another few days). You must then add a new last shard handles IDs (Y+1) to infinity. We call this process "shard cutover", and it is a supported operation in the Jetpants command suite.
+
+
+== Assumptions for shard-split logic
+
+In order for \Jetpants to be able to rebalance your shards efficiently, we also must assume the following:
+
+* No auto_increment columns in any sharded table. Use an external ID generation service instead. Build your own (it's a hundred lines of C) or use Memcached or Redis or Snowflake or any other number of options.
+* Primary key of all sharded tables begins with the sharding key. This takes advantage of InnoDB clustered indexes to allow efficient export/import of data by ranges. This isn't strictly mandatory, but it makes an absolutely huge performance difference versus using a secondary index.
+* Uniform MySQL configuration between masters and slaves: log-slave-updates, unique server-id, generic log-bin and relay-log names, replication user/grants everywhere.
+* Redundant rows temporarily on the wrong shard don’t matter to your application. Hopefully these would only matter if you're doing aggregate queries (like COUNTs) over the whole data set... but if your data set is big enough to shard, these already don't scale, so it shouldn't really matter.
+
+For more information on the shard split process implemented by \Jetpants, including diagrams of each stage of the process, please see {Evan Elias's presentation at Velocity Europe 2011}[https://github.com/tumblr/jetpants/blob/master/doc/VelocityEurope2011Presentation.pdf?raw=true], starting at slide 19.

data/etc/jetpants.yaml.sample

@@ -0,0 +1,58 @@
+# Jetpants supports a global settings file at /etc/jetpants.yaml,
+# as well as user-specific settings files at ~/.jetpants.yaml
+
+mysql_schema:           mydb
+mysql_app_user:         myuser
+mysql_app_password:     foo
+mysql_repl_user:        replication_user
+mysql_repl_password:    bar
+
+# Users created by Jetpants are given access from these IPs
+mysql_grant_ips:        [10.%, 192.168.%]
+
+# Users created by Jetpants are given these MySQL privileges
+mysql_grant_privs:
+    - SELECT
+    - INSERT
+    - UPDATE
+    - DELETE
+    - REFERENCES
+    - CREATE TEMPORARY TABLES
+    - LOCK TABLES
+    - EXECUTE
+
+# Define what directory to put exported files in. If you have /var/lib/mysql
+# on an SSD RAID mount and the rest of your OS on a rotational disk mount,
+# it's fine to put the export_location on the rotational disk, which probably
+# has higher capacity anyway.
+export_location:        /some/path/on/root/partition
+
+# List all tables defined on the shards, along with what column name corresponds
+# to your app's sharding key (to determine which shard a given row lives on),
+# and how many "chunks" to split the data set into when doing an import or
+# export.
+sharded_tables:
+    posts:
+        sharding_key: user_id
+        chunks: 200
+    likes:
+        sharding_key: from_user_id
+        chunks: 100
+    some_small_table:
+        sharding_key: user_id
+        # chunks defaults to 1 if omitted.
+    following:
+        sharding_key: [from_user_id, to_user_id]
+        # multiple sharding key columns: each row may have to exist on two shards
+        # omit chunks since cannot use chunked export/import with multiple sharding keys
+
+plugins:
+    simple_tracker:
+        tracker_data_file_path: /var/jetpants/assets.json
+        app_config_file_path: /var/www/your-web-app/databases.yaml
+
+    plugin_with_no_settings:
+
+    plugin_with_settings:
+        foo: bar
+        flim: flam

data/lib/jetpants.rb

@@ -0,0 +1,100 @@
+require 'sequel'
+require 'net/ssh'
+require 'yaml'
+
+module Jetpants; end
+
+$LOAD_PATH.unshift File.join(File.dirname(__FILE__), 'jetpants'), File.join(File.dirname(__FILE__), '..', 'plugins')
+%w(callback table host db pool topology shard monkeypatch).each {|g| require g}
+
+# Since Jetpants is extremely multithreaded, we need to force uncaught exceptions to
+# kill all threads in order to have any kind of sane error handling.
+Thread.abort_on_exception = true
+
+# Namespace for the Jetpants toolkit.
+module Jetpants
+  
+  # Establish default configuration values, and then merge in whatever we find globally
+  # in /etc/jetpants.yaml and per-user in ~/.jetpants.yaml
+  @config = {
+    'max_concurrency'         =>  40,       # max threads/conns per database
+    'standby_slaves_per_pool' =>  2,        # number of standby slaves in every pool
+    'mysql_schema'            =>  'test',   # database name
+    'mysql_app_user'          =>  false,    # mysql user for application
+    'mysql_app_password'      =>  false,    # mysql password for application
+    'mysql_repl_user'         =>  false,    # mysql user for replication
+    'mysql_repl_password'     =>  false,    # mysql password for replication
+    'mysql_root_password'     =>  false,    # mysql root password. omit if specified in /root/.my.cnf instead.
+    'mysql_grant_ips'         =>  ['192.168.%'],  # mysql user manipulations are applied to these IPs
+    'mysql_grant_privs'       =>  ['ALL'],  # mysql user manipulations grant this set of privileges by default
+    'export_location'         =>  '/tmp',   # directory to use for data dumping
+    'verify_replication'      =>  true,     # raise exception if the 2 repl threads are in different states, or if actual repl topology differs from Jetpants' understanding of it
+    'plugins'                 =>  {},       # hash of plugin name => arbitrary plugin data (usually a nested hash of settings)
+    'ssh_keys'                =>  nil,      # array of SSH key file locations
+    'sharded_tables'          =>  [],       # array of name => {sharding_key=>X, chunks=>Y} hashes
+  }
+  %w(/etc/jetpants.yaml ~/.jetpants.yml ~/.jetpants.yaml).each do |path|
+    overrides = YAML.load_file(File.expand_path path) rescue {}
+    @config.merge! overrides
+  end
+  
+  class << self
+    # A singleton Jetpants::Topology object is accessible from the global 
+    # Jetpants module namespace.
+    attr_reader :topology
+    
+    # Returns true if the specified plugin is enabled, false otherwise.
+    def plugin_enabled?(plugin_name)
+      !!@config['plugins'][plugin_name]
+    end
+    
+    # Returns a hash containing :user => username string, :pass => password string
+    # for the MySQL application user, as found in Jetpants' configuration. Plugins
+    # may freely override this if there's a better way to obtain this password --
+    # for example, if you already distribute an application configuration or
+    # credentials file to all of your servers.
+    def app_credentials
+      {user: @config['mysql_app_user'], pass: @config['mysql_app_password']}
+    end
+    
+    # Returns a hash containing :user => username string, :pass => password string
+    # for the MySQL replication user, as found in Jetpants' configuration. Plugins
+    # may freely override this if there's a better way to obtain this password --
+    # for example, by parsing master.info on a slave in your topology.
+    def replication_credentials
+      {user: @config['mysql_repl_user'], pass: @config['mysql_repl_password']}
+    end
+    
+    # Proxy missing top-level Jetpants methods to the configuration hash,
+    # or failing that, to the Topology singleton.
+    def method_missing(name, *args, &block)
+      if @config.has_key? name.to_s
+        @config[name.to_s]
+      elsif name.to_s[-1] == '=' && @config.has_key?(name.to_s[0..-2])
+        var = name.to_s[0..-2]
+        @config[var] = args[0]
+      elsif @topology.respond_to? name
+        @topology.send name, *args, &block
+      else
+        super
+      end
+    end
+    
+    def respond_to?(name, include_private=false)
+      super || @config[name] || @topology.respond_to?(name)
+    end
+  end
+
+  # Load plugins at this point, to allow them to override any of the previously-defined methods
+  # in any of the loaded classes
+  (@config['plugins'] || {}).each do |name, attributes|
+    begin
+      require "#{name}/#{name}"
+    rescue LoadError
+      require name
+    end
+  end
+  
+  # Finally, initialize topology object
+  @topology = Topology.new
+end