repctl 0.0.1

data/.gitignore

@@ -0,0 +1,4 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in repctl.gemspec
+gemspec

data/README.md

@@ -0,0 +1,80 @@
+# Repctl - Manage Replication among a set of SQL Servers 
+
+ `repctl` is a utility to configure, reconfigure, start, stop, crash, generate
+ workloads, dump, restore, benchmark, and monitor a set of SQL servers for
+ development environments.  Replication relationships can be set up among server
+ instances with a single command. While running a load generator or benchmark,
+ the replication status, including current lag, can be seen in a continuously
+ updated display. A slave can be added to an existing server that already has
+ data.
+
+The `repctl` gem includes a _Thor_ script that makes all the `repctl`
+functionality available at the command line.
+
+## Limitations
+
+Currently only MySQL is supported but PostgresSQL will soon be added.  All the
+server instances must run on a single host.  This restriction may be soon lifted
+as well.
+
+## Installation
+
+You will need to have a local installation installation of MySQL.  You do not
+need to do anything to configure the installation.  For example, if you compile
+MySQL from source, then do `make install`, then you are done!  No MySQL
+post-installation steps are necessary.  All post-install configuration is
+handled by `repctl`.
+
+The top
+
+== Available Commands
+
+    tethys:repctl mbs$ thor list
+    mysql
+    -----
+    thor mysql:change_master MASTER SLAVE FILE POSITION  # Execute CHANGE MASTER TO on the SLAVE.
+    thor mysql:cluster_user INSTANCE                     # Create the cluster user account on a MySQL instance.
+    thor mysql:config INSTANCE                           # Initialize the data directory for a new instance.
+    thor mysql:config_all                                # Initialize the data directories for all instances.           
+    thor mysql:crash INSTANCE                            # Crash a running MySQL server.
+    thor mysql:dump INSTANCE [DUMPFILE]                  # Dump all databases after FLUSH TABLES WITH READ LOCK             
+    thor mysql:repl_user INSTANCE                        # Create the replication user account on a MySQL insta...
+    thor mysql:reset INSTANCE                            # Remove database and restart MySQL server.
+    thor mysql:reset_all                                 # Remove all databases and restart MySQL instances.          
+    thor mysql:restore INSTANCE [DUMPFILE]               # Restore INSTANCE from a 'mysqldump' file DUMPFILE.
+    thor mysql:start_slave SLAVE                         # Issue START SLAVE on the SLAVE MySQL instance.
+    thor mysql:status                                    # Show the status of replication.
+    thor mysql:stop INSTANCE                             # Stop a running MySQL server instance.
+    thor mysql:stop_all                                  # Stop all the MySQL servers.
+
+    setup
+    -----
+    thor setup:add_slave MASTER SLAVE  # Master has some data that is used to initialize the slave.
+    thor setup:repl_pair MASTER SLAVE  # Set up a single master/slave replication pair from the very beginning.
+
+    utils
+    -----
+    thor utils:bench [INSTANCE] [PROPS]                  # Run the Tungsten Bristlecone benchmarker. The INSTAN...
+    thor utils:create_db [INSTANCE] [DBNAME]             #  "Create a database on a MySQL instance. INSTANCE de...
+    thor utils:create_tbl [INSTANCE] [DBNAME] [TBLNAME]  #  Create a database table. INSTANCE defaults to DEFAU...
+    thor utils:gen_rows [INSTANCE], [DBNAME], [TBLNAME]  #  Add rows to a table that was created by "utils:crea...
+
+== Configuring Simple Repctl
+
+This tool needs some configuration before you can use it.
+
+You should have an valid MySQL installation and the Thor gem installed.  Your existing MySQL server will not be affected by the +repctl+ script.  However, binaries from this installation will be reused.  In the +config.rb+ file set the constants:
+
+* MYSQL_HOME -- the location of the local MySQL installation
+* DATA_HOME -- the location of the directory where per-MySQL server data directories are created.
+* DUMP_DIR -- the location where you want dump files to be stored
+* RELAY_LOG -- adjust this according to your hostname
+
+Next, define the potential instances you want to create.  Edit the <tt>servers.yml</tt> file as appropriate.
+
+Finally, edit the existing <tt>my*.cnf*</tt> files to at least have the correct <tt>datadir</tt> defined.
+
+== Using Simple Repctl
+
+You are now ready to rock.  Run <tt>thor mysql:start_all</tt> to start up all the servers listed in your +servers.yml+ file. Instead or subsequently, you can run <tt>thor setup:repl_pair 1 2</tt> to reset everything and create a master/slave replication pair. Start up some load to the MySQL master (at socket +/tmp/mysql1.sock+, by default), then watch the replication status change by running <tt>thor mysql:status -s 1 2 -c 5</tt> to see a continuous update of the status, updated every 5 seconds.  Add a new slave using instance +3+, which may or may not be running and may or may not have its data directory initialized, by running <tt>thor setup:add_slave 1 3</tt>.  This does a dump on the master and a restore on the slave and restarts replication using the proper replication coordinates.
+

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/bin/mysql.thor

@@ -0,0 +1,431 @@
+require 'socket'
+require 'repctl'
+
+# Thor has the 'helpful' property that any Thor task gets executed only
+# once.  The 'Helpers' module makes many Thor tasks available as ordinary
+# Ruby functions for internal use.  The corresponding Thor tasks usually
+# delegate to the corresponding helper function with 'super'.
+module Helpers
+  
+  include Repctl::Config
+  include Repctl::Servers
+  
+  def start(instance)
+    say "Starting instance #{instance}.", :green
+    do_start(instance)
+  end
+  
+  def start_all
+    all_instances.each do |instance|
+      start(instance)
+    end
+  end
+  
+  def stop(instance)
+    say "Stopping instance #{instance}.", :green
+    do_admin(instance, "shutdown")
+  end
+  
+  def stop_all
+    all_live_instances.each do |instance|
+      stop(instance)
+    end
+  end
+  
+  def config(instance)
+    say "Initializing new data directory for instance #{instance}."
+    do_config(instance)
+  end
+  
+  def config_all
+    all_instances.each do |instance|
+       config(instance)
+     end
+  end
+
+  def secure_accounts(instance)
+    do_secure_accounts(instance)
+  end
+
+  def secure_accounts_all
+    Repctl::Servers.all_live_instances.each do |instance|
+       secure_accounts(instance)
+     end
+  end
+  
+  def reset(instance)
+    stop(instance)
+    config(instance)
+    start(instance)
+    secure_accounts(instance)
+  end
+ 
+  def restart(instance)
+    stop(instance)
+    start(instance)
+  end
+
+  def reset_all
+    all_instances.each do |instance|
+       reset(instance)
+     end
+  end
+  
+  def change_master(master, slave, file, position)
+    say "Changing master: master = #{master}, slave = #{slave}, file = #{file}, position = #{position}"
+    do_change_master(master, slave, :file => file, :position => position)
+  end
+      
+  def start_slave(slave)
+    say "Starting slave #{slave}", :green
+    run_mysql_query(slave, "START SLAVE")
+  end
+  
+  def crash(instance)
+    say "Crashing instance #{instance}", :red
+    do_crash(instance)
+  end
+
+  def repl_user(instance)
+    say "Creating replication account on instance #{instance}.", :green
+    do_repl_user(instance)
+  end
+
+  def cluster_user(instance = 1)
+    say "Installing cluster user for instance #{instance}.", :green
+    do_cluster_user(instance)
+  end
+      
+end # Module helpers
+
+class Mysql < Thor
+  
+  include Thor::Actions
+  include Repctl::Config
+  include Repctl::Commands
+  include Repctl::Servers
+  include Helpers
+  
+  desc "start INSTANCE", "Ensure that the given MySQL server instance is running."
+  def start(instance)
+    super
+  end
+  
+  desc "start_all", "Start all the MySQL instances."
+  def start_all
+    super
+  end
+  
+  desc "stop INSTANCE", "Stop a running MySQL server instance."
+  def stop(instance)
+    super
+  end
+  
+  desc "stop_all", "Stop all the MySQL servers."
+  def stop_all
+    super
+  end
+  
+  desc "config INSTANCE", "Initialize the data directory for a new instance."
+  def config(instance)
+    super
+  end
+  
+  desc "config_all", "Initialize the data directories for all instances."
+  def config_all
+    super
+  end 
+  
+  desc "secure_accounts INSTANCE", "Add passwords for root and anonymous accounts."
+  def secure_accounts(instance)
+    super
+  end
+
+  desc "secure_accounts_all", "Add passwords for root and anonymous accounts for all instances."
+  def secure_accounts_all
+    super
+  end
+  
+  desc "reset", "Remove the database data directory, reconfigure and restart the instance."
+  def reset(instance)
+    super
+  end
+
+  desc "reset_all", "Remove all databases and restart MySQL instances."
+  def reset_all
+    super
+  end
+  
+  desc "start_slave SLAVE", "Issue START SLAVE on the SLAVE MySQL instance."
+  def start_slave(slave)
+    super
+  end
+  
+  desc "change_master MASTER SLAVE FILE POSITION", "Execute CHANGE MASTER TO on the SLAVE."
+  def change_master(master, slave, file, position)
+    super
+  end
+  
+  desc "crash INSTANCE", "Crash a running MySQL server."
+  def crash(instance)
+    super
+  end
+  
+  desc "repl_user INSTANCE", "Create the replication user account on a MySQL instance."
+  def repl_user(instance)
+    super
+  end
+  
+  desc "cluster_user INSTANCE", "Create the cluster user account on a MySQL instance."
+  def cluster_user(instance)
+    super
+  end
+  
+  desc "repl_status", "Show the status of replication."
+   method_option :continuous, :aliases => "-c", :type => :numeric,
+    :desc => "Continuous output at specified interval (in seconds)."
+  method_option :servers, :aliases => "-s", :type => :array,
+    :desc => "Only check the status of given servers."
+  def repl_status
+    todos = options[:servers] || all_live_instances
+    unless todos.any?
+      say "No Running Servers.", :blue
+      return
+    end
+    header = sprintf("%-5s%-25s%-25s%-25s%-8s\n",
+      "inst", "master", "received", "applied", "lag")
+      
+    loop do  
+      say header, :blue
+    
+      todos.each do |i|
+        coordinates = get_coordinates(i)
+        slave_status = get_slave_status(i)
+        is_slave = !(slave_status["Error"] == "MySQL server is not a slave.")
+        master_file = coordinates[:file]
+        master_pos =  coordinates[:position]
+        if is_slave
+          recv_file = slave_status["Master_Log_File"]
+          recv_pos = slave_status["Read_Master_Log_Pos"]
+          apply_file = slave_status["Relay_Master_Log_File"]
+          apply_pos = slave_status["Exec_Master_Log_Pos"]
+          lag = slave_status["Seconds_Behind_Master"]
+        end
+          
+        format = "%-5d%16s:%-8d"
+        if is_slave
+          if lag
+            lag = lag.to_s
+          else
+            lag = "-"
+          end
+          format += "%16s:%-8d%16s:%-8d%-8s"
+          str = sprintf(format, i, master_file, master_pos, recv_file, recv_pos,
+            apply_file, apply_pos, lag)
+        else
+          str = sprintf(format, i, master_file, master_pos)
+        end
+      
+        say str + "\n", :yellow
+      end
+      break unless options[:continuous]
+      sleep options[:continuous]
+      say ""
+    end
+  end
+    
+  desc "dump INSTANCE [DUMPFILE]", "Dump all databases after FLUSH TABLES WITH READ LOCK"
+  def dump(instance, dumpfile = DEFAULT_DUMPFILE)
+    coordinates = do_dump(instance, dumpfile)
+    file = coordinates[:file]
+    position = coordinates[:position]
+    puts "(#{file}, #{position})"
+    [file, String(position)]
+  end
+  
+  desc "restore INSTANCE [DUMPFILE]", "Restore INSTANCE from a \'mysqldump\' file DUMPFILE."
+  def restore(slave, dumpfile = DEFAULT_DUMPFILE)
+    do_restore(slave, dumpfile)
+  end
+  
+end
+
+class Utils < Thor
+  
+  include Thor::Actions
+  include Repctl::Config
+  include Repctl::Commands
+  include Repctl::Servers
+  include Helpers
+  
+  DEFAULT_MASTER = 1
+  
+  desc "bench [INSTANCE] [PROPS]", "Run the Tungsten Bristlecone benchmarker.
+  The INSTANCE specifies the instance to perform all operations to, and PROPS
+  is the properties file to use. The INSTANCE defaults to #{DEFAULT_MASTER} and
+  the properties file defaults to #{BENCHMARK_PROPERTIES}."
+  def bench(instance = DEFAULT_MASTER, props = nil)
+    props ||= BENCHMARK_PROPERTIES
+    invoke :create_db, [instance, "widgets"]
+    run("#{BENCHMARK} -props #{props}", :verbose => true, :capture => false)
+  end
+  
+  desc "create_db [INSTANCE] [DBNAME]", <<-EOS
+    "Create a database on a MySQL instance.  INSTANCE defaults to DEFAULT_MASTER,
+    and DBNAME defaults to "widgets".
+  EOS
+  method_option :replace, :type => :boolean, :aliases => "-r",
+    :desc => "drop and recreate the database"
+  def create_db(instance = DEFAULT_MASTER, dbname = "widgets")
+    run_mysql_query(instance, "DROP DATABASE IF EXISTS #{dbname}") if options[:replace]
+    run_mysql_query(instance, "CREATE DATABASE IF NOT EXISTS #{dbname}")
+  end
+  
+  desc "create_tbl [INSTANCE] [DBNAME] [TBLNAME]", <<-EOS
+    Create a database table. INSTANCE defaults to DEFAULT_MASTER, DBNAME defaults
+    to "widgets" and TBLNAME defaults to "users".  The table schema is fixed.
+  EOS
+  method_option :replace, :type => :boolean, :aliases => "-r",
+    :desc => "drop and recreate the table"
+  def create_tbl(instance = DEFAULT_MASTER, dbname = "widgets", tblname = "users")
+    invoke :create_db, [instance, dbname], :replace => false
+    run_mysql_query(instance, 
+      "DROP TABLE IF EXISTS #{dbname}.#{tblname}") if options[:replace]
+    cmd = <<-EOS
+      CREATE TABLE #{dbname}.#{tblname} (
+        id	INT NOT NULL,
+        last_name CHAR(30) NOT NULL,
+        first_name CHAR(30) NOT NULL,
+        credentials VARCHAR(32768) NOT NULL,
+        PRIMARY KEY (id),
+        INDEX name (last_name,first_name)
+      )
+    EOS
+    run_mysql_query(instance, cmd)
+  end
+  
+  desc "gen_rows [INSTANCE], [DBNAME], [TBLNAME]", <<-EOS
+    Add rows to a table that was created by "utils:create_tbl". INSTANCE defaults
+    to DEFAULT_MASTER, DBNAME defaults to "widgets", and TBLNAME defaults to "users".
+  EOS
+  method_option :delay, :type => :numeric, :aliases => "-d", :default => 0, 
+    :desc => "sleep for the specified number of milliseconds between row inserts."
+    method_option :count, :type => :numeric, :aliases => "-c", :default => 1000, 
+       :desc => "number of rows to insert"
+    method_option :size, :type => :numeric, :aliases => "-s", :default => 100, 
+      :desc => "the approximate size of the record to insert (in bytes)."
+    method_option :forever, :type => :boolean, :aliases => "-f",
+      :desc => "run forever, ignoring COUNT option."
+    method_option :verbose, :type => :boolean, :aliases => "-v",
+      :desc => "print a '.' for each row inserted."
+  def gen_rows(instance = DEFAULT_MASTER, dbname = "widgets", tblname = "users")
+    invoke :create_tbl, [instance, dbname], :replace => true
+    size = options[:size]
+    size ||= 100
+    size = [size, 32768].min
+    data = IO.read("#{Mysql::DATA_HOME}/words.txt", size)
+    id = 1
+    count = 0
+    
+    loop do
+      cmd = <<-EOS
+        INSERT INTO #{dbname}.#{tblname} VALUES (
+          #{id},
+          'Fillmore',
+          'Millard',
+          '#{data}'
+        )
+      EOS
+      run_mysql_query(instance, cmd)
+      putc "." if options[:verbose]
+      id += 1
+      count += 1
+      break if (count >= options[:count] and (not options[:forever]))
+      msecs = options[:delay]
+      sleep(msecs / 1000.0) if msecs > 0
+    end
+    
+  end
+  
+end
+
+class Setup < Thor
+  
+  include ::Thor::Actions
+  include Repctl::Config
+  include Repctl::Commands
+  include Repctl::Servers
+  include Helpers
+  
+  #
+  # Setting Up Replication with New Master and Slaves.
+  # Here, we stop all MySQL servers, remove the data directories, reinitialize
+  # the data directories, restart the servers, and set up a master/slave
+  # relationship.
+  #
+  desc "repl_pair MASTER SLAVE",
+    "Set up a single master/slave replication pair from the very beginning."
+  def repl_pair(master, slave)
+    say "master is #{master}, slave is #{slave}", :green
+    reset(master)
+    reset(slave)
+    cluster_user(master)
+    repl_user(master)
+    coordinates = get_coordinates(master)
+    file = coordinates[:file]
+    position = coordinates[:position]
+    say "File is #{file}, Position is #{position}", :green
+    change_master(master, slave, file, position)
+    start_slave(slave)
+  end
+
+  desc "repl_trio MASTER SLAVE1 SLAVE2",
+    "Set up a single master and two slave replication cluster."
+  method_option :reset, :type => :boolean, :default => true
+  def repl_trio(master, slave1, slave2)
+    say "master is #{master}, slaves are #{slave1} and #{slave2}", :green
+    if options[:reset]
+      reset(master)
+      reset(slave1)
+      reset(slave2)
+      cluster_user(master)
+      repl_user(master)
+    else
+      restart(master)
+      restart(slave1)
+      restart(slave2)
+    end
+    coordinates = get_coordinates(master)
+    file = coordinates[:file]
+    position = coordinates[:position]
+    say "File is #{file}, Position is #{position}", :green
+    change_master(master, slave1, file, position)
+    start_slave(slave1)
+    change_master(master, slave2, file, position)
+    start_slave(slave2)
+  end
+    
+  #
+  # Setting Up Replication with Existing Data using the 'mysqldump' utility. The
+  # master has existing data.
+  #
+  desc "add_slave MASTER SLAVE", "Master has some data that is used to initialize the slave."
+  method_option :populate, :type => :boolean, :default => false
+  def add_slave(master, slave)
+    reset(slave)
+    
+    if options[:populate]
+      invoke "utils:bench", [master, "/opt/MySQL/b2.properties"]
+    end
+    file, position = invoke "mysql:dump", [master]
+    
+    # Slave is running, but it is not yet configured as a slave.
+    # Load slave from the dump file...
+    invoke "mysql:restore", [slave]
+    
+    change_master(master, slave, file, position)
+    start_slave(slave)
+  end
+  
+end
+

data/config/bristlecone.properties

@@ -0,0 +1,29 @@
+url=jdbc:mysql://localhost:3307/widgets
+user=cluster
+password=secret
+type=MySQL 5.6.2-m5
+analyzeCmd=select 1
+
+# Benchmark test to compare clustered and non-clustered write performance. 
+#
+# We execute INSERT statements with varying numbers of clients and tables. 
+#
+# To invoke this test try the following command. 
+#   $benchmark.sh -props WriteSimpleScenario.properties
+
+# Scenario name. 
+scenario=com.continuent.bristlecone.benchmark.scenarios.WriteSimpleScenario
+
+# Database connection information.  
+# include=connection_postgresql.properties|connection_pcluster.properties
+
+# Test duration and number of threads.  
+bound=duration
+duration=60
+threads=1|10|20
+
+# Database table information. 
+tables=1|16
+datatype=varchar
+datawidth=100
+datarows=100

data/config/config.rb

@@ -0,0 +1,50 @@
+module Repctl
+  
+  module Config
+    
+    # The location of the local MySQL installation.
+    MYSQL_HOME = "/usr/local/mysql"
+
+    ROOT_PASSWORD = "har526"
+
+    SERVER_CONFIG = File.expand_path("../servers.yml", __FILE__)
+
+    # Define the directory where subdirectores for data will be created
+    # for each MySQL server instance.  This should agree with the 
+    # per server'data_dir' property in the servers.yml file, and it should
+    # also agree with the 'datadir' property in the server's configuration
+    # file (my*.cnf).  
+    DATA_HOME = "/opt/MySQL/instances"
+
+    # The home directory of Continuent's open source replicator.  Eventually,
+    # a command will be added to this script to switch between MySQL native
+    # replication and Continuent's open-source Tungsten replicator.
+    REPLICATOR_HOME = "/opt/continuent/replicator"  
+
+    # For simplicity, we're using the load-generator/benchmarker that comes
+    # in the Continuent source package.  This may be replaced with sql-bench
+    # from the MySQL source distribution.
+    BENCHMARK = "#{REPLICATOR_HOME}/bristlecone/bin/benchmark.sh"
+    BENCHMARK_PROPERTIES = File.expand_path("../bristlecone.properties", __FILE__)
+
+    # Set this to the directory where you want dump files to be stored.
+    DUMP_DIR = "#{DATA_HOME}/dump"
+
+    # The default name of the dump file in the DUMP_DIR directory.
+    DEFAULT_DUMPFILE = "dbdump.db"
+
+    # User name and password for the replication account, used only internally by
+    # the replication processes.
+    REPLICATION_USER = "repl"
+    REPLICATION_PASSWORD = "secret"
+
+    # Replication account is set up as %.#{REPLICATION_DOMAIN}.
+    REPLICATION_DOMAIN = "thirdmode.com"
+
+    # A minor convenience. 
+    DEFAULT_MASTER = 1
+
+    # Typically, this is of the form #{HOSTNAME}-relay-bin'.
+    RELAY_LOG = "deimos-relay-bin"
+  end
+end