sqlup 0.0.1

data/History.txt

@@ -0,0 +1,5 @@
+== 0.0.1 / 2007-06-18
+
+* 1 major enhancement
+  * Birthday!
+

data/Manifest.txt

@@ -0,0 +1,38 @@
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+bin/sqlup
+bin/sqlup_control
+test/unit/mysql_backup/librarian/librarian_test.rb
+test/unit/mysql_backup/librarian/backup_test.rb
+test/unit/mysql_backup/librarian/backup_collection_test.rb
+test/unit/mysql_backup/utilities/test_helper.rb
+test/unit/mysql_backup/storage/test_helper.rb
+test/unit/mysql_backup/storage/s3_test.rb
+test/unit/mysql_backup/entity/files/myisam_test.rb
+test/unit/mysql_backup/entity/files/files_test.rb
+test/unit/mysql_backup/entity/files/innodb_test.rb
+test/unit/mysql_backup/entity/logs_test.rb
+test/unit/mysql_backup/entity/mysqldump_test.rb
+test/unit/mysql_backup/entity/identifier_test.rb
+test/unit/mysql_backup/test_helper.rb
+test/unit/mysql_backup/server_test.rb
+test/unit/test_helper.rb
+lib/mysql_backup/librarian/backup.rb
+lib/mysql_backup/librarian/backup_collection.rb
+lib/mysql_backup/entity/files/myisam.rb
+lib/mysql_backup/entity/files/innodb.rb
+lib/mysql_backup/entity/identifier.rb
+lib/mysql_backup/entity/logs.rb
+lib/mysql_backup/entity/mysqldump.rb
+lib/mysql_backup/entity/files.rb
+lib/mysql_backup/storage/s3.rb
+lib/mysql_backup/utilities/factory_create_method.rb
+lib/mysql_backup/server.rb
+lib/mysql_backup/librarian.rb
+lib/mysql_backup/storage.rb
+lib/mysql_backup/entity.rb
+lib/mysql_backup.rb
+lib/sqlup.rb
+config/environment.rb

data/README.txt

@@ -0,0 +1,122 @@
+sqldump
+    by James Moore
+
+== DESCRIPTION:
+
+sqlup is a set of libraries and utilities to automate backups of a MySQL server running on Amazon's EC2
+service to Amazon's S3 storage service.  
+
+=== Quick start
+
+Create an S3 bucket named 'sqlup' for your backups.  (You'll need to choose a different name for your bucket.)
+
+Install the gem:
+  gem install sqlup
+
+Put your S3 keys in a .sqluprc file in the backup user's home directory:
+  cat ~/.sqluprc
+  access_key_id: xxxxxxxxxxxxxxxx
+  secret_access_key: xxxxxxxxxxxxxxxxx
+
+Backup your database:
+  sqlup binary -bucket sqlup
+  
+See what was written:
+  sqlup ls -bucket sqlup
+  
+Start the backup daemon that will store the binary logs as they're written:
+  sqlup_control start -- -logs_delay 10 log_daemon -bucket sqlup
+  
+Retrieve the backup files (where full:type_mysqldump:log_file_domU-12-31-35-00-35-42-bin.000019:log_position_0000000169 is the name of the full backup you want):
+  sqlup get_logs -bucket sqlup -d /tmp
+  sqlup get -bucket sqlup -d /tmp -name full:type_mysqldump:log_file_domU-12-31-35-00-35-42-bin.000019:log_position_0000000169
+
+=== Usage
+
+Get help:
+  sqlup �h 
+Back up the data files: 
+  sqlup binary -bucket sqlup
+Back up a mysqldump run: 
+  sqlup mysqldump -bucket sqlup
+Start the sqlup daemon to take a backup every 10 seconds, to the bucket 'sqlup', with a pidfile in /tmp:
+  export SQLUP_PID_DIR=/tmp
+  sqlup_control start -- -logs_delay 10 log_daemon -bucket sqlup
+Get a list of the backup files:
+  sqlup ls -bucket sqlup
+Remove a backup file from the bucket 'sqlup':
+  sqlup -bucket sqlup rm -name log:type_complete:log_file_fnord.000002
+Remove obsolete current logs:
+  obsolete_files=`bin/sqlup -bucket sqlup ls -backup_type log_current -skip_most_recent 3`
+  for i in $obsolete_files ; do
+  sqlup -bucket sqlup rm -name $i
+  done
+  
+You need to specify your access codes using the AWS::S3 environment variables:
+
+  export AMAZON_ACCESS_KEY_ID='xxxxxxx'
+  export AMAZON_SECRET_ACCESS_KEY='xxxxxxxxxx'
+
+  
+It�s primarily targeted at MySQL servers running on Amazon�s EC2 virtual server system,
+with backups sent to Amazon�s S3 storage service.
+
+=== What it backs up
+There are three parts to a MySQL backup system:
+
+1.	The actual MySQL data files (by default, the files in /var/lib/mysql).
+2.	Full dumps using the mysqldump tool.
+3.	The binary log files.
+
+Normally, you�d make a full copy of your system using either #1 or #2, and then have sqlup make backups of the binary logs to S3 every N seconds.
+Backups are tarred, gzip�ed, and split to fit into S3 buckets.
+
+=== FAQs
+
+*  Why would I want to make copies of the data files instead of using mysqldump?
+
+Speed.  Recovering mysqldump files can take a while; recovering when you have copies of the data files is much faster.  If you�ve got a small database, though, just use mysqldump.  You can test this yourself; just do a mysqldump of your current database, and run it against a MySQL server on another machine.  If that�s fast enough for you, you don�t need to worry about the binary files.
+
+* Can I use the database while it�s being backed up?
+
+Yes, sort of.  The binary backup is going to lock every table in the system until it�s finished making a tarball of all the data files.  (A future enhancement will be to use a versioning file system.)  As long as no one writes to the database, reads will continue.  However, as soon as someone wants to write to a table, it�s very likely that all future readers will be blocked until the backup is finished writing temporary files to disk.
+
+* Is there an automated recovery system?
+
+Not yet.  There�s a command to get the backups back from S3, but you need to go through the standard mysql recovery process by hand once you have the files.
+
+
+== FEATURES/PROBLEMS:
+
+== TODO:
+
+1.  Improve the recovery process.
+2.  Improve the ability to do backups from slaves.
+
+== REQUIREMENTS:
+
+* You must run MySQL with binary logging enabled.  No logs == no backups.
+
+== INSTALL:
+
+  gem install sqlup
+  
+== LICENSE:
+
+sqlup - a backup tool for MySQL, EC2, and S3
+
+Copyright (C) 2007 James Moore
+
+This program is free software; you can redistribute it and/or
+modify it under the terms of the GNU General Public License
+as published by the Free Software Foundation; either version 2
+of the License, or (at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

data/Rakefile

@@ -0,0 +1,18 @@
+require 'rubygems'
+require 'hoe'
+require './lib/mysql_backup'
+
+Hoe.new('sqlup', MysqlBackup::VERSION) do |p|
+  p.rubyforge_name = 'sqlup'
+  p.author = 'James Moore'
+  p.email = 'banshee@restphone.com'
+  p.summary = "A backup tool for saving MySQL data to Amazon's S3 service"
+  p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
+  p.extra_deps << ['named_arguments', '>= 0.0.5']
+  p.extra_deps << ['optiflag', '>= 0.6.5']
+end
+
+Rake::TestTask.new do |t|
+  t.test_files = FileList['test/**/*_test.rb']
+  t.verbose = true
+end

data/bin/sqlup

@@ -0,0 +1,107 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'optiflag'
+require 'pp'
+require 'logger'
+require File.dirname(__FILE__) + '/../lib/mysql_backup'
+
+module Example extend OptiFlagSet
+  keyword 'ls', :description => "Print a list of all backups."
+  keyword 'mysqldump', :description => 'Backup using mysqldump'
+  keyword 'binary', :description => 'Backup mysql data files'
+  keyword 'logs', :description => 'Backup the binary log files'
+  keyword 'log_daemon', :description => "Backup the binary logs every --logs_delay seconds."
+  keyword 'get', :description => 'Get a binary or a mysqldump backup.  Requires --name'
+  keyword 'get_logs', :description => "Get all the current and complete logs"
+  keyword 'rm', :description => 'Remove a backup (use -name to specify which backup to remove)'
+  flag 'bucket', :description => "The name of the S3 bucket"
+  optional_flag 'backup_type', :description => "(ls) The kind of backup"
+  optional_flag 'skip_most_recent', :description => "(ls) Do not display the most recent N logs.  Usually used to get a list of obsolete current logs."
+  optional_flag 'name', :description => '(get, rm) The name of the backup to process'
+  optional_flag "engine", :description => "The storage engine.  The default (and currently the only option) is s3."
+  optional_flag "get_directory", :alternate_forms => 'd', :description => "The directory to write files for the get command (defaults to the current directory)"
+  optional_flag 'logs_delay', :description => 'The delay (in seconds, as a float) to use for log_daemon.  Defaults to 1.'
+  optional_switch_flag 'verbose', :alternate_forms => %w(v) do
+    description 'Send verbose output'
+  end
+  and_process!
+end 
+
+flags = ARGV.flags
+
+if flags.get && !flags.name
+  raise RuntimeError, "You must specify the name of a backup (--name)"
+end
+
+if flags.verbose?
+  log = Logger.new $stderr
+  log.level = Logger::DEBUG
+else
+  log = nil
+end
+
+engine = flags.engine || 's3'
+
+case engine
+when 's3'
+  # Read the keys from the $HOME/.sqluprc file
+  args = {:log => log, :bucket => flags.bucket}
+  p = Pathname.new(ENV['HOME']) + '.sqluprc'
+  if p.readable?
+    keys = YAML::load p.open
+    keys.each_pair do |k,v|
+      args[k.to_sym] = v
+    end
+  else
+    raise RuntimeError, ("Cannot open #{p.to_s}; that file should have two lines, one for access_key_id: yourkey and the other for secret_access_key: yourotherkey")
+  end
+  engine = MysqlBackup::Storage::S3.new args
+  engine.read_existing_objects
+end
+librarian = MysqlBackup::Librarian.new :log => log, :storage => engine, :user => 'root', :bucket => 'banshee', :sock => '/var/lib/mysql/mysql.sock'
+
+begin
+  if flags.ls?
+    klass = Kernel.const_get flags.backup_type rescue nil
+  klass ||= case flags.backup_type
+    when /log_complete/: MysqlBackup::Librarian::Backup::Log::Complete
+    when /log_current/: MysqlBackup::Librarian::Backup::Log::Current
+    when /log/: MysqlBackup::Librarian::Backup::Log
+    end
+    klass ||= Object
+    range = 0..-1
+    if flags.skip_most_recent
+      range = 0..(-(flags.skip_most_recent.to_i) - 1)
+    end
+    puts librarian.ls(klass).slice(range).join("\n")
+  end
+  if flags.mysqldump?
+    librarian.backup_mysqldump
+  end
+  if flags.binary?
+    librarian.backup_data_files
+  end
+  if flags.rm?
+    librarian.rm flags.name
+  end
+  if flags.logs?
+    librarian.backup_binary_logs
+  end
+  if flags.get?
+    librarian.get flags.name, flags.get_directory || '.'
+  end
+  if flags.get_logs?
+    librarian.get_logs flags.get_directory || '.'
+  end
+  if flags.log_daemon?
+    while true
+      librarian.backup_binary_logs
+      delay = flags.logs_delay.to_f
+      sleep(delay == 0 ? 1 : delay)
+    end
+  end
+rescue Exception => e
+  puts "Exception: #{e.class}: #{e.message}\n\t#{e.backtrace.join("\n\t")}"
+  exit 1
+end

data/bin/sqlup_control

@@ -0,0 +1,25 @@
+#!/usr/bin/env ruby
+#
+# Normally run like so:
+#
+# export SQLUP_PID_DIR=/tmp
+# sqlup_control start -- -logs_delay 10 log_daemon
+#
+# This will back up your log files every 10 seconds.
+
+require 'rubygems'
+require 'daemons'
+require 'optiflag'
+
+options = {}
+
+pid_dir = ENV['SQLUP_PID_DIR']
+
+if pid_dir
+  options[:dir] = pid_dir
+  options[:dir_mode] = :normal
+else
+  options[:dir_mode] = :script
+end
+
+Daemons.run(File.dirname(__FILE__) + '/sqlup', options)

data/lib/mysql_backup/entity/files/innodb.rb

@@ -0,0 +1,77 @@
+require 'mysql_backup/entity/files'
+
+# Used to save all of the files for a MySQL Innodb database.
+# 
+# Normal use is:
+#
+#   i = Mysql::InnodbFiles.new
+#   array_of_pathname_objects = i.tar_files
+#
+# The caller is responsible for removing the files returned
+# from tar_files.
+#
+# See new for a description of the arguments used to create a Mysql::InnodbFiles object
+# matching your mysql layout.
+# 
+# You can create instances of Mysql::InnodbFiles by hand, but normally
+# you'd use MysqlBackup::Server to create them for you.  
+# MysqlBackup::Server
+# objects will detect where your data files are and fill in the correct options for
+# Mysql::InnodbFiles.new.
+#
+# = What is backed up
+#
+# - Innodb data files.  Your innodb files must all start with the same string
+#   and end with a sequence of digits.  +ibdata001+, +ibdata002+, etc. is fine,
+#   but +firstfile+, +secondfile+ is not supported.
+# - MyISAM files.
+# - Log files.  You need to specify the directory where the log files are stored;
+#   there's no way to extract this from a running server. 
+# - +mysqladmindump+ files.
+#
+# For all of these, we track the log positions (the log file/log position pair).
+class MysqlBackup::Entity::Files::Innodb < MysqlBackup::Entity::Files
+  attr_accessor :datadir, :innodb_data_home_dir, :innodb_data_file_path
+
+  # Takes the following arguments:
+  #
+  #   :datadir => The MySQL data directory.
+  #   :innodb_data_home_dir => The directory for innodb files.
+  #   :innodb_data_file_path => The root name for the innodb files.
+  #   :log_bin_dir => The directory containing the log files.
+  #   :log_bin => The prefix of the log files.
+  # 
+  # The following files will be backed up with the default base_dir and ib_basename:
+  #
+  #   /var/lib/mysql/ibdata*
+  #   /var/lib/mysql/*/*.frm - all files in any directory that contain one or more *.frm files
+  #   /var/lib/mysql/*/*.MYD - all files in any directory that contain one or more *.MYD files
+  def initialize args = {}
+    set_path_vars %w(datadir innodb_data_home_dir), args
+  end
+  
+  # Returns a list of Pathname objects that need to be
+  # backed up for a mysql server using innodb.
+  def required_paths
+    result = []
+    result.concat innodb_files
+    result.concat innodb_database_dirs
+    result.uniq
+  end
+  
+  # Returns an array of Pathnames for all databases
+  # in the base directory.
+  #
+  # A database in this case is a directory containing 
+  # any files matching *.frm.
+  def innodb_database_dirs
+    result = Pathname.glob(@innodb_data_home_dir_path + '*/*.frm')
+    result = result.map {|d| d.dirname}
+    result.uniq
+  end
+  
+  def innodb_files
+    ib = @innodb_data_home_dir_path + @innodb_data_file_path
+    Pathname.glob ib.cleanpath.to_s + '*'
+  end
+end

data/lib/mysql_backup/entity/files/myisam.rb

@@ -0,0 +1,62 @@
+require 'mysql_backup/entity/files'
+
+# Used to save all of the files for a MySQL MyISAM database.
+# 
+# Normal use is:
+#
+#   i = Mysql::InnodbFiles.new
+#   array_of_pathname_objects = i.tar_files
+#
+# The caller is responsible for removing the files returned
+# from tar_files.
+#
+# See new for a description of the arguments used to create a Mysql::InnodbFiles object
+# matching your mysql layout.
+# 
+# You can create instances of Mysql::InnodbFiles by hand, but normally
+# you'd use MysqlBackup::Server to create them for you.  
+# MysqlBackup::Server
+# objects will detect where your data files are and fill in the correct options for
+# Mysql::InnodbFiles.new.
+#
+# = What is backed up
+#
+# - MyISAM files.
+#
+# For all of these, we track the log positions (the log file/log position pair).
+class MysqlBackup::Entity::Files::Myisam < MysqlBackup::Entity::Files
+  # Takes the following arguments:
+  #
+  #   :datadir => The MySQL data directory.
+  # 
+  # The following files will be backed up with the default base_dir and ib_basename:
+  #
+  #   /var/lib/mysql/*/*.MYD - all files in any directory that contain one or more *.MYD files
+  
+  attr_accessor :datadir
+  
+  def initialize args = {}
+    super
+    set_path_vars %w(datadir), args
+  end
+  
+  # Returns a list of Pathname objects that need to be
+  # backed up for a mysql server using innodb.
+  def required_paths
+    result = []
+    result.concat myisam_database_dirs
+    result.uniq
+  end
+  
+  # Returns an array of Pathnames for all databases
+  # in the base directory.
+  #
+  # A database in this case is a directory containing 
+  # any files matching *.frm.
+  def myisam_database_dirs
+    result = Pathname.glob(@datadir_path + '*/*.MYD')
+    result.concat Pathname.glob(@datadir_path + '*/*.MYI')
+    result = result.map {|d| d.dirname}
+    result.uniq
+  end
+end

data/lib/mysql_backup/entity/files.rb

@@ -0,0 +1,95 @@
+require 'pathname'
+require 'tempfile'
+
+require 'mysql_backup/entity'
+require 'mysql_backup/entity/identifier'
+
+# See MysqlBackup::Entity::Files::Innodb and MysqlBackup::Entity::Files::Myisam
+# for implementations of this abstract base class.
+class MysqlBackup::Entity::Files < MysqlBackup::Entity
+  def initialize args = {}
+    super()
+  end
+  
+  # Create the tar files to back up a mysql instance.
+  #
+  # Takes the following:
+  # 
+  #   :mysql_server => a MysqlBackup::Server object
+  #   :mysql_files => an array of MysqlBackup::Entity::Files objects
+  # 
+  # Yields a hash:
+  #
+  #   :identifier => a MysqlBackup::Entity::Identifier object
+  #   :file => a Pathname object
+  def self.create_tar_files args = {}, &block
+    begin
+      log_data = build_files args
+      log_data[:files].each do |p|
+        process_file(log_data.merge(:file => p), &block)
+      end
+    ensure
+      #      files.each {|f| f.unlink if f.exist?}
+    end
+  end
+  
+  # Create the actual files in a 
+  # <tt>with_lock</tt> block.
+  def self.build_files args #:nodoc:
+    log_data = nil
+    args[:mysql_server].with_lock do |l|
+      log_data = l
+      mysql_file_objs = args[:mysql_files] 
+      mysql_file_objs.each do |o|
+        o.confirm_required_paths_are_readable
+      end
+      path_strings = (mysql_file_objs.map {|o| o.required_path_strings}).flatten.uniq
+      log_data[:files] = tar_files path_strings
+      log_data[:n_parts] = log_data[:files].length
+    end
+    log_data
+  end
+  
+  # Process each file with the specified block
+  def self.process_file args, &block #:nodoc:
+    part_number = args[:file].to_s[/.*(\d+)$/, 1].to_i
+    identifier = MysqlBackup::Entity::Identifier.create_object args.merge(:category => :full, :type => :binary, :part_number => part_number)
+    block.call(:identifier => identifier, :file => args[:file])
+  end
+  
+  def confirm_required_paths_are_readable
+    required_paths.each do |p|
+      raise RuntimeError, "Not readable: #{p}" unless p.readable?
+    end
+  end
+  
+  # Returns a list of path strings that need to be
+  # backed up.
+  def required_path_strings
+    files = required_paths.map {|p| p.cleanpath.to_s}
+  end
+  
+  def self.do_tar args #:nodoc:
+    log = args[:log]
+    log && log.info("running #{cmd}")
+    system args[:cmd] or raise RuntimeError, "The command failed with status #{$?}"
+  end
+  
+  # Given a set of variable names, create a 
+  # set of matching Pathname objects with
+  # <tt>_path</tt> appended to the name.
+  def set_path_vars var_names, args = {} #:nodoc:
+    args.each_pair do |k,v|
+      send "#{k}=", v
+    end
+    var_names.each do |p|
+      instance_var_name = "@#{p}"
+      instance_var_value = instance_variable_get(instance_var_name)
+      path_instance_var_name = "@#{p}_path"
+      raise RuntimeError, "Must pass :#{p}" unless instance_var_value
+      new_path = Pathname.new(instance_var_value)
+      instance_variable_set path_instance_var_name, new_path
+      raise RuntimeError, "Must provide a readable file for #{instance_var_name}" unless new_path.readable?
+    end
+  end
+end