db-rotator 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ba4a2bd4dffd3e671210454b4e7ba6dbb758ca07
-  data.tar.gz: c9c7f9835d3807f3d302288844138330919a5942
+  metadata.gz: 07ce9c6a6b6f1da73a5e9e232173748c81f84673
+  data.tar.gz: cc3c9b96fe45aa4f715f92c3f4e22cd928e8d695
 SHA512:
-  metadata.gz: 7abcb7f9c2aad35f1d891edb148182cb86e09f338a5b008bb8ab09386b93f25e87ed1d0d430f93447b57dd38ee51b06214e78a8f1f0ffdc662ee1264c5ef327f
-  data.tar.gz: 03d563062c0940986b8f94e422869f1aa9ad22059820b3e354f98835d4db3b8246aaefdc18616790bef8a68fcfc9a9a362a857dccfadfbdd854c11c471747fc9
+  metadata.gz: 1d5a98d23de5eb850f8232e38beb2585f5eb2ade0eb91bb74a4b7562b53b51145451c6f7800fdeeff213f3f4c6e06cec5753b17a4efad43c745cbc6a9264d403
+  data.tar.gz: 883f046aec07acdddbee6cc09f3c9646674c380e910a0f655c5d028f6630f6533b9e6f716fd9b03c5725ec53bb08af50930c2f804e2d95652fd078e998cf6b84

data/README.md

@@ -1,42 +1,58 @@
 # DBRotator
-Easy MySQL database rotation and pruning. Tell this utility where your nightly MySQL backup is, and it downloads, imports, and organizes the dump, pruning all but the newest N databases.
+Easy MySQL database rotation and pruning -- downloads and imports a mysql dump, and then deletes all but N databases, keeping only the newest. This tool is geared toward rotating out achived MySQL backups to your local dev environment. So it works best when you have a most-current MySQL backup as a symlink that updates nightly.
 
 ## Installation
 
 `gem install db-rotator`
 
+## Requirements
+- This tool creates and drops databases, and runs any SQL in your dump (duh), so you have to give it a user that can do all of that. If you're using this on a local dev environment, it's recommended to setup user/pass in `~/.my.cnf` so that "mysql" works without `-u` or `-p`. You can configure DBRotator to work with any credentials, however. See config section.
+- Disk space for N+1 database instances, where N is the amount you want to prune to (maximum DBs).
+
 ## Usage
 
-1. First off, the following is required:
-  - SSH access to your most current MySQL backup as a single endpoint, like a symlink that updates nightly.
-  - root access to destination MySQL instance. Recommended to setup username/password in `~/.my.cnf` so that "mysql" works without `-u` or `-p`.
+**Minimal usage:**
+
+Run: `db-rotator -p 'appdump_' -c 'scp db5:/opt/backups/latest.sql.bz2'`
+
+**Or, from default config file** `~/.db-rotator.yml`:
+
+```
+db_prefix: "appdump_"
+scp_command: "scp db5:/opt/backups/latest.sql.bz2"
+```
+
+Run: `db-rotator`
 
-2. Configure the required options (see below). Configuration can be passed as command line arguments or set in `~/.db-rotator.yml`.
+**Or, from a specific config file** `/whatever/rotator-config.conf`:
 
-  Minimal usage example:
+```
+db_prefix: "appdump_"
+scp_command: "scp db5:/opt/backups/latest.sql.bz2"
+```
 
-  `db-rotator -p 'appdump_' -c 'scp db5:/opt/backups/latest.sql.bz2'`
+Run: `db-rotator -f /whatever/rotator-config.conf`
 
-  Run `db-rotator -h` for the list of configuration options from CLI.
+**Rotate nightly**, so you'll always have a fresh dump during your workday:
 
-  You can also set up configuration at `~/.db-rotator.yml`, and omit any CLI options, like so:
+`0 3 * * * bash -lc "db-rotator -f /whatever/rotator-config.conf >> /some/log/file"`
 
-  ```
-  db_prefix: "appdump_"
-  scp_command: "scp db5:/opt/backups/latest.sql.bz2"
-  ```
+## Configuration
 
-3. Put `db-rotator` in your crontab, and execute it when you're sure your nightly dump has finished.
+Run `db-rotator` without any options to show config options.
 
-  `0 3 * * * db-rotator -p 'appdump_' -c 'scp db5:/opt/backups/latest.sql.bz2' >> /some/log/file`
+### Required
+#### db_prefix (-p)
+Database naming prefix that will apply to all dumps rotated with DBRotator.
+Example: `myproject_`, which might name a DB as myproject_09182013.
 
-## Required Configuration
-  - **db_prefix** (-p). Database naming prefix. Example: `myproject_`, which might name a DB as myproject_09182013.
-  - **scp_command** (-c). Receives second arg of dump path. Example: `scp hostname:/path/to/mysql/backups/backup_filename.sql.bz2`
+#### scp_command (-c)
+How DBRotator retrieves your dumps. This ideally is an scp command, but really can be any command that receives a second argument of the dump destination.
+Example: `scp hostname:/path/to/mysql/backups/backup_filename.sql.bz2`
 
-## Optional Configuration
+### Optional
 
-- **local_dump_destination** (-d). Where to put the dump, as a directory. Won't be deleted after running rotator. Default: `/tmp`
+- **local_dump_destination** (-d). Where to put the dump, as a directory. The dump won't be deleted after running rotator. Default: `/tmp`
 - **mysql_command** (-m). Used for all database management operations. Default: `mysql`
 - **maximum_dbs** (-n). Maximum number of DBs to maintain, or null to disable pruning. Default: 2
 - **unarchive_command** (-u). How to unarchive your dump to standard output. Default: `bzip2 -cd`

data/Rakefile

@@ -0,0 +1,2 @@
+require 'rake/testtask'
+Rake::TestTask.new(:test)

data/bin/db-rotator

@@ -5,6 +5,5 @@ require_relative '../lib/db_rotator.rb'
 
 config = DBRotatorConfig.new
 config.configure
-puts config.config
 rotator = DBRotator.new(config)
 rotator.rotate

data/db-rotator.gemspec

@@ -1,13 +1,13 @@
 Gem::Specification.new do |s|
   s.name        = "db-rotator"
-  s.version     = "0.0.1"
+  s.version     = "0.0.2"
   s.summary     = "Easy MySQL database rotation and pruning"
   s.description = "Easy MySQL database rotation and pruning"
   s.authors     = ["Michael Nelson"]
   s.email       = "michael@nelsonware.com"
   s.executables << "db-rotator"
   s.files       = `git ls-files`.split
-  #s.test_files  = `git ls-files -- test/*`.split
+  s.test_files  = `git ls-files -- test/*`.split
   s.homepage    = "https://github.com/mcnelson/db-rotator"
   s.license     = "LGPL-2.0"
 

data/lib/db_rotator.rb

@@ -63,7 +63,7 @@ class DBRotator
 
   def grant_access
     verbose_message "Granting ..."
-    mysql_exec "GRANT ALL ON #{todays_dbname}.* to `%`@`localhost`"
+    mysql_exec "GRANT ALL ON #{todays_dbname}.* to #{current_user}"
   end
 
   def populate_schemas
@@ -120,6 +120,10 @@ class DBRotator
     mysql_exec "SHOW DATABASES"
   end
 
+  def current_user
+    mysql_exec("SELECT current_user()").split("\n").last
+  end
+
   def bash_exec(cmd, skip_raise = false)
     out = `#{cmd} || echo '__failed'`
     out = out.strip == "__failed" ? nil : out

data/lib/db_rotator_config.rb

@@ -8,26 +8,28 @@ class DBRotatorConfig
   FILE = ".db-rotator.yml"
 
   CONFIG = {
-    db_prefix:              [['-p', "--db-prefix PREFIX", "Database naming prefix"], nil],
-    scp_command:            [['-c', "--scp-command COMMAND", "Command to retrive dump. Receives second arg of dump path"], nil],
+    db_prefix:              [['-p', "--db-prefix PREFIX", "Database naming prefix."], nil],
+    scp_command:            [['-c', "--scp-command COMMAND", "Command to retrive dump. Receives second arg of dump path."], nil],
 
-    local_dump_destination: [['-d', "--local-destination [PATH]", "Where to put the dump, as a directory. Won't be deleted after running rotator.", "Default: /tmp"], "/tmp"],
-    mysql_command:          [['-m', "--mysql-command [COMMAND]", "Used for all database management operations.", "mysql"], "mysql"],
+    local_dump_destination: [['-d', "--local-destination [PATH]", "Where to put the dump, as a directory. The dump won't be deleted after running rotator.", "Default: /tmp"], "/tmp"],
+    mysql_command:          [['-m', "--mysql-command [COMMAND]", "Used for all database management operations.", "Default: mysql"], "mysql"],
     maximum_dbs:            [['-n', "--maximum-dbs [N]", "Maximum number of DBs to maintain, or null to disable pruning.", "Default: 2"], 2],
     unarchive_command:      [['-u', "--unarchive-command [COMMAND]", "How to unarchive your dump to standard output.", "Default: bzip2 -cd"], "bzip2 -cd"],
     unarchive_extra_pipe:   [['-i', "--extra-pipes [SCRIPT1,SCRIPT2]", Array, "Any extra script(s) you want to run between unarchive & import.", "Example: -i 'script1,script2'"], nil],
-    reasonable_diskspace:   [['-s', "--minimum-diskspace [GB]", "Rough estimate of temporary disk space required to import a typical dump, in GB.", "Default: nil"], nil],
+    reasonable_diskspace:   [['-s', "--minimum-diskspace [GB]", "Rough estimate of temporary disk space required to import a typical dump, in GB."], nil],
     rails_db_yaml_path:     [['-y', "--rails-db-yaml [PATH]", "Updates database name in your YAML file when given.", "Default: nil"], nil],
-    rails_environments:     [['-e', "--rails-db-environments [ENV1,ENV2]", Array, "In conjunction with -y, which rails envs to update DB name for.", "Default [development]"], ["development"]],
+    rails_environments:     [['-e', "--rails-db-environments [ENV1,ENV2]", Array, "In conjunction with -y, which rails envs to update DB name for. Default: development"], ["development"]],
 
-    config_file:            [['-f', "--config-file PATH", "Load configuration from .yml file"], nil],
+    config_file:            [['-f', "--config-file PATH", "Runs rotator with configuration from this .yml file."], nil],
   }
 
   REQUIRED = %i(db_prefix scp_command)
+  EXCLUDE_FROM_GENERATE_FILE = %i(config_file dump_filename)
   attr_reader :config
 
   def initialize
     @config = {}
+    @generate_output_file = nil
   end
 
   def configure
@@ -41,6 +43,12 @@ class DBRotatorConfig
       check_required
       add_default_values
       add_derived_values
+
+      if @generate_output_file
+        generate_config
+        exit
+      end
+
     rescue FileNotFoundError => e
       puts "There was a problem loading configuration: #{e.message}"
       puts cli_parser.summarize
@@ -78,6 +86,10 @@ class DBRotatorConfig
         end
       end
 
+      opts.on("-g", "--generate-config FILE", "Generates .yml config file from given arguments.") do |v|
+        @generate_output_file = v
+      end
+
       opts.on("-v", "--verbose") do |v|
         @config[:verbose] = v
       end
@@ -97,6 +109,15 @@ class DBRotatorConfig
     REQUIRED.each { |k| raise "config option '#{k.to_s} is required'" if @config[k].nil? }
   end
 
+  def generate_config
+    File.write(@generate_output_file, config_yaml)
+  end
+
+  def config_yaml
+    require 'pry'; binding.pry
+    @config.reject { |k, _| EXCLUDE_FROM_GENERATE_FILE.include?(k) } .to_yaml
+  end
+
   def add_default_values
     CONFIG.each do |key, pair|
       @config[key] ||= pair.last
@@ -105,7 +126,13 @@ class DBRotatorConfig
 
   def add_derived_values
     # Figure out the dump filename
-    pn = Pathname.new(@config[:scp_command])
+    cmd = if @config[:scp_command].strip.match(/^scp/)
+            @config[:scp_command].split(':').last
+          else
+            @config[:scp_command]
+          end
+
+    pn = Pathname.new(cmd)
     @config[:dump_filename] = pn.basename.to_s
   end
 end

data/test/fixtures/basic_dump.sql

@@ -0,0 +1,54 @@
+-- MySQL dump 10.13  Distrib 5.5.34, for Linux (x86_64)
+--
+-- Host: localhost    Database: whatever
+-- ------------------------------------------------------
+-- Server version	5.5.34-32.0-log
+
+/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
+/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
+/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
+/*!40101 SET NAMES utf8 */;
+/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
+/*!40103 SET TIME_ZONE='+00:00' */;
+/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
+/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
+/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
+/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
+
+--
+-- Table structure for table `test`
+--
+
+DROP TABLE IF EXISTS `test`;
+/*!40101 SET @saved_cs_client     = @@character_set_client */;
+/*!40101 SET character_set_client = utf8 */;
+CREATE TABLE `test` (
+  `id` int(11) NOT NULL AUTO_INCREMENT,
+  `column_varchar` varchar(45) COLLATE utf8_unicode_ci DEFAULT NULL,
+  `column_int` tinyint(1) DEFAULT NULL,
+  `column_date` datetime DEFAULT NULL,
+  `column_nonnull` varchar(45) COLLATE utf8_unicode_ci NOT NULL,
+  PRIMARY KEY (`id`)
+) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
+/*!40101 SET character_set_client = @saved_cs_client */;
+
+--
+-- Dumping data for table `test`
+--
+
+LOCK TABLES `test` WRITE;
+/*!40000 ALTER TABLE `test` DISABLE KEYS */;
+INSERT INTO `test` VALUES (1,'Herpa Derpa',1,'2013-01-01 01:10:20','Blargh'),(2,'Dorp Schlorp',0,'2013-09-18 04:50:12','Rawr');
+/*!40000 ALTER TABLE `test` ENABLE KEYS */;
+UNLOCK TABLES;
+/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
+
+/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
+/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
+/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
+/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
+/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
+/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
+/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
+
+-- Dump completed on 2013-11-20  9:00:47

data/test/helper.rb

@@ -0,0 +1,19 @@
+require 'minitest/autorun'
+require_relative '../lib/db_rotator'
+require_relative '../lib/db_rotator_config'
+
+TEST_SCHEMA_PREFIX = "__minitest_"
+
+def dummy_config(config={})
+  DBRotatorConfig.new.tap do |cfg|
+    cfg.instance_variable_set(:@config, {
+      db_prefix:   TEST_SCHEMA_PREFIX,
+      scp_command: "cp test/fixtures/basic_dump.sql.bz2",
+
+      local_dump_destination: "/tmp"
+    }.merge(config))
+
+    cfg.add_default_values
+    cfg.add_derived_values
+  end
+end

data/test/test_db_rotator.rb

@@ -1,6 +1,45 @@
-
-require 'minitest'
+require_relative '../test/helper'
 
 describe DBRotator do
+  after do
+    `mysql -B -e 'SHOW SCHEMAS;'`.split.each do |schema|
+      if /^#{TEST_SCHEMA_PREFIX}/.match(schema)
+        `mysql -e "DROP SCHEMA #{schema}"`
+      end
+    end
+  end
+
+  def dbname_with(time)
+    "__minitest_#{ time.strftime(DBRotator::TIME_FORMAT) }"
+  end
+
+  describe "integration test" do
+    it "rotates databases correctly" do
+      dbr = DBRotator.new(dummy_config)
+
+      t1 = Time.new(2013, 1, 1)
+      t2 = Time.new(2013, 1, 2)
+      t3 = Time.new(2013, 1, 3)
+
+      Time.stub(:now, t1) do
+        dbr.rotate
+        `mysql -B -e 'SHOW SCHEMAS;'`.must_include(dbname_with(t1))
+        `mysql -B -e 'SHOW TABLES FROM #{dbname_with(t1)};'`.must_include("test")
+      end
+
+      Time.stub(:now, t2) do
+        dbr.rotate
+        `mysql -B -e 'SHOW SCHEMAS;'`.must_include(dbname_with(t2))
+        `mysql -B -e 'SHOW SCHEMAS;'`.must_include(dbname_with(t1))
+      end
+
+      Time.stub(:now, t3) do
+        dbr.rotate
+        `mysql -B -e 'SHOW SCHEMAS;'`.must_include(dbname_with(t3))
+        `mysql -B -e 'SHOW SCHEMAS;'`.must_include(dbname_with(t2))
 
+        `mysql -B -e 'SHOW SCHEMAS;'`.wont_include(dbname_with(t1))
+      end
+    end
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: db-rotator
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
 platform: ruby
 authors:
 - Michael Nelson
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-11-17 00:00:00.000000000 Z
+date: 2013-11-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest
@@ -34,10 +34,16 @@ files:
 - .gitignore
 - LICENSE
 - README.md
+- Rakefile
 - bin/db-rotator
 - db-rotator.gemspec
 - lib/db_rotator.rb
 - lib/db_rotator_config.rb
+- test/fixtures/basic_dump.sql
+- test/fixtures/basic_dump.sql.bz2
+- test/fixtures/basic_dump.sql.gz
+- test/fixtures/basic_dump.sql.zip
+- test/helper.rb
 - test/test_db_rotator.rb
 homepage: https://github.com/mcnelson/db-rotator
 licenses:
@@ -63,4 +69,10 @@ rubygems_version: 2.0.2
 signing_key: 
 specification_version: 4
 summary: Easy MySQL database rotation and pruning
-test_files: []
+test_files:
+- test/fixtures/basic_dump.sql
+- test/fixtures/basic_dump.sql.bz2
+- test/fixtures/basic_dump.sql.gz
+- test/fixtures/basic_dump.sql.zip
+- test/helper.rb
+- test/test_db_rotator.rb