backupgem 0.0.2

data/CHANGELOG

@@ -0,0 +1,6 @@
+= Backup Changelog
+
+== Version 0.0.2
+
+* First stable code in place.
+* Packaged in an gem format

data/README

@@ -0,0 +1,387 @@
+===What is Backup?===
+<tt>Backup</tt> is the easiest and most flexible backup, archive and rotate
+tool.  It's a beginning-to-end solution for scheduled backups in a clean ruby
+package that is simple use and powerful when customized.
+
+Backup allows you to specify each of the following options:
+* what is being archived (files, folders, arbitrary scripts)  
+* how it's being archived (tar gzip, bz2)
+* where the archive is going (multiple backup servers? easy)           
+* how the archive is going to get there (scp, ftp, mv)
+* where is will be stored when it gets there
+* how it's going to be rotated when it gets there (grandfather-father-son, etc)
+* how often will this process happen (customizable cycles)
+* what happens to the working copy after the process (recreate files, folders etc. restart daemons)
+
+Backup is a collection of scripts that is complete enough to save you
+time, but flexible enough to work with any situation.
+
+===Getting Backup===
+====Prerequisites====   
+Backup makes the following assumptions about your machines:
+* server and client understand POSIX commmands
+* passwords and paths are the same on each server
+
+Backup depends on the following libraries:
+* [[Runt]] for describing [[temporal ranges]]
+* [[Net::SSH]] for SSH backups
+* [[Net::FTP]] for FTP backups
+
+These are listed as dependencies in the gem file so you should be prompted to
+install them when you install Backup.
+
+====Using RubyGems====
+If you have [[http://rubygems.rubyforge.org RubyGems]] installed, installing
+Backup is simple:
+
+  sudo gem install backupgem
+
+====Using svn====
+If you prefer, you can checkout backupgem from the [[RubyForge Repository]].
+Feel free to browse the releases or trunk [[here]].
+
+  svn+ssh://blar blar blar
+
+===License Information===
+Backup is made available under either the BSD license, or the same license Ruby
+(which, by extension, also allows the GPL as a permissable license as well).
+You can view the full text of any of these licenses in the <tt>doc</tt> subdirectory
+of the Backup distrubtion. The texts of the BSD and GPL licenses are also
+available online: "BSD":http://www.opensource.org/licenses/bsd-license.php and
+"GPL":http://www.opensource.org/licenses/gpl-license.php.
+
+If you desire permission to use either Backup in a manner incompatible with
+these licenses, please contact the copyright holder
+([[mailto:nate@natemurray.com Nate Murray]] in order to negotiate a more
+compatible license.
+
+===Support===
+Mailing lists, bug trackers, feature requests, and public forums are all
+available courtesty of [[http://rubyforge.org RubyForge]] at the 
+[[http://rubyforge.org/projects/backupgem BackupGem project page]].
+
+====Mailing Lists====
+{|class="wikitable"
+! List Name
+!
+! Desc.
+|---
+| [[http://rubyforge.org/pipermail/backupgem-users backupgem-users]]
+| [[http://rubyforge.org/mailman/listinfo/backupgem-users subscribe / unsubscribe]]
+| The BackupGem users list is devoted to the discussion of and questions about the usage of Backup. If you can't quite figure out how to get a feature of Backup to work, this is the list you would go to in order to ask your questions. 
+|---
+| [[http://rubyforge.org/pipermail/backupgem-devel backupgem-devel]]
+| [[http://rubyforge.org/mailman/listinfo/backupgem-devel subscribe / unsubscribe]]
+| The Backup developers list is devoted to the discussion of Backup's implementation. If you have created a patch that you would like to discuss, or if you would like to discuss a new feature, this is the list for you.
+|}
+
+===About the Author===
+Backup was written by [[mailto:nate@natemurray.com Nate Murray]. 
+Nate currently works at an internet retailer in Southern California.
+Feel free to send him compliments, candy, money, praise, or new feature patches--he likes
+all those things. You can send him questions and suggestions, too, if you
+really want to. However, for bug reports and general feature requests,
+please use the trackers on the [[http://rubyforge.org/projects/backupgem BackupGem project page]].
+
+===Special Thanks===
+* Matt Pulver for help with various technical problems and ideas.
+
+* Jamis Buck for writing [http://weblog.rubyonrails.com/2006/8/30/capistrano-1-1-9-beta Capistrano]. Capistrano provided the inspiration and some code for this work. Additionally, the Net::SSH manual provided the inspiration for this manual. Thanks for the top-notch work Jamis!
+
+* [[mailto:info@digitalclash.com Matthew Lipper]] for writing the Runt Ruby Temporal Expressions Library
+
+==How Backup Works==
+===Intro===
+A basic backup has the following sequence:
+* content 
+* compress 
+* encrypt 
+* deliver 
+* rotate 
+* cleanup
+
+This order is the default, however, like most things it is customizable. 
+Think of it like a pipline: the input of each step is the output of the last
+step.
+
+Each of these things are specified in a <tt>recipe</tt> file which is describe below. 
+
+===CLI===
+
+  Usage: ./backup [options] 
+  Recipe Options -----------------------
+      -r, --recipe RECIPE              A recipe file to load. Multiple recipes
+                                       may be specified, and are loaded in the 
+                                       given order.
+      -g, --global FILE                Specify the global recipe file to work
+                                       with. Defaults to the file <tt>global.rb</tt> 
+                                       in the directory of <tt>recipe</tt> 
+      -s, --set NAME=VALUE             Specify a variable and it's value to 
+                                       set. This will be set after loading all
+                                       recipe files.
+
+==Backup Recipe File Format==
+===Introduction===
+* The Backup Recipe format is pure ruby code. Anything that is valid ruby is valid in the recipe file. There are a number of shortcuts that will make your life easier.
+
+* Each of the steps are specified as an <tt>action</tt>. (An action is really nothing more than a method that becomes defined in the Actor instance. See API docs if you're interestd.)
+
+* You may create "hook" actions for any of the actions. So if you define a method <tt>before_content</tt> it will be called just before <tt>content</tt> is called. A method named <tt>after_rotation</tt> would be called after rotation. This may not always be needed as you can customize the rotation order to be whatever you want. See [[#XXX]] below. 
+
+* Each action has the variable <tt>last_result</tt> available to it. This is the return value of the method that was called previously. Note that this includes the output of the "hook" methods. 
+
+* All configuration variables are available to actions via the hash c[]. For example, the backup path is available to your actions as c[:backup_path].
+
+===Variables===
+Intro on how to set variables. How this works.
+
+Required variables for all configurations.
+{|class="wikitable"
+! Name
+! Desc.
+! Example
+|---
+| :action_order
+| short desc. TODO 
+| set :action_order,      %w{ content compress encrypt deliver rotate cleanup }
+|---
+| :tmp_dir
+| Specify a directory that backup can use as a temporary directory. Default <tt>/tmp</tt>.
+| set :tmp_dir, File.dirname(__FILE__) + "/../tmp"
+|---
+| :backup_path
+| The path to backup on. TODO - if its local the local server if its foreign the foreign server
+| set :backup_path, "/var/local/backups/mediawiki"
+|}
+
+===Content===
+The first step in any backup is the content that is to be backed up. Backup
+provides a couple of shortcuts for common ways to locate content and allows you
+to arbitrarily define your own.
+
+Some typical types of content are:
+* a particular file
+* a particular folder
+* the contents of a particular folder 
+
+These could be specified like so:
+
+  action :content, :is_file   => "/path/to/file"               # content is a single file
+  action :content, :is_file   => "/path/to/error_log", :recreate => true
+  action :content, :is_folder => "/path/to/folder"             # content is the folder itself 
+  action :content, :is_contents_of => "/path/to/other/folder"  # content is folder/* , recursive option
+
+If you want :content to be a series of shell commands just pass "action" a block:
+
+  action(:content) do 
+    sh "echo \"hello $HOSTNAME\""
+    sh "mysqldump -uroot database > /path/to/db.sql" 
+    "/path/to/db.sql" # make sure you return the full path to the folder/file you wish to be the content 
+  end
+
+===Compress===
+Next you may want to compress your content. Again, there are a few one-liners for common cases and you can create your own.
+
+  action :compress, :method => :tar_bz2  # actually calls a method named tar_bz2 with output of ":content" ( or ":after_content" ) 
+  # or
+  action :compress, :method => :tar_gzip 
+
+Again, you can create your own.
+
+  action(:compress) 
+    sh "my_tar  #{last_result} #{last_result}.tar" 
+    sh "my_bzip #{last_result}.tar #{last_result}.tar.bz2"
+    last_result + ".tar.bz2"
+  end
+
+===Encrypt===
+If you wish to use encryption this is available to you. I would recommend that
+you think seriously about how you wish to manage your keys for this backup
+process. If you are backing up encrypted data then you need to backup your keys
+or else you risk losing access to your data. Secure key management is beyond
+the scope of this document, but I recommend the following links:
+* link 1
+* link 2
+
+  set :encrypt, true                              # default is <tt>false</tt>
+  set :gpg_encrypt_options, "--default-recipient" # default is an empty string
+  action :encrypt, :method => :gpg # default, none
+
+or your own:
+
+  action(:encrypt)
+    sh "gpg #{c[:gpg_encrypt_options]} --encrypt #{last_result}"
+    last_result + ".gpg" # ?
+  end
+
+===Delivery===
+====Action====
+Delivery is supported via <tt>scp</tt>, <tt>ftp</tt>, and <tt>mv</tt>
+
+  action :delivery, :method => :scp
+  action :delivery, :method => :ftp
+  action :delivery, :method => :mv
+
+The <tt>:mv</tt> action is defined like any user-defined action:
+
+  action(:mv) do
+    sh "mv #{last_result} #{c[:backup_path]}/"
+    c[:backup_path] <tt> "/" </tt> File.basename(last_result)
+  end
+
+====Variables====
+{|class="wikitable"
+! Name
+! Desc.
+! Example
+|---
+| :servers
+| An array of host names to deliver the data to. TODO this currently only supports 1 server. 
+| set :servers,           %w{ localhost }
+|---
+| :ssh_user
+| The name of the ssh user on the foreign server. Default ENV['USER'].
+| set :ssh_user,          ENV['USER']
+|---
+| :identity_key
+| The path to the key to use when ssh'ing into a foreign server.
+| set :identity_key,      ENV['HOME'] + "/.ssh/id_rsa"
+|}
+
+==Rotate==
+Rotation of your backups is a way to keep snapshot copies of your backups in time while not keeping every single backup for every single day.
+Currently the only form of rotation Backup supports is [[grandfather-father-son]] See Appendix A if you are unfamiliar with how this works. 
+
+  set :rotation_method,  :gfs # this is the default. you don't need to set it, but this is how you could
+
+By deafult, a <tt>son</tt> is created daily, unless it is a day to create a father or
+grandfather.  It is assumed that every time you run Backup you want to create a
+backup. Therefore, if you do not want to a son etc, do not run the program. 
+You can specify when the son is promoted to a father by the following variable.
+
+  set :son_promoted_on,    :fri
+
+You specify when fathers are promoted to grandfathers by something like the following
+
+  set :father_promoted_on, :last_fri_of_the_month
+
+Valid argumetns for specifying these promotions are as follows:
+* :mon-:sun - A symbol of the abbreviation of any day of the week
+* :last_*_of_the_month - A symbol, replacing the * with the abbreviation for the day of the weeks. Such as :last_fri_of_the_month.
+* Any valid Runt object. 
+
+Representing these [[temporal ranges]] is done internally by using Runt. You are, therefore, allowed to pass in your own arbitrarily complex runt object.
+Say for instance that I wanted to promote to fathers on monday, wednesday and friday. I could do something like the following:
+
+  mon_wed_fri = Runt::DIWeek.new(Runt::Mon) | 
+                Runt::DIWeek.new(Runt::Wed) | 
+                Runt::DIWeek.new(Runt::Fri)
+  set :son_promoted_on, mon_wed_fri
+
+See the [[Runt documentation]] for more information on this.
+
+You can set how many of each rank to keep:
+
+set :sons_to_keep,         14
+set :fathers_to_keep,       6
+set :grandfathers_to_keep,  6   
+
+==Examples==
+
+Here we will cover three examples. 
+# a super-simple backup to a local directory, show how easy it is
+# a more complex implementation, show the variables you can set show the customizability and use of foreign server
+# every more complex. define your own method, use a global file to share in the configuration.
+
+===Example One: Backup folder of Logs===
+Our first example will be backing up a folder of logs. Say we have a folder
+'/var/my_logs/' and it is full of log files. It's full. Seriously, it's getting
+stuffy in there. 
+Anyway, what we want is to:
+* move out all the old log files
+* compress them and store them in a local folder
+* store 2 weeks of daily backups (sons)
+* store a weekly backup (father) going back 6 weeks
+* and create a monthly backup on the last friday of every month (grandfather) for 6 months
+
+Thankfully, this is incredibly simple:
+
+  set :backup_path, "/var/local/backups/my_old_logs"
+  set :tmp_dir, "/tmp" # this is the default so you actually dont have to specify it
+  action :content, :is_contents_of => "/var/my_logs"
+
+''In this case, make sure that <tt>:backup_path</tt> and <tt>:tmp_dir</tt> are writable by the user
+that is running the backup script.''
+
+And thats it!
+
+Note a few things here.
+# Each time we <tt>set</tt> a variable that becomes available to the actions as <tt>c[:var]</tt>
+
+===Example Two: SQL Backup===
+Our second example will be backing up a MediaWiki installation. 
+Say we have a MySQL database named 'mediawiki'. 
+What we want is to:
+* create a dump of the database every day
+* compress this backup and store it in a local folder
+* store 2 weeks of daily backups (son) [same as last time]
+* store "father" backups every monday,wednesday and friday going back 6 weeks
+* and create a monthly backup on the last friday of every month (grandfather) for 6 months
+
+Thankfully, this is incredibly simple:
+
+  set :backup_path, "/var/local/backups/mediawiki"
+
+  action(:content) do
+    dump = c[:tmp_dir] + "/mediawiki.sql"
+    sh "mysqldump -uroot mediawiki > #{dump}"
+    dump # make sure you return the name of the file
+  end
+
+  action :delivery, :method => :scp
+  action :rotate,   :method => :via_ssh
+
+  set :servers,           %w{ my.server.com }
+
+  set :son_promoted_on,    :sun
+  set :father_promoted_on, :last_sun_of_the_month
+
+  set :sons_to_keep,         21  
+  set :fathers_to_keep,      12
+  set :grandfathers_to_keep, 12 
+
+===Example Three: Something more complex===
+
+  action :content,  :is_file => "/path/to/file.abc"
+  action :compress, :method  => :my_tar_gzip
+
+  action(:my_tar_gzip) do
+    name = c[:tmp_dir] + "/" + File.basename(last_result) + ".tar.gzip"
+    sh "tar -czv --exclude .DS* --exclude CVS  #{last_result} > #{name}"
+    name # make sure you return the name of the
+  end
+
+  set :encrypt, true
+
+  action :deliver,  :method => :scp  
+  action :rotate,   :method => :via_ssh
+
+  set :ssh_user,          "backup_user"
+  set :identity_key,      ENV['HOME'] + "/.ssh/backup_key"
+
+* how to setup the cron job
+
+==TODO==
+
+what is left to do:
+* start testing it
+* work on the styles
+* lookup setup.rb files
+
+==TODO==
+* Add in better logging 
+
+==BUGS==
+* You can't <tt>return</tt> in the user-defined actions for some reason. I think this
+  has to do with the <tt>instance_eval</tt>. But still, I wouldn't think it would
+  matter. I'd be interested in any suggestions on how to fix this.

data/bin/backup

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+$: << File.dirname(__FILE__) + "/../lib"
+
+begin
+  require 'rubygems'
+rescue LoadError
+  # no rubygems to load, so we fail silently
+end
+
+require 'backup/cli'
+
+Backup::CLI.execute!

data/bin/commands.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+./bin/backup --recipe examples/mediawiki.rb