dated_backup 0.1.0 → 0.2.0

data/README

@@ -0,0 +1,186 @@
+
+= Introduction
+
+Dated Backup is a program which does exactly what it's name says: 
+It creates backups of any directory, timestamping the backups. It then 
+performs incremental backups on every subsequent run.  The really nice thing here 
+is that those backups are fully viewable as snapshots, even though they are 
+also incremental.
+
+This method of backup uses the hard-link technique in combination
+with rsync.  For more information on this technique, see: 
+
+  http://www.mikerubel.org/computers/rsync_snapshots/
+  
+At the moment, this program can be thought of as a limited, Ruby version of
+the popular unix utility rsnapshot.  Dated Backup's feature set already does
+things a little different from rsnapshot, and in the future will diverge widely.
+  
+Dated Backup no longer depends on GNU cp, but instead uses rsync's --link-dest
+option to simulate the hard-link method.
+
+
+== Backup Assumptions
+
+* Your backup *server* is POSIX compliant (a modern day UNIX - Linux, *BSD, Mac OS X)
+* You would like to perform incremental snapshots with timestamps
+
+= Installation:
+
+* sudo gem install 'datedbackup' --include-dependencies
+
+== Dependencies
+
+Dated Backup has the following dependencies:
+
+  * Ruby
+  * Rubygems
+  * Rails' 'ActiveSupport' gem
+  * A copy of rsync, which supports the --link-dest option.  
+
+Rsync is not required on the machine to be backed up - only on the machine which stores
+the backup.  
+
+
+= HOWTO Backup with Dated Backup
+
+Each backup source will correspond to a configuration file, which defines the source 
+directory (or remote location), and the local destination directory.  These two 
+parameters are the only requirements for a backup configuration file to be valid.  
+The script can be run with the executable dbackup:
+
+% dbackup my_script
+
+Other scripts can be run sequentially by listing them in order:
+
+% dbackup my_first_script my_second_script
+
+All of the configuration occurs in the configuration file.
+
+
+== The DSL, or How To Write A Configuration File
+
+Here are the valid key words which can be set in the main section of the configuration file:
+
+  * source
+  * sources
+  * destination
+  * options
+  * user_domain
+  
+The values for these should be strings.  They are specified like so:
+
+  source '/etc'
+  
+Multiple values can also be given:
+
+  sources '/etc', '/home'
+  
+The destination keyword only takes one value, but in the next release (0.3), this should
+be fixed to allow backups to be copied to multiple locations.
+
+The 'source' (or 'sources') keyword and the 'destination' keyword
+are the only ones needed for a valid backup config file.  The 'user_domain' keyword is used 
+when the source is not a local directory (or local file).  The user_domain should be in user@server
+style.  As for the 'options' keyword, this should be specified as a string for extra options to be
+feed into rsync.  It too, is optional.
+
+
+=== Before and After Filters
+
+It is very convenient, and often necessary to perform something before or after a backup script
+runs.  Any actions must be inside a 'before' or 'after' block:
+
+before do
+ ...some before action here
+end
+
+after {
+  ..some other action here
+}
+
+You have the pick of do...end, or { ... }, thanks to Matz.  No doubt, this will be familiar to any
+Ruby programmer.
+
+At the time of this release (0.2), there is only one valid action - 'remove_old'.  Other actions, such
+as running a script (or any number of scripts, in sequence), as well as running a command specified 
+in the configuration file itself, should be coming in subsequent releases.
+
+For some example configuration files, see the examples bundled with RDoc, or in the example_configs
+directory.
+
+
+=== remove_old
+
+The remove_old block takes several different natural language time forms.  All of the statements inside
+a remove_old block must begin with 'keep'.  An example would work best to illustrate how to use this:
+
+after do
+  remove_old do
+    keep this months backups
+    keep last months backups
+    keep monthly backups
+  end
+end
+
+This says the following: After the backup runs, remove all backups that do not conform to the criteria
+given.  The first line will save all backups which have occured this month, regardless of what time 
+they occurred (as long as they occurred in the current month).  The next line says the same, except for
+last month's backups.
+
+The last line, "keep monthly backups", will keep one backup from each month not already kept.  If you perform
+daily backups every day, this would end of keeping the last day's backup from every month (the 29, 30, or 31,
+according to the month)
+
+A few things should be notice here: These config files read very easily, so don't let them fool you.  
+They will delete your backups, forever lost.  I've already burned myself with the following: 
+
+after {
+  remove_old {
+    keep this weeks backups 
+    keep last weeks backups 
+    keep weekly backups from this month # or: keep this months weekly backups
+    keep monthly backups # or: keep all monthly backups
+  }
+}
+
+After a month of backups, the month rolled over, and the next backup on the first of the month deleted
+all backups, except the one just performed, and the one from the last day of the month before.  
+So beware, and think before you specify any remove_old block at all.  
+
+Another thing to notice (for any of you non-ruby programmers out there): The config file must be in valid
+ruby, so specifying a "week's backups" should be written as a 'weeks backups', 
+i.e., don't put any apostrophes in there.
+
+The config file shown above should give you some hints on the possibilities.  Here are the valid keywords:
+
+  Time specifiers:
+    * this
+    * last
+  Time ranges (these can also be pluralized):
+    * day 
+    * month
+    * week
+    * year
+  Incremental time slices (one per * methods):
+    * daily
+    * weekly
+    * monthly
+    * yearly
+  And some placeholders, which have no affect, but allow the file to read nicely:
+    * backup
+    * backups
+    * from
+    * all
+
+Some other keywords, such as 'yesterday', and 'today' will be added in subsequent releases.
+
+And finally, a final warning: If a remove_old block is given, but no keep rules are given, every backup
+will be deleted!  This may change in a subsequent release, but for now, beware!
+
+
+= Contributions
+
+If you are interested in contributing code or documentation, please contact me, 
+Scott Taylor, at scott AT railsnewbie DOT com.
+

data/RELEASES

@@ -0,0 +1,21 @@
+
+= 0.2.0
+=======
+
+- A Script Runner/binary has been added (dbackup)
+- A DSL has been added for configuring backups
+- Removal of old backups can now be specified in the DSL, using
+  natural language forms
+- Before + After actions for the script runner
+- RSpec + RCov = Bug free code
+- Documentation added in the README file
+- Developer Documentation in Rake and RSpec Report
+- A series of example config files have been attached for Documentation
+- Root backup directories are now created automatically
+- Removed dependency on GNU's cp, so now this utility can be used
+  on any *NIX, including Mac OS X
+
+= 0.1.0
+=======
+
+- Initial Release

data/bin/dbackup

@@ -0,0 +1,22 @@
+#!/usr/bin/env ruby
+#
+# DatedBackup, A snapshot backup utility
+# Copyright (C) Scott Taylor (scott@railsnewbie.com), 2007
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+require File.dirname(__FILE__) + "/../lib/dated_backup"
+
+DatedBackup::ExecutionContext.new :main, *ARGV

data/{example_scripts/example.com.rb → example_configs/example.com}

@@ -1,5 +1,3 @@
-#!/usr/bin/env ruby
-
 # A script which will copy the directories /etc and /home
 # into /var/backups/network/backups/example.com/#{date}.  Note that
 # we wanted to get everything - even private keys owned by root
@@ -28,11 +26,7 @@
 # 
 
 
-require "dated_backup"
-
-DatedBackup.new(
-  :user_domain => "nbackup@example.com",
-  :options => '-v -e "ssh -i /root/.ssh/rsync-key" --rsync-path="sudo rsync"',
-  :sources => %w(/etc /home),
-  :destination => "/var/backups/network/backups/example.com"
-).run
+user_domain  "nbackup@example.com"
+options      "-v -e 'ssh -i /root/.ssh/rsync-key' --rsync-path='sudo rsync'"
+sources      "/etc", "/home"
+destination  "/var/backups/network/backups/example.com"

data/example_configs/local_etc_backup

@@ -0,0 +1,10 @@
+# A script to back up etc, locally, in /root/etc_backup
+
+source        '/etc'
+destination   '/root/etc_backup'
+
+after {
+  remove_old {
+    keep monthly backups    
+  }
+}

data/{example_scripts/samba_shares.rb → example_configs/samba_shares}

@@ -1,5 +1,3 @@
-#!/usr/bin/env ruby
-
 # A general purpose script to copy some WinXP/2000 shares
 # on a local network.  The shares are the 'C' drives
 # of the various nodes on the network.  They are mounted
@@ -16,23 +14,21 @@
 # there is no reason that it couldn't also be used for mounted
 # NFS drives, or even for a rudimentary Version Control
 # for any set of files (locally, or remotely)
- 
-require "dated_backup"
 
-puts "* mounting the samba clients"
-%x(mount //teresa2/c)
-%x(mount //jay/c)
-%x(mount //claudio/c)
 
-DatedBackup.new(
-  :source => "/mnt/shares",
-  :destination => "/var/backups/network/backups/shares",
-  :options => "-v" 
-).run
+source       '/mnt/shares'
+destination  '/var/backups/network/backups/shares'
+options      '-v' 
 
-# umount the clients
-puts "* umounting samba clients"
-%x(umount //teresa2/c)
-%x(umount //jay/c)
-%x(umount //claudio/c)
+after do
 
+  # CAREFUL! This will remove all old backups
+  # except the ones specified inside the block by the 'keep' rules
+  remove_old do
+    keep this weeks backups 
+    keep last weeks backups 
+    keep weekly backups from this month # or: keep this months weekly backups
+    keep monthly backups # or: keep all monthly backups
+  end
+  
+end

data/lib/dated_backup/core/backup_remover.rb

@@ -0,0 +1,43 @@
+
+
+module DatedBackup
+  class Core
+    class BackupRemover
+      class << self
+        
+        include DatedBackup::Core::CommandLine
+        
+        def remove!(dir, keep_rules=[])
+          find_removable_sets(dir, keep_rules)
+          
+          unless no_sets_to_remove?
+            execute("rm -rf #{to_remove.map{ |element| "#{element} " }.to_s.strip}")
+          end
+        end
+        
+      private
+
+        def find_removable_sets(dir, rules)
+          complete_set = BackupSet.find_files_in_directory dir
+          @to_remove = set_to_remove(complete_set, rules)
+        end
+        
+        attr_reader :to_remove
+        
+        def no_sets_to_remove?
+          to_remove.empty?
+        end
+        
+        def set_to_remove(set, keep_rules)
+          if keep_rules.empty?
+            set
+          else
+            to_remove = set - set.filter_by_rule(keep_rules.car)
+            return set_to_remove(to_remove, keep_rules.cdr)
+          end
+        end
+      end
+
+    end
+  end
+end

data/lib/dated_backup/core/backup_set.rb

@@ -0,0 +1,118 @@
+
+module DatedBackup
+  class Core
+    
+    class BackupSet < ReverseSortedUniqueArray
+
+      class << self    
+        # Given the base of the backup directories as a string, this method should find all of the Backup Directories,
+        # and return these Directories as a BackupSet
+        def find_files_in_directory(dir)
+          raise InvalidDirectoryError, "A valid directory must be given." unless File.directory?(dir)
+          new(Dir.glob "#{dir}/*")
+        end
+        
+        # Creates the boolean include_*? methods (include_month?, include_year? and so on).
+        # See the notes on the create_per_time_methods, and TimeSymbol.valid_symbols
+        def create_include_time_boolean_methods(*methods)
+          methods.each do |method|
+            define_method "include_#{method}?" do |time_value|
+              truth_value = false
+              self.each_as_time do |t|
+                truth_value = true if t.send(method) == time_value
+              end
+              truth_value
+            end
+          end
+        end    
+
+        # Creates the one_per_* (one_per_month, one_per_year, one_per_week, and one_per_day)
+        # methods.  Each of those methods will call the appropriate include_* methods,
+        # which is also dynamically defined
+        def create_one_per_time_methods(*methods)
+          methods.each do |method|
+            define_method "one_per_#{method}" do 
+              set = BackupSet.new
+              reject_with_string_and_timestamp do |string, timestamp|
+                set.push string unless set.send("include_#{method}?", timestamp.send("#{method}"))
+              end
+              set
+            end      
+          end
+        end
+
+        # Creates the many similar time methods:
+        #
+        # * include_year?
+        # * include_month?
+        # * include_day?
+        # * include_week
+        #
+        # * one_per_year
+        # * one_per_month
+        # * one_per_day
+        # * one_per_week
+        #
+        def create_dynamic_time_methods(time_array=[])
+          create_include_time_boolean_methods *time_array
+          create_one_per_time_methods *time_array
+        end    
+      end
+
+      create_dynamic_time_methods TimeSymbol.valid_symbols
+
+      def filter_by_rule(rule)
+        obj = self.dup
+        obj = obj.filter_by_range(rule[:constraint]) if rule[:constraint]
+        obj = obj.filter_by_scope(rule[:scope]) if rule[:scope]
+        return obj
+      end
+
+      def filter_by_scope(scope)
+        case scope
+        when :yearly
+          one_per_year
+        when :monthly
+          one_per_month
+        when :weekly
+          one_per_week
+        when :daily
+          one_per_day
+        end    
+      end
+
+      def filter_by_range(time_range)
+        reject_with_timestamp do |timestamp| 
+          !(time_range.include? timestamp)
+        end
+      end
+
+      def reject_with_timestamp &blk
+        reject do |element|
+          yield element.to_time
+        end
+      end
+
+      def reject_with_string_and_timestamp &blk
+        reject do |element|
+          yield element, element.to_time
+        end
+      end
+
+      def each_as_time &blk
+        self.each do |obj|
+          yield obj.to_time
+        end
+      end
+      
+      # BackupSet#- should return a new BackupSet,
+      # not an array
+      define_method "-" do |obj|
+        (self.to_a - obj.to_a).to_backup_set
+      end
+      
+    end
+        
+  end
+end
+

data/lib/dated_backup/core/command_line.rb

@@ -0,0 +1,11 @@
+module DatedBackup
+  class Core
+    module CommandLine
+      def execute(cmd, kernel_class=Kernel)
+        kernel_class.puts "* running: #{cmd}"
+        output = kernel_class.send :`, cmd
+        kernel_class.puts output
+      end
+    end      
+  end
+end

data/lib/dated_backup/core/dated_backup.rb

@@ -0,0 +1,71 @@
+
+module DatedBackup 
+  class Core
+
+    BACKUP_REGEXP = /[12][0-9]{3}\-[01][0-9]\-[0-3][0-9]\-[0-2][0-9]h\-[0-6][0-9]m\-[0-6][0-9]s/
+
+    include DatedBackup::Core::CommandLine
+    include DatedBackup::Core::Tasks
+
+    attr_accessor :sources, :destination, :options, :backup_root, :user_domain
+    attr_reader :pre_run_commands, :kernel
+    attr_reader :before_run, :after_run
+
+    def initialize(procs={}, kernel=Kernel)
+      @kernel = kernel
+      @before_run = procs[:before] || Proc.new {}
+      @after_run = procs[:after] || Proc.new {}
+    end
+
+    def set_attributes(h={})
+      parse_command_options(h)
+      @destination = generate_backup_filename    
+      if @user_domain
+        @sources.map! { |src| "#{@user_domain}:#{src}" }
+      end        
+    end
+
+    def check_for_directory_errors
+      if sources.nil? || sources.empty?
+        raise DirectoryError, "No source directory given"
+      elsif backup_root.nil? || backup_root.empty?
+        raise DirectoryError, "No destination directory given"
+      end
+    end
+
+    # create the first backup, if non-existent  
+    # otherwise cp -al (or # replace cp -al a b with cd a && find . -print | cpio -dpl ../b )
+    # and then create the backup of the dirs using rsync -a --delete
+    # the files, in the end, should be read only and undeletable
+    def run
+      DatedBackup::ExecutionContext.new :before, &@before_run
+      run_tasks
+      DatedBackup::ExecutionContext.new :after, &@after_run
+    end
+
+  private
+
+    def generate_backup_filename
+      timestamp = Time.now.strftime "%Y-%m-%d-%Hh-%Mm-%Ss"
+      "#{@backup_root}/#{timestamp}"
+    end                         
+
+  protected
+
+    def parse_command_options(h={})
+      @pre_run_commands = h[:pre_run_commands]
+      @pre_run_commands = h[:pre_run_command].to_a if h[:pre_run_command]
+
+      @backup_root = *h[:destination]
+      @options = h[:options] ? h[:options].map { |e| "#{e} "}.to_s.strip : ""
+
+      @user_domain = h[:user_domain]    
+      @sources = h[:sources] || h[:source]
+      check_for_directory_errors
+    end
+
+  end
+    
+end   
+
+

data/lib/dated_backup/core/tasks.rb

@@ -0,0 +1,44 @@
+
+module DatedBackup
+  class Core
+    module Tasks
+      
+      def run_tasks
+        create_main_backup_directory                     
+        create_backup
+      end
+
+      def create_backup
+        additional_options = has_backup? ? "--link-dest #{latest_backup} " : ""
+        
+        sources.each do |source|        
+          cmd = "rsync -a --delete #{additional_options}#{options} #{source} #{destination}"
+          execute cmd, kernel
+          kernel.puts "\n\n"
+        end
+      end
+
+      def create_main_backup_directory
+        unless File.exists? backup_root
+          kernel.puts "* Creating main backup directory #{backup_root}"
+          Dir.mkdir backup_root 
+        end
+      end
+
+      def find_latest_backup
+        backup_directories.sort.reverse.first
+      end
+      
+      alias :latest_backup :find_latest_backup
+      
+      def backup_directories
+        Dir.glob("#{backup_root}/*").grep(BACKUP_REGEXP)
+      end
+      
+      def has_backup?
+        backup_directories.empty? ? false : true
+      end
+      
+    end
+  end
+end

data/lib/dated_backup/core/warnings.rb

@@ -0,0 +1,15 @@
+
+module DatedBackup
+  module Warnings
+  
+    def execute_silently(&blk)
+      old_warning_level = $VERBOSE
+      $VERBOSE = nil
+      yield
+      $VERBOSE = old_warning_level
+    end
+    
+    module_function :execute_silently
+
+  end
+end

data/lib/dated_backup/core.rb

@@ -0,0 +1,6 @@
+require File.dirname(__FILE__) + "/core/warnings"
+require File.dirname(__FILE__) + "/core/backup_set"
+require File.dirname(__FILE__) + "/core/tasks"
+require File.dirname(__FILE__) + "/core/command_line"
+require File.dirname(__FILE__) + "/core/backup_remover"
+require File.dirname(__FILE__) + "/core/dated_backup"

data/lib/dated_backup/dsl/execution_context.rb

@@ -0,0 +1,56 @@
+
+module DatedBackup
+  class ExecutionContext
+
+    def initialize(name, *params, &blk)  
+      DatedBackup::Warnings.execute_silently do
+        if name == :main
+          params.each do |filename|
+            Main.load filename
+          end
+        elsif name == :before || name == :after
+          Around.new &blk
+        end        
+      end
+    end
+    
+    class Main
+      class << self
+        def load(filename)
+          klass = Class.new
+          klass.send(:include, DSL::Main)
+          instance = klass.new          
+          
+          File.open filename, "r" do |file|
+            instance.instance_eval file.read
+          end
+          
+          @main_instance = DatedBackup::Core.new(instance.procs)
+          @main_instance.set_attributes(instance.hash)
+          @main_instance.run
+        end
+        
+        attr_reader :main_instance
+        alias :core_instance :main_instance
+        alias :instance      :main_instance
+      end
+    end
+
+    class Around
+      def initialize(around=self, &blk)
+        around.instance_eval &blk
+      end
+
+      def remove_old(&blk)
+        klass = Class.new
+        klass.send(:include, DSL::TimeExtensions)
+        instance = klass.new
+
+        instance.instance_eval &blk
+        
+        Core::BackupRemover.remove!(Main.instance.backup_root, instance.kept)
+      end
+    end
+  
+  end
+end