odisk 0.2.0

data/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2012, Peter Ohler
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ - Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+ - Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+ - Neither the name of Peter Ohler nor the names of its contributors may be
+   used to endorse or promote products derived from this software without
+   specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,150 @@
+oDisk
+=====
+
+Remote Encrypted File Synchronization, oDisk
+
+oDisk is a file synchronization application. The need for oDisk came from the
+discontinuation of Apple's iDisk. oDisk provides a means to backup to a remote
+host. oDisk encrypts and compresses backups to save space and provide a layer
+of security not found in iDisk or other backup schemes. Backups are stored as
+individual files to provide finer granularity in recover if everything else
+falls apart.
+
+oDisk is also a demonstration of the [Opee](http://www.ohler.com/opee) gem
+which utilizes an alternative approach to dealing with multiple threads.
+
+## <a name="source">Source</a>
+
+*GitHub* *repo*: https://github.com/ohler55/odisk
+
+*RubyGems* *repo*: https://rubygems.org/gems/odisk
+
+## <a name="links">Links of Interest</a>
+
+[Object-based Parallel Evaluation Environment](http://www.ohler.com/opee) the gem oDisk is built on.
+
+## <a name="release">Release Notes</a>
+
+### Release 0.2.0
+
+ - Renamed Orefs to oDisk.
+
+# Plans and Notes
+
+oDisk has only recently been opened up for public viewing. It is barely ready
+for use. I am using it to backup some financial records but I am also keeping
+copies on more than one computer in case something fails. So far the basic
+backup functionality works with backups encrypted and compressed on a remote
+server. Adding new files and making modifications works just fine. Removing
+files has not been implemented yet and changing ownership or mode only take
+effect if the file is touched as well. Feel free to give it a try and let me
+know when you run into bugs.
+
+ - Support exclusion of file and directories
+  - specify on command line and create .odisk/exclude file
+   - use ::File.fnmatch(pattern, path, ::File::FNM_DOTMATCH)
+  - pass array of excludes to digest creation
+   - loosen up current restriction on any file that begins with .
+
+ - Support file removal
+  - Detect file removal based on previous digest or by calling a remove script
+  - Keep record of removals in digest
+  - Note conflicts if modifications are more recent that removal
+   - Use a script to pick remove or keep
+
+ - Handle changes in mode, owner, and group
+  - Compare to previous digest to detect changes
+   - File modification times are not changes by mode, owner, or group changes
+  - Note conflicts if modifications are more recent than remote
+   - Use a script to pick change or keep local version
+
+ - Add progress tracker
+  - Have component pass info to progress actor which will update progress
+   - Planner sends info on all changes along with sizes and comp/crypt flags to progress actor
+    - come up with algoritm for estimating time based on size and flags
+   - comp, crypt, and transfer send status to progress actor
+  - Optional terminal display
+
+ - Support backgroup application with web front end (much later)
+
+## Installation
+
+Installation requires Ruby 1.9.3. After that, install the odisk gem.
+
+    gem install odisk
+
+The net-ssh and net-sftp gems are also needed as are the oj and opee gems.
+
+    gem install net-ssh
+    gem install net-sftp
+    gem install oj
+    gem install opee
+
+GnuPG must be installed. It can be down loaded from
+[GnuPG.org](http://www.gnupg.org). Follow the instruction on
+[GnuPG.org](http://www.gnupg.org) site for installation.
+
+oDisk is now ready to use. ssh and sftp must be running on the remote site and
+credentials must be installed so that the user is not prompted for a password
+when using ssh or sftp.
+
+## Usage
+
+After a directory has been selected for backing up *odisk* can be run to
+copy the directory to a remote server. For purposes of this description the
+directory to be backed up is *~/backup*.
+
+A passphrase file will also be needed for encryption. The recommended location
+is in a *~/.odisk* directory. The contents of the file will be the passphrase
+for *gpg*.
+
+Make sure you have a remote server that has an sftp and ssh daemon
+running. Your credentials must be set in the authorized_keys file. If you can
+login without a password it is set up correctly.
+
+The first time *odisk* is used to backup a directory information about the
+remote server must be provided. After the first time that information is not
+needed again. Alternatively a *~/.odisk/remotes* file can be set up before
+running *odisk*.
+
+To backup to *my_server.remote.com* for user *me* to the *backup* directory on
+the remote server with a passphrase file of *~/.odisk/backup.pass* the
+following command should be executed.
+
+    odisk -r me@my_server.remote.com:~/.odisk/backup.pass ~/backup
+
+A file named *~/backup/.odisk/remote* will be created with the connection
+information for future invocations so that the next time a backup is made on
+the *~/backup* directory the command only needs to be:
+
+    odisk ~/backup
+
+## License:
+
+    Copyright (c) 2012, Peter Ohler
+    All rights reserved.
+    
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions are met:
+    
+     - Redistributions of source code must retain the above copyright notice, this
+       list of conditions and the following disclaimer.
+    
+     - Redistributions in binary form must reproduce the above copyright notice,
+       this list of conditions and the following disclaimer in the documentation
+       and/or other materials provided with the distribution.
+    
+     - Neither the name of Peter Ohler nor the names of its contributors may be
+       used to endorse or promote products derived from this software without
+       specific prior written permission.
+    
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+    AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+    DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+    SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+    CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+    OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/bin/odisk

@@ -0,0 +1,184 @@
+#!/usr/bin/env ruby -wW1
+# encoding: UTF-8
+
+while (i = ARGV.index('-I'))
+  x,path = ARGV.slice!(i, 2)
+  $: << path
+end
+
+# TBD tmp for testing
+$: << ::File.join(::File.dirname(__FILE__), "../../oj/lib")
+$: << ::File.join(::File.dirname(__FILE__), "../../oj/ext")
+$: << ::File.join(::File.dirname(__FILE__), "../../opee/lib")
+$: << ::File.join(::File.dirname(__FILE__), "../lib")
+
+require 'optparse'
+begin
+  v = $VERBOSE
+  $VERBOSE = false
+  require 'net/ssh'
+  require 'net/sftp'
+  $VERBOSE = v
+end
+require 'opee'
+require 'oj'
+require 'odisk'
+
+$verbose = Logger::WARN
+$dir = '.'
+$digests_only = false
+$dry_run = false
+$master = nil
+$plain = false
+$remote = ::ODisk::Remote.new()
+$crypter_count = 2
+$copier_count = 4
+
+opts = OptionParser.new('Usage: odisk [options] <local_directory>')
+opts.on('-s', 'decrease verbosity')                 { $verbose += 1 unless 5 == $verbose }
+opts.on('-v', 'increase verbosity')                 { $verbose -= 1 unless 0 == $verbose }
+opts.on('-d', 'generate digests only')              { $digests_only = true }
+opts.on('-u', 'unencrypted or plain remote files')  { $plain = true }
+opts.on('-n', 'dry run / no modifications')         { $dry_run = true }
+opts.on('-r', '--remote [user@host:dir:pass_file]',
+        String, 'remote user, host, directory, passphrase file for gpg')  { |r|
+  $remote.update(r)
+}
+opts.on('-m', '--master [local | remote]',
+        String, 'force master to local or remote')  { |m|
+  case m
+  when 'local'
+    $master = ::ODisk::Planner::Step::LOCAL
+  when 'remote'
+    $master = ::ODisk::Planner::Step::REMOTE
+  else
+    puts opts.help
+    Process.exit!(0)
+  end
+}
+opts.on('-e', '--encrypt-count Integer', Integer, 'number of encryption actors') { |i| $crypter_count = i }
+opts.on('-c', '--copier-count Integer', Integer, 'number of copier actors')      { |i| $copier_count = i }
+opts.on('-h', '--help', 'Show this display')                                     { puts opts.help; Process.exit!(0) }
+dirs = opts.parse(ARGV)
+
+if 1 != dirs.size
+  puts opts.help
+  Process.exit!(0)
+end
+$local_top = ::File.expand_path(dirs[0])
+
+# Walk up the directory tree looking for .odisk directories and a remote.json
+# in that directory. If the directory does not exist then stop the walk. If
+# not found check the ~/.odisk/remotes.json file for a matching top.
+top = $local_top
+if $ruser.nil? || $rhost.nil? || $rtop.nil?
+  while ::File.directory?(::File.join(top, '.odisk'))
+    rfile = ::File.join(top, '.odisk', 'remote')
+    if ::File.file?(rfile)
+      rstr = ::File.read(rfile).strip()
+      #$remote.pass_file = ::File.expand_path($remote.pass_file) unless $remote.pass_file.nil? || $remote.pass_file.empty?
+      orig_pass_file = $remote.pass_file
+      $remote.update(rstr)
+      $remote.pass_file = ::File.expand_path($remote.pass_file) unless $remote.pass_file.nil? || $remote.pass_file.empty?
+      if !$remote.dir.nil? && !$remote.dir.empty? && top != $local_top
+        $remote.dir = $remote.dir + $local_top[top.size..-1]
+      end
+      if $remote.pass_file != orig_pass_file && !::File.file?($remote.pass_file)
+        $remote.pass_file = ::File.join(top, '.odisk', $remote.pass_file)
+      end
+      break
+    end
+    top = ::File.dirname(top)
+  end
+end
+
+ODisk.info_from_remotes($local_top, $remote) unless $remote.complete?
+$remote.user = ENV['USER'] if $remote.user.nil?
+$remote.pass_file = nil if $plain
+unless $remote.okay?
+  puts "*** user@host:top_dir not specified on command line, in local .odisk/remote file, or in ~/.odisk/remotes"
+  puts opts.help
+  Process.exit!(0)
+end
+
+Thread.current[:name] = 'main'
+::Opee::Env.logger.formatter = proc { |s,t,p,m|
+  s = '' if s.nil?
+  "#{s[0]} [#{t.strftime('%Y-%m-%dT%H:%M:%S.%6N')} ##{p}]: #{m}\n"
+}
+::Opee::Env.logger.severity = $verbose
+
+if Logger::INFO >= $verbose
+  if $digests_only
+    ::Opee::Env.info(%{
+  Generate Local Digests
+    local directory:  #{::File.expand_path($local_top)}
+})
+  else
+    ::Opee::Env.info(%{
+  Synchronize
+    remote host:      #{$remote.host}
+    remote user:      #{$remote.user}
+    remote directory: #{$remote.dir}
+    local:            #{::File.expand_path($local_top)}
+    dry run:          #{$dry_run}
+    master:           #{$master.nil? ? 'NONE' : (::ODisk::Planner::Step::LOCAL == $master ? 'LOCAL' : 'REMOTE')}
+})
+  end
+end
+
+# If $local_top/.odisk/remote does not exist or is different that what is in $remote, replace it.
+remote_str = $remote.to_s
+top_remote_path = ::File.join($local_top, '.odisk', 'remote')
+if !::File.file?(top_remote_path) || ::File.read(top_remote_path).strip() != remote_str
+  ::Opee::Env.info("Writing #{top_remote_path}")
+  unless $dry_run
+  `mkdir -p #{::File.join($local_top, '.odisk')}`
+  ::File.open(top_remote_path, 'w') { |f| f.write(remote_str + "\n") }
+  end
+end
+
+fetcher = nil
+planner_inputs = [:digester, :starter]
+
+fixer = ::ODisk::StatFixer.new(:name => 'Fixer')
+dir_wq = ::Opee::WorkQueue.new(:method => :start, :name => 'DirWorkQueue')
+if $digests_only
+  copy_wq = nil
+  crypt_wq = nil
+else
+  copy_wq = ::Opee::AskQueue.new(:name => 'CopyQueue')
+  crypt_wq = ::Opee::AskQueue.new(:name => 'CryptQueue')
+  $copier_count.times { |i|
+    ::ODisk::Copier.new(:name => "Copier-#{i}",
+                        :crypt_queue => crypt_wq,
+                        :copy_queue => copy_wq,
+                        :fixer => fixer)
+  }
+  $crypter_count.times { |i|
+    ::ODisk::Crypter.new(:name => "Crypter-#{i}",
+                         :crypt_queue => crypt_wq,
+                         :copy_queue => copy_wq,
+                         :fixer => fixer)
+  }
+  planner_inputs << :fetcher
+end
+planner = ::ODisk::Planner.new(:name => 'Planner',
+                               :dir_queue => dir_wq,
+                               :copy_queue => copy_wq,
+                               :crypt_queue => crypt_wq,
+                               :inputs => planner_inputs,
+                               :fixer => fixer)
+fetcher = ::ODisk::Fetcher.new(:name => 'Fetcher',
+                               :collector => planner) unless $digests_only
+digester = ::ODisk::Digester.new(:name => 'Digester',
+                                 :collector => planner)
+starter = ::ODisk::SyncStarter.new(:name => 'Starter',
+                                   :dir_queue => dir_wq,
+                                   :digester => digester,
+                                   :fetcher => fetcher,
+                                   :collector => planner)
+
+dir_wq.ask(:add, '')
+
+::Opee::Env.wait_close()

data/lib/odisk.rb

@@ -0,0 +1,46 @@
+
+module ODisk
+
+  def self.info_from_remotes(local_dir, remote)
+    orig_pass_file = remote.pass_file
+    local_dir = ::File.expand_path(local_dir)
+    remotes_file = ::File.join(::File.expand_path('~'), '.odisk', 'remotes')
+    if ::File.file?(remotes_file)
+      ::File.readlines(remotes_file).each do |line|
+        l,r = line.split(':', 2)
+        l = ::File.expand_path(l)
+        if l == local_dir || local_dir.start_with?(l + '/')
+          remote.update(r)
+          remote.dir = remote.dir + remote.dir[l.size..-1] if l != local_dir && !remote.dir.nil?
+          if remote.pass_file != orig_pass_file && !::File.file?(remote.pass_file)
+            remote.pass_file = ::File.join(::File.expand_path('~'), '.odisk', remote.pass_file)
+          end
+          break
+        end
+      end
+    end
+  end
+
+end
+
+# data
+require 'odisk/remote'
+require 'odisk/info'
+require 'odisk/dir'
+require 'odisk/link'
+require 'odisk/file'
+require 'odisk/digest'
+require 'odisk/diff'
+# jobs
+require 'odisk/dirsyncjob'
+require 'odisk/statjob'
+require 'odisk/syncjob'
+# collectors
+require 'odisk/statfixer'
+require 'odisk/planner'
+# actors
+require 'odisk/copier'
+require 'odisk/crypter'
+require 'odisk/digester'
+require 'odisk/fetcher'
+require 'odisk/syncstarter'

data/lib/odisk/copier.rb

@@ -0,0 +1,74 @@
+
+module ODisk
+  class Copier < ::Opee::Actor
+
+    def initialize(options={})
+      @ftp = nil
+      @ssh = nil
+      super(options)
+      @copy_queue.ask(:ready, self)
+    end
+    
+    def set_options(options)
+      super(options)
+      @copy_queue = options[:copy_queue]
+      @crypt_queue = options[:crypt_queue]
+      @fixer = options[:fixer]
+    end
+
+    def close()
+      @ftp.close_channel() unless @ftp.nil?
+      @ftp = nil
+      @ssh.close() unless @ssh.nil?
+      super()
+    end
+
+    private
+
+    def upload(local, remote, delete_after=false)
+      ::Opee::Env.info("upload \"#{local}\" to \"#{remote}\"#{delete_after ? ' then delete' : ''}")
+      unless $dry_run
+        @ftp = Net::SFTP.start($remote.host, $remote.user) if @ftp.nil?
+        begin
+          @ftp.upload!(local, remote)
+          `rm "#{local}"` if delete_after
+          ::Opee::Env.warn("Uploaded \"#{local}\"")
+        rescue Net::SFTP::StatusException => e
+          if Net::SFTP::Constants::StatusCodes::FX_NO_SUCH_FILE == e.code
+            assure_dirs_exist(::File.dirname(remote))
+            retry
+          else
+            ::Opee::Env.error("Upload of \"#{local}\" failed: #{e.class}: (#{e.code}) #{e.description}\n #{e.text}\n #{e.response}")
+          end
+        end
+      end
+      @copy_queue.ask(:ready, self)
+    end
+
+    def download(remote, local, decrypt_path=nil)
+      ::Opee::Env.info("download #{remote} to #{local}")
+      @ftp = Net::SFTP.start($remote.host, $remote.user) if @ftp.nil?
+      begin
+        @ftp.download!(remote, local)
+        if decrypt_path.nil?
+          @fixer.ask(:collect, local, :copier) unless @fixer.nil?
+          ::Opee::Env.warn("Downloaded \"#{local}\"")
+        else
+          @crypt_queue.add_method(:decrypt, local, decrypt_path)
+        end
+      rescue Exception => e
+        ::Opee::Env.error("Download of \"#{local}\" failed: #{e.class}: #{e.message}")
+        #::Opee::Env.rescue(e)
+      end
+      @copy_queue.ask(:ready, self)
+    end
+
+    def assure_dirs_exist(dir)
+      ::Opee::Env.info("creating remote dir \"#{dir}\"")
+      @ssh = Net::SSH.start($remote.host, $remote.user) if @ssh.nil?
+      out = @ssh.exec!(%{mkdir -p "#{dir}"})
+      raise out unless out.nil? || out.strip().empty?
+    end
+
+  end # Copier
+end # ODisk