dbox 0.5.3 → 0.6.0

data/History.txt

@@ -0,0 +1,83 @@
+== 0.6.0 / 2011-11-24
+  * Major Enhancements
+    * Upgrade from Dropbox API v0 to v1.
+    * Storage of hashes of local files to alleviate problems with moving/copying directories causing files to be detected as modified.
+    * Handling of writes that would overwrite unsaved data (files are renamed).
+    * SSL support.
+    * Sync API.
+
+== 0.5.3 / 2011-11-04
+  * Major Enhancements
+    * Tracking & returning of files that fail to sync.
+    * Better consistency during failure scenarios.
+  * Minor Enhancements
+    * Returning of exit status of 1 when failures occur.
+  * Bug Fixes
+    * Fixed incorrect handling of some Dropbox server errors.
+
+== 0.5.2 / 2011-10-03
+  * Bug Fixes
+    * Fixed broken database migration.
+
+== 0.5.1 / 2011-10-03
+  * Bug Fixes
+    * Fixed cascading deletes on directory deletion.
+    * Fixed support for moving/renaming local repository.
+
+== 0.5.0 / 2011-09-19
+  * Major Enhancements
+    * Complete rewrite of metadata storage to use SQLite database.
+    * Streaming downloads.
+    * Parallelized downloads and uploads.
+  * Minor Enhancements
+    * Atomic saving of downloaded files to eliminate partial files if killed during download.
+
+== 0.4.4 / 2011-06-28
+  * Minor Enhancements
+    * Detection of corrupt database.
+    * Added gem dependencies.
+  * Bug Fixes
+    * Fixed Ruby 1.9 support.
+    * Fixed DB corruption issue when hard killing process.
+
+== 0.4.3 / 2011-05-18
+  * Minor Enhancements
+    * More robust DB saving to handle errors after partial pushes or pulls.
+  * Bug Fixes
+    * Fixed revision stamping.
+    * Fixed sorting of filenames in changelist.
+
+== 0.4.2 / 2011-05-17
+  * Bug Fixes
+    * Fixed directories with certain characters: []{}*?.
+    * Fixed URL encoding.
+
+== 0.4.1 / 2011-05-16
+  * Minor Enhancements
+    * Added Dbox.exists? to Ruby API.
+    * Improved logging.
+
+== 0.4.0 / 2011-05-16
+  * Major Enhancements
+    * Added move API to rename remote folder.
+
+== 0.3.0 / 2011-05-14
+  * Major Enhancements
+    * Added returning of changelist from push & pull operations.
+    * Improved error handling.
+  * Minor Enhancements
+    * Logging cleanup.
+  * Bug Fixes
+    * Fixed re-download after upload issue.
+
+== 0.2.0 / 2011-05-14
+  * Major Enhancements
+    * Logging.
+    * Tests.
+
+== 0.1.0 / 2011-05-13
+  * Inaugural Release
+    * Ruby API.
+    * Command-line client.
+    * Create, clone, pull, push support.
+    * Metadata database to track synchronized content.

data/README.md

@@ -1,9 +1,13 @@
 dbox
 ====
 
-An easy way to push and pull your Dropbox folders, with fine-grained control over what folder you are syncing, where you are syncing it to, and when you are doing it.
+Dropbox integration made easy.
 
-**IMPORTANT:** This is **not** an automated Dropbox client. It will exit after sucessfully pushing/pulling, so if you want regular updates, you can run it in cron, a loop, etc. If you do want to run it in a loop, take a look at [sample_polling_script.rb](http://github.com/kenpratt/dbox/blob/master/sample_polling_script.rb).
+Push and pull your Dropbox folders, with fine-grained control over what folder you are syncing, where you are syncing it to, and when you are doing it.
+
+Take your pick: a command-line client or a Ruby API.
+
+**IMPORTANT:** This is **not** an automated Dropbox client. It will exit after sucessfully pushing/pulling, so if you want regular updates, you can run it in cron, a loop, etc. If you do want to run it in a loop, take a look at [sample_polling_script.rb](http://github.com/kenpratt/dbox/blob/master/sample_polling_script.rb). You get deterministic control over what you want Dropbox to do and when you want it to happen.
 
 
 Installation
@@ -17,7 +21,7 @@ $ gem install dbox
 
 ### Get developer keys
 
-* Follow the instructions at https://www.dropbox.com/developers/quickstart to create a Dropbox development application, and copy the application keys. Unless you get your app approved for production status, these keys will only work with the account you create them under, so make sure you are logged in with the account you want to access from dbox.
+* Follow the instructions at https://www.dropbox.com/developers/quickstart to create a Dropbox development application, and copy the application keys. Unless you get your app approved for production status, these keys will only work with the account you create them under, so make sure you are logged in with the account you want to access from ```dbox```.
 
 * Now either set the keys as environment variables:
 
@@ -26,7 +30,7 @@ $ export DROPBOX_APP_KEY=cmlrrjd3j0gbend
 $ export DROPBOX_APP_SECRET=uvuulp75xf9jffl
 ```
 
-* Or include them in calls to dbox:
+* Or include them in calls to ```dbox```:
 
 ```sh
 $ DROPBOX_APP_KEY=cmlrrjd3j0gbend DROPBOX_APP_SECRET=uvuulp75xf9jffl dbox ...
@@ -53,7 +57,7 @@ $ export DROPBOX_AUTH_KEY=v4d7l1rez1czksn
 $ export DROPBOX_AUTH_SECRET=pqej9rmnj0i1gcxr4
 ```
 
-* Or include them in calls to dbox:
+* Or include them in calls to ```dbox```:
 
 ```sh
 $ DROPBOX_AUTH_KEY=v4d7l1rez1czksn DROPBOX_AUTH_SECRET=pqej9rmnj0i1gcxr4 dbox ...
@@ -97,6 +101,12 @@ $ dbox pull [<local_path>]
 $ dbox push [<local_path>]
 ```
 
+#### Sync (pull changes from Dropbox, then push changes to Dropbox)
+
+```sh
+$ dbox sync [<local_path>]
+```
+
 #### Move (move/rename the Dropbox folder)
 
 ```sh
@@ -146,12 +156,20 @@ Oh, Hello
 Using dbox from Ruby
 --------------------
 
-The Ruby clone, pull, and push APIs return a hash listing the changes made during that pull/push. If any failures were encountered while uploading or downloading from Dropbox, they will be shown in the ```:failed``` entry in the hash. Often, trying your operation again will resolve the failures as the Dropbox API returns errors for valid operations on occasion.
+The Ruby clone, pull, and push APIs return a hash of the changes made during that operation. If any failures were encountered while uploading or downloading from Dropbox, they will be shown in the ```:failed``` entry in the hash. Often, trying your operation again will resolve the failures as the Dropbox API occasionally returns errors for valid operations.
 
 ```ruby
-{ :created => ["foo.txt"], :deleted => [], :updated => [], :failed => [] }
+{ :created => ["foo.txt"], :deleted => [], :updated => [] :failed => [] }
 ```
 
+If any conflicts occur where file contents would be lost, the conflicting file is renamed and the resulting hash has a ```:conflicts``` entry. On a push operation, the conflicting file being pushed will be renamed. On a pull, the existing file that would have been overwritten will be renamed and the downloaded file will take the name (as that will keep multiple clients in sync).
+
+```ruby
+{ :created => [], :updated => [], :deleted => [], :conflicts => [{ :original => "foo.txt", :renamed => "foo (1).txt" }], :failed => [] }
+```
+
+The sync API returns a hash with two entries: ```:push``` and ```:pull```, which contain the change hashes for the two operations.
+
 ### Usage
 
 #### Setup
@@ -186,6 +204,12 @@ Dbox.pull(local_path)
 Dbox.push(local_path)
 ```
 
+#### Sync (pull changes from Dropbox, then push changes to Dropbox)
+
+```ruby
+Dbox.sync(local_path)
+```
+
 #### Move (move/rename the Dropbox folder)
 
 ```ruby

data/TODO.txt

@@ -1,6 +1,6 @@
+* Make Dbox case insentive on case sensitive filesystems, as Dropbox is case insensitive.
 * Refactor threading in change detection in pull to use ParallelTasks?
 * Saving SQLite db changes dir timestamp -- try to preserve it?
-* Add a "sync" command that pushes and pulls in one go
 * See if prepared statements speed up operations on large repos much
 * Look down directory tree until you hit a .dbox.sqlite3 file so you can use commands from anywhere inside a tree (like git)
 * Add support for partial push/pull (subtree push/pull)?

data/VERSION

@@ -1 +1 @@
-0.5.3
+0.6.0

data/bin/dbox

@@ -14,6 +14,7 @@ Commands:
   clone <remote_path> [<local_path>]    Clone an existing Dropbox folder
   pull [<local_path>]                   Pull chonges from Dropbox
   push [<local_path>]                   Push changes to Dropbox
+  sync [<local_path>]                   Sync changes to Dropbox
   move <new_remote_path> [<local_path>] Move the remote Dropbox folder to a new location
 
 Environment varables needed for everything:
@@ -53,7 +54,7 @@ when "create", "clone"
 
   res = Dbox.send(command, remote_path, local_path)
   exit 1 if res[:failed].size > 0
-when "pull", "push"
+when "pull", "push", "sync"
   # default to current directory
   local_path = args[0] || "."
 

data/dbox.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = "dbox"
-  s.version = "0.5.3"
+  s.version = "0.6.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Ken Pratt"]
-  s.date = "2011-11-04"
+  s.date = "2011-11-24"
   s.description = "An easy-to-use Dropbox client with fine-grained control over syncs."
   s.email = "ken@kenpratt.net"
   s.executables = ["dbox"]
@@ -19,6 +19,7 @@ Gem::Specification.new do |s|
   ]
   s.files = [
     ".document",
+    "History.txt",
     "LICENSE.txt",
     "README.md",
     "Rakefile",
@@ -28,23 +29,24 @@ Gem::Specification.new do |s|
     "dbox.gemspec",
     "lib/dbox.rb",
     "lib/dbox/api.rb",
+    "lib/dbox/cacert.pem",
     "lib/dbox/database.rb",
     "lib/dbox/db.rb",
     "lib/dbox/loggable.rb",
     "lib/dbox/parallel_tasks.rb",
     "lib/dbox/syncer.rb",
+    "lib/dbox/utils.rb",
     "sample_polling_script.rb",
     "spec/dbox_spec.rb",
     "spec/spec_helper.rb",
-    "vendor/dropbox-client-ruby/LICENSE",
-    "vendor/dropbox-client-ruby/README",
-    "vendor/dropbox-client-ruby/Rakefile",
-    "vendor/dropbox-client-ruby/config/testing.json.example",
-    "vendor/dropbox-client-ruby/lib/dropbox.rb",
-    "vendor/dropbox-client-ruby/manifest",
-    "vendor/dropbox-client-ruby/test/authenticator_test.rb",
-    "vendor/dropbox-client-ruby/test/client_test.rb",
-    "vendor/dropbox-client-ruby/test/util.rb"
+    "vendor/dropbox-ruby-sdk/CHANGELOG",
+    "vendor/dropbox-ruby-sdk/LICENSE",
+    "vendor/dropbox-ruby-sdk/README",
+    "vendor/dropbox-ruby-sdk/cli_example.rb",
+    "vendor/dropbox-ruby-sdk/dropbox_controller.rb",
+    "vendor/dropbox-ruby-sdk/gemspec.rb",
+    "vendor/dropbox-ruby-sdk/lib/dropbox_sdk.rb",
+    "vendor/dropbox-ruby-sdk/web_file_browser.rb"
   ]
   s.homepage = "http://github.com/kenpratt/dbox"
   s.licenses = ["MIT"]

data/lib/dbox.rb

@@ -1,8 +1,8 @@
 ROOT_PATH = File.expand_path(File.join(File.dirname(__FILE__), ".."))
 $:.unshift File.join(ROOT_PATH, "lib")
-$:.unshift File.join(ROOT_PATH, "vendor", "dropbox-client-ruby", "lib")
+$:.unshift File.join(ROOT_PATH, "vendor", "dropbox-ruby-sdk", "lib")
 
-require "dropbox"
+require "dropbox_sdk"
 require "fileutils"
 require "time"
 require "yaml"
@@ -12,6 +12,7 @@ require "sqlite3"
 require "active_support/core_ext/hash/indifferent_access"
 
 require "dbox/loggable"
+require "dbox/utils"
 require "dbox/api"
 require "dbox/database"
 require "dbox/db"
@@ -53,6 +54,14 @@ module Dbox
     Dbox::Syncer.push(local_path)
   end
 
+  def self.sync(local_path)
+    log.debug "Syncing (local: #{local_path})"
+    res = {}
+    res[:pull] = pull(local_path)
+    res[:push] = push(local_path)
+    res
+  end
+
   def self.move(new_remote_path, local_path)
     log.debug "Moving (new remote: #{new_remote_path}, local: #{local_path})"
     new_remote_path = clean_remote_path(new_remote_path)

data/lib/dbox/api.rb

@@ -9,19 +9,24 @@ module Dbox
     include Loggable
 
     def self.authorize
-      auth = Authenticator.new(conf)
-      authorize_url = auth.get_request_token
-      puts "Please visit the following URL in your browser, log into Dropbox, and authorize the app you created.\n\n#{authorize_url}\n\nWhen you have done so, press [ENTER] to continue."
+      app_key = ENV["DROPBOX_APP_KEY"]
+      app_secret = ENV["DROPBOX_APP_SECRET"]
+
+      raise(ConfigurationError, "Please set the DROPBOX_APP_KEY environment variable to a Dropbox application key") unless app_key
+      raise(ConfigurationError, "Please set the DROPBOX_APP_SECRET environment variable to a Dropbox application secret") unless app_secret
+
+      auth = DropboxSession.new(app_key, app_secret)
+      puts "Please visit the following URL in your browser, log into Dropbox, and authorize the app you created.\n\n#{auth.get_authorize_url}\n\nWhen you have done so, press [ENTER] to continue."
       STDIN.readline
       res = auth.get_access_token
-      puts "export DROPBOX_AUTH_KEY=#{res.token}"
+      puts "export DROPBOX_AUTH_KEY=#{res.key}"
       puts "export DROPBOX_AUTH_SECRET=#{res.secret}"
       puts
       puts "This auth token will last for 10 years, or when you choose to invalidate it, whichever comes first."
       puts
       puts "Now either include these constants in yours calls to dbox, or set them as environment variables."
       puts "In bash, including them in calls looks like:"
-      puts "$ DROPBOX_AUTH_KEY=#{res.token} DROPBOX_AUTH_SECRET=#{res.secret} dbox ..."
+      puts "$ DROPBOX_AUTH_KEY=#{res.key} DROPBOX_AUTH_SECRET=#{res.secret} dbox ..."
     end
 
     def self.connect
@@ -35,7 +40,6 @@ module Dbox
     # IMPORTANT: API.new is private. Please use API.authorize or API.connect as the entry point.
     private_class_method :new
     def initialize
-      @conf = self.class.conf
     end
 
     def initialize_copy(other)
@@ -43,45 +47,57 @@ module Dbox
     end
 
     def connect
+      app_key = ENV["DROPBOX_APP_KEY"]
+      app_secret = ENV["DROPBOX_APP_SECRET"]
       auth_key = ENV["DROPBOX_AUTH_KEY"]
       auth_secret = ENV["DROPBOX_AUTH_SECRET"]
 
+      raise(ConfigurationError, "Please set the DROPBOX_APP_KEY environment variable to a Dropbox application key") unless app_key
+      raise(ConfigurationError, "Please set the DROPBOX_APP_SECRET environment variable to a Dropbox application secret") unless app_secret
       raise(ConfigurationError, "Please set the DROPBOX_AUTH_KEY environment variable to an authenticated Dropbox session key") unless auth_key
       raise(ConfigurationError, "Please set the DROPBOX_AUTH_SECRET environment variable to an authenticated Dropbox session secret") unless auth_secret
 
-      @auth = Authenticator.new(@conf, auth_key, auth_secret)
-      @client = DropboxClient.new(@conf["server"], @conf["content_server"], @conf["port"], @auth)
+      @session = DropboxSession.new(app_key, app_secret)
+      @session.set_access_token(auth_key, auth_secret)
+      @client = DropboxClient.new(@session, 'dropbox')
     end
 
     def run(path)
       begin
         res = yield
-        case res
-        when Hash
-          HashWithIndifferentAccess.new(res)
-        when String
-          res
-        when Net::HTTPNotFound
-          raise RemoteMissing, "#{path} does not exist on Dropbox"
-        when Net::HTTPForbidden
-          raise RequestDenied, "Operation on #{path} denied"
-        when Net::HTTPNotModified
-          :not_modified
-        when true
-          true
-        else
-          raise RuntimeError, "Unexpected result: #{res.inspect}"
-        end
+        handle_response(path, res) { raise RuntimeError, "Unexpected result: #{res.inspect}" }
+      rescue DropboxNotModified => e
+        :not_modified
+      rescue DropboxAuthError => e
+        raise e
       rescue DropboxError => e
-        log.debug e.inspect
-        raise ServerError, "Server error -- might be a hiccup, please try your request again (#{e.message})"
+        handle_response(path, e.http_response) { raise ServerError, "Server error -- might be a hiccup, please try your request again (#{e.message})" }
+      end
+    end
+
+    def handle_response(path, res, &else_proc)
+      case res
+      when Hash
+        HashWithIndifferentAccess.new(res)
+      when String
+        res
+      when Net::HTTPNotFound
+        raise RemoteMissing, "#{path} does not exist on Dropbox"
+      when Net::HTTPForbidden
+        raise RequestDenied, "Operation on #{path} denied"
+      when Net::HTTPNotModified
+        :not_modified
+      when true
+        true
+      else
+        else_proc.call()
       end
     end
 
-    def metadata(path = "/", hash = nil)
+    def metadata(path = "/", hash = nil, list=true)
       log.debug "Fetching metadata for #{path}"
       run(path) do
-        res = @client.metadata(@conf["root"], escape_path(path), 10000, hash)
+        res = @client.metadata(path, 10000, list, hash)
         log.debug res.inspect
         res
       end
@@ -90,11 +106,14 @@ module Dbox
     def create_dir(path)
       log.info "Creating #{path}"
       run(path) do
-        case res = @client.file_create_folder(@conf["root"], path)
-        when Net::HTTPForbidden
-          raise RemoteAlreadyExists, "Either the directory at #{path} already exists, or it has invalid characters in the name"
-        else
-          res
+        begin
+          @client.file_create_folder(path)
+        rescue DropboxError => e
+          if e.http_response.kind_of?(Net::HTTPForbidden)
+            raise RemoteAlreadyExists, "Either the directory at #{path} already exists, or it has invalid characters in the name"
+          else
+            raise e
+          end
         end
       end
     end
@@ -102,67 +121,112 @@ module Dbox
     def delete_dir(path)
       log.info "Deleting #{path}"
       run(path) do
-        @client.file_delete(@conf["root"], path)
+        @client.file_delete(path)
       end
     end
 
-    def get_file(path, output_file_obj)
+    def get_file(path, file_obj, stream=false)
       log.info "Downloading #{path}"
-      run(path) do
-        @client.get_file(@conf["root"], escape_path(path), output_file_obj)
+      unless stream
+        # just download directly using the get_file API
+        res = run(path) { @client.get_file(path) }
+        if res.kind_of?(String)
+          file_obj << res
+          true
+        else
+          raise DropboxError.new("Invalid response #{res.inspect}")
+        end
+      else
+        # use the media API to get a URL that we can stream from, and
+        # then stream the file to disk
+        res = run(path) { @client.media(path) }
+        url = res[:url] if res && res.kind_of?(Hash)
+        if url
+          streaming_download(url, file_obj)
+        else
+          get_file(path, file_obj, false)
+        end
       end
     end
 
-    def put_file(path, file_obj)
+    def put_file(path, file_obj, previous_revision=nil)
       log.info "Uploading #{path}"
+
+      # temporary workaround for bug in Dropbox /put_files API where
+      # passing a valid older parent_rev causes a 500
+      if previous_revision
+        m = metadata(path)
+        if m && m[:rev] && m[:rev] != previous_revision
+          previous_revision = nil
+        end
+      end
+
       run(path) do
-        dir = File.dirname(path)
-        name = File.basename(path)
-        @client.put_file(@conf["root"], escape_path(dir), name, file_obj)
+        @client.put_file(path, file_obj, false, previous_revision)
       end
     end
 
     def delete_file(path)
       log.info "Deleting #{path}"
       run(path) do
-        @client.file_delete(@conf["root"], path)
+        @client.file_delete(path)
       end
     end
 
     def move(old_path, new_path)
       log.info "Moving #{old_path} to #{new_path}"
       run(old_path) do
-        case res = @client.file_move(@conf["root"], old_path, new_path)
-        when Net::HTTPBadRequest
-          raise RemoteAlreadyExists, "Error during move -- there may already be a Dropbox folder at #{new_path}"
-        else
-          res
+        begin
+          @client.file_move(old_path, new_path)
+        rescue DropboxError => e
+          if e.http_response.kind_of?(Net::HTTPForbidden)
+            raise RemoteAlreadyExists, "Error during move -- there may already be a Dropbox folder at #{new_path}"
+          else
+            raise e
+          end
         end
       end
     end
 
-    def escape_path(path)
-      path.split("/").map {|s| CGI.escape(s).gsub("+", "%20") }.join("/")
+    def streaming_download(url, io)
+      url = URI.parse(url)
+      http = Net::HTTP.new(url.host, url.port)
+      http.use_ssl = true
+
+      req = Net::HTTP::Get.new(url.request_uri)
+      req["User-Agent"] = "OfficialDropboxRubySDK/#{Dropbox::SDK_VERSION}"
+
+      http.request(req) do |res|
+        if res.kind_of?(Net::HTTPSuccess)
+          # stream into given io
+          res.read_body {|chunk| io.write(chunk) }
+          true
+        else
+          raise DropboxError.new("Invalid response #{res}\n#{res.body}")
+        end
+      end
     end
+  end
+end
 
-    def self.conf
-      app_key = ENV["DROPBOX_APP_KEY"]
-      app_secret = ENV["DROPBOX_APP_SECRET"]
+# monkey-patch DropboxSession to add SSL certificate checking, since the
+# Dropbox Ruby SDK doesn't do it and doesn't have a place to hook into.
+class DropboxSession
+  private
+  def do_http(uri, auth_token, request) # :nodoc:
+    http = Net::HTTP.new(uri.host, uri.port)
+    http.use_ssl = true
 
-      raise(ConfigurationError, "Please set the DROPBOX_APP_KEY environment variable to a Dropbox application key") unless app_key
-      raise(ConfigurationError, "Please set the DROPBOX_APP_SECRET environment variable to a Dropbox application secret") unless app_secret
+    # IMPORTANT: other than these two extra lines, this should be
+    # identical to the definition in dropbox_sdk.rb
+    http.verify_mode = OpenSSL::SSL::VERIFY_PEER
+    http.ca_file = File.join(File.dirname(__FILE__), "cacert.pem")
 
-      {
-        "server"            => "api.dropbox.com",
-        "content_server"    => "api-content.dropbox.com",
-        "port"              => 80,
-        "request_token_url" => "http://api.dropbox.com/0/oauth/request_token",
-        "access_token_url"  => "http://api.dropbox.com/0/oauth/access_token",
-        "authorization_url" => "http://www.dropbox.com/0/oauth/authorize",
-        "root"              => "dropbox",
-        "consumer_key"      => app_key,
-        "consumer_secret"   => app_secret
-      }
-    end
+    request.add_field('Authorization', build_auth_header(auth_token))
+
+    #We use this to better understand how developers are using our SDKs.
+    request['User-Agent'] =  "OfficialDropboxRubySDK/#{Dropbox::SDK_VERSION}"
+
+    http.request(request)
   end
 end