worochi 0.0.14 → 0.0.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a16bf51d367b8b62f009a425dd929ed96d38f376
-  data.tar.gz: 9e24cbc3ebee89c808a6f1c9e5622ab78b67680f
+  metadata.gz: 791d8d31548bebe42edf49ba96f3e6a64dea0d14
+  data.tar.gz: 207531dc87e9d1158668c259fde956728fdc4195
 SHA512:
-  metadata.gz: 2cbfcb2ba4fc6d306bedfde432eae0f8cc66586a84f7a06b3b156235cf8762ada3da0ef09ee56d9dea665755b10ccea8d572bda49f76e5f8b336186b9709a8e3
-  data.tar.gz: d28e49ee2f24e2eae8d982d3706407c4f76019d51f80f118c771e8833829d0bfa7e1dd2cab9effa292bd07563e4180e0bc57c92d006bfb2c230f2bf0241c4911
+  metadata.gz: b8a5a0a98112ac258077771b0a9cb8298a3c030893633a555c3ff60780470091f3321e3f4c50887b4b8dcd579e9288b4bbc6cd315c0dc621952d11bb07219b1b
+  data.tar.gz: d0596891d2b02b451facb7d6e243b98c3dd0f713b8aca53515a2b29ee0b93af73ed9cc81ac9ee190cdfca8d5ac102fab76edc67d97e7bc3dbe12425b42e9fac6

data/README.md

@@ -1,6 +1,6 @@
 # Worochi
 
-![Coverage Status](https://coveralls.io/repos/Pixelapse/worochi/badge.png?branch=master)
+[![Coverage Status](https://coveralls.io/repos/Pixelapse/worochi/badge.png?branch=master)](https://coveralls.io/r/Pixelapse/worochi?branch=master)
 
 Worochi provides a standard way to interface with Ruby API wrappers provided
 by various cloud storage services such as Dropbox and Google Drive.
@@ -20,20 +20,20 @@ Worochi can be installed as a gem.
 ## Basic Usage
 
 Pushing files is easy. Just create an agent using the OAuth authorization
-token for the user and then call {Worochi::Agent#push} or {Worochi.push}. File
+token for the user and then call `Worochi::Agent#push` or `Worochi.push`. File
 origins can be local, HTTP, or Amazon S3 paths.
 
-Pushing files to Dropbox:
+### Pushing files to Dropbox
 
 ```ruby
 token = '982n3b989az'
 agent = Worochi.create(:dropbox, token)
 agent.push('test.txt')
-agent.files
+agent.files # lists the files in the default directory
 # => ['test.txt']
 ```
 
-Pushing multiple files:
+### Pushing multiple files
 
 ```ruby
 agent.push(['a.txt', 'folder1/b.txt', 'http://example.com/c.txt'])
@@ -41,7 +41,7 @@ agent.files
 # => ['a.txt', 'b.txt', 'c.txt']
 ```
 
-Pushing files to more than one agent at the same time:
+###  Pushing to more than one service
 
 ```ruby
 a = Worochi.create(:dropbox, 'hxhrerx')
@@ -53,26 +53,34 @@ b.files
 # => ['test.txt']
 ```
 
-Pushing to a specific folder:
+### Specifying remote paths
 
-```ruby
-agent = Worochi.create(:dropbox, token, { dir: '/folder1' })
-agent.push('a.txt')
+Instead of pushing all the files to the default directory at `/`, you can
+specify the default path and also the path of every file individually.
 
-agent.files
-# => ['a.txt']
+```ruby
+agent = Worochi.create(:dropbox, token, { dir: '/parent' }) # default path
+agent.push([
+  { source: 'a.txt', path: 'A.txt' },
+  { source: 'b.txt', path: 'folder1/B.txt' },
+  { source: 'c.txt', path: '/C.txt' } # absolute remote path
+])
 
-agent.set_dir('/')
-agent.push('b.txt')
 agent.files
-# => ['b.txt']
+# => ['A.txt']
 
 agent.files_and_folders
-# => ['folder1', 'b.txt']
-agent.files('/folder1')
-# => ['a.txt']
-```
+# => ['A.txt', 'folder1']
+
+agent.files('/parent') # same as default directory
+# => ['A.txt']
 
+agent.files('/parent/folder1')
+# => ['B.txt']
+
+agent.files('/') # root
+# => ['C.txt']
+```
 ## Amazon S3 Support
 
 Files can be retrieved directly from their Amazon S3 location either using the
@@ -98,7 +106,7 @@ this to work.
 
 Worochi provides helper methods to assist with the OAuth2 authorization flow.
 
-Example Rails controller:
+### Example in Rails
 
 ```ruby
 class ApiTokensController < ApplicationController
@@ -130,9 +138,57 @@ Service-specific settings for OAuth2 are predefined in the gem, so the
 framework just needs to handle verification of session state (this is usually
 optional) and storing the retrieved access token value.
 
+### Refresh Token
+
+Retrieved tokens can be refreshed if `refresh_token` is supported by the
+service.
+
+```ruby
+token = oauth.flow_end(code)
+
+new_token = oauth.refresh(token)
+```
+
+Tokens are hashes and `refresh` expects a hash containing the field
+`refresh_token`. It raises an error if `refresh_token` is invalid.
+
+## Supported Services
+
+Currently these services are fully supported:
+
+**Google Drive**
+  - Service name `:google_drive`
+  - Env variables `GOOGLE_ID` `GOOGLE_SECRET` `GOOGLE_TEST_TOKEN`
+
+**Dropbox**
+  - Service name `:dropbox`
+  - Env variables `DROPBOX_ID` `DROPBOX_SECRET` `DROPBOX_TEST_TOKEN`
+
+**GitHub**
+  - Service name `:github`
+  - Env variables `GITHUB_ID` `GITHUB_SECRET` `GITHUB_TEST_TOKEN`
+
+### Environmental Variables
+
+`ID` and `SECRET` variables are only needed for retrieving access tokens and
+can be omitted if you are using other OAuth2 libraries for that purpose.
+
+`TEST_TOKEN` is a valid user access token used for RSpec testing. The user
+account being used for testing should contain these [test files]
+(https://github.com/darkmirage/test) at the directory specified by the tests.
+
+## MIME Types
+
+Some services such as Google Drive require Worochi to provide MIME types
+for the files being uploaded. Worochi will attempt to use the file name
+to determine the MIME type, but this does not work well. You can use
+`ruby-filemagic` for better MIME type detection using magic numbers.
+
+    gem install ruby-filemagic
+
 ## Development
 
-Each service is implemented as an {Worochi::Agent} object. Below is an
+Each service is implemented as an `Worochi::Agent` object. Below is an
 overview of the files necessary for defining an agent to support a new
 service.
 
@@ -152,10 +208,12 @@ Test file:
 Use underscore for filenames and corresponding mixed case for class name. The
 class name and service name symbol for the above example would be:
 
-    class Worochi::Agent::FooBar < Worochi::Agent
-    end
+```ruby
+class Worochi::Agent::FooBar < Worochi::Agent
+end
 
-    Worochi.create(:foo_bar, token)
+Worochi.create(:foo_bar, token)
+```
 
 RSpec tests use the [VCR](https://github.com/vcr/vcr) gem to record and
 playback real HTTP interactions. Remember to filter out API tokens in the

data/lib/worochi/agent/#example.rb

@@ -52,6 +52,12 @@ class Worochi
       end
     end
 
+    # Deletes the file at path. Raises {Error} if file deletion is not
+    # supported by the service.
+    def delete(path)
+      raise Error, 'Deletion is not supported'
+    end
+
     # Service specific methods
   end
 end

data/lib/worochi/agent/dropbox.rb

@@ -23,7 +23,8 @@ class Worochi
       if chunked = item.size > options[:chunk_size]
         push_item_chunked(item)
       else
-        @client.put_file(full_path(item), item.content, options[:overwrite])
+        path = full_path(item.path)
+        @client.put_file(path, item.content, options[:overwrite])
       end
       Worochi::Log.debug "Uploaded"
       chunked
@@ -35,11 +36,7 @@ class Worochi
     # @param path [String] path to list instead of the current directory
     # @return [Array<Hash>] list of files and subdirectories
     def list(path=nil)
-      if path
-        remote_path = path[0] == '/' ? path : File.join(options[:dir], path)
-      else
-        remote_path = options[:dir]
-      end
+      remote_path = list_path(path)
       
       begin
         response = @client.metadata(remote_path)
@@ -74,7 +71,7 @@ class Worochi
           end
           Worochi::Log.debug "Uploaded #{uploader.offset} bytes"
       end
-      uploader.finish(full_path(item), options[:overwrite])
+      uploader.finish(full_path(item.path), options[:overwrite])
       nil
     end
 
@@ -83,8 +80,9 @@ class Worochi
     # @param path [String] path relative to current directory
     # @return [Boolean] `true` if a file was actually deleted
     def delete(path)
+      abs_path = full_path(path)
       begin
-        @client.file_delete(File.join(options[:dir], path))
+        @client.file_delete(abs_path)
       rescue DropboxError
         false
       end

data/lib/worochi/agent/github.rb

@@ -46,7 +46,7 @@ class Worochi
     # @param sha [String] list a different branch than the `:source`
     # @return [Array<Hash>] list of files and subdirectories
     def list(path=nil, sha=nil)
-      remote_path = (path || options[:dir]).sub(/^\//, '').sub(/\/$/, '')
+      remote_path = list_path(path).sub(/^\//, '').sub(/\/$/, '')
 
       result = @client.tree(repo, sha || source_branch, recursive: true).tree
       result.sort! do |x, y|
@@ -89,6 +89,11 @@ class Worochi
       repos.map { |repo| repo[:full_name] } unless opts[:details]
     end
 
+    # Deletion is not supported by GitHub, so raises {Error}.
+    def delete(path)
+      raise Error, 'Cannot delete from GitHub'
+    end
+
   private
     # Appends an item to the existing tree.
     #
@@ -97,7 +102,7 @@ class Worochi
     # @return [String] SHA1 checksum of the resulting tree
     def tree_append(tree_sha, item)
       child = {
-        path: full_path(item).gsub(/^\//, ''),
+        path: full_path(item.path).gsub(/^\//, ''),
         sha: push_blob(item),
         type: 'blob',
         mode: '100644'

data/lib/worochi/agent/google_drive.rb

@@ -0,0 +1,272 @@
+require 'google/api_client'
+require 'awesome_print'
+
+class Worochi
+  # The {Agent} for Google Drive API.
+  # @see https://github.com/google/google-api-ruby-client
+  class Agent::GoogleDrive < Agent
+
+    # Initializes the Google Drive API client.
+    #
+    # @return [ApiClient]
+    # @see Agent#initialize
+    def init_client
+      clear_cache
+      disable_write
+      @client = Google::APIClient.new(
+        application_name: 'Worochi',
+        application_version: Worochi::VERSION)
+      @client.authorization.access_token = options[:token]
+      @drive = @client.discovered_api('drive', 'v2')
+      @client
+    end
+
+    # Pushes an individual item to Google Drive.
+    #
+    # @return [String] resource ID of the uploaded file.
+    def push_item(item)
+      enable_write
+      abs_path = full_path(item.path)
+      parent_id = get_parent_id(abs_path)
+      id = insert_file(item, parent_id)
+      disable_write
+      id
+    end
+
+    # Returns a list of files and subdirectories at the remote path specified
+    # by `options[:dir]`.
+    #
+    # @param path [String] path to list instead of the current directory
+    # @return [Array<Hash>] list of files and subdirectories
+    def list(path=nil)
+      remote_path = list_path(path)
+      result, folder_id = navigate_to(remote_path)
+      result.map do |elem|
+        if elem.mimeType == 'application/vnd.google-apps.folder'
+          file_type = 'folder'
+        else
+          file_type = 'file'
+        end
+        {
+          name: elem.title,
+          path: File.join(remote_path, elem.title),
+          type: file_type,
+          id: elem.id,
+          parent_id: folder_id
+        }
+      end
+    end
+
+    # Deletes the file at `path` from Google Drive.
+    #
+    # @param path [String] path relative to current directory
+    # @return [Boolean] `true` if a file was actually deleted
+    def delete(path)
+      abs_path = full_path(path)
+      item_id = get_item_id(abs_path)
+      return false if item_id.nil?
+      delete_by_id(item_id)
+    end
+
+    # Clears the internal resource ID cache
+    def clear_cache
+      @id_cache = {}
+    end
+
+  private
+    # Saves a resource ID in the cache.
+    #
+    # @param abs_path [String] absolute path to the file/folder
+    # @param id [String] resource ID of the file/folder
+    # @return [nil]
+    def set_cache(abs_path, id)
+      abs_path ='/' if abs_path == ''
+      @id_cache[abs_path] = { id: id, time: Time.now }
+      nil
+    end
+
+    # Retrieves a saved cache entry.
+    #
+    # @param abs_path [String] absolute path to the file/folder
+    # @return [String] resource ID of the file/folder
+    def get_cached(abs_path)
+      return nil unless @id_cache.include?(abs_path)
+      @id_cache[abs_path][:id]
+    end
+
+    # Recursive directory lookup automatically creates missing directories.
+    def enable_write
+      @write = true
+    end
+
+    # Disable auto-creation of missing directories.
+    def disable_write
+      @write = false
+    end
+
+    # @return [Boolean] `true` if auto-creation is enabled.
+    def write?
+      @write
+    end
+
+    # @param id [String] resource ID of file/folder to be deleted
+    # @return [Boolean] successfully deleted
+    def delete_by_id(id)
+      response = @client.execute(
+        api_method: @drive.files.delete,
+        parameters: { 'fileId' => id })
+      response.status == 204
+    end
+
+    # @param abs_path [String] absolute path of the child
+    # @return [String] parent resource ID
+    def get_parent_id(abs_path)
+      parent_path = File.dirname(abs_path)
+      get_item_id(parent_path)
+    end
+
+    # @param abs_path [String] absolute path to the item
+    # @return [String] Google Drive source ID for the item
+    def get_item_id(abs_path)
+      return get_cached(abs_path) if get_cached(abs_path)
+
+      file_name = File.basename(abs_path)
+      folder_path = File.dirname(abs_path)
+      return 'root' if ['/', '.', ''].include?(file_name)
+
+      parent_items, parent_id = navigate_to(folder_path)
+      item = find_item_by_name(parent_items, file_name)
+      if item
+        item_id = item.id
+      else
+        item_id = write? ? insert_file(file_name, parent_id) : nil
+      end
+
+      set_cache(abs_path, item_id)
+      item_id
+    end
+
+    # Uploads files or creates new directories.
+    #
+    # @overload insert_file(title, parent_id)
+    #   @param title [String] new directory name
+    #   @param parent_id [String] parent resource ID
+    # @overload insert_file(item, parent_id)
+    #   @param item [Worochi::Item] file to be uploaded
+    #   @param parent_id [String] parent resource ID
+    # @return [String] resource ID of the created file
+    def insert_file(arg, parent_id)
+      if arg.respond_to?(:filename)
+        item = arg
+        title = arg.filename
+      else
+        item = false
+        title = arg
+      end
+
+      body = { 'title' => title, 'parents' => [{ 'id' => parent_id }]}
+      opts = { api_method: @drive.files.insert }
+
+      if item
+        opts[:parameters] = { 'uploadType' => 'resumable' }
+        opts[:media] = Google::APIClient::UploadIO.new(item.content,
+          item.content_type, item.content.path)
+      else
+        body['mimeType'] = 'application/vnd.google-apps.folder'
+      end
+
+      opts[:body_object] = @drive.files.insert.request_schema.new(body)
+
+      begin
+        response = @client.execute(opts)
+      rescue NoMethodError
+        # Google API client does not fail correctly given an invalid token
+        # Need to catch and handle it here
+        raise Error, 'Google Error: Invalid credentials'
+      end
+
+      # if item
+      #   retries = 0
+      #   while !response.resumable_upload.complete? && retries < 10
+      #     retries += 1
+      #     sleep(rand(6))
+      #     ap "retry #{retries}"
+      #     response = @client.execute(response.resumable_upload)
+      #   end
+      # end
+
+      response.data ? response.data.id : nil
+    end
+
+    # List the files and folders at the target absolute path.
+    #
+    # @param abs_path [String] absolute path
+    # @return [Array<Google::APIClient::Schema::Drive::V2::File>] children
+    def navigate_to(abs_path)
+      folders = abs_path.split('/').reject(&:empty?)
+      get_folder_items(folders)
+    end
+
+    # Recursively navigate to the target folder.
+    #
+    # @param folders [Array<String>] list of folder names
+    # @param parent_id [String] parent resource ID
+    # @param curr_path [String] absolute path to current parent
+    # @return [Array<Google::APIClient::Schema::Drive::V2::File>, String]
+    #   children list and resource ID of parent
+    def get_folder_items(folders, parent_id='root', curr_path='')
+      set_cache(curr_path, parent_id)
+      items = retrieve_items(parent_id)
+      return items, parent_id if folders.empty?
+
+      name = folders.shift
+      item = find_item_by_name(items, name)
+      if item
+        item_id = item.id
+      else
+        raise Error, 'Invalid path specified' unless write?
+        item_id = insert_file(name, parent_id)
+      end
+
+      get_folder_items(folders, item_id, "#{curr_path}/#{name}")
+    end
+
+    # Returns a single item from a list of items by matching its file name.
+    #
+    # @param items [Array<Google::APIClient::Schema::Drive::V2::File>] items
+    # @param name [String] file name
+    # @return [Google::APIClient::Schema::Drive::V2::File]
+    def find_item_by_name(items, name)
+      found = items.select { |elem| elem.title == name }
+      raise Error, "Ambiguous file name #{name}" if found.size > 1
+      found.first
+    end
+
+    # Retrieves all the children of the specified directory.
+    #
+    # @param id [String] Google Drive resource ID of the directory
+    # @return [Array<Google::APIClient::Schema::Drive::V2::File>] children
+    def retrieve_items(id)
+      fields = 'items(id,mimeType,title),nextPageToken'
+      q = "'#{id}' in parents and trashed = false"
+      page_token = nil
+      result = []
+      begin
+        params = { 'q' => q, 'fields' => fields }
+        params['pageToken'] = page_token unless page_token.to_s == ''
+        response = @client.execute(
+          api_method: @drive.files.list,
+          parameters: params)
+        if response.status == 200
+          elems = response.data
+          result.concat(elems.items)
+          page_token = elems.next_page_token
+        else
+          error_msg = response.data['error']['message']
+          raise Worochi::Error, "Google Error: #{error_msg}"
+        end
+      end while page_token.to_s != ''
+      result
+    end
+  end
+end

data/lib/worochi/agent.rb

@@ -164,9 +164,30 @@ class Worochi
       result
     end
 
+    # Returns the full remote target path for the file being pushed. If the
+    # specified path starts with / then this is assumed to be the full
+    # absolute path. If not, join the path with the configured directory.
+    #
+    # @param path [String] path to the file
     # @return [String] full path combining remote directory and item path
-    def full_path(item)
-      File.join(options.dir, item.path)
+    def full_path(path)
+      if path[0] == '/'
+        path
+      else
+        File.join(options.dir, path)
+      end
+    end
+
+    # Path used for listing. Defaults to `options[:dir]` if not specified.
+    #
+    # @param path [String] relative/absolute path or nil
+    # @return [String] absolute path to list
+    def list_path(path)
+      if path.nil? || path.empty?
+        options.dir
+      else
+        return full_path(path)
+      end
     end
 
     # Agents should either override this or have a YAML config file at

data/lib/worochi/config/google_drive.yml

@@ -5,7 +5,9 @@ dir: /
 oauth:
   id_env: GOOGLE_ID
   secret_env: GOOGLE_SECRET
-  token_url: /1/oauth2/token
+  token_url: /o/oauth2/token
   authorize_url: /o/oauth2/auth
   site: https://accounts.google.com
   scope: https://www.googleapis.com/auth/drive
+  access_type: offline
+  approval_prompt: force

data/lib/worochi/item.rb

@@ -1,4 +1,11 @@
 require 'tempfile'
+require 'mime/types'
+
+# Install ruby-filemagic if you want better mime-type identification
+begin
+  require 'filemagic'
+rescue LoadError
+end
 
 class Worochi
   # This represents a single file that is being pushed. The {#content}
@@ -21,9 +28,15 @@ class Worochi
     def initialize(path, content)
       @path = path
       @content = content
+      detect_type
       @content.rewind
     end
 
+    # @return [String] the filename
+    def filename
+      File.basename(path)
+    end
+
     # The total size of the content in bytes.
     #
     # @return [Integer]
@@ -47,6 +60,35 @@ class Worochi
       content.rewind
     end
 
+    # @return [String] mime-type of the content.
+    def content_type
+      @type
+    end
+
+  private
+    # Detects the mime-type of the file. Uses file name matching if
+    # ruby-filemagic is not installed.
+    #
+    # @param simplified [Boolean] whether to use simplified mime-types
+    # @return [String] mime-type
+    def detect_type(simplified=true)
+      if defined?(FileMagic)
+        # Magic number matching
+        fm = FileMagic.mime
+        fm.simplified = true if simplified
+        @type = fm.file(@content.path)
+      else
+        # File name matching
+        types = MIME::Types.type_for(path)
+        if types.empty?
+          @type = 'application/octet-stream'
+        else
+          @type = simplified ? types[0].content_type : types[0].simplified
+        end
+      end
+      @type
+    end
+
     class << self
       # Takes in either a single file entry or a list of file entries and
       # parses them into a list of {Item} objects. Each entry can be either a
@@ -93,7 +135,6 @@ class Worochi
       end
 
     private
-
       # Retrieves the file content from `source`.
       #
       # @param source [String] local or remote location of the file content