defog 0.1.1 → 0.2.0

data/README.rdoc

@@ -1,7 +1,7 @@
 = defog
 
-Defog wraps the [fog](https://rubygems.org/gems/fog) gem (specifically,
-[Fog::Storage](http://fog.io/1.3.1/storage/)), providing access to files
+Defog wraps the fog[https://rubygems.org/gems/fog] gem (specifically,
+{Fog::Storage}[http://fog.io/1.3.1/storage/]), providing access to files
 stored in the cloud via proxy files on the local file system.
 A proxy file can be
 * Read-only:  A local cached copy of a cloud file.
@@ -10,7 +10,7 @@ A proxy file can be
 
 Defog thus lets you use ordinary programmatic tools to access and
 manipulate your cloud data.  Thanks to the magic of
-[fog](https://rubygems.org/gems/fog) it works across cloud providers, and
+fog[https://rubygems.org/gems/fog] it works across cloud providers, and
 it also works with the local file system as a "provider" so that you can,
 e.g. use the local file system for development and the cloud for
 production.
@@ -100,38 +100,83 @@ In addition, the handle allows you to look up the path where the local proxy fil
 
 === Persistence
 
-By default, the local proxy files are deleted when closed.  However, it is
-possible to persist the local proxy, so that it if the remote is accessed
-again-- whether in the same program execution or at a later time or by a
-different program--the data will not need to be transferred again.  Do so
-via
+By default, Defog will delete the local proxy when closing a file.
+However, it is possible to keep the local proxy file so that it if the
+remote is accessed again the data will not need to be transferred again.
+(This is true even between executions of the program: a Defog::Proxy
+instance can start with proxy files already in place, and it will use them.)
+
+Persistence can be enabled by default for the Defog::Proxy instance via:
+
+        defog = Defog::Proxy.new(:provider => ..., :persist => true)
+
+And/or persistence can be overridden on a per-file basis at proxy open time:
 
         file = defog.file("key/of/file", mode, :persist => true)
 
-or
+or at proxy close time:
 
         file.close(:persist => true)
 
 When opening a file whose local proxy has been persisted, Defog checks to see if
-the local proxy is out of date and if so replaces it.
+the local proxy is out of date and if so replaces it (via MD5 digests).
+
+== Local proxy file cache
+
+For basic usage, you don't need to worry about the cache, the default
+settings work fine.  But if you will be persisting proxy files you may want to
+manage the cache more carefully.
+
+=== Cache location
+
+The cache for a given Defog::Proxy is rooted at a directory on the local
+file system.  You can set and query the root via
+
+    defog = Defog::Proxy.new(:provider => ..., :proxy_root => "/my/chosen/root")
+    defog.proxy_root    # => returns a Pathname
 
-Currently, Defog does not natively support a way to explictly delete the
-locally persisted file (other than opening and closing it again without
-:persist => true).  But it's fair game to delete it outside of Defog, such
-as via a cron job that cleans out old files.
+If you don't specify a root, Defog uses one of two defaults:
 
-=== Local Proxy File Location
+    {Rails.root}/tmp/defog/{provider}-{location}  # if Rails is defined
+    {Dir.tmpdir}/defog/{provider}-{location}      # if Rails is not defined
 
-Local proxy files are stored by default in
+In these, <code>location</code> disambiguates between Defog::Proxy instances.
+For :AWS it's the bucket name and for :local it's the
+<code>local_root</code> directory path with slashes replaced with dashes.
 
-        #{tmproot}/defog/#{provider}/#{location}/#{key}
+[Why cache local files, you ask?  Why not bypass this whole cache thing if
+using :local?  Well, the motivation for supporting :local is to use it in
+development and use :AWS in production.  So, to more faithfully mimic
+production behavior, :local mode goes through the same code path and same
+caching mechanism.]
 
-where <code>tmproot</code> is <code>Rails.root+"tmp"</code> if Rails is
-defined, otherwise <code>Dir.tmpdir()</code>.  For :AWS,
-<code>location</code> is the bucket name, and for :local it's the
-<code>local_root</code> directory path with slashes replaced with underscores.
+Within the cache, indvidiual proxy files are located by treating the key as
+a path relative to the proxy root (with slashes in the key indicating
+subdirectories in the path).
 
-See the documentation for configuring other locations.
+=== Cache size management
+
+Defog can perform simple size management of the local proxy file cache.  This is
+of course useful mostly when persisting files.
+
+You can specify a maximum cache size via:
+
+        defog = Defog::Proxy.new(:provider => ..., :max_cache_size => size-in-bytes)
+
+If a maximum size is set, then before downloading data to create a proxy,
+Defog will check the space available and delete persisted proxy files as needed
+in LRU order.  Does not delete files for proxies that are currently open.
+If this would not free up enough space (because of open proxies or just
+because the remote is larger than the cache), raises
+Defog::Error::CacheFull and doesn't actually delete anything.
+
+You can also manually delete an individual persisted file, such as via:
+
+        defog.file("key").proxy_path.unlink
+
+And it's fair game to delete proxy files outside of Defog, such as via a
+cron job.  Of course in these cases it's up to you to make sure not to
+unintentionally delete a proxy file that's currently open.
 
 == Installation
 
@@ -141,9 +186,8 @@ Gemfile:
 == Compatibility
 
 Defog has (so far) been tested on MRI 1.9.3 using
-[fog](https://rubygems.org/gems/fog) storage providers :local and :AWS
+fog[https://rubygems.org/gems/fog] storage providers :local and :AWS
 
 == Copyright
 
 Released under the MIT License.  See LICENSE for details.
-

data/lib/defog/error.rb

@@ -3,6 +3,9 @@ module Defog
 
     class NoCloudFile < Error
     end
+
+    class CacheFull < Error
+    end
   end
 
 end

data/lib/defog/file.rb

@@ -31,35 +31,40 @@ module Defog
   # Upon closing the proxy file, in normal use the cloud storage gets synchronized and
   # the proxy deleted.  See File#close for more details.
   class File < ::File
-    def self.get(opts={}, &block) #:nodoc:
-      opts = opts.keyword_args(:handle => :required, :mode => :required, :persist => :optional)
 
-      handle = opts.handle
-      key = handle.key
-      proxy_path = handle.proxy_path
+    def initialize(opts={}, &block) #:nodoc:
+      opts = opts.keyword_args(:handle => :required, :mode => :required, :persist => :optional)
+      @handle = opts.handle
+      @persist = opts.persist
 
+      key = @handle.key
+      proxy_path = @handle.proxy_path
       proxy_path.dirname.mkpath
-
       case opts.mode
-      when "r" then
-        handle.proxy.fog_wrapper.get_file(key, proxy_path)
-      when "w", "w+" then
-        opts[:upload] = true
-      when "r+", "a", "a+" then
-        handle.proxy.fog_wrapper.get_file(key, proxy_path)
-        opts[:upload] = true
+      when "r"
+        create_proxy
+      when "w", "w+"
+        @upload = true
+      when "r+", "a", "a+"
+        create_proxy
+        @upload = true
       else
         raise ArgumentError, "Invalid mode #{opts.mode.inspect}"
       end
 
-      self.open(opts, &block)
+      super(proxy_path, opts.mode, &block)
     end
 
-    def initialize(opts={}, &block) #:nodoc:
-      @defog = opts.keyword_args(:handle => :required, :mode => :required, :upload => :optional, :persist => :optional)
-      super(@defog.handle.proxy_path, @defog.mode, &block)
+    def create_proxy
+      @handle.proxy.open_proxy_file(@handle)
+      @handle.proxy.fog_wrapper.get_file(@handle.key, @handle.proxy_path)
+    end
+
+    def upload_proxy
+      @handle.proxy.fog_wrapper.put_file(@handle.key, @handle.proxy_path) 
     end
 
+
     # Closes the proxy file and synchronizes the cloud storage (if it was
     # opened as writeable) then deletes the proxy file.
     #
@@ -75,13 +80,13 @@ module Defog
     # (This will override the setting of <code>:persist</code> passed to Proxy#file)
     #
     def close(opts={})
-      opts = opts.keyword_args(:persist => @defog.persist, :synchronize => true)
+      opts = opts.keyword_args(:persist => @persist, :synchronize => true)
       super()
-      handle = @defog.handle
-      if handle.proxy_path.exist?
-        handle.proxy.fog_wrapper.put_file(handle.key, handle.proxy_path) if @defog.upload and opts.synchronize
-        handle.proxy_path.unlink unless opts.persist
+      if @handle.proxy_path.exist?
+        upload_proxy if @upload and opts.synchronize
+        @handle.proxy_path.unlink unless opts.persist
       end
+      @handle.proxy.close_proxy_file(@handle)
     end
   end
 end

data/lib/defog/handle.rb

@@ -93,8 +93,8 @@ module Defog
     # to suppress deleting the file and so maintain the file after closing.   See File#close for more
     # details.
     def open(mode, opts={}, &block)
-      opts = opts.keyword_args(:persist)
-      File.get(opts.merge(:handle => self, :mode => mode), &block)
+      opts = opts.keyword_args(:persist => @proxy.persist)
+      File.open(opts.merge(:handle => self, :mode => mode), &block)
     end
 
   end

data/lib/defog/proxy.rb

@@ -1,17 +1,20 @@
 require "hash_keyword_args"
-require "tmpdir"
 require "pathname"
+require "set"
+require "tmpdir"
 
 module Defog
   class Proxy
 
-    attr_reader :proxy_root     # 
+    attr_reader :proxy_root
+    attr_reader :persist
+    attr_reader :max_cache_size
     attr_reader :fog_wrapper    # :nodoc:
 
     # Opens a <code>Fog</code> cloud storage connection to map to a corresponding proxy
     # directory.  Use via, e.g.,
     #
-    #     Defog::Proxy.new(:provider => :AWS, :aws_access_key_id => access_key, ...)
+    #     defog = Defog::Proxy.new(:provider => :AWS, :aws_access_key_id => access_key, ...)
     #
     # The <code>:provider</code> and its corresponding options must be
     # specified as per <code>Fog::Storage.new</code>.  Currently, only
@@ -31,10 +34,27 @@ module Defog
     # to worry about it.  But if you do care, you can specify the option:
     #   :proxy_root => "/root/for/this/proxy/files"
     #
+    # You can turn on persistence of local proxy files by specifying
+    #   :persist => true
+    # The persistence behavior can be overriden on a per-file basis when
+    # opening a proxy (see Defog::Handle#open)
+    #
+    # You can enable cache management by specifying a max cache size in
+    # bytes, e.g.
+    #    :max_cache_size => 3.gigabytes
+    # See the README for discussion.  [Number#gigabytes is defined in
+    # Rails' ActiveSupport core extensions]
     def initialize(opts={})
-      opts = opts.keyword_args(:provider => :required, :proxy_root => :optional, :OTHERS => :optional)
+      opts = opts.keyword_args(:provider => :required,
+                               :proxy_root => :optional,
+                               :persist => :optional,
+                               :max_cache_size => :optional,
+                               :OTHERS => :optional)
 
       @proxy_root = Pathname.new(opts.delete(:proxy_root)) if opts.proxy_root
+      @persist = opts.delete(:persist)
+      @max_cache_size = opts.delete(:max_cache_size)
+      @open_proxy_paths = Set.new
 
       @fog_wrapper = FogWrapper.connect(opts)
 
@@ -69,12 +89,13 @@ module Defog
       @fog_wrapper.fog_directory
     end
 
-    # Proxy a remote cloud file.  Returns a Defog::Handle object that
+    # Proxy a remote cloud file.  Returns or yields a Defog::Handle object that
     # represents the file. 
     #
-    # If a <code>mode</code> is specified given opens a proxy file via
+    # If a <code>mode</code> is given, opens a proxy file via
     # Defog::Handle#open (passing it the mode and other options and
-    # optional block), returning instead the Defog::File object.
+    # optional block), returning or yielding instead the Defog::File object.
+    #
     #
     # Thus 
     #    proxy.file("key", mode, options, &block)
@@ -90,5 +111,54 @@ module Defog
       end
     end
 
+    def open_proxy_file(handle) #:nodoc:
+      manage_cache(handle) if max_cache_size
+      @open_proxy_paths << handle.proxy_path
+    end
+
+    def close_proxy_file(handle) #:nodoc:
+      @open_proxy_paths.delete handle.proxy_path
+    end
+
+    private
+
+    def manage_cache(handle)
+      remote_size = handle.size
+      proxy_path = handle.proxy_path
+
+      # find available space (not counting current proxy)
+      available = max_cache_size
+      proxy_root.find { |path| available -= path.size if path.file? and path != proxy_path}
+      return if available >= remote_size
+
+      space_needed = remote_size - available
+
+      # find all paths in the cache that aren't currently open (not
+      # counting current proxy)
+      candidates = []
+      proxy_root.find { |path| candidates << path if path.file? and not @open_proxy_paths.include?(path) and path != proxy_path}
+
+      # take candidates in LRU order until that would be enough space
+      would_free = 0
+      candidates = Set.new(candidates.sort_by(&:atime).take_while{|path| (would_free < space_needed).tap{|condition| would_free += path.size}})
+
+      # still not enough...?
+      raise Error::CacheFull, "No room in cache for #{handle.key.inspect}: size=#{remote_size} available=#{available} can_free=#{would_free}, max_cache_size=#{max_cache_size}" if would_free < space_needed
+
+      # LRU order may have taken more than needed, if last file was a big
+      # chunk.  So take another pass, eliminating files that aren't needed.
+      # Do this in reverse size order, since we want to keep big files in
+      # the cache if possible since they're most expensive to replace.
+      candidates.sort_by(&:size).reverse.each do |path|
+        if (would_free - path.size) > space_needed
+          candidates.delete path
+          would_free -= path.size
+        end
+      end
+
+      # free the remaining candidates
+      candidates.each(&:unlink)
+    end
+
   end
 end

data/lib/defog/version.rb

@@ -1,3 +1,3 @@
 module Defog
-  VERSION = "0.1.1"
+  VERSION = "0.2.0"
 end

data/spec/proxy_spec.rb

@@ -2,25 +2,7 @@ require 'spec_helper'
 
 shared_examples "a proxy" do |args|
 
-  it "should default proxy root to tmpdir/defog" do
-    proxy = Defog::Proxy.new(args)
-    proxy.proxy_root.should == Pathname.new(Dir.tmpdir) + "defog" + proxy.provider.to_s + proxy.location
-  end
-
-  it "should default proxy root to Rails.root" do
-    with_rails_defined do
-      proxy = Defog::Proxy.new(args)
-      proxy.proxy_root.should == Rails.root + "tmp/defog" + proxy.provider.to_s + proxy.location
-    end
-  end
-
-  it "should accept proxy root parameter" do
-    path = Pathname.new("/a/random/path")
-    proxy = Defog::Proxy.new(args.merge(:proxy_root => path))
-    proxy.proxy_root.should == path
-  end
-
-  context do
+  context "basic features" do
     before(:each) do
       @proxy = Defog::Proxy.new(args)
     end
@@ -57,6 +39,123 @@ shared_examples "a proxy" do |args|
     end
   end
 
+  context "proxy root location" do
+    it "should default proxy root to tmpdir/defog" do
+      proxy = Defog::Proxy.new(args)
+      proxy.proxy_root.should == Pathname.new(Dir.tmpdir) + "defog" + proxy.provider.to_s + proxy.location
+    end
+
+    it "should default proxy root to Rails.root" do
+      with_rails_defined do
+        proxy = Defog::Proxy.new(args)
+        proxy.proxy_root.should == Rails.root + "tmp/defog" + proxy.provider.to_s + proxy.location
+      end
+    end
+
+    it "should accept proxy root parameter" do
+      path = Pathname.new("/a/random/path")
+      proxy = Defog::Proxy.new(args.merge(:proxy_root => path))
+      proxy.proxy_root.should == path
+    end
+  end
+
+  context "cache management" do
+    before(:each) do
+      @proxy = Defog::Proxy.new(args.merge(:max_cache_size => 100, :persist => true))
+      @proxy.proxy_root.rmtree
+      @proxy.proxy_root.mkpath
+    end
+
+    it "should raise an error trying to proxy a file larger than the cache" do
+      create_remote("x" * 101)
+      expect { @proxy.file(key, "r") }.should raise_error(Defog::Error::CacheFull)
+      proxy_path.should_not be_exist
+    end
+
+    it "should not count existing proxy in total" do
+      create_proxy("y" * 70)
+      create_remote("x" * 70)
+      expect { @proxy.file(key, "r") do end }.should_not raise_error(Defog::Error::CacheFull)
+      proxy_path.should be_exist
+      proxy_path.read.should == remote_body
+    end
+
+    it "should delete proxies to make room" do
+      create_other_proxy("a", 10)
+      create_other_proxy("b", 30)
+      create_other_proxy("c", 40)
+      create_remote("x" * 80)
+      expect { @proxy.file(key, "r") do end }.should_not raise_error(Defog::Error::CacheFull)
+      proxy_path.should be_exist
+      other_proxy_path("a").should be_exist
+      other_proxy_path("b").should_not be_exist
+      other_proxy_path("c").should_not be_exist
+    end
+
+    it "should not delete proxies that are open" do
+      create_other_proxy("a", 20)
+      create_other_proxy("b", 20)
+      create_other_remote("R", 30)
+      create_other_remote("S", 30)
+      create_remote("x" * 30)
+      @proxy.file("R", "r") do
+        @proxy.file("S", "r") do
+          expect { @proxy.file(key, "r") do end }.should_not raise_error(Defog::Error::CacheFull)
+          proxy_path.should be_exist
+          other_proxy_path("a").should_not be_exist
+          other_proxy_path("b").should_not be_exist
+          other_proxy_path("R").should be_exist
+          other_proxy_path("S").should be_exist
+        end
+      end
+    end
+
+    it "should delete proxies that are no longer open" do
+      create_other_remote("R", 60)
+      create_remote("z" * 60)
+      @proxy.file("R", "r") do end
+      other_proxy_path("R").should be_exist
+      expect { @proxy.file(key, "r") do end }.should_not raise_error(Defog::Error::CacheFull)
+      proxy_path.should be_exist
+      other_proxy_path("R").should_not be_exist
+    end
+
+    it "should not delete proxies if there wouldn't be enough space" do
+      create_other_proxy("a", 20)
+      create_other_proxy("b", 20)
+      create_other_remote("r", 30)
+      create_other_remote("s", 30)
+      create_remote("z" * 50)
+      @proxy.file("r", "r") do
+        @proxy.file("s", "r") do
+          expect { @proxy.file(key, "r") do end }.should raise_error(Defog::Error::CacheFull)
+          proxy_path.should_not be_exist
+          other_proxy_path("a").should be_exist
+          other_proxy_path("b").should be_exist
+          other_proxy_path("r").should be_exist
+          other_proxy_path("s").should be_exist
+        end
+      end
+    end
+
+    private
+
+    def create_other_proxy(otherkey, size)
+      other_proxy_path(otherkey).open("w") do |f|
+        f.write("x" * size)
+      end
+    end
+
+    def other_proxy_path(otherkey)
+      @proxy.file(otherkey).proxy_path
+    end
+
+    def create_other_remote(otherkey, size)
+      @proxy.fog_directory.files.create(:key => otherkey, :body => "x" * size)
+    end
+
+  end
+
 end
 
 describe Defog::Proxy do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: defog
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -13,7 +13,7 @@ date: 2012-04-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog
-  requirement: &70159750978980 !ruby/object:Gem::Requirement
+  requirement: &70234406646360 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70159750978980
+  version_requirements: *70234406646360
 - !ruby/object:Gem::Dependency
   name: hash_keyword_args
-  requirement: &70159750978560 !ruby/object:Gem::Requirement
+  requirement: &70234406645920 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70159750978560
+  version_requirements: *70234406645920
 - !ruby/object:Gem::Dependency
   name: fastandand
-  requirement: &70159750978140 !ruby/object:Gem::Requirement
+  requirement: &70234406645400 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70159750978140
+  version_requirements: *70234406645400
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70159750977700 !ruby/object:Gem::Requirement
+  requirement: &70234406644900 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70159750977700
+  version_requirements: *70234406644900
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70159750977280 !ruby/object:Gem::Requirement
+  requirement: &70234406644280 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70159750977280
+  version_requirements: *70234406644280
 - !ruby/object:Gem::Dependency
   name: simplecov
-  requirement: &70159750976820 !ruby/object:Gem::Requirement
+  requirement: &70234406643420 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70159750976820
+  version_requirements: *70234406643420
 - !ruby/object:Gem::Dependency
   name: simplecov-gem-adapter
-  requirement: &70159750976360 !ruby/object:Gem::Requirement
+  requirement: &70234406643000 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -87,7 +87,7 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70159750976360
+  version_requirements: *70234406643000
 description: Wrapper to fog gem, proxying access to cloud files as local files.
 email:
 - ronen@barzel.org