defog 0.7.1 → 0.7.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +249 -0
- data/defog.gemspec +1 -0
- data/lib/defog/proxy.rb +9 -1
- data/lib/defog/version.rb +1 -1
- data/spec/proxy_spec.rb +10 -1
- metadata +21 -10
- data/README.rdoc +0 -237
data/README.md
ADDED
@@ -0,0 +1,249 @@
|
|
1
|
+
# defog
|
2
|
+
|
3
|
+
[<img
|
4
|
+
src="https://secure.travis-ci.org/ronen/defog.png"/>](http://travis-ci.org/ron
|
5
|
+
en/defog) [<img src="https://gemnasium.com/ronen/defog.png" alt="Dependency
|
6
|
+
Status" />](https://gemnasium.com/ronen/defog)
|
7
|
+
|
8
|
+
Defog wraps the [fog](https://rubygems.org/gems/fog) gem (specifically,
|
9
|
+
[Fog::Storage](http://fog.io/1.3.1/storage/)), providing access to files
|
10
|
+
stored in the cloud via proxy files on the local file system. A proxy file can
|
11
|
+
be
|
12
|
+
* Read-only: A local cached copy of a cloud file.
|
13
|
+
* Write-only: A local file that will be uploaded to the cloud.
|
14
|
+
* Read-Write: A local file that mirrors a cloud file, propogating changes
|
15
|
+
back to the cloud.
|
16
|
+
|
17
|
+
|
18
|
+
Defog thus lets you use ordinary programmatic tools to access and manipulate
|
19
|
+
your cloud data. Thanks to the magic of [fog](https://rubygems.org/gems/fog)
|
20
|
+
it works across cloud providers, and it also works with the local file system
|
21
|
+
as a "provider" so that you can, e.g. use the local file system for
|
22
|
+
development and the cloud for production.
|
23
|
+
|
24
|
+
Defog also provides a few simple remote-file management methods to minimize
|
25
|
+
the need to dig down into the Fog layer; but full access to the underlying fog
|
26
|
+
objects is available should it be needed.
|
27
|
+
|
28
|
+
## Usage Summary
|
29
|
+
|
30
|
+
Full Rdoc is available at http://rubydoc.info/gems/defog
|
31
|
+
|
32
|
+
### Create proxy connection
|
33
|
+
|
34
|
+
Connect to the remote storage by creating a `Defog::Proxy` object, which
|
35
|
+
proxies files in a specific remote location, e.g.:
|
36
|
+
|
37
|
+
defog = Defog::Proxy.new(:provider => :AWS,
|
38
|
+
:aws_access_key_id => "yourid",
|
39
|
+
:aws_secret_access_key => "yoursecret",
|
40
|
+
:region => "optional-s3-region",
|
41
|
+
:bucket => "s3-bucket-name")
|
42
|
+
|
43
|
+
defog = Defog::Proxy.new(:provider => :Local,
|
44
|
+
:local_root => "/path/to/directory")
|
45
|
+
|
46
|
+
For complete options, see Defog::Proxy.new RDOC
|
47
|
+
|
48
|
+
### Proxy a file
|
49
|
+
|
50
|
+
Open a proxy to a remote file by creating a `Defog::File` object:
|
51
|
+
|
52
|
+
file = defog.file("key/of/file", mode)
|
53
|
+
# ... access file ...
|
54
|
+
file.close
|
55
|
+
|
56
|
+
defog.file("key/of/file", mode) do |file|
|
57
|
+
# ... access file ...
|
58
|
+
end
|
59
|
+
|
60
|
+
`mode` can be "r", "r+", "w", "w+", "a", or "a+" with the usual semantics, and
|
61
|
+
can be suffixed with "b" or with ":" and encoding as usual.
|
62
|
+
|
63
|
+
When opened in a readable mode ("r", "r+", "w+", "a+"), Defog first caches the
|
64
|
+
cloud file in the local proxy. When opened in a writeable mode ("r+", "w",
|
65
|
+
"w+", "a", "a+"), Defog arranges to upload the changes back to the cloud file
|
66
|
+
at close time.
|
67
|
+
|
68
|
+
The `Defog::File` class inherits from `::File`. So you can use it directly
|
69
|
+
for I/O operations, such as
|
70
|
+
|
71
|
+
defog.file("key", "r") do |file|
|
72
|
+
file.readlines
|
73
|
+
end
|
74
|
+
|
75
|
+
You can also access the proxy file via its path, allowing things such as
|
76
|
+
|
77
|
+
defog.file("image100x100.jpg", "w") do |file|
|
78
|
+
system("convert souce.png -scale 100x100 #{file.path}")
|
79
|
+
end
|
80
|
+
|
81
|
+
(Note that the proxy file path has the same file extension as the cloud key
|
82
|
+
string.)
|
83
|
+
|
84
|
+
Closing the file object (explicitly or implicitly at the end of the block)
|
85
|
+
synchronizes the local proxy with the remote storage if needed, and then (by
|
86
|
+
default) deletes the local proxy file.
|
87
|
+
|
88
|
+
To suppress deleting the local proxy file, use `:persist => true` (see
|
89
|
+
Persistence below). To suppress updating the remote storage, delete the local
|
90
|
+
proxy file before closing (e.g. via `File.unlink(file.path)`) or pass
|
91
|
+
`:synchronize => false` to the `#close` method.
|
92
|
+
|
93
|
+
### Proxy handle
|
94
|
+
|
95
|
+
Calling Defog::Proxy#file without a mode returns a Defog::Handle object that
|
96
|
+
supports cloud file query and manipulation:
|
97
|
+
|
98
|
+
handle = defog.file("key")
|
99
|
+
handle.exist? # => true if the cloud file exists
|
100
|
+
handle.delete # deletes the cloud file
|
101
|
+
handle.size # => size of the cloud file
|
102
|
+
handle.last_modified # => modification date of the cloud file
|
103
|
+
|
104
|
+
In fact, `defog.file("key", mode, options, &block)` is really just shorthand
|
105
|
+
for
|
106
|
+
|
107
|
+
defog.file("key").open(mode, options, &block)
|
108
|
+
|
109
|
+
In addition, the handle allows you to look up the path where the local proxy
|
110
|
+
file will be if/when you open the proxy (but without actually doing the
|
111
|
+
proxying).
|
112
|
+
|
113
|
+
defog.file("key").proxy_path # => Pathname where proxy file is, was, or will be
|
114
|
+
|
115
|
+
You can also iterate through handles of all cloud files, e.g.:
|
116
|
+
|
117
|
+
defog.each { |handle| puts handle.key }
|
118
|
+
defog.each.select { |handle| handle.last_modified < 12.hours.ago }
|
119
|
+
|
120
|
+
### Persistence
|
121
|
+
|
122
|
+
By default, Defog will delete the local proxy when closing a file. However, it
|
123
|
+
is possible to keep the local proxy file so that it if the remote is accessed
|
124
|
+
again the data will not need to be transferred again. (This is true even
|
125
|
+
between executions of the program: a Defog::Proxy instance can start with
|
126
|
+
proxy files already in place, and it will use them.)
|
127
|
+
|
128
|
+
Persistence can be enabled by default for the Defog::Proxy instance via:
|
129
|
+
|
130
|
+
defog = Defog::Proxy.new(:provider => ..., :persist => true)
|
131
|
+
|
132
|
+
And/or persistence can be overridden on a per-file basis at proxy open time:
|
133
|
+
|
134
|
+
file = defog.file("key/of/file", mode, :persist => true)
|
135
|
+
|
136
|
+
or at proxy close time:
|
137
|
+
|
138
|
+
file.close(:persist => true)
|
139
|
+
|
140
|
+
When opening a file whose local proxy has been persisted, Defog checks to see
|
141
|
+
if the local proxy is out of date and if so replaces it (via MD5 digests).
|
142
|
+
|
143
|
+
## Local proxy file cache
|
144
|
+
|
145
|
+
For basic usage, you don't need to worry about the cache, the default settings
|
146
|
+
work fine. But if you will be persisting proxy files you may want to manage
|
147
|
+
the cache more carefully.
|
148
|
+
|
149
|
+
### Cache location
|
150
|
+
|
151
|
+
The cache for a given Defog::Proxy is rooted at a directory on the local file
|
152
|
+
system. You can set and query the root via
|
153
|
+
|
154
|
+
defog = Defog::Proxy.new(:provider => ..., :proxy_root => "/my/chosen/root")
|
155
|
+
defog.proxy_root # => returns a Pathname
|
156
|
+
|
157
|
+
If you don't specify a root, Defog uses one of two defaults:
|
158
|
+
|
159
|
+
{Rails.root}/tmp/defog/{provider}-{location} # if Rails is defined
|
160
|
+
{Dir.tmpdir}/defog/{provider}-{location} # if Rails is not defined
|
161
|
+
|
162
|
+
In these, `location` disambiguates between Defog::Proxy instances. For :AWS
|
163
|
+
it's the bucket name and for :local it's the `local_root` directory path with
|
164
|
+
slashes replaced with dashes.
|
165
|
+
|
166
|
+
[Why cache local files, you ask? Why not bypass this whole cache thing if
|
167
|
+
using :local? Well, the motivation for supporting :local is to use it in
|
168
|
+
development and use :AWS in production. So, to more faithfully mimic
|
169
|
+
production behavior, :local mode goes through the same code path and same
|
170
|
+
caching mechanism.]
|
171
|
+
|
172
|
+
Within the cache, indvidiual proxy files are located by treating the key as a
|
173
|
+
path relative to the proxy root (with slashes in the key indicating
|
174
|
+
subdirectories in the path).
|
175
|
+
|
176
|
+
### Cache size management
|
177
|
+
|
178
|
+
Defog can perform simple size management of the local proxy file cache. This
|
179
|
+
is of course useful mostly when persisting files.
|
180
|
+
|
181
|
+
You can specify a maximum cache size via:
|
182
|
+
|
183
|
+
defog = Defog::Proxy.new(:provider => ..., :max_cache_size => size-in-bytes)
|
184
|
+
|
185
|
+
If a maximum size is set, then before downloading data to create a proxy,
|
186
|
+
Defog will check the space available and delete persisted proxy files as
|
187
|
+
needed in LRU order. Does not delete files for proxies that are currently
|
188
|
+
open. If this would not free up enough space (because of open proxies or just
|
189
|
+
because the remote is larger than the cache), raises Defog::Error::CacheFull
|
190
|
+
and doesn't actually delete anything.
|
191
|
+
|
192
|
+
For writeable proxes, of course Defog doesn't know in advance the size of the
|
193
|
+
data you will write into proxy file. As a crude estimate, if the remote file
|
194
|
+
already exists, Defog will reserve the same amount of space. Instead, you can
|
195
|
+
tell Defog the expected size via:
|
196
|
+
|
197
|
+
defog.file("key", "w", :size_hint => size-in-bytes)
|
198
|
+
|
199
|
+
You can also manage the cache manually, by explicitly deleting an individual
|
200
|
+
persisted proxy files, such as via:
|
201
|
+
|
202
|
+
defog.file("key").proxy_path.unlink
|
203
|
+
|
204
|
+
And it's fair game to delete proxy files outside of Defog, such as via a cron
|
205
|
+
job. Of course in these cases it's up to you to make sure not to
|
206
|
+
unintentionally delete a proxy file that's currently open.
|
207
|
+
|
208
|
+
## Accessing Fog
|
209
|
+
|
210
|
+
You can access the underlying fog objects as needed:
|
211
|
+
|
212
|
+
defog = Defog::Proxy.new(:provider => ...)
|
213
|
+
|
214
|
+
defog.fog_connection # => the Fog::Storage object
|
215
|
+
defog.fog_directory # => the fog directory that contains the files being proxied
|
216
|
+
defog.file("key").fog_model # => the fog model for the cloud file
|
217
|
+
|
218
|
+
## Installation
|
219
|
+
|
220
|
+
Gemfile:
|
221
|
+
gem 'defog'
|
222
|
+
|
223
|
+
## Compatibility
|
224
|
+
|
225
|
+
Defog is currently known to work on:
|
226
|
+
|
227
|
+
* Ruby: MRI 1.9.2, MRI 1.9.3
|
228
|
+
* Fog Storage: :local, :AWS
|
229
|
+
|
230
|
+
|
231
|
+
The above storage providers are what the author uses. Please fork and add
|
232
|
+
others! (There's just a very small amount of provider-specific code in one
|
233
|
+
file, https://github.com/ronen/defog/blob/master/lib/defog/fog_wrapper.rb,
|
234
|
+
plus appropriate rspec examples.)
|
235
|
+
|
236
|
+
## History
|
237
|
+
|
238
|
+
Release Notes:
|
239
|
+
|
240
|
+
* 0.7.2 - Bug fix: don't fail when clearing cache if another process clears it first
|
241
|
+
* 0.7.1 - Add key info to message if there's an exception when getting file
|
242
|
+
* 0.7.0 - Add :query option to Handle#url
|
243
|
+
* 0.6.1 - Bug fix (caching)
|
244
|
+
* 0.6.0 - Add logging
|
245
|
+
|
246
|
+
|
247
|
+
## Copyright
|
248
|
+
|
249
|
+
Released under the MIT License. See LICENSE for details.
|
data/defog.gemspec
CHANGED
@@ -20,6 +20,7 @@ Gem::Specification.new do |gem|
|
|
20
20
|
gem.add_dependency 'fastandand'
|
21
21
|
|
22
22
|
gem.add_development_dependency 'rake'
|
23
|
+
gem.add_development_dependency 'rdoc'
|
23
24
|
gem.add_development_dependency 'rspec'
|
24
25
|
gem.add_development_dependency 'simplecov'
|
25
26
|
gem.add_development_dependency 'simplecov-gem-adapter'
|
data/lib/defog/proxy.rb
CHANGED
@@ -222,7 +222,15 @@ module Defog
|
|
222
222
|
end
|
223
223
|
|
224
224
|
# free the remaining candidates
|
225
|
-
candidates.each
|
225
|
+
candidates.each do |candidate|
|
226
|
+
begin
|
227
|
+
candidate.unlink
|
228
|
+
rescue Errno::ENOENT
|
229
|
+
# some other process has deleted the while we were looking at it.
|
230
|
+
# nothing to do.
|
231
|
+
end
|
232
|
+
end
|
233
|
+
|
226
234
|
end
|
227
235
|
|
228
236
|
end
|
data/lib/defog/version.rb
CHANGED
data/spec/proxy_spec.rb
CHANGED
@@ -35,7 +35,7 @@ shared_examples "a proxy" do |args|
|
|
35
35
|
end
|
36
36
|
|
37
37
|
it "should return fog storage" do
|
38
|
-
@proxy.fog_connection.should == @proxy.fog_directory.
|
38
|
+
@proxy.fog_connection.should == @proxy.fog_directory.service
|
39
39
|
end
|
40
40
|
|
41
41
|
it "should return fog directory" do
|
@@ -167,6 +167,15 @@ shared_examples "a proxy" do |args|
|
|
167
167
|
other_proxy_path("c").should_not be_exist
|
168
168
|
end
|
169
169
|
|
170
|
+
it "should not fail when proxies get deleted by another process" do
|
171
|
+
create_other_proxy("a", 10)
|
172
|
+
create_other_proxy("b", 30)
|
173
|
+
create_other_proxy("c", 40)
|
174
|
+
create_remote("x" * 80)
|
175
|
+
Pathname.any_instance.should_receive(:unlink).and_raise Errno::ENOENT
|
176
|
+
expect { @proxy.file(key, "r") do end }.to_not raise_error(Errno::ENOENT)
|
177
|
+
end
|
178
|
+
|
170
179
|
it "should delete proxies to make room for hinted size" do
|
171
180
|
create_other_proxy("a", 10)
|
172
181
|
create_other_proxy("b", 30)
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: defog
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.7.
|
4
|
+
version: 0.7.2
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2013-03-27 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: fog
|
@@ -75,6 +75,22 @@ dependencies:
|
|
75
75
|
- - ! '>='
|
76
76
|
- !ruby/object:Gem::Version
|
77
77
|
version: '0'
|
78
|
+
- !ruby/object:Gem::Dependency
|
79
|
+
name: rdoc
|
80
|
+
requirement: !ruby/object:Gem::Requirement
|
81
|
+
none: false
|
82
|
+
requirements:
|
83
|
+
- - ! '>='
|
84
|
+
- !ruby/object:Gem::Version
|
85
|
+
version: '0'
|
86
|
+
type: :development
|
87
|
+
prerelease: false
|
88
|
+
version_requirements: !ruby/object:Gem::Requirement
|
89
|
+
none: false
|
90
|
+
requirements:
|
91
|
+
- - ! '>='
|
92
|
+
- !ruby/object:Gem::Version
|
93
|
+
version: '0'
|
78
94
|
- !ruby/object:Gem::Dependency
|
79
95
|
name: rspec
|
80
96
|
requirement: !ruby/object:Gem::Requirement
|
@@ -134,7 +150,7 @@ files:
|
|
134
150
|
- .travis.yml
|
135
151
|
- Gemfile
|
136
152
|
- LICENSE
|
137
|
-
- README.
|
153
|
+
- README.md
|
138
154
|
- Rakefile
|
139
155
|
- defog.gemspec
|
140
156
|
- lib/defog.rb
|
@@ -162,21 +178,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
162
178
|
- - ! '>='
|
163
179
|
- !ruby/object:Gem::Version
|
164
180
|
version: '0'
|
165
|
-
segments:
|
166
|
-
- 0
|
167
|
-
hash: -4067277694003183593
|
168
181
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
169
182
|
none: false
|
170
183
|
requirements:
|
171
184
|
- - ! '>='
|
172
185
|
- !ruby/object:Gem::Version
|
173
186
|
version: '0'
|
174
|
-
segments:
|
175
|
-
- 0
|
176
|
-
hash: -4067277694003183593
|
177
187
|
requirements: []
|
178
188
|
rubyforge_project:
|
179
|
-
rubygems_version: 1.8.
|
189
|
+
rubygems_version: 1.8.25
|
180
190
|
signing_key:
|
181
191
|
specification_version: 3
|
182
192
|
summary: Wrapper to fog gem, proxying access to cloud files as local files. Access
|
@@ -187,3 +197,4 @@ test_files:
|
|
187
197
|
- spec/proxy_spec.rb
|
188
198
|
- spec/spec_helper.rb
|
189
199
|
- spec/support/helpers.rb
|
200
|
+
has_rdoc:
|
data/README.rdoc
DELETED
@@ -1,237 +0,0 @@
|
|
1
|
-
= defog
|
2
|
-
|
3
|
-
{<img src="https://secure.travis-ci.org/ronen/defog.png"/>}[http://travis-ci.org/ronen/defog]
|
4
|
-
{<img src="https://gemnasium.com/ronen/defog.png" alt="Dependency Status" />}[https://gemnasium.com/ronen/defog]
|
5
|
-
|
6
|
-
Defog wraps the fog[https://rubygems.org/gems/fog] gem (specifically,
|
7
|
-
{Fog::Storage}[http://fog.io/1.3.1/storage/]), providing access to files
|
8
|
-
stored in the cloud via proxy files on the local file system.
|
9
|
-
A proxy file can be
|
10
|
-
* Read-only: A local cached copy of a cloud file.
|
11
|
-
* Write-only: A local file that will be uploaded to the cloud.
|
12
|
-
* Read-Write: A local file that mirrors a cloud file, propogating changes back to the cloud.
|
13
|
-
|
14
|
-
Defog thus lets you use ordinary programmatic tools to access and
|
15
|
-
manipulate your cloud data. Thanks to the magic of
|
16
|
-
fog[https://rubygems.org/gems/fog] it works across cloud providers, and
|
17
|
-
it also works with the local file system as a "provider" so that you can,
|
18
|
-
e.g. use the local file system for development and the cloud for
|
19
|
-
production.
|
20
|
-
|
21
|
-
Defog also provides a few simple remote-file management methods to minimize
|
22
|
-
the need to dig down into the Fog layer; but full access to the underlying
|
23
|
-
fog objects is available should it be needed.
|
24
|
-
|
25
|
-
== Usage Summary
|
26
|
-
|
27
|
-
Full Rdoc is available at http://rubydoc.info/gems/defog
|
28
|
-
|
29
|
-
=== Create proxy connection
|
30
|
-
|
31
|
-
Connect to the remote storage by creating a <code>Defog::Proxy</code>
|
32
|
-
object, which proxies files in a specific remote location, e.g.:
|
33
|
-
|
34
|
-
defog = Defog::Proxy.new(:provider => :AWS,
|
35
|
-
:aws_access_key_id => "yourid",
|
36
|
-
:aws_secret_access_key => "yoursecret",
|
37
|
-
:region => "optional-s3-region",
|
38
|
-
:bucket => "s3-bucket-name")
|
39
|
-
|
40
|
-
defog = Defog::Proxy.new(:provider => :Local,
|
41
|
-
:local_root => "/path/to/directory")
|
42
|
-
|
43
|
-
For complete options, see Defog::Proxy.new RDOC
|
44
|
-
|
45
|
-
=== Proxy a file
|
46
|
-
|
47
|
-
Open a proxy to a remote file by creating a <code>Defog::File</code> object:
|
48
|
-
|
49
|
-
file = defog.file("key/of/file", mode)
|
50
|
-
# ... access file ...
|
51
|
-
file.close
|
52
|
-
|
53
|
-
defog.file("key/of/file", mode) do |file|
|
54
|
-
# ... access file ...
|
55
|
-
end
|
56
|
-
|
57
|
-
<code>mode</code> can be "r", "r+", "w", "w+", "a", or "a+" with the usual
|
58
|
-
semantics, and can be suffixed with "b" or with ":" and encoding as usual.
|
59
|
-
|
60
|
-
When opened in a readable mode ("r", "r+", "w+", "a+"), Defog first caches the
|
61
|
-
cloud file in the local proxy. When opened in a writeable mode ("r+", "w",
|
62
|
-
"w+", "a", "a+"), Defog arranges to upload the changes back to the cloud file
|
63
|
-
at close time.
|
64
|
-
|
65
|
-
The <code>Defog::File</code> class inherits from <code>::File</code>. So
|
66
|
-
you can use it directly for I/O operations, such as
|
67
|
-
|
68
|
-
defog.file("key", "r") do |file|
|
69
|
-
file.readlines
|
70
|
-
end
|
71
|
-
|
72
|
-
You can also access the proxy file via its path, allowing things such as
|
73
|
-
|
74
|
-
defog.file("image100x100.jpg", "w") do |file|
|
75
|
-
system("convert souce.png -scale 100x100 #{file.path}")
|
76
|
-
end
|
77
|
-
|
78
|
-
(Note that the proxy file path has the same file extension as the cloud key string.)
|
79
|
-
|
80
|
-
Closing the file object (explicitly or implicitly at the end of the block)
|
81
|
-
synchronizes the local proxy with the remote storage if needed, and then (by
|
82
|
-
default) deletes the local proxy file.
|
83
|
-
|
84
|
-
To suppress deleting the local proxy file, use <code>:persist => true</code> (see
|
85
|
-
Persistence below). To suppress updating the remote storage, delete the local proxy file before
|
86
|
-
closing (e.g. via <code>File.unlink(file.path)</code>) or pass
|
87
|
-
<code>:synchronize => false</code> to the <code>#close</code> method.
|
88
|
-
|
89
|
-
=== Proxy handle
|
90
|
-
|
91
|
-
Calling Defog::Proxy#file without a mode returns a Defog::Handle object that supports cloud file query and manipulation:
|
92
|
-
|
93
|
-
handle = defog.file("key")
|
94
|
-
handle.exist? # => true if the cloud file exists
|
95
|
-
handle.delete # deletes the cloud file
|
96
|
-
handle.size # => size of the cloud file
|
97
|
-
handle.last_modified # => modification date of the cloud file
|
98
|
-
|
99
|
-
In fact, <code>defog.file("key", mode, options, &block)</code> is really just shorthand for
|
100
|
-
|
101
|
-
defog.file("key").open(mode, options, &block)
|
102
|
-
|
103
|
-
In addition, the handle allows you to look up the path where the local proxy file will be if/when you open the proxy (but without actually doing the proxying).
|
104
|
-
|
105
|
-
defog.file("key").proxy_path # => Pathname where proxy file is, was, or will be
|
106
|
-
|
107
|
-
You can also iterate through handles of all cloud files, e.g.:
|
108
|
-
|
109
|
-
defog.each { |handle| puts handle.key }
|
110
|
-
defog.each.select { |handle| handle.last_modified < 12.hours.ago }
|
111
|
-
|
112
|
-
=== Persistence
|
113
|
-
|
114
|
-
By default, Defog will delete the local proxy when closing a file.
|
115
|
-
However, it is possible to keep the local proxy file so that it if the
|
116
|
-
remote is accessed again the data will not need to be transferred again.
|
117
|
-
(This is true even between executions of the program: a Defog::Proxy
|
118
|
-
instance can start with proxy files already in place, and it will use them.)
|
119
|
-
|
120
|
-
Persistence can be enabled by default for the Defog::Proxy instance via:
|
121
|
-
|
122
|
-
defog = Defog::Proxy.new(:provider => ..., :persist => true)
|
123
|
-
|
124
|
-
And/or persistence can be overridden on a per-file basis at proxy open time:
|
125
|
-
|
126
|
-
file = defog.file("key/of/file", mode, :persist => true)
|
127
|
-
|
128
|
-
or at proxy close time:
|
129
|
-
|
130
|
-
file.close(:persist => true)
|
131
|
-
|
132
|
-
When opening a file whose local proxy has been persisted, Defog checks to see if
|
133
|
-
the local proxy is out of date and if so replaces it (via MD5 digests).
|
134
|
-
|
135
|
-
== Local proxy file cache
|
136
|
-
|
137
|
-
For basic usage, you don't need to worry about the cache, the default
|
138
|
-
settings work fine. But if you will be persisting proxy files you may want to
|
139
|
-
manage the cache more carefully.
|
140
|
-
|
141
|
-
=== Cache location
|
142
|
-
|
143
|
-
The cache for a given Defog::Proxy is rooted at a directory on the local
|
144
|
-
file system. You can set and query the root via
|
145
|
-
|
146
|
-
defog = Defog::Proxy.new(:provider => ..., :proxy_root => "/my/chosen/root")
|
147
|
-
defog.proxy_root # => returns a Pathname
|
148
|
-
|
149
|
-
If you don't specify a root, Defog uses one of two defaults:
|
150
|
-
|
151
|
-
{Rails.root}/tmp/defog/{provider}-{location} # if Rails is defined
|
152
|
-
{Dir.tmpdir}/defog/{provider}-{location} # if Rails is not defined
|
153
|
-
|
154
|
-
In these, <code>location</code> disambiguates between Defog::Proxy instances.
|
155
|
-
For :AWS it's the bucket name and for :local it's the
|
156
|
-
<code>local_root</code> directory path with slashes replaced with dashes.
|
157
|
-
|
158
|
-
[Why cache local files, you ask? Why not bypass this whole cache thing if
|
159
|
-
using :local? Well, the motivation for supporting :local is to use it in
|
160
|
-
development and use :AWS in production. So, to more faithfully mimic
|
161
|
-
production behavior, :local mode goes through the same code path and same
|
162
|
-
caching mechanism.]
|
163
|
-
|
164
|
-
Within the cache, indvidiual proxy files are located by treating the key as
|
165
|
-
a path relative to the proxy root (with slashes in the key indicating
|
166
|
-
subdirectories in the path).
|
167
|
-
|
168
|
-
=== Cache size management
|
169
|
-
|
170
|
-
Defog can perform simple size management of the local proxy file cache. This is
|
171
|
-
of course useful mostly when persisting files.
|
172
|
-
|
173
|
-
You can specify a maximum cache size via:
|
174
|
-
|
175
|
-
defog = Defog::Proxy.new(:provider => ..., :max_cache_size => size-in-bytes)
|
176
|
-
|
177
|
-
If a maximum size is set, then before downloading data to create a proxy,
|
178
|
-
Defog will check the space available and delete persisted proxy files as needed
|
179
|
-
in LRU order. Does not delete files for proxies that are currently open.
|
180
|
-
If this would not free up enough space (because of open proxies or just
|
181
|
-
because the remote is larger than the cache), raises
|
182
|
-
Defog::Error::CacheFull and doesn't actually delete anything.
|
183
|
-
|
184
|
-
For writeable proxes, of course Defog doesn't know in advance the size of
|
185
|
-
the data you will write into proxy file. As a crude estimate, if the
|
186
|
-
remote file already exists, Defog will reserve the same amount of space.
|
187
|
-
Instead, you can tell Defog the expected size via:
|
188
|
-
|
189
|
-
defog.file("key", "w", :size_hint => size-in-bytes)
|
190
|
-
|
191
|
-
You can also manage the cache manually, by explicitly deleting an individual persisted proxy files, such as via:
|
192
|
-
|
193
|
-
defog.file("key").proxy_path.unlink
|
194
|
-
|
195
|
-
And it's fair game to delete proxy files outside of Defog, such as via a
|
196
|
-
cron job. Of course in these cases it's up to you to make sure not to
|
197
|
-
unintentionally delete a proxy file that's currently open.
|
198
|
-
|
199
|
-
== Accessing Fog
|
200
|
-
|
201
|
-
You can access the underlying fog objects as needed:
|
202
|
-
|
203
|
-
defog = Defog::Proxy.new(:provider => ...)
|
204
|
-
|
205
|
-
defog.fog_connection # => the Fog::Storage object
|
206
|
-
defog.fog_directory # => the fog directory that contains the files being proxied
|
207
|
-
defog.file("key").fog_model # => the fog model for the cloud file
|
208
|
-
|
209
|
-
== Installation
|
210
|
-
|
211
|
-
Gemfile:
|
212
|
-
gem 'defog'
|
213
|
-
|
214
|
-
== Compatibility
|
215
|
-
|
216
|
-
Defog is currently known to work on:
|
217
|
-
|
218
|
-
* Ruby: MRI 1.9.2, MRI 1.9.3
|
219
|
-
* Fog Storage: :local, :AWS
|
220
|
-
|
221
|
-
The above storage providers are what the author uses. Please fork and add
|
222
|
-
others! (There's just a very small amount of provider-specific code in
|
223
|
-
one file, https://github.com/ronen/defog/blob/master/lib/defog/fog_wrapper.rb,
|
224
|
-
plus appropriate rspec examples.)
|
225
|
-
|
226
|
-
== History
|
227
|
-
|
228
|
-
Release Notes:
|
229
|
-
|
230
|
-
* 0.7.1 - Add key info to message if there's an exception when getting file
|
231
|
-
* 0.7.0 - Add :query option to Handle#url
|
232
|
-
* 0.6.1 - Bug fix (caching)
|
233
|
-
* 0.6.0 - Add logging
|
234
|
-
|
235
|
-
== Copyright
|
236
|
-
|
237
|
-
Released under the MIT License. See LICENSE for details.
|