shrine 2.10.1 → 2.11.0
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of shrine might be problematic. Click here for more details.
- checksums.yaml +5 -5
- data/CHANGELOG.md +25 -1
- data/README.md +241 -393
- data/doc/advantages.md +346 -0
- data/doc/attacher.md +1 -1
- data/doc/carrierwave.md +9 -9
- data/doc/creating_storages.md +172 -84
- data/doc/design.md +1 -1
- data/doc/direct_s3.md +98 -85
- data/doc/metadata.md +213 -0
- data/doc/migrating_storage.md +1 -1
- data/doc/multiple_files.md +4 -3
- data/doc/paperclip.md +4 -4
- data/doc/processing.md +415 -0
- data/doc/refile.md +23 -23
- data/doc/testing.md +47 -51
- data/doc/validation.md +148 -0
- data/lib/shrine.rb +45 -4
- data/lib/shrine/plugins/add_metadata.rb +35 -14
- data/lib/shrine/plugins/determine_mime_type.rb +7 -5
- data/lib/shrine/plugins/direct_upload.rb +3 -1
- data/lib/shrine/plugins/infer_extension.rb +1 -1
- data/lib/shrine/plugins/metadata_attributes.rb +2 -2
- data/lib/shrine/plugins/presign_endpoint.rb +27 -17
- data/lib/shrine/plugins/rack_response.rb +4 -4
- data/lib/shrine/plugins/signature.rb +1 -1
- data/lib/shrine/plugins/store_dimensions.rb +10 -18
- data/lib/shrine/plugins/upload_endpoint.rb +22 -0
- data/lib/shrine/plugins/versions.rb +10 -14
- data/lib/shrine/storage/linter.rb +11 -0
- data/lib/shrine/storage/s3.rb +57 -30
- data/lib/shrine/version.rb +2 -2
- data/shrine.gemspec +3 -3
- metadata +11 -7
data/doc/advantages.md
ADDED
@@ -0,0 +1,346 @@
|
|
1
|
+
# Advantages of Shrine
|
2
|
+
|
3
|
+
There are many popular file upload solutions for Ruby out there – [Paperclip],
|
4
|
+
[CarrierWave], [Dragonfly], [Refile], and [Active Storage], to name the most
|
5
|
+
popular ones. This guide will attempt to cover some of the main advantages that
|
6
|
+
Shrine offers compared to these alternatives.
|
7
|
+
|
8
|
+
## Generality
|
9
|
+
|
10
|
+
Many alternative file upload solutions are coupled to either Rails (Active
|
11
|
+
Storage) or Active Record itself (Paperclip, CarrierWave, Dragonfly). This is
|
12
|
+
not ideal, as Rails-specific solutions fragment the Ruby community between
|
13
|
+
developers that use Rails and developers that don't. There are many great web
|
14
|
+
frameworks ([Sinatra], [Roda], [Cuba], [Hanami], [Grape] etc.) and database
|
15
|
+
libraries ([Sequel], [ROM], [Hanami::Model] etc.) out there that people use
|
16
|
+
instead of Rails and Active Record.
|
17
|
+
|
18
|
+
Shrine, on the other hand, doesn't make any assumptions about which web
|
19
|
+
framework or ORM you're using. Any HTTP-specific functionality is implemented
|
20
|
+
on top of [Rack], the Ruby web server interface that powers all the popular
|
21
|
+
Ruby web frameworks (including Rails). The integrations for specific ORMs are
|
22
|
+
provided as plugins.
|
23
|
+
|
24
|
+
```rb
|
25
|
+
# Rack-based plugins
|
26
|
+
Shrine.plugin :upload_endpoint
|
27
|
+
Shrine.plugin :presign_endpoint
|
28
|
+
Shrine.plugin :download_endpoint
|
29
|
+
Shrine.plugin :rack_response
|
30
|
+
Shrine.plugin :rack_file
|
31
|
+
|
32
|
+
# ORM plugins
|
33
|
+
Shrine.plugin :activerecord
|
34
|
+
Shrine.plugin :sequel
|
35
|
+
Shrine.plugin :mongoid # https://github.com/shrinerb/shrine-mongoid
|
36
|
+
Shrine.plugin :hanami # https://github.com/katafrakt/hanami-shrine
|
37
|
+
```
|
38
|
+
|
39
|
+
## Modularity
|
40
|
+
|
41
|
+
Shrine uses a [plugin system] that allows you to pick and choose the features
|
42
|
+
that you want, which makes it very flexible. Moreover, you're only loading the
|
43
|
+
code for features that you use, which means that Shrine will generally load
|
44
|
+
very fast.
|
45
|
+
|
46
|
+
```rb
|
47
|
+
Shrine.plugin :logging # loads the logging feature
|
48
|
+
```
|
49
|
+
|
50
|
+
### Dependencies
|
51
|
+
|
52
|
+
Shrine is very diligent when it comes to dependencies. It has only one
|
53
|
+
mandatory dependency - [Down], a gem for streaming downloads from a URL. Some
|
54
|
+
Shrine plugins require more dependencies, but you only need to load them if
|
55
|
+
you're using those plugins. Moreover, for the same task you can often choose
|
56
|
+
between different dependencies.
|
57
|
+
|
58
|
+
For example, if you want to determine MIME type from file content, upon loading
|
59
|
+
the `determine_mime_type` plugin you can choose whether you want to use the
|
60
|
+
[`file`] command, [FileMagic], [FastImage], [MimeMagic] or [Marcel] gem to do
|
61
|
+
that. For determining MIME type from file extension you can choose between
|
62
|
+
[mime-types] or [mini_mime] gems. Likewise, for `store_dimensions` plugin you
|
63
|
+
can choose between [FastImage], [MiniMagick] or [ruby-vips] gems for extracting
|
64
|
+
image dimensions.
|
65
|
+
|
66
|
+
```rb
|
67
|
+
Shrine.plugin :determine_mime_type, analyzer: :marcel
|
68
|
+
Shrine.plugin :store_dimensions, analyzer: :mini_magick
|
69
|
+
```
|
70
|
+
|
71
|
+
With this approach you have control over your dependencies and are free to
|
72
|
+
choose the combination that best suit your needs.
|
73
|
+
|
74
|
+
## Simplicity
|
75
|
+
|
76
|
+
Shrine was designed with simplicity in mind. Where other solutions favour
|
77
|
+
complex class-level DSLs, Shrine chooses simple instance-level interfaces where
|
78
|
+
you can write regular Ruby code.
|
79
|
+
|
80
|
+
There are also no `CarrierWave::Uploader::Base` and `Paperclip::Attachment` [God
|
81
|
+
objects]. The `Shrine` class is responsible for uploading files to the storage
|
82
|
+
(which are simple Ruby classes), `Shrine::UploadedFile` exposes extracted
|
83
|
+
metadata and can retrieve the file from the storage, and `Shrine::Attacher`
|
84
|
+
wraps those two classes to provide an interface for attaching files to database
|
85
|
+
records.
|
86
|
+
|
87
|
+
```rb
|
88
|
+
photo.image #=> #<Shrine::UploadedFile>
|
89
|
+
photo.image.storage #=> #<Shrine::Storage::S3>
|
90
|
+
photo.image.uploader #=> #<Shrine>
|
91
|
+
photo.image_attacher #=> #<Shrine::Attacher>
|
92
|
+
```
|
93
|
+
|
94
|
+
Special care was taken to make integrating new ORMs and storages possible with
|
95
|
+
minimal amount of code.
|
96
|
+
|
97
|
+
## Inheritance
|
98
|
+
|
99
|
+
Shrine is designed to handle any types of files. If you're accepting uploads of
|
100
|
+
multiple types of files, such as videos and images, chances are that the logic
|
101
|
+
for handling them will be very different:
|
102
|
+
|
103
|
+
* images can be processed on-the-fly, while videos should be transcoded on upload
|
104
|
+
* you might want to store images on one service and videos on another
|
105
|
+
* tools for extracting image metadata are different than ones for video metadata
|
106
|
+
|
107
|
+
With Shrine you can create isolated uploaders for each type of file. Plugins
|
108
|
+
that you want to be applied to both uploaders can be applied globally, while
|
109
|
+
other plugins would be loaded only for a specific uploader.
|
110
|
+
|
111
|
+
```rb
|
112
|
+
Shrine.plugin :activerecord
|
113
|
+
Shrine.plugin :logging
|
114
|
+
```
|
115
|
+
```rb
|
116
|
+
class ImageUploader < Shrine
|
117
|
+
plugin :store_dimensions
|
118
|
+
end
|
119
|
+
```
|
120
|
+
```rb
|
121
|
+
class VideoUploader < Shrine
|
122
|
+
plugin :default_storage, store: :vimeo
|
123
|
+
end
|
124
|
+
```
|
125
|
+
|
126
|
+
## Processing
|
127
|
+
|
128
|
+
Instead of writing yet another vendored solution for generating image
|
129
|
+
thumbnails, the **[ImageProcessing]** gem was created to be used with Shrine,
|
130
|
+
but it can also be used with any other file upload library. It's very flexible
|
131
|
+
and takes care of many details for you, such as [auto orienting] the input
|
132
|
+
image and [sharpening] the thumbnails after they are resized.
|
133
|
+
|
134
|
+
```rb
|
135
|
+
require "image_processing"
|
136
|
+
|
137
|
+
thumbnail = ImageProcessing::MiniMagick
|
138
|
+
.source(image)
|
139
|
+
.resize_to_limit(400, 400)
|
140
|
+
.call # convert input.jpg -auto-orient -resize 400x400> -sharpen 0x1 output.jpg
|
141
|
+
|
142
|
+
thumbnail #=> #<Tempfile:/var/folders/.../image_processing20180316-18446-1j247h6.png>
|
143
|
+
```
|
144
|
+
|
145
|
+
### libvips
|
146
|
+
|
147
|
+
Probably the biggest ImageProcessing feature is the support for **[libvips]**.
|
148
|
+
Libvips is an image processing library which can process images multiple times
|
149
|
+
faster than ImageMagick and has significantly lower memory usage (see [Why is
|
150
|
+
libvips quick]). The `ImageProcessing::Vips` backend implements the same API as
|
151
|
+
`ImageProcessing::MiniMagick`, so you can easily swap one for the other.
|
152
|
+
|
153
|
+
```rb
|
154
|
+
require "image_processing/mini_magick"
|
155
|
+
require "image_processing/vips"
|
156
|
+
require "open-uri"
|
157
|
+
|
158
|
+
original = open("https://upload.wikimedia.org/wikipedia/commons/3/36/Hopetoun_falls.jpg")
|
159
|
+
|
160
|
+
ImageProcessing::MiniMagick.resize_to_fit(800, 800).call(original)
|
161
|
+
#=> 1.0s
|
162
|
+
|
163
|
+
ImageProcessing::Vips.resize_to_fit(800, 800).call(original)
|
164
|
+
#=> 0.2s (5x faster)
|
165
|
+
```
|
166
|
+
|
167
|
+
### Other processors
|
168
|
+
|
169
|
+
Using other processors for other types of files doesn't require remembering any
|
170
|
+
specific API, you just call them in the same processing block where you're
|
171
|
+
calling ImageProcessing. Shrine's processing block acts as a functional
|
172
|
+
transformation: you get the original file on the input, and you're expected to
|
173
|
+
return processed file(s) on the output.
|
174
|
+
|
175
|
+
```rb
|
176
|
+
class VideoUploader < Shrine
|
177
|
+
plugin :processing
|
178
|
+
|
179
|
+
process(:store) do |io, context|
|
180
|
+
# define your processing
|
181
|
+
end
|
182
|
+
end
|
183
|
+
```
|
184
|
+
|
185
|
+
### On-the-fly processing
|
186
|
+
|
187
|
+
Shrine is primarily designed for processing files on upload, since that's
|
188
|
+
applicable to all types of files, while on-the-fly processing that [Dragonfly],
|
189
|
+
[Refile], and [Active Storage] offer makes sense only for images.
|
190
|
+
|
191
|
+
However, there are many specialized solutions that provide on-the-fly
|
192
|
+
processing functionality, both open source and commercial, and it's fairly easy
|
193
|
+
to apply them to files uploaded by Shrine.
|
194
|
+
|
195
|
+
```rb
|
196
|
+
Dragonfly.app
|
197
|
+
.fetch_url(photo.image_url) # image uploaded by Shrine
|
198
|
+
.thumb("800x800")
|
199
|
+
.url #=> "/attachments/W1siZnUiLCJodHRwOi8vd3d3LnB1YmxpY2RvbWFpbn..."
|
200
|
+
```
|
201
|
+
|
202
|
+
## Metadata
|
203
|
+
|
204
|
+
Shrine automatically [extracts metadata][metadata] from each uploaded file,
|
205
|
+
including derivates like image thumbnails, and saves them into the database
|
206
|
+
column. In addition to filename, filesize, and MIME type that are extracted by
|
207
|
+
default, you can also extract [image dimensions][store_dimensions], or your own
|
208
|
+
[custom metadata][add_metadata]. This metadata can additionally be
|
209
|
+
[validated][validation].
|
210
|
+
|
211
|
+
```rb
|
212
|
+
photo.image.metadata #=>
|
213
|
+
# {
|
214
|
+
# "size" => 42487494,
|
215
|
+
# "filename" => "nature.jpg",
|
216
|
+
# "mime_type" => "image/jpeg",
|
217
|
+
# "width" => 600,
|
218
|
+
# "height" => 400,
|
219
|
+
# ...
|
220
|
+
# }
|
221
|
+
```
|
222
|
+
|
223
|
+
## Direct Uploads
|
224
|
+
|
225
|
+
Instead of submitting selected files synchronously via the form, it's better to
|
226
|
+
start uploading files asynchronously as soon as they're selected. Shrine
|
227
|
+
streamlines this workflow, allowing you to upload directly [to your
|
228
|
+
app][upload_endpoint] or [to the cloud][presign_endpoint].
|
229
|
+
|
230
|
+
[Refile] and [Active Storage] provide this functionality as well, and they also
|
231
|
+
ship with a custom plug-and-play JavaScript solution for integrating these
|
232
|
+
endpoints. In contrast, Shrine doesn't ship with any custom JavaScript but
|
233
|
+
instead recommends using **[Uppy]**. Uppy is a flexible JavaScript file upload
|
234
|
+
library that allows uploading to a [custom endpoint][XHRUpload], to [AWS
|
235
|
+
S3][AwsS3], or even to a [resumable endpoint][Tus], with the possibility to use
|
236
|
+
UI components such as a simple [status bar][StatusBar] or a complete
|
237
|
+
[dashboard][Dashboard]. Since Uppy is maintained by the whole JavaScript
|
238
|
+
community, it will generally be better than any homegrown solution.
|
239
|
+
|
240
|
+
## Backgrounding
|
241
|
+
|
242
|
+
Unlike most other file upload solutions, where background processing is an
|
243
|
+
afterthought, Shrine was designed with this feature in mind from day one. It
|
244
|
+
is supported via the `backgrounding` plugin, which provides a flexible API
|
245
|
+
that allows using [any backgrounding library][backgrounding libraries].
|
246
|
+
|
247
|
+
## Large Files
|
248
|
+
|
249
|
+
Some applications need to handle large files such as videos, and Shrine doesn't
|
250
|
+
fall short in this department. It uses and encourages streaming uploads and
|
251
|
+
downloads, where only a small part of the file is loaded into memory at any
|
252
|
+
given time. This keeps memory usage very low regardless of the size of the
|
253
|
+
files.
|
254
|
+
|
255
|
+
Shrine storages also automatically support [partial downloads][Down streaming]
|
256
|
+
(provided by the [Down] gem), which allows you to read only a portion of the
|
257
|
+
file. This can be useful for extracting metadata, because common information such
|
258
|
+
as MIME type or image dimensions are typically written in the beginning of the
|
259
|
+
file, so it's enough to download just the first few kilobytes of the file.
|
260
|
+
|
261
|
+
### Resumable uploads
|
262
|
+
|
263
|
+
Another challenge with large files is that it can be difficult for users
|
264
|
+
to upload them to your app, especially on flaky internet connections. With
|
265
|
+
a simple HTTP request, should there be any interruption during the execution,
|
266
|
+
the whole upload needs to be retried from the beginning.
|
267
|
+
|
268
|
+
To fix this problem, the community has created an open HTTP-based protocol for
|
269
|
+
resumable uploads – **[tus]**. There even exists a [Ruby server
|
270
|
+
implementation][tus-ruby-server] for this protocol ready to use. Finally, for
|
271
|
+
attaching files uploaded via the tus protocol you can use the [shrine-tus] gem.
|
272
|
+
|
273
|
+
## Security
|
274
|
+
|
275
|
+
It's important to care about security when handling file uploads, and
|
276
|
+
Shrine bakes in many good practices. For starters, it uses a separate
|
277
|
+
"temporary" storage for direct uploads, making it easy to periodically clear
|
278
|
+
uploads that didn't end up being attached and difficult for the attacker to
|
279
|
+
flood the main storage.
|
280
|
+
|
281
|
+
File processing and upload to permanent storage is done outside of a database
|
282
|
+
transaction, and only after the file has been successfully validated. The
|
283
|
+
`determine_mime_type` plugin determines MIME type from the file content (rather
|
284
|
+
than relying on the `Content-Type` request header), preventing exploits like
|
285
|
+
[ImageTragick].
|
286
|
+
|
287
|
+
The `remote_url` plugin requires specifying a `:max_size` option, which limits
|
288
|
+
the maximum allowed size of the remote file. The [Down] gem which the
|
289
|
+
`remote_url` plugin uses will immediately terminate the download if it reads
|
290
|
+
from the `Content-Length` response header that the file will be too large. For
|
291
|
+
chunked responses (where `Content-Length` header is absent) the download will
|
292
|
+
will be terminated as soon as the received content surpasses the specified
|
293
|
+
limit.
|
294
|
+
|
295
|
+
[Paperclip]: https://github.com/thoughtbot/paperclip
|
296
|
+
[CarrierWave]: https://github.com/carrierwaveuploader/carrierwave
|
297
|
+
[Dragonfly]: http://markevans.github.io/dragonfly/
|
298
|
+
[Refile]: https://github.com/refile/refile
|
299
|
+
[Active Storage]: https://github.com/rails/rails/tree/master/activestorage#active-storage
|
300
|
+
[Rack]: https://rack.github.io
|
301
|
+
[Sinatra]: http://sinatrarb.com
|
302
|
+
[Roda]: http://roda.jeremyevans.net
|
303
|
+
[Cuba]: http://cuba.is
|
304
|
+
[Hanami]: http://hanamirb.org
|
305
|
+
[Grape]: https://github.com/ruby-grape/grape
|
306
|
+
[Sequel]: http://sequel.jeremyevans.net
|
307
|
+
[ROM]: http://rom-rb.org
|
308
|
+
[Hanami::Model]: https://github.com/hanami/model
|
309
|
+
[plugin system]: https://twin.github.io/the-plugin-system-of-sequel-and-roda/
|
310
|
+
[Down]: https://github.com/janko-m/down
|
311
|
+
[`file`]: http://linux.die.net/man/1/file
|
312
|
+
[FileMagic]: https://github.com/blackwinter/ruby-filemagic
|
313
|
+
[FastImage]: https://github.com/sdsykes/fastimage
|
314
|
+
[MimeMagic]: https://github.com/minad/mimemagic
|
315
|
+
[Marcel]: https://github.com/basecamp/marcel
|
316
|
+
[mime-types]: https://github.com/mime-types/ruby-mime-types
|
317
|
+
[mini_mime]: https://github.com/discourse/mini_mime
|
318
|
+
[MiniMagick]: https://github.com/minimagick/minimagick
|
319
|
+
[ruby-vips]: https://github.com/jcupitt/ruby-vips
|
320
|
+
[God objects]: https://en.wikipedia.org/wiki/God_object
|
321
|
+
[ImageMagick]: https://www.imagemagick.org
|
322
|
+
[refile-mini_magick]: https://github.com/refile/refile-mini_magick
|
323
|
+
[ImageProcessing]: https://github.com/janko-m/image_processing
|
324
|
+
[auto orienting]: https://www.imagemagick.org/script/command-line-options.php#auto-orient
|
325
|
+
[sharpening]: https://photography.tutsplus.com/tutorials/what-is-image-sharpening--cms-26627
|
326
|
+
[libvips]: http://jcupitt.github.io/libvips/
|
327
|
+
[Why is libvips quick]: https://github.com/jcupitt/libvips/wiki/Why-is-libvips-quick
|
328
|
+
[metadata]: https://shrinerb.com/rdoc/files/doc/metadata_md.html
|
329
|
+
[store_dimensions]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/StoreDimensions.html
|
330
|
+
[add_metadata]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/AddMetadata.html
|
331
|
+
[validation]: https://shrinerb.com/rdoc/files/doc/validation_md.html
|
332
|
+
[upload_endpoint]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/UploadEndpoint.html
|
333
|
+
[presign_endpoint]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/PresignEndpoint.html
|
334
|
+
[Uppy]: https://uppy.io
|
335
|
+
[XHRUpload]: https://uppy.io/docs/xhrupload/
|
336
|
+
[AwsS3]: https://uppy.io/docs/aws-s3/
|
337
|
+
[Tus]: https://uppy.io/docs/tus/
|
338
|
+
[StatusBar]: https://uppy.io/examples/statusbar/
|
339
|
+
[Dashboard]: https://uppy.io/examples/dashboard/
|
340
|
+
[background job]: http://shrinerb.com/rdoc/classes/Shrine/Plugins/Backgrounding.html
|
341
|
+
[backgrounding libraries]: https://github.com/shrinerb/shrine/wiki/Backgrounding-libraries
|
342
|
+
[Down streaming]: https://github.com/janko-m/down#streaming
|
343
|
+
[tus]: https://tus.io
|
344
|
+
[tus-ruby-server]: https://github.com/janko-m/tus-ruby-server
|
345
|
+
[shrine-tus]: https://github.com/shrinerb/shrine-tus
|
346
|
+
[ImageTragick]: https://imagetragick.com
|
data/doc/attacher.md
CHANGED
@@ -233,4 +233,4 @@ attacher.delete!(stored_file) # delegates to `Shrine#delete`
|
|
233
233
|
The `#cache!` and `#store!` only upload the file to the storage, they don't
|
234
234
|
write to record's data column.
|
235
235
|
|
236
|
-
[file migrations]:
|
236
|
+
[file migrations]: https://shrinerb.com/rdoc/files/doc/migrating_storage_md.html
|
data/doc/carrierwave.md
CHANGED
@@ -40,7 +40,7 @@ via [direct uploads]):
|
|
40
40
|
```rb
|
41
41
|
Shrine.storages = {
|
42
42
|
cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
|
43
|
-
store: Shrine::Storage::S3.new(
|
43
|
+
store: Shrine::Storage::S3.new(**s3_options),
|
44
44
|
}
|
45
45
|
```
|
46
46
|
|
@@ -570,11 +570,11 @@ that you can use for retaining the cached file:
|
|
570
570
|
```rb
|
571
571
|
Shrine.plugin :cached_attachment_data
|
572
572
|
```
|
573
|
-
```
|
574
|
-
|
575
|
-
|
576
|
-
|
577
|
-
|
573
|
+
```rb
|
574
|
+
form_for @user do |f|
|
575
|
+
f.hidden_field :avatar, value: @user.cached_avatar_data
|
576
|
+
f.file_field :avatar
|
577
|
+
end
|
578
578
|
```
|
579
579
|
|
580
580
|
#### `#remote_<attachment>_url`
|
@@ -711,9 +711,9 @@ Shrine::Storage::S3.new(endpoint: "https://s3-accelerate.amazonaws.com", **optio
|
|
711
711
|
|
712
712
|
[image_processing]: https://github.com/janko-m/image_processing
|
713
713
|
[demo app]: https://github.com/shrinerb/shrine/tree/master/demo
|
714
|
-
[Reprocessing versions]:
|
714
|
+
[Reprocessing versions]: https://shrinerb.com/rdoc/files/doc/regenerating_versions_md.html
|
715
715
|
[shrine-fog]: https://github.com/shrinerb/shrine-fog
|
716
|
-
[direct uploads]:
|
717
|
-
[`Shrine::Storage::S3`]:
|
716
|
+
[direct uploads]: https://shrinerb.com/rdoc/files/doc/direct_s3_md.html
|
717
|
+
[`Shrine::Storage::S3`]: https://shrinerb.com/rdoc/classes/Shrine/Storage/S3.html
|
718
718
|
[`Shrine::Storage::GoogleCloudStorage`]: https://github.com/renchap/shrine-google_cloud_storage
|
719
719
|
[`Shrine::Storage::Fog`]: https://github.com/shrinerb/shrine-fog
|
data/doc/creating_storages.md
CHANGED
@@ -1,10 +1,8 @@
|
|
1
1
|
# Creating a New Storage
|
2
2
|
|
3
|
-
## Essentials
|
4
|
-
|
5
3
|
Shrine ships with the FileSystem and S3 storages, but it's also easy to create
|
6
|
-
your own. A storage is a class which needs to implement
|
7
|
-
methods
|
4
|
+
your own. A storage is a class which needs to implement `#upload`, `#url`,
|
5
|
+
`#open`, `#exists?`, and `#delete` methods.
|
8
6
|
|
9
7
|
```rb
|
10
8
|
class Shrine
|
@@ -14,16 +12,16 @@ class Shrine
|
|
14
12
|
# uploads `io` to the location `id`, can accept upload options
|
15
13
|
end
|
16
14
|
|
17
|
-
def
|
18
|
-
# returns
|
15
|
+
def open(id, **options)
|
16
|
+
# returns the remote file as an IO-like object
|
19
17
|
end
|
20
18
|
|
21
|
-
def
|
22
|
-
# returns the remote file
|
19
|
+
def url(id, **options)
|
20
|
+
# returns URL to the remote file, can accept URL options
|
23
21
|
end
|
24
22
|
|
25
23
|
def exists?(id)
|
26
|
-
#
|
24
|
+
# returns whether the file exists on storage
|
27
25
|
end
|
28
26
|
|
29
27
|
def delete(id)
|
@@ -36,10 +34,27 @@ end
|
|
36
34
|
|
37
35
|
## Upload
|
38
36
|
|
39
|
-
The
|
40
|
-
|
41
|
-
|
42
|
-
|
37
|
+
The `#upload` storage method is called by `Shrine#upload`, it accepts an IO
|
38
|
+
object (`io`) and upload location (`id`) and is expected to upload the IO
|
39
|
+
content to the specified location. It's also given `:shrine_metadata` that was
|
40
|
+
extracted from the IO, which can be used for specifying request headers on
|
41
|
+
upload. The storage can also support custom upload options (which can be
|
42
|
+
utilized with the `upload_options` plugin).
|
43
|
+
|
44
|
+
```rb
|
45
|
+
class MyStorage
|
46
|
+
# ...
|
47
|
+
def upload(io, id, shrine_metadata: {}, **upload_options)
|
48
|
+
# uploads `io` to the location `id`, can accept upload options
|
49
|
+
end
|
50
|
+
# ...
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
54
|
+
Unless you're already using a Ruby SDK, it's recommended to use [HTTP.rb] for
|
55
|
+
uploading. It accepts any IO object that responds to `#read` (not just file
|
56
|
+
objects), and it streams the request body directly to the TCP socket, both for
|
57
|
+
raw and multipart uploads, making it suitable for large uploads.
|
43
58
|
|
44
59
|
```rb
|
45
60
|
require "http"
|
@@ -63,8 +78,9 @@ def upload(io, id, shrine_metadata: {}, **upload_options)
|
|
63
78
|
end
|
64
79
|
```
|
65
80
|
|
66
|
-
Likewise, if you need to save some information into the metadata after upload
|
67
|
-
you can modify the
|
81
|
+
Likewise, if you need to save some information into the metadata after upload
|
82
|
+
(e.g. if the MIME type of the file changes on upload), you can modify the
|
83
|
+
metadata hash:
|
68
84
|
|
69
85
|
```rb
|
70
86
|
def upload(io, id, shrine_metadata: {}, **upload_options)
|
@@ -73,99 +89,175 @@ def upload(io, id, shrine_metadata: {}, **upload_options)
|
|
73
89
|
end
|
74
90
|
```
|
75
91
|
|
92
|
+
## Open
|
93
|
+
|
94
|
+
The `#open` storage method is called by various `Shrine::UploadedFile` methods
|
95
|
+
that retrieve uploaded file content. It accepts the file location and is
|
96
|
+
expected to return an IO-like object (that implements `#read`, `#size`,
|
97
|
+
`#rewind`, `#eof?`, and `#close`) that represents the uploaded file.
|
98
|
+
|
99
|
+
|
100
|
+
```rb
|
101
|
+
class MyStorage
|
102
|
+
# ...
|
103
|
+
def open(id, **options)
|
104
|
+
# returns the remote file as an IO-like object
|
105
|
+
end
|
106
|
+
# ...
|
107
|
+
end
|
108
|
+
```
|
109
|
+
|
110
|
+
Ideally, the returned IO object should lazily retrieve uploaded content, so
|
111
|
+
that in cases where metadata needs to be extracted from an uploaded file, only
|
112
|
+
a small portion of the file will be downloaded.
|
113
|
+
|
114
|
+
It's recommended to use the [Down] gem for this. If the storage exposes its
|
115
|
+
files over HTTP, you can use `Down.open`, otherwise if it's possible to stream
|
116
|
+
chunks of content from the storage, that can be wrapped in a `Down::ChunkedIO`.
|
117
|
+
It's recommended to use the [`Down::Http`] backend, as the [HTTP.rb] gem
|
118
|
+
allocates an order of magnitude less memory when reading the response body
|
119
|
+
compared to `Net::HTTP`.
|
120
|
+
|
121
|
+
The storage can support additional options to customize how the file will be
|
122
|
+
opened, `Shrine::UploadedFile#open` and `Shrine::UploadedFile#download` will
|
123
|
+
forward any given options to `#open`.
|
124
|
+
|
76
125
|
## Download
|
77
126
|
|
78
|
-
Shrine
|
79
|
-
|
80
|
-
|
127
|
+
`Shrine::UploadedFile#download` by default uses the `#open` storage method to
|
128
|
+
stream file content to a Tempfile. However, if you would like to use your own
|
129
|
+
custom way of downloading to a file, you can define `#download` on the storage
|
130
|
+
and `Shrine::UploadedFile#download` will automatically call that instead.
|
81
131
|
|
82
132
|
```rb
|
83
|
-
class
|
84
|
-
|
85
|
-
|
86
|
-
|
133
|
+
class MyStorage
|
134
|
+
# ...
|
135
|
+
def download(id, **options)
|
136
|
+
# download the uploaded file to a Tempfile
|
137
|
+
end
|
138
|
+
# ...
|
139
|
+
end
|
140
|
+
```
|
87
141
|
|
88
|
-
|
89
|
-
|
90
|
-
|
142
|
+
The storage can support additional options to customize how the file will be
|
143
|
+
downloaded, `Shrine::UploadedFile#download` will forward any given options to
|
144
|
+
`#download`.
|
91
145
|
|
92
|
-
|
93
|
-
|
146
|
+
## Url
|
147
|
+
|
148
|
+
The `#url` storage method is called by `Shrine::UploadedFile#url`, it accepts a
|
149
|
+
file location and is expected to return a resolvable URL to the uploaded file.
|
150
|
+
Custom URL options can be supported if needed, `Shrine::UploadedFile#url` will
|
151
|
+
forward any given options to `#url`.
|
152
|
+
|
153
|
+
```rb
|
154
|
+
class MyStorage
|
155
|
+
# ...
|
156
|
+
def url(id, **options)
|
157
|
+
# returns URL to the remote file, can accept URL options
|
94
158
|
end
|
159
|
+
# ...
|
95
160
|
end
|
96
161
|
```
|
97
162
|
|
98
|
-
|
163
|
+
If the storage does not have uploaded files accessible via HTTP, the `#url`
|
164
|
+
method should return `nil`. Note that in this case users can use the
|
165
|
+
`download_endpoint` or `rack_response` plugins to create a downloadable link,
|
166
|
+
which are implemented in terms of `#open`.
|
99
167
|
|
100
|
-
|
101
|
-
additional information from the server, you can implement a `#presign` method,
|
102
|
-
which will be used by the `presign_endpoint` plugin. The method should return an
|
103
|
-
object which responds to
|
168
|
+
## Exists
|
104
169
|
|
105
|
-
|
106
|
-
|
107
|
-
|
170
|
+
The `#exists?` storage method is called by `Shrine::UploadedFile#exists?`, it
|
171
|
+
accepts a file location and should return `true` if the file exists on the
|
172
|
+
storage and `false` otherwise.
|
108
173
|
|
109
174
|
```rb
|
110
|
-
class
|
111
|
-
|
112
|
-
|
113
|
-
|
175
|
+
class MyStorage
|
176
|
+
# ...
|
177
|
+
def exists?(id)
|
178
|
+
# returns whether the file exists on storage
|
179
|
+
end
|
180
|
+
# ...
|
181
|
+
end
|
182
|
+
```
|
114
183
|
|
115
|
-
|
116
|
-
# returns an object which responds to #url and #presign
|
117
|
-
end
|
184
|
+
## Delete
|
118
185
|
|
119
|
-
|
120
|
-
|
186
|
+
The `#delete` storage method is called by `Shrine::UploadedFile#delete`, it
|
187
|
+
accepts a file location and is expected to delete the file from the storage.
|
188
|
+
|
189
|
+
```rb
|
190
|
+
class MyStorage
|
191
|
+
# ...
|
192
|
+
def delete(id)
|
193
|
+
# deletes the file from the storage
|
121
194
|
end
|
195
|
+
# ...
|
122
196
|
end
|
123
197
|
```
|
124
198
|
|
125
|
-
|
199
|
+
For convenience of use, this method should not raise an exception if the file
|
200
|
+
doesn't exist.
|
201
|
+
|
202
|
+
## Presign
|
126
203
|
|
127
|
-
If
|
128
|
-
|
204
|
+
If the storage service supports direct uploads, and requires fetching
|
205
|
+
additional information from the server, you can implement a `#presign` method,
|
206
|
+
which will be called by the `presign_endpoint` plugin. The `#presign` method
|
207
|
+
should return a Hash with the following keys:
|
208
|
+
|
209
|
+
* `:method` – HTTP verb that should be used
|
210
|
+
* `:url` – URL to which the file should be uploaded to
|
211
|
+
* `:fields` – Hash of request parameters that should be used for the upload (optional)
|
212
|
+
* `:headers` – Hash of request headers that should be used for the upload (optional)
|
129
213
|
|
130
214
|
```rb
|
131
|
-
class
|
132
|
-
|
133
|
-
|
134
|
-
|
215
|
+
class MyStorage
|
216
|
+
# ...
|
217
|
+
def presign(id, **options)
|
218
|
+
# returns a Hash with :method, :url, :fields, and :headers keys
|
219
|
+
end
|
220
|
+
# ...
|
221
|
+
end
|
222
|
+
```
|
135
223
|
|
136
|
-
|
137
|
-
|
138
|
-
|
224
|
+
The storage can support additional options to customize how the presign will be
|
225
|
+
generated, those can be forwarded via the `:presign_options` option on the
|
226
|
+
`presign_endpoint` plugin.
|
139
227
|
|
140
|
-
|
141
|
-
# whether the given `io` is movable to the location `id`
|
142
|
-
end
|
228
|
+
## Move
|
143
229
|
|
144
|
-
|
145
|
-
|
230
|
+
If your storage can move files, you can add the additional `#move` and
|
231
|
+
`#movable?` methods, and they will automatically get used if the `moving`
|
232
|
+
plugin is loaded.
|
233
|
+
|
234
|
+
```rb
|
235
|
+
class MyStorage
|
236
|
+
# ...
|
237
|
+
def move(io, id, **upload_options)
|
238
|
+
# does the moving of the `io` to the location `id`
|
239
|
+
end
|
240
|
+
|
241
|
+
def movable?(io, id)
|
242
|
+
# whether the given `io` is movable to the location `id`
|
146
243
|
end
|
244
|
+
# ...
|
147
245
|
end
|
148
246
|
```
|
149
247
|
|
150
|
-
##
|
248
|
+
## Clear
|
151
249
|
|
152
250
|
While this method is not used by Shrine, it is good to give users the
|
153
251
|
possibility to delete all files in a storage, and the conventional name for
|
154
|
-
this method is `#clear
|
252
|
+
this method is `#clear!`.
|
155
253
|
|
156
254
|
```rb
|
157
|
-
class
|
158
|
-
|
159
|
-
|
160
|
-
|
161
|
-
|
162
|
-
def clear!
|
163
|
-
# deletes all files in the storage
|
164
|
-
end
|
165
|
-
|
166
|
-
# ...
|
167
|
-
end
|
255
|
+
class MyStorage
|
256
|
+
# ...
|
257
|
+
def clear!
|
258
|
+
# deletes all files in the storage
|
168
259
|
end
|
260
|
+
# ...
|
169
261
|
end
|
170
262
|
```
|
171
263
|
|
@@ -175,18 +267,12 @@ If your storage supports updating data of existing files (e.g. some metadata),
|
|
175
267
|
the convention is to create an `#update` method:
|
176
268
|
|
177
269
|
```rb
|
178
|
-
class
|
179
|
-
|
180
|
-
|
181
|
-
|
182
|
-
|
183
|
-
def update(id, options = {})
|
184
|
-
# update data of the file
|
185
|
-
end
|
186
|
-
|
187
|
-
# ...
|
188
|
-
end
|
270
|
+
class MyStorage
|
271
|
+
# ...
|
272
|
+
def update(id, **options)
|
273
|
+
# update data of the file
|
189
274
|
end
|
275
|
+
# ...
|
190
276
|
end
|
191
277
|
```
|
192
278
|
|
@@ -226,4 +312,6 @@ tests for your storage. There will likely be some edge cases that won't be
|
|
226
312
|
tested by the linter.
|
227
313
|
|
228
314
|
[HTTP.rb]: https://github.com/httprb/http
|
229
|
-
[fake IO]: https://github.com/shrinerb/shrine
|
315
|
+
[fake IO]: https://github.com/shrinerb/shrine/blob/master/test/support/fakeio.rb
|
316
|
+
[Down]: https://github.com/janko-m/down
|
317
|
+
[`Down::Http`]: https://github.com/janko-m/down#httprb
|