shrine 3.1.0 → 3.2.0
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of shrine might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/CHANGELOG.md +18 -0
- data/README.md +3 -3
- data/doc/advantages.md +3 -3
- data/doc/carrierwave.md +17 -8
- data/doc/design.md +134 -85
- data/doc/external/articles.md +1 -0
- data/doc/external/extensions.md +38 -35
- data/doc/getting_started.md +135 -79
- data/doc/metadata.md +79 -43
- data/doc/paperclip.md +13 -4
- data/doc/plugins/add_metadata.md +92 -35
- data/doc/plugins/backgrounding.md +12 -2
- data/doc/plugins/default_url.md +3 -0
- data/doc/plugins/derivatives.md +35 -27
- data/doc/plugins/included.md +25 -5
- data/doc/plugins/remove_invalid.md +9 -1
- data/doc/plugins/type_predicates.md +96 -0
- data/doc/plugins/url_options.md +2 -2
- data/doc/plugins/validation.md +5 -4
- data/doc/processing.md +286 -121
- data/doc/refile.md +9 -9
- data/doc/release_notes/3.2.0.md +96 -0
- data/doc/securing_uploads.md +1 -1
- data/doc/storage/memory.md +19 -0
- data/doc/storage/s3.md +102 -81
- data/doc/testing.md +2 -2
- data/doc/upgrading_to_3.md +94 -32
- data/lib/shrine.rb +3 -2
- data/lib/shrine/attacher.rb +14 -9
- data/lib/shrine/plugins/add_metadata.rb +13 -0
- data/lib/shrine/plugins/derivatives.rb +8 -7
- data/lib/shrine/plugins/determine_mime_type.rb +3 -3
- data/lib/shrine/plugins/model.rb +1 -1
- data/lib/shrine/plugins/refresh_metadata.rb +2 -2
- data/lib/shrine/plugins/remove_invalid.rb +10 -5
- data/lib/shrine/plugins/type_predicates.rb +113 -0
- data/lib/shrine/plugins/validation.rb +8 -6
- data/lib/shrine/storage/s3.rb +34 -28
- data/lib/shrine/version.rb +1 -1
- metadata +7 -3
data/doc/metadata.md
CHANGED
@@ -15,6 +15,18 @@ uploaded_file.metadata #=>
|
|
15
15
|
# }
|
16
16
|
```
|
17
17
|
|
18
|
+
Under the hood, `Shrine#upload` calls `Shrine#extract_metadata`, which you can
|
19
|
+
also use directly to extract metadata from any IO object:
|
20
|
+
|
21
|
+
```rb
|
22
|
+
uploader.extract_metadata(io) #=>
|
23
|
+
# {
|
24
|
+
# "size" => 345993,
|
25
|
+
# "filename" => "matrix.mp4",
|
26
|
+
# "mime_type" => "video/mp4",
|
27
|
+
# }
|
28
|
+
```
|
29
|
+
|
18
30
|
The following metadata is extracted by default:
|
19
31
|
|
20
32
|
| Key | Default source |
|
@@ -23,7 +35,9 @@ The following metadata is extracted by default:
|
|
23
35
|
| `mime_type` | extracted from `io.content_type` |
|
24
36
|
| `size` | extracted from `io.size` |
|
25
37
|
|
26
|
-
|
38
|
+
## Accessing metadata
|
39
|
+
|
40
|
+
You can access the stored metadata in three ways:
|
27
41
|
|
28
42
|
```rb
|
29
43
|
# via methods (if they're defined)
|
@@ -42,17 +56,7 @@ uploaded_file["filename"]
|
|
42
56
|
uploaded_file["mime_type"]
|
43
57
|
```
|
44
58
|
|
45
|
-
|
46
|
-
also use directly to extract metadata from any IO object:
|
47
|
-
|
48
|
-
```rb
|
49
|
-
uploader.extract_metadata(io) #=>
|
50
|
-
# {
|
51
|
-
# "size" => 345993,
|
52
|
-
# "filename" => "matrix.mp4",
|
53
|
-
# "mime_type" => "video/mp4",
|
54
|
-
# }
|
55
|
-
```
|
59
|
+
## Controlling extraction
|
56
60
|
|
57
61
|
`Shrine#upload` accepts a `:metadata` option which accepts the following values:
|
58
62
|
|
@@ -85,11 +89,11 @@ By default, the `mime_type` metadata will be copied over from the
|
|
85
89
|
`#content_type` attribute of the input file (if present). However, since
|
86
90
|
`#content_type` value comes from the `Content-Type` header of the upload
|
87
91
|
request, it's *not guaranteed* to hold the actual MIME type of the file (browser
|
88
|
-
determines this header based on file extension).
|
89
|
-
|
90
|
-
`
|
91
|
-
|
92
|
-
|
92
|
+
determines this header based on file extension).
|
93
|
+
|
94
|
+
Moreover, only `ActionDispatch::Http::UploadedFile`, `Shrine::RackFile`, and
|
95
|
+
`Shrine::DataFile` objects have `#content_type` defined, so when uploading
|
96
|
+
objects such as `File`, the `mime_type` value will be nil by default.
|
93
97
|
|
94
98
|
To remedy that, Shrine comes with a
|
95
99
|
[`determine_mime_type`][determine_mime_type] plugin which is able to extract
|
@@ -112,15 +116,14 @@ You can choose different analyzers, and even mix-and-match them. See the
|
|
112
116
|
|
113
117
|
## Image Dimensions
|
114
118
|
|
115
|
-
Shrine comes with a `store_dimensions` plugin for
|
116
|
-
It adds `width` and `height` metadata values, and
|
117
|
-
`#height`, and `#dimensions` methods to the
|
118
|
-
|
119
|
-
have it use [MiniMagick] or [ruby-vips]:
|
119
|
+
Shrine comes with a [`store_dimensions`][store_dimensions] plugin for
|
120
|
+
extracting image dimensions. It adds `width` and `height` metadata values, and
|
121
|
+
also adds `#width`, `#height`, and `#dimensions` methods to the
|
122
|
+
`Shrine::UploadedFile` object.
|
120
123
|
|
121
124
|
```rb
|
122
125
|
# Gemfile
|
123
|
-
gem "fastimage"
|
126
|
+
gem "fastimage" # default analyzer
|
124
127
|
```
|
125
128
|
```rb
|
126
129
|
Shrine.plugin :store_dimensions
|
@@ -136,12 +139,17 @@ uploaded_file.height #=> 900
|
|
136
139
|
uploaded_file.dimensions #=> [1600, 900]
|
137
140
|
```
|
138
141
|
|
142
|
+
By default, the plugin uses [FastImage] to analyze dimensions, but you can also
|
143
|
+
have it use [MiniMagick] or [ruby-vips]. See the
|
144
|
+
[`store_dimensions`][store_dimensions] plugin docs for more details.
|
145
|
+
|
139
146
|
## Custom metadata
|
140
147
|
|
141
148
|
In addition to the built-in metadata, Shrine allows you to extract and store
|
142
|
-
any custom metadata, using the `add_metadata` plugin (which
|
143
|
-
`Shrine#extract_metadata`).
|
144
|
-
|
149
|
+
any custom metadata, using the [`add_metadata`][add_metadata] plugin (which
|
150
|
+
internally extends `Shrine#extract_metadata`).
|
151
|
+
|
152
|
+
For example, you might want to extract EXIF data from images:
|
145
153
|
|
146
154
|
```rb
|
147
155
|
# Gemfile
|
@@ -203,9 +211,20 @@ uploaded_file.metadata #=>
|
|
203
211
|
|
204
212
|
The yielded `io` object will not always be an object that responds to `#path`.
|
205
213
|
For example, with the `data_uri` plugin the `io` can be a `StringIO` wrapper,
|
206
|
-
with `restore_cached_data` or `refresh_metadata` plugins the `io` might
|
207
|
-
`Shrine::UploadedFile` object. So we're using `Shrine.with_file` to
|
208
|
-
have a file object.
|
214
|
+
while with `restore_cached_data` or `refresh_metadata` plugins the `io` might
|
215
|
+
be a `Shrine::UploadedFile` object. So, we're using `Shrine.with_file` to
|
216
|
+
ensure we have a file object.
|
217
|
+
|
218
|
+
### Adding metadata
|
219
|
+
|
220
|
+
If you wish to add metadata to an already attached file, you can do it as
|
221
|
+
follows:
|
222
|
+
|
223
|
+
```rb
|
224
|
+
photo.image_attacher.add_metadata("foo" => "bar")
|
225
|
+
photo.image.metadata #=> { ..., "foo" => "bar" }
|
226
|
+
photo.save # persist changes
|
227
|
+
```
|
209
228
|
|
210
229
|
## Metadata columns
|
211
230
|
|
@@ -225,15 +244,20 @@ photo.image_type #=> "image/jpeg"
|
|
225
244
|
When attaching files that were uploaded directly to the cloud or a [tus
|
226
245
|
server], Shrine won't automatically extract metadata from them, instead it will
|
227
246
|
copy any existing metadata that was set on the client side. The reason why this
|
228
|
-
is the default behaviour is because
|
229
|
-
|
230
|
-
|
231
|
-
|
247
|
+
is the default behaviour is because metadata extraction requires (at least
|
248
|
+
partially) retrieving file content from the storage, which could potentially be
|
249
|
+
expensive depending on the storage and the type of metadata being extracted.
|
250
|
+
|
251
|
+
```rb
|
252
|
+
# no additional metadata will be extracted in this assignment by default
|
253
|
+
photo.image = '{"id":"9e6581a4ea1.jpg","storage":"cache","metadata":{...}}'
|
254
|
+
```
|
232
255
|
|
233
|
-
|
234
|
-
|
235
|
-
you want to
|
236
|
-
|
256
|
+
### Extracting on attachment
|
257
|
+
|
258
|
+
If you want metadata to be automatically extracted on assignment (which is
|
259
|
+
useful if you want to validate the extracted metadata or have it immediately
|
260
|
+
available for any other reason), you can load the `restore_cached_data` plugin:
|
237
261
|
|
238
262
|
```rb
|
239
263
|
Shrine.plugin :restore_cached_data # automatically extract metadata from cached files on assignment
|
@@ -248,7 +272,9 @@ photo.image.metadata #=>
|
|
248
272
|
# }
|
249
273
|
```
|
250
274
|
|
251
|
-
###
|
275
|
+
### Extracting in the background
|
276
|
+
|
277
|
+
#### A) Extracting with promotion
|
252
278
|
|
253
279
|
If you're using [backgrounding], you can extract metadata during background
|
254
280
|
promotion using the `refresh_metadata` plugin (which the `restore_cached_data`
|
@@ -271,12 +297,14 @@ class PromoteJob
|
|
271
297
|
record = Object.const_get(record_class).find(record_id) # if using Active Record
|
272
298
|
|
273
299
|
attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
|
274
|
-
attacher.refresh_metadata!
|
300
|
+
attacher.refresh_metadata! # extract metadata
|
275
301
|
attacher.atomic_promote
|
276
302
|
end
|
277
303
|
end
|
278
304
|
```
|
279
305
|
|
306
|
+
#### B) Extracting separately from promotion
|
307
|
+
|
280
308
|
You can also extract metadata in the background separately from promotion:
|
281
309
|
|
282
310
|
```rb
|
@@ -303,11 +331,13 @@ class MetadataJob
|
|
303
331
|
end
|
304
332
|
```
|
305
333
|
|
334
|
+
### Combining foreground and background
|
335
|
+
|
306
336
|
If you have some metadata that you want to extract in the foreground and some
|
307
337
|
that you want to extract in the background, you can use the uploader context:
|
308
338
|
|
309
339
|
```rb
|
310
|
-
class
|
340
|
+
class VideoUploader < Shrine
|
311
341
|
plugin :add_metadata
|
312
342
|
|
313
343
|
add_metadata do |io, **options|
|
@@ -324,7 +354,7 @@ class MyUploader < Shrine
|
|
324
354
|
end
|
325
355
|
```
|
326
356
|
```rb
|
327
|
-
class
|
357
|
+
class PromoteJob
|
328
358
|
include Sidekiq::Worker
|
329
359
|
|
330
360
|
def perform(attacher_class, record_class, record_id, name, file_data)
|
@@ -332,12 +362,16 @@ class MetadataJob
|
|
332
362
|
record = Object.const_get(record_class).find(record_id) # if using Active Record
|
333
363
|
|
334
364
|
attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
|
335
|
-
attacher.refresh_metadata!(background: true)
|
336
|
-
attacher.
|
365
|
+
attacher.refresh_metadata!(background: true) # specify the flag
|
366
|
+
attacher.atomic_promote
|
337
367
|
end
|
338
368
|
end
|
339
369
|
```
|
340
370
|
|
371
|
+
Now triggering metadata extraction in the controller on attachment (using
|
372
|
+
`restore_cached_data` or `refresh_metadata` plugin) will skip the video
|
373
|
+
metadata block, which will be triggered later in the background job.
|
374
|
+
|
341
375
|
### Optimizations
|
342
376
|
|
343
377
|
If you want to do both metadata extraction and file processing during
|
@@ -389,4 +423,6 @@ end
|
|
389
423
|
[ruby-vips]: https://github.com/libvips/ruby-vips
|
390
424
|
[tus server]: https://github.com/janko/tus-ruby-server
|
391
425
|
[determine_mime_type]: https://shrinerb.com/docs/plugins/determine_mime_type
|
426
|
+
[store_dimensions]: https://shrinerb.com/docs/plugins/store_dimensions
|
427
|
+
[add_metadata]: https://shrinerb.com/docs/plugins/add_metadata
|
392
428
|
[backgrounding]: https://shrinerb.com/docs/plugins/backgrounding
|
data/doc/paperclip.md
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
---
|
2
|
-
title:
|
2
|
+
title: Upgrading from Paperclip
|
3
3
|
---
|
4
4
|
|
5
5
|
This guide is aimed at helping Paperclip users transition to Shrine, and it
|
@@ -136,7 +136,7 @@ into separate columns.
|
|
136
136
|
While Paperclip works only with Active Record, Shrine is designed to integrate
|
137
137
|
with any persistence library (there are integrations for [Active
|
138
138
|
Record][activerecord], [Sequel][sequel], [ROM][rom], [Hanami][hanami] and
|
139
|
-
[
|
139
|
+
[Mongoid][mongoid]), and can also be used standalone:
|
140
140
|
|
141
141
|
```rb
|
142
142
|
attacher = ImageUploader::Attacher.new
|
@@ -511,11 +511,20 @@ Alternatively, if you want to generate locations yourself you can override the
|
|
511
511
|
|
512
512
|
```rb
|
513
513
|
class ImageUploader < Shrine
|
514
|
-
def generate_location(io, **
|
515
|
-
|
514
|
+
def generate_location(io, record: nil, name: nil, **)
|
515
|
+
[ storage_key,
|
516
|
+
record && record.class.name.underscore,
|
517
|
+
record && record.id,
|
518
|
+
super,
|
519
|
+
io.original_filename ].compact.join("/")
|
516
520
|
end
|
517
521
|
end
|
518
522
|
```
|
523
|
+
```
|
524
|
+
cache/user/123/2feff8c724e7ce17/nature.jpg
|
525
|
+
store/user/456/7f99669fde1e01fc/kitten.jpg
|
526
|
+
...
|
527
|
+
```
|
519
528
|
|
520
529
|
#### `:validate_media_type`
|
521
530
|
|
data/doc/plugins/add_metadata.md
CHANGED
@@ -2,76 +2,110 @@
|
|
2
2
|
title: Add Metadata
|
3
3
|
---
|
4
4
|
|
5
|
-
The [`add_metadata`][add_metadata] plugin
|
6
|
-
|
5
|
+
The [`add_metadata`][add_metadata] plugin allows adding custom metadata to
|
6
|
+
uploaded files.
|
7
7
|
|
8
8
|
```rb
|
9
|
-
plugin :add_metadata
|
9
|
+
Shrine.plugin :add_metadata
|
10
|
+
```
|
11
|
+
|
12
|
+
## Metadata block
|
10
13
|
|
11
|
-
add_metadata
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
14
|
+
The `Shrine.add_metadata` method allows you to register a block that will get
|
15
|
+
executed on upload, where you can return custom metadata:
|
16
|
+
|
17
|
+
```rb
|
18
|
+
require "pdf-reader" # https://github.com/yob/pdf-reader
|
19
|
+
|
20
|
+
class PdfUploader < Shrine
|
21
|
+
add_metadata :page_count do |io|
|
22
|
+
reader = PDF::Reader.new(io)
|
23
|
+
reader.page_count
|
16
24
|
end
|
17
25
|
end
|
18
26
|
```
|
19
27
|
|
20
|
-
The above will add
|
21
|
-
reader method on `Shrine::UploadedFile`.
|
28
|
+
The above will add `page_count` key to the metadata hash, and also create the
|
29
|
+
`#page_count` reader method on the `Shrine::UploadedFile`.
|
22
30
|
|
23
31
|
```rb
|
24
|
-
|
32
|
+
uploaded_file.metadata["page_count"] #=> 30
|
25
33
|
# or
|
26
|
-
|
34
|
+
uploaded_file.page_count #=> 30
|
27
35
|
```
|
28
36
|
|
29
|
-
|
37
|
+
### Multiple values
|
30
38
|
|
31
39
|
You can also extract multiple metadata values at once, by using `add_metadata`
|
32
40
|
without an argument and returning a hash of metadata.
|
33
41
|
|
34
42
|
```rb
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
43
|
+
require "exif" # https://github.com/tonytonyjan/exif
|
44
|
+
|
45
|
+
class ImageUploader < Shrine
|
46
|
+
add_metadata do |io|
|
47
|
+
begin
|
48
|
+
data = Exif::Data.new(io)
|
49
|
+
rescue Exif::NotReadable # not a valid image
|
50
|
+
next {}
|
51
|
+
end
|
52
|
+
|
53
|
+
{ "date_time" => data.date_time,
|
54
|
+
"flash" => data.flash,
|
55
|
+
"focal_length" => data.focal_length,
|
56
|
+
"exposure_time" => data.exposure_time }
|
40
57
|
end
|
41
|
-
|
42
|
-
{ date_time: data.date_time,
|
43
|
-
flash: data.flash,
|
44
|
-
focal_length: data.focal_length,
|
45
|
-
exposure_time: data.exposure_time }
|
46
58
|
end
|
47
59
|
```
|
60
|
+
```rb
|
61
|
+
uploaded_file.metadata #=>
|
62
|
+
# {
|
63
|
+
# ...
|
64
|
+
# "date_time" => "2019:07:20 16:16:08",
|
65
|
+
# "flash" => 16,
|
66
|
+
# "focal_length" => 26/1,
|
67
|
+
# "exposure_time" => 1/500,
|
68
|
+
# }
|
69
|
+
```
|
48
70
|
|
49
71
|
In this case Shrine won't automatically create reader methods for the extracted
|
50
|
-
metadata
|
51
|
-
`#metadata_method`.
|
72
|
+
metadata, but you can create them via `Shrine.metadata_method`:
|
52
73
|
|
53
74
|
```rb
|
54
|
-
|
75
|
+
class ImageUploader < Shrine
|
76
|
+
# ...
|
77
|
+
metadata_method :date_time, :flash
|
78
|
+
end
|
79
|
+
```
|
80
|
+
```rb
|
81
|
+
uploaded_file.date_time #=> "2019:07:20 16:16:08"
|
82
|
+
uploaded_file.flash #=> 16
|
55
83
|
```
|
56
84
|
|
57
|
-
|
85
|
+
### Ensuring file
|
58
86
|
|
59
87
|
The `io` might not always be a file object, so if you're using an analyzer
|
60
88
|
which requires the source file to be on disk, you can use `Shrine.with_file` to
|
61
89
|
ensure you have a file object.
|
62
90
|
|
63
91
|
```rb
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
92
|
+
require "streamio-ffmpeg" # https://github.com/streamio/streamio-ffmpeg
|
93
|
+
|
94
|
+
class VideoUploader < Shrine
|
95
|
+
add_metadata do |io|
|
96
|
+
movie = Shrine.with_file(io) do |file|
|
97
|
+
FFMPEG::Movie.new(file.path)
|
98
|
+
end
|
99
|
+
|
100
|
+
{ "duration" => movie.duration,
|
101
|
+
"bitrate" => movie.bitrate,
|
102
|
+
"resolution" => movie.resolution,
|
103
|
+
"frame_rate" => movie.frame_rate }
|
104
|
+
end
|
71
105
|
end
|
72
106
|
```
|
73
107
|
|
74
|
-
|
108
|
+
### Uploader options
|
75
109
|
|
76
110
|
Uploader options are also yielded to the block, you can access them for more
|
77
111
|
context:
|
@@ -89,6 +123,8 @@ add_metadata do |io, **options|
|
|
89
123
|
end
|
90
124
|
```
|
91
125
|
|
126
|
+
#### Metadata
|
127
|
+
|
92
128
|
The `:metadata` option holds metadata that was extracted so far:
|
93
129
|
|
94
130
|
```rb
|
@@ -116,4 +152,25 @@ add_metadata :bar do |io, metadata:, **|
|
|
116
152
|
end
|
117
153
|
```
|
118
154
|
|
155
|
+
## Updating metadata
|
156
|
+
|
157
|
+
If you just wish to add some custom metadata to existing uploads, you can do it
|
158
|
+
with `UploadedFile#add_metadata` (and write the changes back to the model):
|
159
|
+
|
160
|
+
```rb
|
161
|
+
attacher.file.add_metadata("foo" => "bar")
|
162
|
+
attacher.write # write changes to the model attribute
|
163
|
+
|
164
|
+
attacher.file.metadata #=> { ..., "foo" => "bar" }
|
165
|
+
```
|
166
|
+
|
167
|
+
You can also use the `Attacher#add_metadata` shorthand, which also takes care
|
168
|
+
of syncing the model:
|
169
|
+
|
170
|
+
```rb
|
171
|
+
attacher.add_metadata("foo" => "bar")
|
172
|
+
|
173
|
+
attacher.file.metadata #=> { ..., "foo" => "bar" }
|
174
|
+
```
|
175
|
+
|
119
176
|
[add_metadata]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/add_metadata.rb
|
@@ -187,12 +187,22 @@ and make the execution synchronous, you can override them on the attacher level
|
|
187
187
|
and call the default behaviour:
|
188
188
|
|
189
189
|
```rb
|
190
|
-
photo.image_attacher.promote_block
|
191
|
-
photo.image_attacher.destroy_block
|
190
|
+
photo.image_attacher.promote_block { promote } # promote synchronously
|
191
|
+
photo.image_attacher.destroy_block { destroy } # destroy synchronously
|
192
192
|
|
193
193
|
# ... now promotion and deletion will be synchronous ...
|
194
194
|
```
|
195
195
|
|
196
|
+
You can also do this on the class level if you want to disable backgrounding
|
197
|
+
that was set up by a superclass:
|
198
|
+
|
199
|
+
```rb
|
200
|
+
class MyUploader < Shrine
|
201
|
+
Attacher.promote_block { promote } # promote synchronously
|
202
|
+
Attacher.destroy_block { destroy } # destroy synchronously
|
203
|
+
end
|
204
|
+
```
|
205
|
+
|
196
206
|
[backgrounding]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/backgrounding.rb
|
197
207
|
[derivatives]: https://shrinerb.com/docs/plugins/derivatives
|
198
208
|
[atomic_helpers]: https://shrinerb.com/docs/plugins/atomic_helpers
|