shrine 2.19.4 → 3.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +485 -43
- data/LICENSE.txt +1 -1
- data/README.md +81 -977
- data/doc/advantages.md +231 -204
- data/doc/attacher.md +304 -153
- data/doc/carrierwave.md +297 -226
- data/doc/changing_derivatives.md +308 -0
- data/doc/changing_location.md +102 -21
- data/doc/changing_storage.md +110 -0
- data/doc/creating_persistence_plugins.md +132 -0
- data/doc/creating_plugins.md +43 -23
- data/doc/creating_storages.md +19 -5
- data/doc/design.md +147 -97
- data/doc/direct_s3.md +38 -28
- data/doc/external/articles.md +63 -0
- data/doc/external/extensions.md +53 -0
- data/doc/external/misc.md +32 -0
- data/doc/getting_started.md +1115 -0
- data/doc/metadata.md +190 -109
- data/doc/multiple_files.md +62 -34
- data/doc/paperclip.md +384 -262
- data/doc/plugins/activerecord.md +177 -46
- data/doc/plugins/add_metadata.md +139 -38
- data/doc/plugins/atomic_helpers.md +217 -0
- data/doc/plugins/backgrounding.md +156 -98
- data/doc/plugins/cached_attachment_data.md +7 -5
- data/doc/plugins/column.md +121 -0
- data/doc/plugins/data_uri.md +23 -22
- data/doc/plugins/default_storage.md +36 -10
- data/doc/plugins/default_url.md +30 -13
- data/doc/plugins/delete_raw.md +4 -2
- data/doc/plugins/derivation_endpoint.md +162 -101
- data/doc/plugins/derivatives.md +829 -0
- data/doc/plugins/determine_mime_type.md +4 -2
- data/doc/plugins/download_endpoint.md +64 -8
- data/doc/plugins/dynamic_storage.md +5 -3
- data/doc/plugins/entity.md +263 -0
- data/doc/plugins/form_assign.md +55 -0
- data/doc/plugins/included.md +31 -8
- data/doc/plugins/infer_extension.md +21 -10
- data/doc/plugins/instrumentation.md +38 -16
- data/doc/plugins/keep_files.md +14 -17
- data/doc/plugins/metadata_attributes.md +42 -13
- data/doc/plugins/mirroring.md +118 -0
- data/doc/plugins/model.md +210 -0
- data/doc/plugins/module_include.md +4 -2
- data/doc/plugins/multi_cache.md +24 -0
- data/doc/plugins/persistence.md +101 -0
- data/doc/plugins/presign_endpoint.md +9 -4
- data/doc/plugins/pretty_location.md +16 -3
- data/doc/plugins/processing.md +4 -2
- data/doc/plugins/rack_file.md +8 -2
- data/doc/plugins/rack_response.md +6 -2
- data/doc/plugins/recache.md +4 -2
- data/doc/plugins/refresh_metadata.md +49 -9
- data/doc/plugins/remote_url.md +84 -47
- data/doc/plugins/remove_attachment.md +27 -6
- data/doc/plugins/remove_invalid.md +21 -6
- data/doc/plugins/restore_cached_data.md +11 -3
- data/doc/plugins/sequel.md +159 -35
- data/doc/plugins/signature.md +16 -5
- data/doc/plugins/store_dimensions.md +14 -2
- data/doc/plugins/tempfile.md +4 -2
- data/doc/plugins/type_predicates.md +96 -0
- data/doc/plugins/upload_endpoint.md +13 -13
- data/doc/plugins/upload_options.md +6 -4
- data/doc/plugins/{default_url_options.md → url_options.md} +9 -7
- data/doc/plugins/validation.md +97 -0
- data/doc/plugins/validation_helpers.md +16 -13
- data/doc/plugins/versions.md +15 -19
- data/doc/processing.md +438 -221
- data/doc/refile.md +185 -167
- data/doc/release_notes/1.0.0.md +4 -0
- data/doc/release_notes/1.1.0.md +6 -2
- data/doc/release_notes/1.2.0.md +4 -0
- data/doc/release_notes/1.3.0.md +4 -0
- data/doc/release_notes/1.4.0.md +4 -0
- data/doc/release_notes/1.4.1.md +4 -0
- data/doc/release_notes/1.4.2.md +4 -0
- data/doc/release_notes/2.0.0.md +4 -0
- data/doc/release_notes/2.0.1.md +4 -0
- data/doc/release_notes/2.1.0.md +4 -0
- data/doc/release_notes/2.1.1.md +4 -0
- data/doc/release_notes/2.10.0.md +4 -0
- data/doc/release_notes/2.10.1.md +4 -0
- data/doc/release_notes/2.11.0.md +4 -0
- data/doc/release_notes/2.12.0.md +4 -0
- data/doc/release_notes/2.13.0.md +4 -0
- data/doc/release_notes/2.14.0.md +5 -1
- data/doc/release_notes/2.15.0.md +11 -7
- data/doc/release_notes/2.16.0.md +4 -0
- data/doc/release_notes/2.17.0.md +4 -0
- data/doc/release_notes/2.18.0.md +4 -0
- data/doc/release_notes/2.19.0.md +6 -3
- data/doc/release_notes/2.2.0.md +4 -0
- data/doc/release_notes/2.3.0.md +4 -0
- data/doc/release_notes/2.3.1.md +4 -0
- data/doc/release_notes/2.4.0.md +4 -0
- data/doc/release_notes/2.4.1.md +4 -0
- data/doc/release_notes/2.5.0.md +4 -0
- data/doc/release_notes/2.6.0.md +4 -0
- data/doc/release_notes/2.6.1.md +4 -0
- data/doc/release_notes/2.7.0.md +4 -0
- data/doc/release_notes/2.8.0.md +4 -0
- data/doc/release_notes/2.9.0.md +4 -0
- data/doc/release_notes/3.0.0.md +981 -0
- data/doc/release_notes/3.0.1.md +22 -0
- data/doc/release_notes/3.1.0.md +73 -0
- data/doc/release_notes/3.2.0.md +96 -0
- data/doc/release_notes/3.2.1.md +31 -0
- data/doc/release_notes/3.2.2.md +14 -0
- data/doc/release_notes/3.3.0.md +105 -0
- data/doc/release_notes/3.4.0.md +35 -0
- data/doc/retrieving_uploads.md +4 -1
- data/doc/securing_uploads.md +60 -37
- data/doc/storage/file_system.md +20 -3
- data/doc/storage/memory.md +19 -0
- data/doc/storage/s3.md +117 -83
- data/doc/testing.md +124 -144
- data/doc/upgrading_to_3.md +710 -0
- data/doc/validation.md +54 -90
- data/lib/shrine/attacher.rb +287 -171
- data/lib/shrine/attachment.rb +13 -46
- data/lib/shrine/plugins/_persistence.rb +93 -0
- data/lib/shrine/plugins/activerecord.rb +77 -34
- data/lib/shrine/plugins/add_metadata.rb +25 -17
- data/lib/shrine/plugins/atomic_helpers.rb +119 -0
- data/lib/shrine/plugins/backgrounding.rb +77 -113
- data/lib/shrine/plugins/cached_attachment_data.rb +6 -15
- data/lib/shrine/plugins/column.rb +102 -0
- data/lib/shrine/plugins/data_uri.rb +38 -36
- data/lib/shrine/plugins/default_storage.rb +45 -15
- data/lib/shrine/plugins/default_url.rb +12 -24
- data/lib/shrine/plugins/default_url_options.rb +3 -30
- data/lib/shrine/plugins/delete_raw.rb +10 -16
- data/lib/shrine/plugins/derivation_endpoint.rb +89 -134
- data/lib/shrine/plugins/derivatives.rb +637 -0
- data/lib/shrine/plugins/determine_mime_type.rb +9 -21
- data/lib/shrine/plugins/download_endpoint.rb +109 -133
- data/lib/shrine/plugins/dynamic_storage.rb +5 -11
- data/lib/shrine/plugins/entity.rb +152 -0
- data/lib/shrine/plugins/form_assign.rb +108 -0
- data/lib/shrine/plugins/included.rb +6 -6
- data/lib/shrine/plugins/infer_extension.rb +13 -20
- data/lib/shrine/plugins/instrumentation.rb +54 -42
- data/lib/shrine/plugins/keep_files.rb +3 -15
- data/lib/shrine/plugins/metadata_attributes.rb +28 -19
- data/lib/shrine/plugins/mirroring.rb +142 -0
- data/lib/shrine/plugins/model.rb +158 -0
- data/lib/shrine/plugins/module_include.rb +3 -3
- data/lib/shrine/plugins/multi_cache.rb +27 -0
- data/lib/shrine/plugins/presign_endpoint.rb +18 -22
- data/lib/shrine/plugins/pretty_location.rb +15 -9
- data/lib/shrine/plugins/processing.rb +22 -9
- data/lib/shrine/plugins/rack_file.rb +2 -42
- data/lib/shrine/plugins/rack_response.rb +15 -10
- data/lib/shrine/plugins/recache.rb +6 -5
- data/lib/shrine/plugins/refresh_metadata.rb +13 -11
- data/lib/shrine/plugins/remote_url.rb +49 -49
- data/lib/shrine/plugins/remove_attachment.rb +10 -6
- data/lib/shrine/plugins/remove_invalid.rb +19 -8
- data/lib/shrine/plugins/restore_cached_data.rb +13 -7
- data/lib/shrine/plugins/sequel.rb +86 -36
- data/lib/shrine/plugins/signature.rb +10 -16
- data/lib/shrine/plugins/store_dimensions.rb +35 -40
- data/lib/shrine/plugins/tempfile.rb +1 -3
- data/lib/shrine/plugins/type_predicates.rb +113 -0
- data/lib/shrine/plugins/upload_endpoint.rb +25 -23
- data/lib/shrine/plugins/upload_options.rb +14 -15
- data/lib/shrine/plugins/url_options.rb +31 -0
- data/lib/shrine/plugins/validation.rb +80 -0
- data/lib/shrine/plugins/validation_helpers.rb +34 -57
- data/lib/shrine/plugins/versions.rb +107 -87
- data/lib/shrine/plugins.rb +22 -0
- data/lib/shrine/storage/file_system.rb +46 -64
- data/lib/shrine/storage/linter.rb +42 -7
- data/lib/shrine/storage/memory.rb +49 -0
- data/lib/shrine/storage/s3.rb +154 -158
- data/lib/shrine/uploaded_file.rb +28 -30
- data/lib/shrine/version.rb +3 -3
- data/lib/shrine.rb +86 -149
- data/shrine.gemspec +9 -10
- metadata +79 -83
- data/doc/migrating_storage.md +0 -76
- data/doc/plugins/backup.md +0 -31
- data/doc/plugins/copy.md +0 -24
- data/doc/plugins/delete_promoted.md +0 -12
- data/doc/plugins/direct_upload.md +0 -172
- data/doc/plugins/hooks.md +0 -58
- data/doc/plugins/logging.md +0 -42
- data/doc/plugins/migration_helpers.md +0 -60
- data/doc/plugins/moving.md +0 -19
- data/doc/plugins/multi_delete.md +0 -20
- data/doc/plugins/parallelize.md +0 -16
- data/doc/plugins/parsed_json.md +0 -23
- data/doc/regenerating_versions.md +0 -143
- data/lib/shrine/plugins/background_helpers.rb +0 -5
- data/lib/shrine/plugins/backup.rb +0 -90
- data/lib/shrine/plugins/copy.rb +0 -50
- data/lib/shrine/plugins/delete_promoted.rb +0 -20
- data/lib/shrine/plugins/direct_upload.rb +0 -217
- data/lib/shrine/plugins/hooks.rb +0 -90
- data/lib/shrine/plugins/logging.rb +0 -142
- data/lib/shrine/plugins/migration_helpers.rb +0 -70
- data/lib/shrine/plugins/moving.rb +0 -57
- data/lib/shrine/plugins/multi_delete.rb +0 -32
- data/lib/shrine/plugins/parallelize.rb +0 -78
- data/lib/shrine/plugins/parsed_json.rb +0 -29
@@ -0,0 +1,308 @@
|
|
1
|
+
---
|
2
|
+
id: changing-derivatives
|
3
|
+
title: Managing Derivatives
|
4
|
+
---
|
5
|
+
|
6
|
+
This guide shows how to add, create, update, and remove [derivatives] for an
|
7
|
+
app in production already handling file attachments, with zero downtime.
|
8
|
+
|
9
|
+
Let's assume we have a `Photo` model with an `image` file attachment. The
|
10
|
+
examples will be showing image thumbnails, but the advice applies to any kind
|
11
|
+
of derivatives.
|
12
|
+
|
13
|
+
```rb
|
14
|
+
Shrine.plugin :derivatives
|
15
|
+
Shrine.plugin :activerecord
|
16
|
+
```
|
17
|
+
```rb
|
18
|
+
class ImageUploader < Shrine
|
19
|
+
# ...
|
20
|
+
end
|
21
|
+
```
|
22
|
+
```rb
|
23
|
+
class Photo < ActiveRecord::Base
|
24
|
+
include ImageUploader::Attachment(:image)
|
25
|
+
end
|
26
|
+
```
|
27
|
+
|
28
|
+
## Adding derivatives
|
29
|
+
|
30
|
+
*Scenario: Your app is currently working only with original files, and you want
|
31
|
+
to introduce derivatives.*
|
32
|
+
|
33
|
+
You'll first want to start creating the derivatives in production, without yet
|
34
|
+
generating URLs for them (because existing attachments won't yet have
|
35
|
+
derivatives generated). Let's assume you're generating image thumbnails:
|
36
|
+
|
37
|
+
```rb
|
38
|
+
# Gemfile
|
39
|
+
gem "image_processing", "~> 1.8"
|
40
|
+
```
|
41
|
+
```rb
|
42
|
+
require "image_processing/mini_magick"
|
43
|
+
|
44
|
+
class ImageUploader < Shrine
|
45
|
+
Attacher.derivatives do |original|
|
46
|
+
magick = ImageProcessing::MiniMagick.source(original)
|
47
|
+
|
48
|
+
# generate the thumbnails you want here
|
49
|
+
{
|
50
|
+
small: magick.resize_to_limit!(300, 300),
|
51
|
+
medium: magick.resize_to_limit!(500, 500),
|
52
|
+
large: magick.resize_to_limit!(800, 800),
|
53
|
+
}
|
54
|
+
end
|
55
|
+
end
|
56
|
+
```
|
57
|
+
```rb
|
58
|
+
photo = Photo.new(photo_params)
|
59
|
+
photo.image_derivatives! # generate derivatives
|
60
|
+
photo.save
|
61
|
+
```
|
62
|
+
|
63
|
+
Once we've deployed this to production, we can run the following script to
|
64
|
+
generate derivatives for all existing attachments in production. It fetches the
|
65
|
+
records in batches, downloads attachments on permanent storage, creates
|
66
|
+
derivatives, and persists the changes.
|
67
|
+
|
68
|
+
```rb
|
69
|
+
Photo.find_each do |photo|
|
70
|
+
attacher = photo.image_attacher
|
71
|
+
|
72
|
+
next unless attacher.stored?
|
73
|
+
|
74
|
+
attacher.create_derivatives
|
75
|
+
|
76
|
+
begin
|
77
|
+
attacher.atomic_persist # persist changes if attachment has not changed in the meantime
|
78
|
+
rescue Shrine::AttachmentChanged, # attachment has changed
|
79
|
+
ActiveRecord::RecordNotFound # record has been deleted
|
80
|
+
attacher.delete_derivatives # delete now orphaned derivatives
|
81
|
+
end
|
82
|
+
end
|
83
|
+
```
|
84
|
+
|
85
|
+
Now all attachments should have correctly generated derivatives. You can update
|
86
|
+
the attachment URLs to use derivatives as needed.
|
87
|
+
|
88
|
+
## Reprocessing all derivatives
|
89
|
+
|
90
|
+
*Scenario: The processing logic has changed for all or most derivatives, and
|
91
|
+
now you want to reprocess them for existing attachments.*
|
92
|
+
|
93
|
+
Let's assume we've made the following change and have deployed it to production:
|
94
|
+
|
95
|
+
```diff
|
96
|
+
Attacher.derivatives do |original|
|
97
|
+
magick = ImageProcessing::MiniMagick.source(original)
|
98
|
+
+ .saver(quality: 85)
|
99
|
+
|
100
|
+
{
|
101
|
+
small: magick.resize_to_limit!(300, 300),
|
102
|
+
medium: magick.resize_to_limit!(500, 500),
|
103
|
+
large: magick.resize_to_limit!(800, 800),
|
104
|
+
}
|
105
|
+
end
|
106
|
+
```
|
107
|
+
|
108
|
+
We can now run the following script to reprocess derivatives for all existing
|
109
|
+
records. It fetches the records in batches, downloads attachments on permanent
|
110
|
+
storage, reprocesses new derivatives, persists the changes, and deletes old
|
111
|
+
derivatives.
|
112
|
+
|
113
|
+
```rb
|
114
|
+
Photo.find_each do |photo|
|
115
|
+
attacher = photo.image_attacher
|
116
|
+
|
117
|
+
next unless attacher.stored?
|
118
|
+
|
119
|
+
old_derivatives = attacher.derivatives
|
120
|
+
|
121
|
+
attacher.set_derivatives({}) # clear derivatives
|
122
|
+
attacher.create_derivatives # reprocess derivatives
|
123
|
+
|
124
|
+
begin
|
125
|
+
attacher.atomic_persist # persist changes if attachment has not changed in the meantime
|
126
|
+
attacher.delete_derivatives(old_derivatives) # delete old derivatives
|
127
|
+
rescue Shrine::AttachmentChanged, # attachment has changed
|
128
|
+
ActiveRecord::RecordNotFound # record has been deleted
|
129
|
+
attacher.delete_derivatives # delete now orphaned derivatives
|
130
|
+
end
|
131
|
+
end
|
132
|
+
```
|
133
|
+
|
134
|
+
## Reprocessing certain derivatives
|
135
|
+
|
136
|
+
*Scenario: The processing logic has changed for specific derivatives, and now
|
137
|
+
you want to reprocess them for existing attachments.*
|
138
|
+
|
139
|
+
Let's assume we've made a following change and have deployed it to production:
|
140
|
+
|
141
|
+
```diff
|
142
|
+
Attacher.derivatives do |original|
|
143
|
+
magick = ImageProcessing::MiniMagick.source(original)
|
144
|
+
|
145
|
+
{
|
146
|
+
small: magick.resize_to_limit!(300, 300),
|
147
|
+
- medium: magick.resize_to_limit!(500, 500),
|
148
|
+
+ medium: magick.resize_to_limit!(600, 600),
|
149
|
+
large: magick.resize_to_limit!(800, 800),
|
150
|
+
}
|
151
|
+
end
|
152
|
+
```
|
153
|
+
|
154
|
+
We can now run the following script to reprocess the derivative for all
|
155
|
+
existing records. It fetches the records in batches, downloads attachments with
|
156
|
+
derivatives, reprocesses the specific derivative, persists the change, and
|
157
|
+
deletes old derivative.
|
158
|
+
|
159
|
+
```rb
|
160
|
+
Photo.find_each do |photo|
|
161
|
+
attacher = photo.image_attacher
|
162
|
+
|
163
|
+
next unless attacher.derivatives.key?(:medium)
|
164
|
+
|
165
|
+
old_medium = attacher.derivatives[:medium]
|
166
|
+
new_medium = attacher.file.download do |original|
|
167
|
+
ImageProcessing::MiniMagick
|
168
|
+
.source(original)
|
169
|
+
.resize_to_limit!(600, 600)
|
170
|
+
end
|
171
|
+
|
172
|
+
attacher.add_derivative(:medium, new_medium)
|
173
|
+
|
174
|
+
begin
|
175
|
+
attacher.atomic_persist # persist changes if attachment has not changed in the meantime
|
176
|
+
old_medium.delete # delete old derivative
|
177
|
+
rescue Shrine::AttachmentChanged, # attachment has changed
|
178
|
+
ActiveRecord::RecordNotFound # record has been deleted
|
179
|
+
attacher.derivatives[:medium].delete # delete now orphaned derivative
|
180
|
+
end
|
181
|
+
end
|
182
|
+
```
|
183
|
+
|
184
|
+
## Adding new derivatives
|
185
|
+
|
186
|
+
*Scenario: A new derivative has been added to the processor, and now
|
187
|
+
you want to add it to existing attachments.*
|
188
|
+
|
189
|
+
Let's assume we've made a following change and have deployed it to production:
|
190
|
+
|
191
|
+
```diff
|
192
|
+
Attacher.derivatives do |original|
|
193
|
+
magick = ImageProcessing::MiniMagick.source(original)
|
194
|
+
|
195
|
+
{
|
196
|
+
+ square: magick.resize_to_fill!(150, 150),
|
197
|
+
small: magick.resize_to_limit!(300, 300),
|
198
|
+
medium: magick.resize_to_limit!(600, 600),
|
199
|
+
large: magick.resize_to_limit!(800, 800),
|
200
|
+
}
|
201
|
+
end
|
202
|
+
```
|
203
|
+
|
204
|
+
We can now run following script to add the new derivative for all existing
|
205
|
+
records. It fetches the records in batches, downloads attachments on permanent
|
206
|
+
storage, creates the new derivative, and persists the changes.
|
207
|
+
|
208
|
+
```rb
|
209
|
+
Photo.find_each do |photo|
|
210
|
+
attacher = photo.image_attacher
|
211
|
+
|
212
|
+
next unless attacher.stored?
|
213
|
+
|
214
|
+
square = attacher.file.download do |original|
|
215
|
+
ImageProcessing::MiniMagick
|
216
|
+
.source(original)
|
217
|
+
.resize_to_fill!(150, 150)
|
218
|
+
end
|
219
|
+
|
220
|
+
attacher.add_derivative(:square, square)
|
221
|
+
|
222
|
+
begin
|
223
|
+
attacher.atomic_persist # persist changes if attachment has not changed in the meantime
|
224
|
+
rescue Shrine::AttachmentChanged, # attachment has changed
|
225
|
+
ActiveRecord::RecordNotFound # record has been deleted
|
226
|
+
attacher.derivatives[:square].delete # delete now orphaned derivative
|
227
|
+
end
|
228
|
+
end
|
229
|
+
```
|
230
|
+
|
231
|
+
Now all attachments should have the new derivative and you can start generating
|
232
|
+
URLs for it.
|
233
|
+
|
234
|
+
## Removing derivatives
|
235
|
+
|
236
|
+
*Scenario: A derivative isn't being used anymore, so we want to delete it for
|
237
|
+
existing attachments.*
|
238
|
+
|
239
|
+
Let's assume we've made the following change and have deployed it to production:
|
240
|
+
|
241
|
+
```diff
|
242
|
+
Attacher.derivatives do |original|
|
243
|
+
magick = ImageProcessing::MiniMagick.source(original)
|
244
|
+
|
245
|
+
{
|
246
|
+
- square: magick.resize_to_fill!(150, 150),
|
247
|
+
small: magick.resize_to_limit!(300, 300),
|
248
|
+
medium: magick.resize_to_limit!(600, 600),
|
249
|
+
large: magick.resize_to_limit!(800, 800),
|
250
|
+
}
|
251
|
+
end
|
252
|
+
```
|
253
|
+
|
254
|
+
We can now run following script to remove the unused derivative for all
|
255
|
+
existing record. It fetches the records in batches, removes and deletes the
|
256
|
+
unused derivative, and persists the changes.
|
257
|
+
|
258
|
+
```rb
|
259
|
+
Photo.find_each do |photo|
|
260
|
+
attacher = photo.image_attacher
|
261
|
+
|
262
|
+
next unless attacher.derivatives.key?(:square)
|
263
|
+
|
264
|
+
attacher.remove_derivative(:square, delete: true)
|
265
|
+
|
266
|
+
begin
|
267
|
+
attacher.atomic_persist # persist changes if attachment has not changed in the meantime
|
268
|
+
rescue Shrine::AttachmentChanged, # attachment has changed
|
269
|
+
ActiveRecord::RecordNotFound # record has been deleted
|
270
|
+
end
|
271
|
+
end
|
272
|
+
```
|
273
|
+
|
274
|
+
## Backgrounding
|
275
|
+
|
276
|
+
For faster migration, we can also delay any of the operations above into a
|
277
|
+
background job:
|
278
|
+
|
279
|
+
```rb
|
280
|
+
Photo.find_each do |photo|
|
281
|
+
attacher = photo.image_attacher
|
282
|
+
|
283
|
+
next unless attacher.stored?
|
284
|
+
|
285
|
+
MakeChangeJob.perform_async(
|
286
|
+
attacher.class.name,
|
287
|
+
attacher.record.class.name,
|
288
|
+
attacher.record.id,
|
289
|
+
attacher.name,
|
290
|
+
attacher.file_data,
|
291
|
+
)
|
292
|
+
end
|
293
|
+
```
|
294
|
+
```rb
|
295
|
+
class MakeChangeJob
|
296
|
+
include Sidekiq::Worker
|
297
|
+
|
298
|
+
def perform(attacher_class, record_class, record_id, name, file_data)
|
299
|
+
attacher_class = Object.const_get(attacher_class)
|
300
|
+
record = Object.const_get(record_class).find(record_id) # if using Active Record
|
301
|
+
|
302
|
+
attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
|
303
|
+
# ... make our change ...
|
304
|
+
end
|
305
|
+
end
|
306
|
+
```
|
307
|
+
|
308
|
+
[derivatives]: https://shrinerb.com/docs/plugins/derivatives
|
data/doc/changing_location.md
CHANGED
@@ -1,31 +1,112 @@
|
|
1
|
-
|
1
|
+
---
|
2
|
+
id: changing-location
|
3
|
+
title: Migrating File Locations
|
4
|
+
---
|
2
5
|
|
3
|
-
|
4
|
-
|
5
|
-
for you.
|
6
|
+
This guide shows how to migrate the location of uploaded files on the same
|
7
|
+
storage in production, with zero downtime.
|
6
8
|
|
7
|
-
|
8
|
-
with the pretty_location plugin, and deploy that change. This will make any new
|
9
|
-
files upload to the desired location, attachments on old locations will still
|
10
|
-
continue to work normally.
|
11
|
-
|
12
|
-
The next step is to run a script that will move old files to new locations. The
|
13
|
-
easiest way to do that is to reupload them and delete them. Shrine has a method
|
14
|
-
exactly for that, `Attacher#promote`, which also handles the situation when
|
15
|
-
someone attaches a new file during "moving" (since we're running this script on
|
16
|
-
live production).
|
9
|
+
Let's assume we have a `Photo` model with an `image` file attachment:
|
17
10
|
|
18
11
|
```rb
|
19
|
-
Shrine.plugin :
|
12
|
+
Shrine.plugin :activerecord
|
13
|
+
```
|
14
|
+
```rb
|
15
|
+
class ImageUploader < Shrine
|
16
|
+
# ...
|
17
|
+
end
|
18
|
+
```
|
19
|
+
```rb
|
20
|
+
class Photo < ActiveRecord::Base
|
21
|
+
include ImageUploader::Attachment(:image)
|
22
|
+
end
|
23
|
+
```
|
24
|
+
|
25
|
+
## 1. Update the location generation
|
20
26
|
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
27
|
+
Since Shrine generates the location only once during upload, it is safe to change
|
28
|
+
the `Shrine#generate_location` method. All the existing files will still continue
|
29
|
+
to work with the previously stored urls because the files have not been migrated.
|
30
|
+
|
31
|
+
```rb
|
32
|
+
class ImageUploader < Shrine
|
33
|
+
def generate_location(io, **options)
|
34
|
+
# change location generation
|
35
|
+
end
|
25
36
|
end
|
26
37
|
```
|
27
38
|
|
28
|
-
|
29
|
-
|
39
|
+
We can now deploy this change to production so new file uploads will be stored in
|
40
|
+
the new location.
|
41
|
+
|
42
|
+
## 2. Move existing files
|
43
|
+
|
44
|
+
To move existing files to new location, run the following script. It fetches
|
45
|
+
the photos in batches, downloads the image, and re-uploads it to the new location.
|
46
|
+
We only need to migrate the files in `:store` storage need to be migrated as the files
|
47
|
+
in `:cache` storage will be uploaded to the new location on promotion.
|
48
|
+
|
49
|
+
```rb
|
50
|
+
Photo.find_each do |photo|
|
51
|
+
attacher = photo.image_attacher
|
52
|
+
|
53
|
+
next unless attacher.stored? # move only attachments uploaded to permanent storage
|
54
|
+
|
55
|
+
old_attacher = attacher.dup
|
56
|
+
current_file = old_attacher.file
|
57
|
+
|
58
|
+
attacher.set attacher.upload(attacher.file) # reupload file
|
59
|
+
attacher.set_derivatives attacher.upload_derivatives(attacher.derivatives) # reupload derivatives if you have derivatives
|
60
|
+
|
61
|
+
begin
|
62
|
+
attacher.atomic_persist(current_file) # persist changes if attachment has not changed in the meantime
|
63
|
+
old_attacher.destroy_attached # delete files on old location
|
64
|
+
rescue Shrine::AttachmentChanged, # attachment has changed during reuploading
|
65
|
+
ActiveRecord::RecordNotFound # record has been deleted during reuploading
|
66
|
+
attacher.destroy_attached # delete now orphaned files
|
67
|
+
end
|
68
|
+
end
|
69
|
+
```
|
30
70
|
|
31
71
|
Now all your existing attachments should be happily living on new locations.
|
72
|
+
|
73
|
+
### Backgrounding
|
74
|
+
|
75
|
+
For faster migration, we can also delay moving files into a background job:
|
76
|
+
|
77
|
+
```rb
|
78
|
+
Photo.find_each do |photo|
|
79
|
+
attacher = photo.image_attacher
|
80
|
+
|
81
|
+
next unless attacher.stored? # move only attachments uploaded to permanent storage
|
82
|
+
|
83
|
+
MoveFilesJob.perform_async(
|
84
|
+
attacher.class.name,
|
85
|
+
attacher.record.class.name,
|
86
|
+
attacher.record.id,
|
87
|
+
attacher.name,
|
88
|
+
attacher.file_data,
|
89
|
+
)
|
90
|
+
end
|
91
|
+
```
|
92
|
+
```rb
|
93
|
+
class MoveFilesJob
|
94
|
+
include Sidekiq::Worker
|
95
|
+
|
96
|
+
def perform(attacher_class, record_class, record_id, name, file_data)
|
97
|
+
attacher_class = Object.const_get(attacher_class)
|
98
|
+
record = Object.const_get(record_class).find(record_id) # if using Active Record
|
99
|
+
|
100
|
+
attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
|
101
|
+
old_attacher = attacher.dup
|
102
|
+
|
103
|
+
attacher.set attacher.upload(attacher.file)
|
104
|
+
attacher.set_derivatives attacher.upload_derivatives(attacher.derivatives)
|
105
|
+
|
106
|
+
attacher.atomic_persist
|
107
|
+
old_attacher.destroy_attached
|
108
|
+
rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound
|
109
|
+
attacher&.destroy_attached
|
110
|
+
end
|
111
|
+
end
|
112
|
+
```
|
@@ -0,0 +1,110 @@
|
|
1
|
+
---
|
2
|
+
id: changing-storage
|
3
|
+
title: Migrating File Storage
|
4
|
+
---
|
5
|
+
|
6
|
+
This guides shows how to move file attachments to a different storage in
|
7
|
+
production, with zero downtime.
|
8
|
+
|
9
|
+
Let's assume we have a `Photo` model with an `image` file attachment stored
|
10
|
+
in AWS S3 storage:
|
11
|
+
|
12
|
+
```rb
|
13
|
+
Shrine.storages = {
|
14
|
+
cache: Shrine::Storage::S3.new(...),
|
15
|
+
store: Shrine::Storage::S3.new(...),
|
16
|
+
}
|
17
|
+
|
18
|
+
Shrine.plugin :activerecord
|
19
|
+
```
|
20
|
+
```rb
|
21
|
+
class ImageUploader < Shrine
|
22
|
+
# ...
|
23
|
+
end
|
24
|
+
```
|
25
|
+
```rb
|
26
|
+
class Photo < ActiveRecord::Base
|
27
|
+
include ImageUploader::Attachment(:image)
|
28
|
+
end
|
29
|
+
```
|
30
|
+
|
31
|
+
Let's also assume that we're migrating from AWS S3 to Google Cloud Storage, and
|
32
|
+
we've added the new storage to `Shrine.storages`:
|
33
|
+
|
34
|
+
```rb
|
35
|
+
Shrine.storages = {
|
36
|
+
...
|
37
|
+
store: Shrine::Storage::S3.new(...),
|
38
|
+
gcs: Shrine::Storage::GoogleCloudStorage.new(...),
|
39
|
+
}
|
40
|
+
```
|
41
|
+
|
42
|
+
## 1. Mirror upload and delete operations
|
43
|
+
|
44
|
+
The first step is to start mirroring uploads and deletes made on your current
|
45
|
+
storage to the new storage. We can do this by loading the `mirroring` plugin:
|
46
|
+
|
47
|
+
```rb
|
48
|
+
Shrine.plugin :mirroring, mirror: { store: :gcs }
|
49
|
+
```
|
50
|
+
|
51
|
+
Put the above code in an initializer and deploy it.
|
52
|
+
|
53
|
+
You can additionally delay the mirroring into a [background job][mirroring
|
54
|
+
backgrounding] for better performance.
|
55
|
+
|
56
|
+
## 2. Copy the files
|
57
|
+
|
58
|
+
Next step is to copy all remaining files from current storage into the new
|
59
|
+
storage using the following script. It fetches the photos in batches, downloads
|
60
|
+
the image, and re-uploads it to the new storage.
|
61
|
+
|
62
|
+
```rb
|
63
|
+
Photo.find_each do |photo|
|
64
|
+
attacher = photo.image_attacher
|
65
|
+
|
66
|
+
next unless attacher.stored?
|
67
|
+
|
68
|
+
attacher.file.trigger_mirror_upload
|
69
|
+
|
70
|
+
# if using derivatives
|
71
|
+
attacher.map_derivative(attacher.derivatives) do |_, derivative|
|
72
|
+
derivative.trigger_mirror_upload
|
73
|
+
end
|
74
|
+
end
|
75
|
+
```
|
76
|
+
|
77
|
+
Now the new storage should have all files the current storage has, and new
|
78
|
+
uploads will continue being mirrored to the new storage.
|
79
|
+
|
80
|
+
## 3. Update storage
|
81
|
+
|
82
|
+
Once all the files are copied over to the new storage, everything should be
|
83
|
+
ready for us to update the storage in the Shrine configuration. We can keep
|
84
|
+
mirroring, in case the change would need to reverted.
|
85
|
+
|
86
|
+
```rb
|
87
|
+
Shrine.storages = {
|
88
|
+
...
|
89
|
+
store: Shrine::Storage::GoogleCloudStorage.new(...),
|
90
|
+
s3: Shrine::Storage::S3.new(...),
|
91
|
+
}
|
92
|
+
|
93
|
+
Shrine.plugin :mirroring, mirror: { store: :s3 } # mirror to :s3 storage
|
94
|
+
```
|
95
|
+
|
96
|
+
## 4. Remove mirroring
|
97
|
+
|
98
|
+
Once everything is looking good, we can remove the mirroring:
|
99
|
+
|
100
|
+
```diff
|
101
|
+
Shrine.storages = {
|
102
|
+
...
|
103
|
+
store: Shrine::Storage::GoogleCloudStorage.new(...),
|
104
|
+
- s3: Shrine::Storage::S3.new(...),
|
105
|
+
}
|
106
|
+
|
107
|
+
- Shrine.plugin :mirroring, mirror: { store: :s3 } # mirror to :s3 storage
|
108
|
+
```
|
109
|
+
|
110
|
+
[mirroring backgrounding]: https://shrinerb.com/docs/plugins/mirroring#backgrounding
|
@@ -0,0 +1,132 @@
|
|
1
|
+
---
|
2
|
+
id: creating-persistence-plugins
|
3
|
+
title: Writing a Persistence Plugin
|
4
|
+
---
|
5
|
+
|
6
|
+
This guide explains some conventions for writing Shrine plugins that integrate
|
7
|
+
with persistence libraries such as Active Record, Sequel, ROM and Mongoid. It
|
8
|
+
assumes you've read the [Writing a Plugin] guide.
|
9
|
+
|
10
|
+
Let's say we're writing a plugin for a persistence library called "Raptor":
|
11
|
+
|
12
|
+
```rb
|
13
|
+
# lib/shrine/plugins/raptor.rb
|
14
|
+
|
15
|
+
require "raptor"
|
16
|
+
|
17
|
+
class Shrine
|
18
|
+
module Plugins
|
19
|
+
module Raptor
|
20
|
+
# ...
|
21
|
+
end
|
22
|
+
|
23
|
+
register_plugin(:raptor, Raptor)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
```
|
27
|
+
|
28
|
+
## Attachment
|
29
|
+
|
30
|
+
If your database library uses the [Active Record pattern], it's recommended to
|
31
|
+
load the [`model`][model] plugin as a dependency.
|
32
|
+
|
33
|
+
```rb
|
34
|
+
module Shrine::Plugins::Raptor
|
35
|
+
def self.load_dependencies(uploader, **)
|
36
|
+
uploader.plugin :model # for Active Record pattern
|
37
|
+
end
|
38
|
+
# ...
|
39
|
+
end
|
40
|
+
```
|
41
|
+
|
42
|
+
Otherwise if it uses the [Repository pattern], you can load the
|
43
|
+
[`entity`][entity] plugin as a dependency.
|
44
|
+
|
45
|
+
```rb
|
46
|
+
module Shrine::Plugins::Raptor
|
47
|
+
def self.load_dependencies(uploader, **)
|
48
|
+
uploader.plugin :entity # for Repository pattern
|
49
|
+
end
|
50
|
+
# ...
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
54
|
+
If you want to add library-specific integration when `Shrine::Attachment` is
|
55
|
+
included into a model/entity, it's recommended to do this in the `included`
|
56
|
+
hook, so that you can perform this logic only for models/entities that belong
|
57
|
+
to that persistence library.
|
58
|
+
|
59
|
+
```rb
|
60
|
+
module Shrine::Plugins::Raptor
|
61
|
+
# ...
|
62
|
+
module AttachmentMethods
|
63
|
+
def included(klass)
|
64
|
+
super
|
65
|
+
|
66
|
+
return unless klass < ::Raptor::Model
|
67
|
+
|
68
|
+
# library specific integration
|
69
|
+
end
|
70
|
+
end
|
71
|
+
# ...
|
72
|
+
end
|
73
|
+
```
|
74
|
+
|
75
|
+
## Attacher
|
76
|
+
|
77
|
+
To help define persistence methods on the `Attacher` according to the
|
78
|
+
convention, load the `_persistence` plugin as a dependency:
|
79
|
+
|
80
|
+
```rb
|
81
|
+
module Shrine::Plugins::Raptor
|
82
|
+
def self.load_dependencies(uploader, **)
|
83
|
+
# ...
|
84
|
+
uploader.plugin :_persistence, plugin: self
|
85
|
+
end
|
86
|
+
# ...
|
87
|
+
end
|
88
|
+
```
|
89
|
+
|
90
|
+
This will define the following attacher methods:
|
91
|
+
|
92
|
+
* `Attacher#persist`
|
93
|
+
* `Attacher#atomic_persist`
|
94
|
+
* `Attacher#atomic_promote`
|
95
|
+
|
96
|
+
For those methods to work, we'll need to implement the following methods:
|
97
|
+
|
98
|
+
* `Attacher#<library>_persist`
|
99
|
+
* `Attacher#<library>_reload`
|
100
|
+
* `Attacher#<library>?`
|
101
|
+
|
102
|
+
```rb
|
103
|
+
module Shrine::Plugins::Raptor
|
104
|
+
# ...
|
105
|
+
module AttacherMethods
|
106
|
+
# ...
|
107
|
+
private
|
108
|
+
|
109
|
+
def raptor_persist
|
110
|
+
# persist attached file to the record
|
111
|
+
end
|
112
|
+
|
113
|
+
def raptor_reload
|
114
|
+
# yield reloaded record (see atomic_helpers plugin)
|
115
|
+
end
|
116
|
+
|
117
|
+
def raptor?
|
118
|
+
# returns whether current model/entity belongs to Raptor
|
119
|
+
end
|
120
|
+
end
|
121
|
+
end
|
122
|
+
```
|
123
|
+
|
124
|
+
[Writing a Plugin]: https://shrinerb.com/docs/creating-plugins
|
125
|
+
[Active Record pattern]: https://www.martinfowler.com/eaaCatalog/activeRecord.html
|
126
|
+
[model]: https://shrinerb.com/docs/plugins/model
|
127
|
+
[entity]: https://shrinerb.com/docs/plugins/entity
|
128
|
+
[Repository pattern]: https://martinfowler.com/eaaCatalog/repository.html
|
129
|
+
[backgrounding]: https://shrinerb.com/docs/plugins/backgrounding
|
130
|
+
[atomic_helpers]: https://shrinerb.com/docs/plugins/atomic_helpers
|
131
|
+
[activerecord]: /lib/shrine/plugins/activerecord.rb
|
132
|
+
[sequel]: /lib/shrine/plugins/sequel.rb
|