shrine 2.19.3 → 3.6.0

data/doc/changing_derivatives.md

@@ -0,0 +1,308 @@
+---
+id: changing-derivatives
+title: Managing Derivatives
+---
+
+This guide shows how to add, create, update, and remove [derivatives] for an 
+app in production already handling file attachments, with zero downtime.
+
+Let's assume we have a `Photo` model with an `image` file attachment. The 
+examples will be showing image thumbnails, but the advice applies to any kind 
+of derivatives. 
+
+```rb
+Shrine.plugin :derivatives
+Shrine.plugin :activerecord
+```
+```rb
+class ImageUploader < Shrine
+  # ...
+end
+```
+```rb
+class Photo < ActiveRecord::Base
+  include ImageUploader::Attachment(:image)
+end
+```
+
+## Adding derivatives
+
+*Scenario: Your app is currently working only with original files, and you want
+to introduce derivatives.*
+
+You'll first want to start creating the derivatives in production, without yet
+generating URLs for them (because existing attachments won't yet have
+derivatives generated). Let's assume you're generating image thumbnails:
+
+```rb
+# Gemfile
+gem "image_processing", "~> 1.8"
+```
+```rb
+require "image_processing/mini_magick"
+
+class ImageUploader < Shrine
+  Attacher.derivatives do |original|
+    magick = ImageProcessing::MiniMagick.source(original)
+
+    # generate the thumbnails you want here
+    {
+      small:  magick.resize_to_limit!(300, 300),
+      medium: magick.resize_to_limit!(500, 500),
+      large:  magick.resize_to_limit!(800, 800),
+    }
+  end
+end
+```
+```rb
+photo = Photo.new(photo_params)
+photo.image_derivatives! # generate derivatives
+photo.save
+```
+
+Once we've deployed this to production, we can run the following script to
+generate derivatives for all existing attachments in production. It fetches the
+records in batches, downloads attachments on permanent storage, creates
+derivatives, and persists the changes.
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored?
+
+  attacher.create_derivatives
+
+  begin
+    attacher.atomic_persist            # persist changes if attachment has not changed in the meantime
+  rescue Shrine::AttachmentChanged,    # attachment has changed
+         ActiveRecord::RecordNotFound  # record has been deleted
+    attacher.delete_derivatives        # delete now orphaned derivatives
+  end
+end
+```
+
+Now all attachments should have correctly generated derivatives. You can update
+the attachment URLs to use derivatives as needed.
+
+## Reprocessing all derivatives
+
+*Scenario: The processing logic has changed for all or most derivatives, and
+now you want to reprocess them for existing attachments.*
+
+Let's assume we've made the following change and have deployed it to production:
+
+```diff
+ Attacher.derivatives do |original|
+   magick = ImageProcessing::MiniMagick.source(original)
++   .saver(quality: 85)
+
+   {
+     small:  magick.resize_to_limit!(300, 300),
+     medium: magick.resize_to_limit!(500, 500),
+     large:  magick.resize_to_limit!(800, 800),
+   }
+ end
+```
+
+We can now run the following script to reprocess derivatives for all existing
+records. It fetches the records in batches, downloads attachments on permanent
+storage, reprocesses new derivatives, persists the changes, and deletes old
+derivatives. 
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored?
+
+  old_derivatives = attacher.derivatives
+
+  attacher.set_derivatives({})                    # clear derivatives
+  attacher.create_derivatives                     # reprocess derivatives
+
+  begin
+    attacher.atomic_persist                       # persist changes if attachment has not changed in the meantime
+    attacher.delete_derivatives(old_derivatives)  # delete old derivatives
+  rescue Shrine::AttachmentChanged,               # attachment has changed
+         ActiveRecord::RecordNotFound             # record has been deleted
+    attacher.delete_derivatives                   # delete now orphaned derivatives
+  end
+end
+```
+
+## Reprocessing certain derivatives
+
+*Scenario: The processing logic has changed for specific derivatives, and now
+you want to reprocess them for existing attachments.*
+
+Let's assume we've made a following change and have deployed it to production:
+
+```diff
+ Attacher.derivatives do |original|
+   magick = ImageProcessing::MiniMagick.source(original)
+
+   {
+     small:  magick.resize_to_limit!(300, 300),
+-    medium: magick.resize_to_limit!(500, 500),
++    medium: magick.resize_to_limit!(600, 600),
+     large:  magick.resize_to_limit!(800, 800),
+   }
+ end
+```
+
+We can now run the following script to reprocess the derivative for all
+existing records. It fetches the records in batches, downloads attachments with
+derivatives, reprocesses the specific derivative, persists the change, and
+deletes old derivative.
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.derivatives.key?(:medium)
+
+  old_medium = attacher.derivatives[:medium]
+  new_medium = attacher.file.download do |original|
+    ImageProcessing::MiniMagick
+      .source(original)
+      .resize_to_limit!(600, 600)
+  end
+
+  attacher.add_derivative(:medium, new_medium)
+
+  begin
+    attacher.atomic_persist               # persist changes if attachment has not changed in the meantime
+    old_medium.delete                     # delete old derivative
+  rescue Shrine::AttachmentChanged,       # attachment has changed
+         ActiveRecord::RecordNotFound     # record has been deleted
+    attacher.derivatives[:medium].delete  # delete now orphaned derivative
+  end
+end
+```
+
+## Adding new derivatives
+
+*Scenario: A new derivative has been added to the processor, and now
+you want to add it to existing attachments.*
+
+Let's assume we've made a following change and have deployed it to production:
+
+```diff
+ Attacher.derivatives do |original|
+   magick = ImageProcessing::MiniMagick.source(original)
+
+   {
++    square: magick.resize_to_fill!(150, 150),
+     small:  magick.resize_to_limit!(300, 300),
+     medium: magick.resize_to_limit!(600, 600),
+     large:  magick.resize_to_limit!(800, 800),
+   }
+ end
+```
+
+We can now run following script to add the new derivative for all existing
+records. It fetches the records in batches, downloads attachments on permanent
+storage, creates the new derivative, and persists the changes.
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored?
+
+  square = attacher.file.download do |original|
+    ImageProcessing::MiniMagick
+      .source(original)
+      .resize_to_fill!(150, 150)
+  end
+
+  attacher.add_derivative(:square, square)
+
+  begin
+    attacher.atomic_persist               # persist changes if attachment has not changed in the meantime
+  rescue Shrine::AttachmentChanged,       # attachment has changed
+         ActiveRecord::RecordNotFound     # record has been deleted
+    attacher.derivatives[:square].delete  # delete now orphaned derivative
+  end
+end
+```
+
+Now all attachments should have the new derivative and you can start generating
+URLs for it.
+
+## Removing derivatives
+
+*Scenario: A derivative isn't being used anymore, so we want to delete it for
+existing attachments.*
+
+Let's assume we've made the following change and have deployed it to production:
+
+```diff
+ Attacher.derivatives do |original|
+   magick = ImageProcessing::MiniMagick.source(original)
+
+   {
+-    square: magick.resize_to_fill!(150, 150),
+     small:  magick.resize_to_limit!(300, 300),
+     medium: magick.resize_to_limit!(600, 600),
+     large:  magick.resize_to_limit!(800, 800),
+   }
+ end
+```
+
+We can now run following script to remove the unused derivative for all
+existing record. It fetches the records in batches, removes and deletes the
+unused derivative, and persists the changes.
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.derivatives.key?(:square)
+
+  attacher.remove_derivative(:square, delete: true)
+
+  begin
+    attacher.atomic_persist            # persist changes if attachment has not changed in the meantime
+  rescue Shrine::AttachmentChanged,    # attachment has changed
+         ActiveRecord::RecordNotFound  # record has been deleted
+  end
+end
+```
+
+## Backgrounding
+
+For faster migration, we can also delay any of the operations above into a
+background job:
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored?
+
+  MakeChangeJob.perform_async(
+    attacher.class.name,
+    attacher.record.class.name,
+    attacher.record.id,
+    attacher.name,
+    attacher.file_data,
+  )
+end
+```
+```rb
+class MakeChangeJob
+  include Sidekiq::Worker
+
+  def perform(attacher_class, record_class, record_id, name, file_data)
+    attacher_class = Object.const_get(attacher_class)
+    record         = Object.const_get(record_class).find(record_id) # if using Active Record
+
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    # ... make our change ...
+  end
+end
+```
+
+[derivatives]: https://shrinerb.com/docs/plugins/derivatives

data/doc/changing_location.md

@@ -1,31 +1,113 @@
-# Migrating to Different Location
+---
+id: changing-location
+title: Migrating File Locations
+---
 
-You have a production app with already uploaded attachments. However, you've
-realized that the existing store folder structure for attachments isn't working
-for you.
+This guide shows how to migrate the location of uploaded files on the same 
+storage in production, with zero downtime.
 
-The first step is to change the location, by overriding `#generate_location` or
-with the pretty_location plugin, and deploy that change. This will make any new
-files upload to the desired location, attachments on old locations will still
-continue to work normally.
-
-The next step is to run a script that will move old files to new locations. The
-easiest way to do that is to reupload them and delete them. Shrine has a method
-exactly for that, `Attacher#promote`, which also handles the situation when
-someone attaches a new file during "moving" (since we're running this script on
-live production).
+Let's assume we have a `Photo` model with an `image` file attachment:
 
 ```rb
-Shrine.plugin :delete_promoted
+Shrine.plugin :activerecord
+```
+```rb
+class ImageUploader < Shrine
+  # ...
+end
+```
+```rb
+class Photo < ActiveRecord::Base
+  include ImageUploader::Attachment(:image)
+end
+```
+
+## 1. Update the location generation
 
-User.paged_each do |user|
-  attacher = user.avatar_attacher
-  attacher.promote(action: :migrate) if attacher.stored?
-  # use `attacher._promote(action: :migrate)` if you want promoting to be backgrounded
+Since Shrine generates the location only once during upload, it is safe to change
+the `Shrine#generate_location` method. All the existing files will still continue
+to work with the previously stored urls because the files have not been migrated.
+
+```rb
+class ImageUploader < Shrine
+  def generate_location(io, **options)
+    # change location generation
+  end
 end
 ```
 
-The `:action` is not mandatory, it's just for better introspection when
-monitoring background jobs and logs.
+We can now deploy this change to production so new file uploads will be stored in 
+the new location.
+
+## 2. Move existing files
+
+To move existing files to new location, run the following script. It fetches
+the photos in batches, downloads the image, and re-uploads it to the new location.
+We only need to migrate the files in `:store` storage need to be migrated as the files
+in `:cache` storage will be uploaded to the new location on promotion.
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored? # move only attachments uploaded to permanent storage
+
+  old_attacher = attacher.dup
+  current_file = old_attacher.file
+
+  attacher.set             attacher.upload(attacher.file)                    # reupload file
+  attacher.set_derivatives attacher.upload_derivatives(attacher.derivatives) # reupload derivatives if you have derivatives
+
+  begin
+    attacher.atomic_persist(current_file) # persist changes if attachment has not changed in the meantime
+    old_attacher.destroy_attached         # delete files on old location
+  rescue Shrine::AttachmentChanged,       # attachment has changed during reuploading
+         ActiveRecord::RecordNotFound     # record has been deleted during reuploading
+    attacher.destroy_attached             # delete now orphaned files
+  end
+end
+```
 
 Now all your existing attachments should be happily living on new locations.
+
+### Backgrounding
+
+For faster migration, we can also delay moving files into a background job:
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored? # move only attachments uploaded to permanent storage
+
+  MoveFilesJob.perform_async(
+    attacher.class.name,
+    attacher.record.class.name,
+    attacher.record.id,
+    attacher.name,
+    attacher.file_data,
+  )
+end
+```
+```rb
+class MoveFilesJob
+  include Sidekiq::Worker
+
+  def perform(attacher_class, record_class, record_id, name, file_data)
+    attacher_class = Object.const_get(attacher_class)
+    record         = Object.const_get(record_class).find(record_id) # if using Active Record
+
+    attacher     = attacher_class.retrieve(model: record, name: name, file: file_data)
+    old_attacher = attacher.dup
+    current_file = old_attacher.file
+
+    attacher.set             attacher.upload(attacher.file)
+    attacher.set_derivatives attacher.upload_derivatives(attacher.derivatives)
+
+    attacher.atomic_persist(current_file)
+    old_attacher.destroy_attached
+  rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound
+    attacher&.destroy_attached
+  end
+end
+```

data/doc/changing_storage.md

@@ -0,0 +1,110 @@
+---
+id: changing-storage
+title: Migrating File Storage
+---
+
+This guides shows how to move file attachments to a different storage in 
+production, with zero downtime.
+
+Let's assume we have a `Photo` model with an `image` file attachment stored
+in AWS S3 storage:
+
+```rb
+Shrine.storages = {
+  cache: Shrine::Storage::S3.new(...),
+  store: Shrine::Storage::S3.new(...),
+}
+
+Shrine.plugin :activerecord
+```
+```rb
+class ImageUploader < Shrine
+  # ...
+end
+```
+```rb
+class Photo < ActiveRecord::Base
+  include ImageUploader::Attachment(:image)
+end
+```
+
+Let's also assume that we're migrating from AWS S3 to Google Cloud Storage, and
+we've added the new storage to `Shrine.storages`:
+
+```rb
+Shrine.storages = {
+  ...
+  store: Shrine::Storage::S3.new(...),
+  gcs:   Shrine::Storage::GoogleCloudStorage.new(...),
+}
+```
+
+## 1. Mirror upload and delete operations
+
+The first step is to start mirroring uploads and deletes made on your current
+storage to the new storage. We can do this by loading the `mirroring` plugin:
+
+```rb
+Shrine.plugin :mirroring, mirror: { store: :gcs }
+```
+
+Put the above code in an initializer and deploy it.
+
+You can additionally delay the mirroring into a [background job][mirroring
+backgrounding] for better performance.
+
+## 2. Copy the files
+
+Next step is to copy all remaining files from current storage into the new
+storage using the following script. It fetches the photos in batches, downloads 
+the image, and re-uploads it to the new storage. 
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored?
+
+  attacher.file.trigger_mirror_upload
+
+  # if using derivatives
+  attacher.map_derivative(attacher.derivatives) do |_, derivative|
+    derivative.trigger_mirror_upload
+  end
+end
+```
+
+Now the new storage should have all files the current storage has, and new
+uploads will continue being mirrored to the new storage.
+
+## 3. Update storage
+
+Once all the files are copied over to the new storage, everything should be
+ready for us to update the storage in the Shrine configuration. We can keep
+mirroring, in case the change would need to reverted.
+
+```rb
+Shrine.storages = {
+  ...
+  store: Shrine::Storage::GoogleCloudStorage.new(...),
+  s3:    Shrine::Storage::S3.new(...),
+}
+
+Shrine.plugin :mirroring, mirror: { store: :s3 } # mirror to :s3 storage
+```
+
+## 4. Remove mirroring
+
+Once everything is looking good, we can remove the mirroring:
+
+```diff
+Shrine.storages = {
+  ...
+  store: Shrine::Storage::GoogleCloudStorage.new(...),
+- s3:    Shrine::Storage::S3.new(...),
+}
+
+- Shrine.plugin :mirroring, mirror: { store: :s3 } # mirror to :s3 storage
+```
+
+[mirroring backgrounding]: https://shrinerb.com/docs/plugins/mirroring#backgrounding

data/doc/creating_persistence_plugins.md

@@ -0,0 +1,132 @@
+---
+id: creating-persistence-plugins
+title: Writing a Persistence Plugin
+---
+
+This guide explains some conventions for writing Shrine plugins that integrate
+with persistence libraries such as Active Record, Sequel, ROM and Mongoid. It
+assumes you've read the [Writing a Plugin] guide.
+
+Let's say we're writing a plugin for a persistence library called "Raptor":
+
+```rb
+# lib/shrine/plugins/raptor.rb
+
+require "raptor"
+
+class Shrine
+  module Plugins
+    module Raptor
+      # ...
+    end
+
+    register_plugin(:raptor, Raptor)
+  end
+end
+```
+
+## Attachment
+
+If your database library uses the [Active Record pattern], it's recommended to
+load the [`model`][model] plugin as a dependency.
+
+```rb
+module Shrine::Plugins::Raptor
+  def self.load_dependencies(uploader, **)
+    uploader.plugin :model # for Active Record pattern
+  end
+  # ...
+end
+```
+
+Otherwise if it uses the [Repository pattern], you can load the
+[`entity`][entity] plugin as a dependency.
+
+```rb
+module Shrine::Plugins::Raptor
+  def self.load_dependencies(uploader, **)
+    uploader.plugin :entity # for Repository pattern
+  end
+  # ...
+end
+```
+
+If you want to add library-specific integration when `Shrine::Attachment` is
+included into a model/entity, it's recommended to do this in the `included`
+hook, so that you can perform this logic only for models/entities that belong
+to that persistence library.
+
+```rb
+module Shrine::Plugins::Raptor
+  # ...
+  module AttachmentMethods
+    def included(klass)
+      super
+
+      return unless klass < ::Raptor::Model
+
+      # library specific integration
+    end
+  end
+  # ...
+end
+```
+
+## Attacher
+
+To help define persistence methods on the `Attacher` according to the
+convention, load the `_persistence` plugin as a dependency:
+
+```rb
+module Shrine::Plugins::Raptor
+  def self.load_dependencies(uploader, **)
+    # ...
+    uploader.plugin :_persistence, plugin: self
+  end
+  # ...
+end
+```
+
+This will define the following attacher methods:
+
+* `Attacher#persist`
+* `Attacher#atomic_persist`
+* `Attacher#atomic_promote`
+
+For those methods to work, we'll need to implement the following methods:
+
+* `Attacher#<library>_persist`
+* `Attacher#<library>_reload`
+* `Attacher#<library>?`
+
+```rb
+module Shrine::Plugins::Raptor
+  # ...
+  module AttacherMethods
+    # ...
+    private
+
+    def raptor_persist
+      # persist attached file to the record
+    end
+
+    def raptor_reload
+      # yield reloaded record (see atomic_helpers plugin)
+    end
+
+    def raptor?
+      # returns whether current model/entity belongs to Raptor
+    end
+  end
+end
+```
+
+[Writing a Plugin]: https://shrinerb.com/docs/creating-plugins
+[Active Record pattern]: https://www.martinfowler.com/eaaCatalog/activeRecord.html
+[model]: https://shrinerb.com/docs/plugins/model
+[entity]: https://shrinerb.com/docs/plugins/entity
+[Repository pattern]: https://martinfowler.com/eaaCatalog/repository.html
+[backgrounding]: https://shrinerb.com/docs/plugins/backgrounding
+[atomic_helpers]: https://shrinerb.com/docs/plugins/atomic_helpers
+[activerecord]: /lib/shrine/plugins/activerecord.rb
+[sequel]: /lib/shrine/plugins/sequel.rb