shrine 3.0.0.beta2 → 3.0.0.beta3

data/doc/changing_location.md

@@ -1,31 +1,102 @@
-# Migrating to Different Location
+# Migrating File Locations
 
-You have a production app with already uploaded attachments. However, you've
-realized that the existing store folder structure for attachments isn't working
-for you.
+This guide shows how to migrate the location of uploaded files on the same 
+storage in production, with zero downtime.
 
-The first step is to change the location, by overriding `#generate_location` or
-with the pretty_location plugin, and deploy that change. This will make any new
-files upload to the desired location, attachments on old locations will still
-continue to work normally.
-
-The next step is to run a script that will move old files to new locations. The
-easiest way to do that is to reupload them and delete them. Shrine has a method
-exactly for that, `Attacher#promote`, which also handles the situation when
-someone attaches a new file during "moving" (since we're running this script on
-live production).
+Let's assume we have a `Photo` model with an `image` file attachment:
 
 ```rb
-Shrine.plugin :delete_promoted
+Shrine.plugin :activerecord
+```
+```rb
+class ImageUploader < Shrine
+  # ...
+end
+```
+```rb
+class Photo < ActiveRecord::Base
+  include ImageUploader::Attachment(:image)
+end
+```
+
+## 1. Update the location generation
 
-User.paged_each do |user|
-  attacher = user.avatar_attacher
-  attacher.promote(action: :migrate) if attacher.stored?
-  # use `attacher._promote(action: :migrate)` if you want promoting to be backgrounded
+Since Shrine generates the location only once during upload, it is safe to change
+the `Shrine#generate_location` method. All the existing files will still continue
+to work with the previously stored urls because the files have not been migrated.
+
+```rb
+class ImageUploader < Shrine
+  def generate_location(io, **options)
+    # change location generation
+  end
 end
 ```
 
-The `:action` is not mandatory, it's just for better introspection when
-monitoring background jobs and logs.
+We can now deploy this change to production so new file uploads will be stored in 
+the new location.
+
+## 2. Move existing files
+
+To move existing files to new location, run the following script. It fetches
+the photos in batches, downloads the image, and re-uploads it to the new location.
+We only need to migrate the files in `:store` storage need to be migrated as the files
+in `:cache` storage will be uploaded to the new location on promotion.
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored? # move only attachments uploaded to permanent storage
+
+  old_attacher = attacher.dup
+
+  attacher.set             attacher.upload(attacher.file)                    # reupload file
+  attacher.set_derivatives attacher.upload_derivatives(attacher.derivatives) # reupload derivatives if you have derivatives
+
+  begin
+    attacher.atomic_persist           # persist changes if attachment has not changed in the meantime
+    old_attacher.destroy              # delete files on old location
+  rescue Shrine::AttachmentChanged,   # attachment has changed during reuploading
+         ActiveRecord::RecordNotFound # record has been deleted during reuploading
+    attacher.destroy                  # delete now orphaned files
+  end
+end
+```
 
 Now all your existing attachments should be happily living on new locations.
+
+### Backgrounding
+
+For faster migration, we can also delay moving files into a background job:
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored? # move only attachments uploaded to permanent storage
+
+  MoveFilesJob.perform_later(
+    attacher.class,
+    attacher.record,
+    attacher.name,
+    attacher.file_data,
+  )
+end
+```
+```rb
+class MoveFilesJob < ActiveJob::Base
+  def perform(attacher_class, record, name, file_data)
+    attacher     = attacher_class.retrieve(model: record, name: name, file: file_data)
+    old_attacher = attacher.dup
+
+    attacher.set             attacher.upload(attacher.file)
+    attacher.set_derivatives attacher.upload_derivatives(attacher.derivatives)
+
+    attacher.atomic_persist
+    old_attacher.destroy
+  rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound
+    attacher&.destroy
+  end
+end
+```

data/doc/changing_storage.md

@@ -0,0 +1,107 @@
+# Migrating File Storage
+
+This guides shows how to move file attachments to a different storage in 
+production, with zero downtime.
+
+Let's assume we have a `Photo` model with an `image` file attachment stored
+in AWS S3 storage:
+
+```rb
+Shrine.storages = {
+  cache: Shrine::Storage::S3.new(...),
+  store: Shrine::Storage::S3.new(...),
+}
+
+Shrine.plugin :activerecord
+```
+```rb
+class ImageUploader < Shrine
+  # ...
+end
+```
+```rb
+class Photo < ActiveRecord::Base
+  include ImageUploader::Attachment(:image)
+end
+```
+
+Let's also assume that we're migrating from AWS S3 to Google Cloud Storage, and
+we've added the new storage to `Shrine.storages`:
+
+```rb
+Shrine.storages = {
+  ...
+  store: Shrine::Storage::S3.new(...),
+  gcs:   Shrine::Storage::GoogleCloudStorage.new(...),
+}
+```
+
+## 1. Mirror upload and delete operations
+
+The first step is to start mirroring uploads and deletes made on your current
+storage to the new storage. We can do this by loading the `mirroring` plugin:
+
+```rb
+Shrine.plugin :mirroring, mirror: { store: :gcs }
+```
+
+Put the above code in an initializer and deploy it.
+
+You can additionally delay the mirroring into a [background job][mirroring
+backgrounding] for better performance.
+
+## 2. Copy the files
+
+Next step is to copy all remaining files from current storage into the new
+storage using the following script. It fetches the photos in batches, downloads 
+the image, and re-uploads it to the new storage. 
+
+```rb
+Photo.find_each do |photo|
+  attacher = photo.image_attacher
+
+  next unless attacher.stored?
+
+  attacher.file.trigger_mirror_upload
+
+  # if using derivatives
+  attacher.map_derivative(attacher.derivatives) do |_, derivative|
+    derivative.trigger_mirror_upload
+  end
+end
+```
+
+Now the new storage should have all files the current storage has, and new
+uploads will continue being mirrored to the new storage.
+
+## 3. Update storage
+
+Once all the files are copied over to the new storage, everything should be
+ready for us to update the storage in the Shrine configuration. We can keep
+mirroring, in case the change would need to reverted.
+
+```rb
+Shrine.storages = {
+  ...
+  store: Shrine::Storage::GoogleCloudStorage.new(...),
+  s3:    Shrine::Storage::S3.new(...),
+}
+
+Shrine.plugin :mirroring, mirror: { store: :s3 } # mirror to :s3 storage
+```
+
+## 4. Remove mirroring
+
+Once everything is looking good, we can remove the mirroring:
+
+```diff
+Shrine.storages = {
+  ...
+  store: Shrine::Storage::GoogleCloudStorage.new(...),
+- s3:    Shrine::Storage::S3.new(...),
+}
+
+- Shrine.plugin :mirroring, mirror: { store: :s3 } # mirror to :s3 storage
+```
+
+[mirroring backgrounding]: /doc/plugins/mirroring.md#backgrounding

data/doc/creating_plugins.md

@@ -94,7 +94,7 @@ If your plugin depends on other plugins, you can load them inside of
 ```rb
 module MyPlugin
   def self.load_dependencies(uploader, *)
-    uploader.plugin :versions # depends on the versions plugin
+    uploader.plugin :derivatives # depends on the derivatives plugin
   end
 end
 ```

data/doc/design.md

@@ -63,14 +63,13 @@ the `#upload` storage method. First the storage needs to be registered under a
 name:
 
 ```rb
-Shrine.storages[:file_system] = Shrine::Storage::FileSystem.new("uploads")
+Shrine.storages[:disk] = Shrine::Storage::FileSystem.new("uploads")
 ```
 
-Now we can instantiate an uploader with this identifier and upload files:
+Now we can upload files to the registered storage:
 
 ```rb
-uploader = Shrine.new(:file_system)
-uploaded_file = uploader.upload(file)
+uploaded_file = Shrine.upload(file, :disk)
 uploaded_file #=> #<Shrine::UploadedFile>
 ```
 
@@ -155,14 +154,14 @@ A `Shrine::Attacher` is instantiated with a model instance and an attachment
 name (an "image" attachment will be saved to `image_data` field):
 
 ```rb
-attacher = Shrine::Attacher.new(photo, :image)
+attacher = Shrine::Attacher.from_model(photo, :image)
 
 attacher.assign(file)
-attacher.get #=> #<Shrine::UploadedFile>
+attacher.file #=> #<Shrine::UploadedFile @storage_key=:cache ...>
 attacher.record.image_data #=> "{\"storage\":\"cache\",\"id\":\"9260ea09d8effd.jpg\",\"metadata\":{...}}"
 
-attacher._promote
-attacher.get #=> #<Shrine::UploadedFile>
+attacher.finalize
+attacher.file #=> #<Shrine::UploadedFile @storage_key=:store ...>
 attacher.record.image_data #=> "{\"storage\":\"store\",\"id\":\"ksdf02lr9sf3la.jpg\",\"metadata\":{...}}"
 ```
 
@@ -196,7 +195,7 @@ We can include this module to a model:
 
 ```rb
 class Photo
-  include Shrine::Attachment.new(:image)
+  include Shrine::Attachment(:image)
 end
 ```
 ```rb

data/doc/direct_s3.md

@@ -308,8 +308,9 @@ backgrounding library to perform the job with a delay:
 ```rb
 Shrine.plugin :backgrounding
 
-Shrine::Attacher.promote do |data|
-  PromoteJob.perform_in(3, data) # tells a Sidekiq worker to perform in 3 seconds
+Shrine::Attacher.promote_block do
+  # tells a Sidekiq worker to perform in 3 seconds
+  PromoteJob.perform_in(3, self.class, record.class, record.id, name, file_data)
 end
 ```
 

data/doc/metadata.md

@@ -84,10 +84,10 @@ By default, the `mime_type` metadata will be copied over from the
 `#content_type` value comes from the `Content-Type` header of the upload
 request, it's *not guaranteed* to hold the actual MIME type of the file (browser
 determines this header based on file extension). Moreover, only
-`ActionDispatch::Http::UploadedFile`, `Shrine::Plugins::RackFile::UploadedFile`,
-and `Shrine::Plugins::DataUri::DataFile` objects have `#content_type` defined,
-so, when uploading simple file objects, `mime_type` will be nil. That makes
-relying on `#content_type` both a security risk and limiting.
+`ActionDispatch::Http::UploadedFile`, `Shrine::RackFile`, and
+`Shrine::DataFile` objects have `#content_type` defined, so, when uploading
+simple file objects, `mime_type` will be nil. That makes relying on
+`#content_type` both a security risk and limiting.
 
 To remedy that, Shrine comes with a
 [`determine_mime_type`][determine_mime_type] plugin which is able to extract
@@ -117,7 +117,11 @@ default, the plugin uses [FastImage] to analyze dimensions, but you can also
 have it use [MiniMagick] or [ruby-vips]:
 
 ```rb
-Shrine.plugin :store_dimensions, analyzer: :mini_magick
+# Gemfile
+gem "fastimage"
+```
+```rb
+Shrine.plugin :store_dimensions
 ```
 ```rb
 uploaded_file = uploader.upload(image)
@@ -138,18 +142,18 @@ any custom metadata, using the `add_metadata` plugin (which extends
 from images:
 
 ```rb
-require "mini_magick"
+# Gemfile
+gem "exiftool"
+```
+```rb
+require "exiftool"
 
 class ImageUploader < Shrine
   plugin :add_metadata
 
   add_metadata :exif do |io, context|
     Shrine.with_file(io) do |file|
-      begin
-        MiniMagick::Image.new(file.path).exif
-      rescue MiniMagick::Error
-        # not a valid image
-      end
+      Exiftool.new(file.path).to_hash
     end
   end
 end
@@ -163,6 +167,10 @@ uploaded_file.exif             #=> {...}
 Or, if you're uploading videos, you might want to extract some video-specific
 meatadata:
 
+```rb
+# Gemfile
+gem "streamio-ffmpeg"
+```
 ```rb
 require "streamio-ffmpeg"
 
@@ -192,11 +200,10 @@ uploaded_file.metadata #=>
 ```
 
 The yielded `io` object will not always be an object that responds to `#path`.
-If you're using the `data_uri` plugin, the `io` will be a `StringIO` wrapper.
-With `restore_cached_data` or `refresh_metadata` plugins, `io` might be a
-`Shrine::UploadedFile` object. If you're using a metadata analyzer that
-requires the source file to be on disk, you can use `Shrine.with_file` to
-ensure you have a file object.
+For example, with the `data_uri` plugin the `io` can be a `StringIO` wrapper,
+with `restore_cached_data` or `refresh_metadata` plugins the `io` might be a
+`Shrine::UploadedFile` object. So we're using `Shrine.with_file` to ensure we
+have a file object.
 
 ## Metadata columns
 
@@ -227,9 +234,7 @@ you want to validate the extracted metadata or have it immediately available
 for any other reason), you can load the `restore_cached_data` plugin:
 
 ```rb
-class ImageUploader < Shrine
-  plugin :restore_cached_data # automatically extract metadata from cached files on assignment
-end
+Shrine.plugin :restore_cached_data # automatically extract metadata from cached files on assignment
 ```
 ```rb
 photo.image = '{"id":"ks9elsd.jpg","storage":"cache","metadata":{}}' # metadata is extracted
@@ -241,100 +246,113 @@ photo.image.metadata #=>
 # }
 ```
 
-On the other hand, if you're using backgrounding, you can extract metadata
-during background promotion using the `refresh_metadata` plugin (which the
-`restore_cached_data` plugin uses internally):
+### Backgrounding
+
+If you're using [backgrounding], you can extract metadata during background
+promotion using the `refresh_metadata` plugin (which the `restore_cached_data`
+plugin uses internally):
 
 ```rb
-class ImageUploader < Shrine
-  plugin :refresh_metadata
-  plugin :processing
+Shrine.plugin :refresh_metadata # allow re-extracting metadata
+Shrine.plugin :backgrounding
+Shrine::Attacher.promote_block { PromoteJob.perform_later(self.class, record, name, file_data) }
+```
+```rb
+class PromoteJob < ActiveJob::Base
+  def perform(attacher_class, record, name, file_data)
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.refresh_metadata!
+    attacher.atomic_promote
+  end
+end
+```
 
-  # this will be called in the background if using backgrounding plugin
-  process(:store) do |io, context|
-    io.refresh_metadata!(context) # extracts metadata and updates `io.metadata`
-    io
+You can also extract metadata in the background separately from promotion:
+
+```rb
+MetadataJob.perform_later(
+  attacher.class,
+  attacher.record,
+  attacher.name,
+  attacher.file_data,
+)
+```
+```rb
+class MetadataJob < ActiveJob::Base
+  def perform(attacher_class, record, name, file_data)
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.refresh_metadata!
+    attacher.atomic_persist
   end
 end
 ```
 
-If you have metadata that is cheap to extract in the foreground, but also have
-additional metadata that can be extracted asynchronously, you can combine the
-two approaches. For example, if you're attaching video files, you might want to
-extract MIME type upfront and video-specific metadata in a background job, which
-can be done as follows (provided that `backgrounding` plugin is used):
+If you have some metadata that you want to extract in the foreground and some
+that you want to extract in the background, you can use the uploader context:
 
 ```rb
 class MyUploader < Shrine
-  plugin :determine_mime_type # this will be called in the foreground
-  plugin :restore_cached_data
-  plugin :refresh_metadata
   plugin :add_metadata
-  plugin :processing
 
-  # this will be called in the background if using backgrounding plugin
-  process(:store) do |io, context|
-    io.refresh_metadata!(context)
-    io
-  end
+  add_metadata do |io, **options|
+    next unless options[:background] # proceed only when `background: true` was specified
 
-  add_metadata do |io, context|
-    next unless context[:action] == :store # this will be the case during promotion
-
-    Shrine.with_file(io) do |file|
-      # example of metadata extraction
-      movie = FFMPEG::Movie.new(file.path) # uses the streamio-ffmpeg gem
+    # example of metadata extraction
+    movie = Shrine.with_file(io) { |file| FFMPEG::Movie.new(file.path) }
 
-      { "duration"   => movie.duration,
-        "bitrate"    => movie.bitrate,
-        "resolution" => movie.resolution,
-        "frame_rate" => movie.frame_rate }
-    end
+    { "duration"   => movie.duration,
+      "bitrate"    => movie.bitrate,
+      "resolution" => movie.resolution,
+      "frame_rate" => movie.frame_rate }
+  end
+end
+```
+```rb
+class MetadataJob < ActiveJob::Base
+  def perform(record, name, file_data)
+    attacher = Shrine::Attacher.retrieve(model: record, name: name, file: file_data)
+    attacher.refresh_metadata!(background: true)
+    attacher.atomic_persist
   end
 end
 ```
 
+### Optimizations
+
 If you want to do both metadata extraction and file processing during
 promotion, you can wrap both in an `UploadedFile#open` block to make
 sure the file content is retrieved from the storage only once.
 
 ```rb
-class MyUploader < Shrine
-  plugin :refresh_metadata
-  plugin :processing
-
-  process(:store) do |io, context|
-    io.open do |io, context|
-      io.refresh_metadata!(context)
+class PromoteJob < ActiveJob::Base
+  def perform(record, name, file_data)
+    attacher = Shrine::Attacher.retrieve(model: record, name: name, file: file_data)
 
-      original = io.download # reuses already open uploaded file
-      # ... processing ...
+    attacher.file.open do
+      attacher.refresh_metadata!
+      attacher.create_derivatives
     end
+
+    attacher.atomic_promote
   end
 end
 ```
 
-If you're dealing with large files, it's recommended to also use the `tempfile`
-plugin to make sure the same copy of the uploaded file is used for metadata
-extraction (`Shrine.with_file`) and processing (`UploadedFile#tempfile`).
+If you're dealing with large files and have metadata extractors that use
+`Shrine.with_file`, you might want to use the `tempfile` plugin to make sure
+the same copy of the uploaded file is reused for both metadata extraction and
+file processing.
 
 ```rb
 Shrine.plugin :tempfile # load it globally so that it overrides `Shrine.with_file`
 ```
 ```rb
-class MyUploader < Shrine
-  plugin :refresh_metadata
-  plugin :processing
-
-  process(:store) do |io, context|
-    io.open do |io, context|
-      io.refresh_metadata!(context)
-
-      original = io.tempfile # used the cached tempfile
-      # ... processing ...
-    end
-  end
+# ...
+attacher.file.open do
+  attacher.refresh_metadata!
+  attacher.create_derivatives(attacher.file.tempfile)
 end
+# ...
 ```
 
 [`file`]: http://linux.die.net/man/1/file
@@ -345,3 +363,4 @@ end
 [ruby-vips]: https://github.com/libvips/ruby-vips
 [tus server]: https://github.com/janko/tus-ruby-server
 [determine_mime_type]: /doc/plugins/determine_mime_type.md#readme
+[backgrounding]: /doc/plugins/backgrounding.md#readme