shrine 3.0.0.beta2 → 3.0.0.beta3

data/doc/plugins/remote_url.md

@@ -48,11 +48,26 @@ gem "http"
 ```rb
 require "down/http"
 
-down = Down::Http.new do |client|
-  client.follow(max_hops: 2).timeout(connect: 2, read: 2)
-end
+plugin :remote_url, downloader: -> (url, **options) {
+  Down::Http.download(url, **options) do |client|
+    client.follow(max_hops: 2).timeout(connect: 2, read: 2)
+  end
+}
+```
+
+Any `Down::NotFound` and `Down::TooLarge` exceptions will be rescued and
+converted into validation errors. If you want to convert any other exceptions
+into validation errors, you can raise them as
+`Shrine::Plugins::RemoteUrl::DownloadError`:
 
-plugin :remote_url, downloader: down.method(:download), ...
+```rb
+plugin :remote_url, downloader: -> (url, **options) {
+  begin
+    RestClient.get(url)
+  rescue RestClient::ExceptionWithResponse => error
+    raise Shrine::Plugins::RemoteUrl::DownloadError, "remote file not found"
+  end
+}
 ```
 
 ## Uploader options

data/doc/plugins/validation.md

@@ -0,0 +1,83 @@
+# Validation
+
+The [`validation`][validation] plugin provides a framework for validating
+attached files. For some useful validators, see the
+[`validation_helpers`][validation_helpers] plugin.
+
+```rb
+plugin :validation
+```
+
+The `Attacher.validate` method is used to register a validation block, which
+is called on attachment:
+
+```rb
+class VideoUploader < Shrine
+  Attacher.validate do
+    if file.duration > 5*60*60
+      errors << "duration must not be longer than 5 hours"
+    end
+  end
+end
+```
+```rb
+attacher.assign(file)
+attacher.errors #=> ["duration must not be longer than 5 hours"]
+```
+
+The validation block is executed in context of a `Shrine::Attacher` instance:
+
+```rb
+class VideoUploader < Shrine
+  Attacher.validate do
+    self #=> #<VideoUploader::Attacher>
+
+    record #=> #<Movie>
+    name   #=> :video
+    file   #=> #<Shrine::UploadedFile>
+  end
+end
+```
+
+## Inheritance
+
+If you're subclassing an uploader that has validations defined, you can call
+those validations via `super()`:
+
+```rb
+class ApplicationUploader < Shrine
+  Attacher.validate { validate_max_size 5.megabytes }
+end
+```
+```rb
+class ImageUploader < ApplicationUploader
+  Attacher.validate do
+    super() # empty parentheses are required
+    validate_mime_type %w[image/jpeg image/png image/webp]
+  end
+end
+```
+
+## Validation options
+
+You can pass options to the validator via the `:validate` option:
+
+```rb
+attacher.assign(file, validate: { foo: "bar" })
+```
+```rb
+class MyUploader < Shrine
+  Attacher.validate do |**options|
+    options #=> { foo: "bar" }
+  end
+end
+```
+
+You can also skip validation by passing `validate: false`:
+
+```rb
+attacher.assign(file, validate: false) # skips validation
+```
+
+[validation]: /lib/shrine/plugins/validation.rb
+[validation_helpers]: /doc/plugins/validation_helpers.md#readme

data/doc/processing.md

@@ -1,19 +1,9 @@
 # File Processing
 
-Shrine allows you to process files in two ways. One is processing "[on
-upload](#processing-on-upload)", where the processing gets triggered when the file is
-attached to a record. The other is "[on-the-fly](#on-the-fly-processing)"
-processing, where the processing is performed lazily at the moment the file is
-requested.
-
-With both ways you need to define some kind of processing block, which accepts
-a source file and is expected to return the processed result file.
-
-```rb
-some_process_block do |source_file|
- # process source file and return the result
-end
-```
+Shrine allows you to process attached files up front or on-the-fly. For
+example, if your app is accepting image uploads, you can generate a predefined
+set of of thumbnails when the image is attached to a record, or you can have
+thumbnails generated dynamically as they're needed.
 
 How you're going to implement processing is entirely up to you. For images it's
 recommended to use the **[ImageProcessing]** gem, which provides wrappers for
@@ -24,12 +14,10 @@ Here is an example of generating a thumbnail with ImageProcessing:
 ```sh
 $ brew install imagemagick
 ```
-
 ```rb
 # Gemfile
-gem "image_processing", "~> 1.0"
+gem "image_processing", "~> 1.8"
 ```
-
 ```rb
 require "image_processing/mini_magick"
 
@@ -40,107 +28,147 @@ thumbnail = ImageProcessing::MiniMagick
 thumbnail #=> #<Tempfile:...> (a 600x400 thumbnail of the source image)
 ```
 
-## Processing on upload
-
-Shrine allows you to process files before they're uploaded to a storage. It's
-generally best to process cached files when they're being promoted to permanent
-storage, because (a) at that point the file has already been successfully
-[validated][validation], (b) the parent record has been saved and the database
-transaction has been committed, and (c) this can be delayed into a [background
-job][backgrounding].
+## Processing up front
 
-You can define processing using the `processing` plugin, which we'll use to
-hook into the `:store` phase (when cached file is uploaded to permanent
-storage).
+Let's say we're handling images, and want to generate a predefined set of
+thumbnails with various dimensions. We can use the `derivatives` plugin to
+upload and save the processed files:
 
 ```rb
-class ImageUploader < Shrine
-  plugin :processing
-
-  process(:store) do |io, context|
-    io      #=> #<Shrine::UploadedFile ...>
-    context #=> {:record=>#<Photo...>,:name=>:image,...}
+Shrine.plugin :derivatives
+```
+```rb
+require "image_processing/mini_magick"
 
-    # ...
+class ImageUploader < Shrine
+  Attacher.derivatives_processor do |original|
+    magick = ImageProcessing::MiniMagick.source(original)
+
+    {
+      large:  magick.resize_to_limit!(800, 800),
+      medium: magick.resize_to_limit!(500, 500),
+      small:  magick.resize_to_limit!(300, 300),
+    }
   end
 end
 ```
+```rb
+photo = Photo.new
+photo.image_derivatives! # calls derivatives processor
+photo.save
+```
 
-The processing block yields two arguments: a [`Shrine::UploadedFile`] object
-representing the file uploaded to temporary storage, and a Hash containing
-additional data such as the model instance and attachment name. The block
-result should be file(s) that will be uploaded to permanent storage.
-
-### Versions
-
-Let's say we're handling images, and want to generate thumbnails of various
-dimensions. In this case we can use the ImageProcessing gem to generate the
-thumbnails, and return a hash of processed files at the end of the block. We'll
-need to load the `versions` plugin which extends Shrine with the ability to
-handle collections of files inside the same attachment.
+After the processed files are uploaded, their data is saved into the
+`<attachment>_data` column. You can then retrieve the derivatives as
+[`Shrine::UploadedFile`][uploaded file] objects:
 
 ```rb
-require "image_processing/mini_magick"
+photo.image(:large)            #=> #<Shrine::UploadedFile ...>
+photo.image(:large).url        #=> "/uploads/store/lg043.jpg"
+photo.image(:large).size       #=> 5825949
+photo.image(:large).mime_type  #=> "image/jpeg"
+```
 
-class ImageUploader < Shrine
-  plugin :processing # allows hooking into promoting
-  plugin :versions   # enable Shrine to handle a hash of files
-  plugin :delete_raw # delete processed files after uploading
+### Backgrounding
 
-  process(:store) do |io, context|
-    versions = { original: io } # retain original
+Since file processing can be time consuming, it's recommended to move it into a
+background job.
 
-    # download the uploaded file from the temporary storage
-    io.download do |original|
-      pipeline = ImageProcessing::MiniMagick.source(original)
+#### With promotion
 
-      versions[:large]  = pipeline.resize_to_limit!(800, 800)
-      versions[:medium] = pipeline.resize_to_limit!(500, 500)
-      versions[:small]  = pipeline.resize_to_limit!(300, 300)
-    end
+The simplest way is to use the [`backgrounding`][backgrounding] plugin to move
+promotion into a background job, and then create derivatives as part of
+promotion:
 
-    versions # return the hash of processed files
+```rb
+Shrine.plugin :backgrounding
+Shrine::Attacher.promote_block { PromoteJob.perform_later(self.class, record, name, file_data) }
+```
+```rb
+class PromoteJob < ActiveJob::Base
+  def perform(attacher_class, record, name, file_data)
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.create_derivatives # calls derivatives processor
+    attacher.atomic_promote
+  rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound
+    # attachment has changed or the record has been deleted, nothing to do
   end
 end
 ```
 
-**NOTE: It's recommended to always keep the original file, just in case you'll
-ever need to reprocess it.**
-
-### Conditional processing
-
-The process block yields the attached file uploaded to temporary storage, so we
-have information like file extension and MIME type available. Together with
-ImageProcessing's chainable API, it's easy to do conditional proccessing.
+#### Separate from promotion
 
-For example, let's say we want our thumbnails to be either JPEGs or PNGs, and
-we also want to save JPEGs as progressive (interlaced). Here's how the code for
-this might look like:
+Derivatives don't need to be created as part of the attachment flow, you can
+create them at any point after promotion:
 
 ```rb
-process(:store) do |io, context|
-  versions = { original: io }
+DerivativesJob.perform_later(
+  attacher.class,
+  attacher.record,
+  attacher.name,
+  attacher.file_data,
+)
+```
+```rb
+class DerivativesJob < ActiveJob::Base
+  def perform(attacher_class, record, name, file_data)
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.create_derivatives # calls derivatives processor
+    attacher.atomic_persist
+  rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound
+    attacher&.destroy_attached # delete now orphaned derivatives
+  end
+end
+```
 
-  io.download do |original|
-    pipeline = ImageProcessing::Vips.source(original)
+#### Concurrent processing
 
-    # Shrine::UploadedFile object contains information about the MIME type
-    unless io.mime_type == "image/png"
-      pipeline = pipeline
-        .convert("jpeg")
-        .saver(interlace: true)
-    end
+You can also generate derivatives concurrently:
 
-    versions[:large]  = pipeline.resize_to_limit!(800, 800)
-    versions[:medium] = pipeline.resize_to_limit!(500, 500)
-    versions[:small]  = pipeline.resize_to_limit!(300, 300)
+```rb
+class ImageUploader < Shrine
+  THUMBNAILS = {
+    large:  [800, 800],
+    medium: [500, 500],
+    small:  [300, 300],
+  }
+
+  Attacher.derivatives_processor do |original, name:|
+    thumbnail = ImageProcessing::MiniMagick
+      .source(original)
+      .resize_to_limit!(*THUMBNAILS.fetch(name))
+
+    { name => thumbnail }
+  end
+end
+```
+```rb
+ImageUploader::THUMBNAILS.each_key do |derivative_name|
+  DerivativeJob.perform_later(
+    attacher.class,
+    attacher.record,
+    attacher.name,
+    attacher.file_data,
+    derivative_name,
+  )
+end
+```
+```rb
+class DerivativeJob < ActiveJob::Base
+  def perform(attacher_class, record, name, file_data, derivative_name)
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.create_derivatives(name: derivative_name)
+    attacher.atomic_persist do |reloaded_attacher|
+      # make sure we don't override derivatives created in other jobs
+      attacher.merge_derivatives(reloaded_attacher.derivatives)
+    end
+  rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound
+    attacher.derivatives[derivative_name].delete # delete now orphaned derivative
   end
-
-  versions
 end
 ```
 
-### Processing other file types
+### Processing other filetypes
 
 So far we've only been talking about processing images. However, there is
 nothing image-specific in Shrine's processing API, you can just as well process
@@ -151,32 +179,23 @@ generic gem.
 To demonstrate, here is an example of transcoding videos using
 [streamio-ffmpeg]:
 
+```rb
+# Gemfile
+gem "streamio-ffmpeg"
+```
 ```rb
 require "streamio-ffmpeg"
-require "tempfile"
 
 class VideoUploader < Shrine
-  plugin :processing
-  plugin :versions
-  plugin :delete_raw
-
-  process(:store) do |io, context|
-    versions = { original: io }
+  Attacher.derivatives_processor do |original|
+    transcoded = Tempfile.new ["transcoded", ".mp4"]
+    screenshot = Tempfile.new ["screenshot", ".jpg"]
 
-    io.download do |original|
-      transcoded = Tempfile.new(["transcoded", ".mp4"], binmode: true)
-      screenshot = Tempfile.new(["screenshot", ".jpg"], binmode: true)
-
-      movie = FFMPEG::Movie.new(original.path)
-      movie.transcode(transcoded.path)
-      movie.screenshot(screenshot.path)
-
-      [transcoded, screenshot].each(&:open) # refresh file descriptors
-
-      versions.merge!(transcoded: transcoded, screenshot: screenshot)
-    end
+    movie = FFMPEG::Movie.new(original.path)
+    movie.transcode(transcoded.path)
+    movie.screenshot(screenshot.path)
 
-    versions
+    { transcoded: transcoded, screenshot: screenshot }
   end
 end
 ```
@@ -185,13 +204,13 @@ end
 
 Generating image thumbnails on upload can be a pain to maintain, because
 whenever you need to add a new version or change an existing one, you need to
-retroactively apply it to all existing uploads (see the [Reprocessing Versions]
+retroactively apply it to all existing uploads (see the [Managing Derivatives]
 guide for more details).
 
 As an alternative, it's very common to instead generate thumbnails dynamically
 as they're requested, and then cache them for future requests. This strategy is
-known as "on-the-fly processing", and it's suitable for generating thumbnails
-or document previews.
+known as "on-the-fly processing", and it's suitable for short-running
+processing such as creating image thumbnails or document previews.
 
 Shrine provides on-the-fly processing functionality via the
 [`derivation_endpoint`][derivation_endpoint] plugin. The basic setup is the
@@ -257,7 +276,7 @@ $ brew install vips
 
 ```rb
 # Gemfile
-gem "image_processing", "~> 1.0"
+gem "image_processing", "~> 1.8"
 ```
 
 ```rb
@@ -271,30 +290,27 @@ thumbnail = ImageProcessing::Vips
 thumbnail #=> #<Tempfile:...> (a 600x400 thumbnail of the source image)
 ```
 
-### Optimizing thumbnails
+### Parallelize uploading
 
-If you're generating image thumbnails, you can additionally use the
-[image_optim] gem to further reduce their filesize:
+If you're generating derivatives, you can parallelize the uploads using the
+[concurrent-ruby] gem:
 
 ```rb
 # Gemfile
-gem "image_processing", "~> 1.0"
-gem "image_optim"
-gem "image_optim_pack" # precompiled binaries
+gem "concurrent-ruby"
 ```
-
 ```rb
-require "image_processing/mini_magick"
+require "concurrent"
 
-thumbnail = ImageProcessing::MiniMagick
-  .source(image)
-  .resize_to_limit!(600, 400)
+derivatives = attacher.process_derivatives
 
-image_optim = ImageOptim.new
-image_optim.optimize_image!(thumbnail.path)
+tasks = derivatives.map do |name, file|
+  Concurrent::Promises.future(name, file) do |name, file|
+    attacher.add_derivative(name, file)
+  end
+end
 
-thumbnail.open # refresh file descriptor
-thumbnail
+Concurrent::Promises.zip(*tasks).wait!
 ```
 
 ### External processing
@@ -319,7 +335,7 @@ class ImageUploader < Shrine
     prefix:     "derivations/image",
     download:   false
 
-  derivation :thumbnail do |source, width, height|
+  derivation :thumbnail do |width, height|
     # generate thumbnails using ImageOptim.com
     down = Down::Http.new(method: :post)
     down.download("https://im2.io/<USERNAME>/#{width}x#{height}/#{source.url}")
@@ -359,7 +375,7 @@ Now when we upload our images to Cloudinary, we can generate URLs with various
 processing parameters:
 
 ```rb
-photo.image.url(width: 100, height: 100, crop: :fit)
+photo.image_url(width: 100, height: 100, crop: :fit)
 #=> "http://res.cloudinary.com/myapp/image/upload/w_100,h_100,c_fit/nature.jpg"
 ```
 
@@ -369,15 +385,15 @@ photo.image.url(width: 100, height: 100, crop: :fit)
 [GraphicsMagick]: http://www.graphicsmagick.org
 [libvips]: http://libvips.github.io/libvips/
 [Why is libvips quick]: https://github.com/libvips/libvips/wiki/Why-is-libvips-quick
-[image_optim]: https://github.com/toy/image_optim
 [ImageOptim.com]: https://imageoptim.com/api
 [streamio-ffmpeg]: https://github.com/streamio/streamio-ffmpeg
-[Reprocessing Versions]: /doc/regenerating_versions.md#readme
+[Managing Derivatives]: /doc/changing_derivatives.md#readme
 [Cloudinary]: https://cloudinary.com
 [shrine-cloudinary]: https://github.com/shrinerb/shrine-cloudinary
 [backgrounding]: /doc/plugins/backgrounding.md#readme
-[validation]: /doc/validation.md#readme
 [ruby-vips]: https://github.com/libvips/ruby-vips
 [MiniMagick]: https://github.com/minimagick/minimagick
 [derivation_endpoint]: /doc/plugins/derivation_endpoint.md#readme
 [derivation_endpoint performance]: /doc/plugins/derivation_endpoint.md#performance
+[derivatives]: /doc/plugins/derivatives.md#readme
+[concurrent-ruby]: https://github.com/ruby-concurrency/concurrent-ruby