shrine 2.19.3 → 3.6.0

data/doc/plugins/derivation_endpoint.md

@@ -1,34 +1,12 @@
-# Derivation Endpoint
+---
+title: Derivation Endpoint
+---
 
 The [`derivation_endpoint`][derivation_endpoint] plugin provides a Rack app for
 dynamically processing uploaded files on request. This allows you to create
 URLs to files that might not have been generated yet, and have the endpoint
 process them on-the-fly.
 
-## Contents
-
-* [Quick start](#quick-start)
-* [How it works](#how-it-works)
-  - [Performance](#performance)
-* [Derivation response](#derivation-response)
-* [Dynamic settings](#dynamic-settings)
-* [Host](#host)
-* [Prefix](#prefix)
-* [Expiration](#expiration)
-* [Response headers](#response-headers)
-  - [Content Type](#content-type)
-  - [Content Disposition](#content-disposition)
-  - [Cache Control](#cache-control)
-* [Uploading](#uploading)
-  - [Redirecting](#redirecting)
-  - [Deleting derivatives](#deleting-derivatives)
-* [Cache busting](#cache-busting)
-* [Accessing source file](#accessing-source-file)
-* [Downloading](#downloading)
-  - [Skipping download](#skipping-download)
-* [Derivation API](#derivation-api)
-* [Plugin Options](#plugin-options)
-
 ## Quick start
 
 We first load the plugin, providing a secret key and a path prefix to where the
@@ -57,7 +35,7 @@ apply to an attached file. For example, we can generate image thumbnails using
 the [ImageProcessing] gem:
 
 ```rb
-gem "image_processing", "~> 1.2"
+gem "image_processing", "~> 1.8"
 ```
 ```rb
 require "image_processing/mini_magick"
@@ -76,7 +54,7 @@ Now we can generate "derivation" URLs from attached files, which on request
 will call the derivation block we defined.
 
 ```rb
-photo.image.derivation_url(:thumbnail, "600", "400")
+photo.image.derivation_url(:thumbnail, 600, 400)
 #=> "/derivations/image/thumbnail/600/400/eyJpZCI6ImZvbyIsInN0b3JhZ2UiOiJzdG9yZSJ9?signature=..."
 ```
 
@@ -102,23 +80,26 @@ derivation URLs, preventing potential DoS attacks.
 
 The derivation endpoint then extracts the source file data, derivation name and
 arguments from the request URL, and calls the corresponding derivation block,
-passing the downloaded source file and derivation arguments.
+passing the downloaded source file and derivation arguments. The derivation
+block is evaluated within the context of a
+[`Shrine::Derivation`](#derivation-api) instance.
 
 ```rb
 derivation :thumbnail do |file, arg1, arg2, ...|
+  self #=> #<Shrine::Derivation>
+
   file #=> #<Tempfile:...> (source file downloaded to disk)
   arg1 #=> "600" (first derivation argument in #derivation_url)
   arg2 #=> "400" (second derivation argument in #derivation_url)
 
   # ... do processing ...
 
-  # return result as a File/Tempfile object or String/Pathname path
+  # return result as a File or Tempfile object
 end
 ```
 
-The derivation block is expected to return the processed file is a
-`File`/`Tempfile` object or a `String`/`Pathname` path. The resulting file is
-then rendered in the HTTP response.
+The derivation block is expected to return the processed file as a `File` or
+`Tempfile` object. The resulting file is then rendered in the HTTP response.
 
 ### Performance
 
@@ -216,7 +197,8 @@ Rails.application.routes.draw do
     end
   end
 end
-
+```
+```rb
 # app/controllers/photos_controller.rb
 class PhotosController < ApplicationController
   def thumbnail
@@ -349,6 +331,29 @@ uploaded_file.derivation_url(:thumbnail, expires_in: 90)
 #=> ".../thumbnail/eyJpZCI6ImZvbyIsInN?expires_at=1547843568&signature=..."
 ```
 
+## Custom signer
+
+The derivation URLs are signed by default, and the signature is checked when
+the URLs are requested, which prevents tampering. If you have URL expiration
+turned on, this may prevent your CDN from caching the response.
+
+In this case, you may need to do custom CDN-specific URL signing. You can
+bypass Shrine's default signing by passing a custom signer:
+
+```rb
+require "aws-sdk-cloudfront"
+signer = Aws::CloudFront::UrlSigner.new(key_pair_id: "...", private_key: "...")
+
+plugin :derivation_endpoint,
+  expires_in: 90,
+  signer: -> (url, expires_in:) do
+    signer.signed_url(url, expires: Time.now.to_i + expires_in)
+  end
+```
+
+When `:signer` option is used, the `:secret_key` option is not required, as
+that secret is only used for default signing.
+
 ## Response headers
 
 ### Content Type
@@ -430,6 +435,8 @@ fetch the original uploaded file, call the derivation block, upload the
 derivative to the storage, and serve the derivative. If the derivative does
 exist on checking, the endpoint will download the derivative and serve it.
 
+### Upload location
+
 The default upload location for derivatives is `<source id>/<name>-<args>`.
 This can be changed with the `:upload_location` option:
 
@@ -445,6 +452,8 @@ response won't know the appropriate `Content-Type` header value to set, and the
 generic `application/octet-stream` will be used. It's recommended to use the
 [`:type`](#content-type) option to set the appropriate `Content-Type` value.
 
+### Upload storage
+
 The target storage used is the same as for the source uploaded file. The
 `:upload_storage` option can be used to specify a different Shrine storage:
 
@@ -453,6 +462,8 @@ plugin :derivation_endpoint, upload: true,
                              upload_storage: :thumbnail_storage
 ```
 
+### Upload options
+
 Additional storage-specific upload options can be passed via `:upload_options`:
 
 ```rb
@@ -460,6 +471,8 @@ plugin :derivation_endpoint, upload: true,
                              upload_options: { acl: "public-read" }
 ```
 
+### Upload open options
+
 Additional storage-specific download options for the uploaded derivation result
 can be passed via `:upload_open_options`:
 
@@ -480,8 +493,7 @@ plugin :derivation_endpoint, upload: true,
                              upload_redirect: true
 ```
 
-In that case additional storage-specific URL options can be passed in for the
-redirect URL:
+Additional storage-specific URL options can be passed in for the redirect URL:
 
 ```rb
 plugin :derivation_endpoint, upload: true,
@@ -489,20 +501,54 @@ plugin :derivation_endpoint, upload: true,
                              upload_redirect_url_options: { public: true }
 ```
 
-If you are using the local filesystem storage, then redirecting does not make
-sense.
+Note that redirecting only makes sense if you're using remote storage services
+such as AWS S3 or Google Cloud Storage.
 
 ### Deleting derivatives
 
-When you use the `:upload` options and upload the derivatives to storage, there
-will come a time when you need to delete the derivatives because the original
-file is being replaced, deleted, or some other reason. In this case, you can
-delete the derivatives before deleting the record or updating the record with
-the new original file.
+When the original attachment is deleted, its uploaded derivatives will not be
+automatically deleted, you will need to do the deletion manually. If you're
+using [backgrounding], you can do this in your `DestroyJob`.
+
+If your storage implements `#delete_prefixed`, and you're using the default
+[`:upload_location`](#upload-location), you can delete the directory containing
+derivatives:
+
+```rb
+class DestroyJob < ActiveJob::Base
+  def perform(attacher_class, data)
+    # ... destroy attached file ...
+
+    derivatives_directory = attacher.file.id + "/"
+    storage               = attacher.store.storage
+
+    storage.delete_prefixed(derivatives_directory)
+  end
+end
+```
+
+Alternatively, you can delete each derivative individually:
 
 ```rb
-# photo is the model and image is the file attachment
-photo.image.derivation(:thumbnail).delete
+class ImageUploader < Shrine
+  DERIVATIONS = [
+    [:thumbnail, 800, 800],
+    [:thumbnail, 600, 400],
+    [:thumbnail, 400, 300],
+    ...
+  ]
+end
+```
+```rb
+class DestroyJob < ActiveJob::Base
+  def perform(attacher_class, data)
+    # ... destroy attached file ...
+
+    attacher.shrine_class::DERIVATIONS.each do |args|
+      attacher.file.derivation(*args).delete
+    end
+  end
+end
 ```
 
 ## Cache busting
@@ -538,50 +584,38 @@ uploaded_file.derivation_url(:thumbnail, version: 1)
 
 ## Accessing source file
 
-If you want to access the source `UploadedFile` object when deriving, you can
-set `:include_uploaded_file` to `true`.
-
-```rb
-plugin :derivation_endpoint, include_uploaded_file: true
-```
-
-Now the source `UploadedFile` will be passed as the second argument of the
-derivation block:
+Inside the derivation block we can access the source `UploadedFile` object via
+`Shrine::Derivation#source`:
 
 ```rb
-derivation :thumbnail do |file, uploaded_file, width, height|
-  uploaded_file             #=> #<Shrine::UploadedFile>
-  uploaded_file.id          #=> "9a7d1bfdad24a76f9cfaff137fe1b5c7.jpg"
-  uploaded_file.storage_key #=> "store"
-  uploaded_file.metadata    #=> {}
+derivation :thumbnail do |file, width, height|
+  source             #=> #<Shrine::UploadedFile>
+  source.id          #=> "9a7d1bfdad24a76f9cfaff137fe1b5c7.jpg"
+  source.storage_key #=> :store
+  source.metadata    #=> {}
 
   # ...
 end
 ```
 
-By default original metadata that were extracted on attachment won't be
-available in the derivation block. This is because metadata we want to have
-available would need to be serialized into the derivation URL, which would make
-it longer. However, you can opt in for the metadata you need with the
-`:metadata` option:
+By default, when using the derivation endpoint, original metadata of the source
+file won't be available in the derivation block. This is because any metadata
+we would want to have available would need to be serialized into the derivation
+URL, which would make it longer. Instead, you can opt in for the metadata you
+want to have available:
 
 ```rb
 plugin :derivation_endpoint, metadata: ["filename", "mime_type"]
-```
-
-Now `filename` and `mime_type` metadata values will be available in the
-derivation block:
 
-```rb
-derivation :thumbnail do |file, uploaded_file, width, height|
-  uploaded_file.metadata #=>
+derivation :thumbnail do |file, width, height|
+  source.metadata #=>
   # {
   #  "filename" => "nature.jpg",
   #  "mime_type" => "image/jpeg"
   # }
 
-  uploaded_file.original_filename #=> "nature.jpg"
-  uploaded_file.mime_type         #=> "image/jpeg"
+  source.original_filename #=> "nature.jpg"
+  source.mime_type         #=> "image/jpeg"
 
   # ...
 end
@@ -602,17 +636,9 @@ plugin :derivation_endpoint, download_options: {
 }
 ```
 
-If the source file has been deleted, the error the storage raises when
-attempting to download it will be propagated by default. For
-`Shrine.derivation_endpoint` and `Shrine.derivation_response` you can have
-these errors converted to 404 responses by adding them to `:download_errors`:
-
-```rb
-plugin :derivation_endpoint, download_errors: [
-  Errno::ENOENT,             # raised by Shrine::Storage::FileSystem
-  Aws::S3::Errors::NotFound, # raised by Shrine::Storage::S3
-]
-```
+If the source file was not found, `Shrine::Derivation::SourceNotFound`
+exception is raised. In a derivation response this is converted into a `404 Not
+Found` response.
 
 ### Skipping download
 
@@ -621,15 +647,8 @@ disk for you, you can set `:download` to `false`.
 
 ```rb
 plugin :derivation_endpoint, download: false
-```
-
-In this case the `UploadedFile` object is yielded to the derivation block
-instead of the raw file:
-
-```rb
-derivation :thumbnail do |uploaded_file, width, height|
-  uploaded_file #=> #<Shrine::UploadedFile>
 
+derivation :thumbnail do |width, height| # source file is not downloaded
   # ...
 end
 ```
@@ -639,10 +658,10 @@ One use case for this is delegating processing to a 3rd-party service:
 ```rb
 require "down/http"
 
-derivation :thumbnail do |uploaded_file, width, height|
+derivation :thumbnail do |width, height|
   # generate the thumbnail using ImageOptim.com
   down = Down::Http.new(method: :post)
-  down.download("https://im2.io/<USERNAME>/#{width}x#{height}/#{uploaded_file.url}")
+  down.download("https://im2.io/<USERNAME>/#{width}x#{height}/#{source.url}")
 end
 ```
 
@@ -735,13 +754,20 @@ uploaded_file    #=> #<Shrine::UploadedFile>
 uploaded_file.id #=> "bcfd0d67e4a8ec2dc9a6d7ddcf3825a1/thumbnail-500-500"
 ```
 
-If not given any arguments, it generates the derivative before uploading it.
+It can also be called without arguments, in which case it will generate a new
+derivative and upload it.
+
+```rb
+derivation.upload # generates derivative and uploads it
+```
+
+Any additional options will be passed to the uploader.
 
 ### `#retrieve`
 
-`Derivation#retrieve` method returns the uploaded derivative file. If the file
-exists on the storage, it returns an `UploadedFile` object, otherwise `nil` is
-returned.
+`Derivation#retrieve` method returns `Shrine::UploadedFile` object pointing to
+the uploaded derivative if it exists. If the uploaded derivative does not
+exist, `nil` is returned.
 
 ```rb
 uploaded_file = derivation.retrieve
@@ -749,6 +775,18 @@ uploaded_file    #=> #<Shrine::UploadedFile>
 uploaded_file.id #=> "bcfd0d67e4a8ec2dc9a6d7ddcf3825a1/thumbnail-500-500"
 ```
 
+### `#opened`
+
+`Derivation#opened` method returns opened `Shrine::UploadedFile` object pointing
+to the uploaded derivative if it exists. If the uploaded derivative does not
+exist, `nil` is returned.
+
+```rb
+uploaded_file = derivation.opened
+uploaded_file    #=> #<Shrine::UploadedFile>
+uploaded_file.id #=> "bcfd0d67e4a8ec2dc9a6d7ddcf3825a1/thumbnail-500-500"
+```
+
 ### `#delete`
 
 `Derivation#delete` method deletes the uploaded derivative file from the
@@ -774,15 +812,14 @@ derivation.option(:upload_location)
 | `:cache_control`               | Hash of directives for the `Cache-Control` response header                                                                            | `{ public: true, max_age: 365*24*60*60 }`            |
 | `:disposition`                 | Whether the browser should attempt to render the derivative (`inline`) or prompt the user to download the file to disk (`attachment`) | `inline`                                             |
 | `:download`                    | Whether the source uploaded file should be downloaded to disk when the derivation block is called                                     | `true`                                               |
-| `:download_errors`             | List of error classes that will be converted to a `404 Not Found` response by the derivation endpoint                                 | `[]`                                                 |
 | `:download_options`            | Additional options to pass when downloading the source uploaded file                                                                  | `{}`                                                 |
 | `:expires_in`                  | Number of seconds after which the URL will not be available anymore                                                                   | `nil`                                                |
 | `:filename`                    | Filename the browser will assume when the derivative is downloaded to disk                                                            | `<name>-<args>-<source id basename>`                 |
-| `:host`                        | URL host to use when generated URLs                                                                                                   | `nil`                                                |
-| `:include_uploaded_file`       | Whether to include the source uploaded file in the derivation block arguments                                                         | `false`                                              |
+| `:host`                        | URL host to use when for URLs                                                                                                         | `nil`                                                |
 | `:metadata`                    | List of metadata keys the source uploaded file should include in the derivation block                                                 | `[]`                                                 |
 | `:prefix`                      | Path prefix added to the URLs                                                                                                         | `nil`                                                |
 | `:secret_key`                  | Key used to sign derivation URLs in order to prevent tampering                                                                        | required                                             |
+| `:signer`                      | Proc accepting URL and query params used for custom signing of URLs.                                                                  | `nil`                                                |
 | `:type`                        | Media type returned in the `Content-Type` response header in the derivation response                                                  | determined from derivative's extension when possible |
 | `:upload`                      | Whether the generated derivatives will be cached on the storage                                                                       | `false`                                              |
 | `:upload_location`             | Location to which the derivatives will be uploaded on the storage                                                                     | `<source id>/<name>-<args>`                          |
@@ -793,8 +830,56 @@ derivation.option(:upload_location)
 | `:upload_storage`              | Storage to which the derivations will be uploaded                                                                                     | same storage as the source file                      |
 | `:version`                     | Version number to append to the URL for cache busting                                                                                 | `nil`                                                |
 
-[derivation_endpoint]: /lib/shrine/plugins/derivation_endpoint.rb
+## Instrumentation
+
+If the `instrumentation` plugin has been loaded, the `determine_mime_type` plugin
+adds instrumentation around derivation processing.
+
+```rb
+# instrumentation plugin needs to be loaded *before* derivation_endpoint
+plugin :instrumentation
+plugin :derivation_endpoint
+```
+
+Derivation processing will trigger a `derivation.shrine` event with the
+following payload:
+
+| Key           | Description                                     |
+| :--           | :----                                           |
+| `:derivation` | `Shrine::Derivation` object for this processing |
+| `:uploader`   | The uploader class that sent the event          |
+
+A default log subscriber is added as well which logs these events:
+
+```
+Derivation (492ms) – {:name=>:thumbnail, :args=>[600, 600], :uploader=>Shrine}
+```
+
+You can also use your own log subscriber:
+
+```rb
+plugin :derivation_endpoint, log_subscriber: -> (event) {
+  Shrine.logger.info JSON.generate(
+    name:     event.name,
+    duration: event.duration,
+    name:     event[:derivation].name,
+    args:     event[:derivation].args,
+  )
+}
+```
+```
+{"name":"derivation","duration":492,"name":"thumbnail","args":[600,600],"uploader":"Shrine"}
+```
+
+Or disable logging altogether:
+
+```rb
+plugin :derivation_endpoint, log_subscriber: nil
+```
+
+[derivation_endpoint]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/derivation_endpoint.rb
 [ImageProcessing]: https://github.com/janko/image_processing
 [`Content-Type`]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type
 [`Content-Disposition`]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition
 [`Cache-Control`]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
+[backgrounding]: https://shrinerb.com/docs/plugins/backgrounding