shrine 2.14.0 → 2.15.0

data/README.md

@@ -6,7 +6,7 @@ Shrine is a toolkit for file attachments in Ruby applications. Some highlights:
 * **Memory friendly** – streaming uploads and downloads make it work great with large files
 * **Cloud storage** – store files on [disk][FileSystem], [AWS S3][S3], [Google Cloud][GCS], [Cloudinary] and others
 * **ORM integrations** – works with [Sequel][sequel plugin], [ActiveRecord][activerecord plugin], [Hanami::Model][hanami plugin] and [Mongoid][mongoid plugin]
-* **Flexible processing** – generate thumbnails with [ImageMagick] or [libvips] using the [ImageProcessing][image_processing] gem
+* **Flexible processing** – generate thumbnails [on upload] or [on-the-fly] using [ImageMagick][ImageProcessing::MiniMagick] or [libvips][ImageProcessing::Vips]
 * **Metadata validation** – [validate files][validation_helpers plugin] based on [extracted metadata][Extracting Metadata]
 * **Direct uploads** – upload asynchronously [to your app][upload_endpoint plugin] or [to the cloud][presign_endpoint plugin] using [Uppy]
 * **Resumable uploads** – make large file uploads [resumable][tus] by pointing [Uppy][uppy tus] to a [resumable endpoint][tus-ruby-server]
@@ -16,12 +16,14 @@ If you're curious how it compares to other file attachment libraries, see the [A
 
 ## Resources
 
-- Documentation: [shrinerb.com](https://shrinerb.com)
-- Demo code: [Roda][roda demo] / [Rails][rails demo]
-- Source: [github.com/shrinerb/shrine](https://github.com/shrinerb/shrine)
-- Wiki: [github.com/shrinerb/shrine/wiki](https://github.com/shrinerb/shrine/wiki)
-- Bugs: [github.com/shrinerb/shrine/issues](https://github.com/shrinerb/shrine/issues)
-- Help & Discussion: [groups.google.com/group/ruby-shrine](https://groups.google.com/forum/#!forum/ruby-shrine)
+| Resource          | URL                                                                                        |
+| :-------          | :---                                                                                       |
+| Website           | [shrinerb.com](https://shrinerb.com)                                                       |
+| Demo code         | [Roda][roda demo] / [Rails][rails demo]                                                    |
+| Source            | [github.com/shrinerb/shrine](https://github.com/shrinerb/shrine)                           |
+| Wiki              | [github.com/shrinerb/shrine/wiki](https://github.com/shrinerb/shrine/wiki)                 |
+| Bugs              | [github.com/shrinerb/shrine/issues](https://github.com/shrinerb/shrine/issues)             |
+| Help & Discussion | [groups.google.com/group/ruby-shrine](https://groups.google.com/forum/#!forum/ruby-shrine) |
 
 ## Quick start
 
@@ -509,13 +511,15 @@ uploaded_file.metadata["foo"]   #=> "bar"
 
 ## Processing
 
-Shrine's `processing` plugin allows you to intercept when the cached file is
-being uploaded to permanent storage, and do any file processing your might want.
+Shrine allows you to processing attached files either "on upload" or
+"on-the-fly". For example, if your app is accepting image uploads, you can
+generate a pre-defined set of of thumbnails as soon as the image is attached to
+a record ("on upload"), or you can generate necessary thumbnails dynamically as
+they're needed ("on-the-fly").
 
-If you're uploading images, it's common to want to generate various thumbnails.
-It's recommended to use the **[ImageProcessing][image_processing]** gem for
-this, which provides a convenient API over [ImageMagick] and [libvips]. You
-also need to load the `versions` plugin to be able to save multiple files.
+In both cases, for image processing you can use the **[ImageProcessing]** gem.
+It provides a convenient unified API for processing with [ImageMagick] or
+[libvips]. Here is an example of generating a thumbnail with ImageProcessing:
 
 ```sh
 $ brew install imagemagick
@@ -524,6 +528,28 @@ $ brew install imagemagick
 # Gemfile
 gem "image_processing", "~> 1.0"
 ```
+```rb
+require "image/mini_magick"
+
+thumbnail = ImageProcessing::MiniMagick
+  .source(original_image)
+  .resize_to_limit!(600, 400)
+
+thumbnail #=> #<Tempfile:...> (thumbnail limited to 600x400)
+```
+
+### On upload
+
+Shrine allows you intercept when a cached file is being uploaded to permanent
+storage, and perform any file processing you might want. The result of
+processing can also be multiple files, such as thumbnails of various
+dimensions. This processing can additionaly be delayed into a [background
+job](#backgrounding).
+
+The promotion hook is provided by the `processing` plugin, while the ability
+to save multiple files is provided by the `versions` plugin. Let's set up our
+uploader to generate some thumbnails from the attached image:
+
 ```rb
 require "image_processing/mini_magick"
 
@@ -553,6 +579,8 @@ After these files have been uploaded, their data will all be saved to the
 of `Shrine::UploadedFile` objects.
 
 ```rb
+photo = Photo.create(image: image)
+
 photo.image_data #=>
 # '{
 #   "original": {"id":"9sd84.jpg", "storage":"store", "metadata":{...}},
@@ -581,8 +609,53 @@ The `versions` plugin also expands `#<attachment>_url` to accept version names:
 photo.image_url(:large) #=> "https://..."
 ```
 
-For more details, including examples of how to do custom and on-the-fly
-processing, see the [File Processing] guide.
+For more details see the [File Processing] guide.
+
+### On-the-fly
+
+On-the-fly processing is provided by the
+[`derivation_endpoint`][derivation_endpoint plugin] plugin. It provides a
+mountable Rack app, which on request will call the processing we defined.
+
+We start by loading the plugin with a secret key and a path prefix to where
+we'll mount the Rack app, and defining a "derivation" we want the app to call:
+
+```rb
+require "image_processing/mini_magick"
+
+class ImageUploader < Shrine
+  plugin :derivation_endpoint,
+    secret_key: "<YOUR SECRET KEY>",
+    prefix:     "derivations/image"
+
+  derivation :thumbnail do |file, width, height|
+    ImageProcessing::MiniMagick
+      .source(file)
+      .resize_to_limit!(width.to_i, height.to_i)
+  end
+end
+```
+
+Then we can mount the Rack app into our router (for web frameworks other than
+Rails see [Mounting Endpoints] wiki):
+
+```rb
+# config/routes.rb (Rails)
+Rails.application.routes.draw do
+  mount ImageUploader.derivation_endpoint => "derivations/image"
+end
+```
+
+Now we can generate URLs from attached files that on request will call the
+processing we defined:
+
+```rb
+photo.image.derivation_url(:thumbnail, "600", "400")
+#=> "/derivations/image/thumbnail/600/400/eyJpZCI6ImZvbyIsInN0b3JhZ2UiOiJzdG9yZSJ9?signature=..."
+```
+
+The `derivation_endpoint` plugin is highly customizable, for more details see
+its [documentation][derivation_endpoint plugin].
 
 ## Context
 
@@ -638,8 +711,8 @@ user.valid? #=> false
 user.errors.to_hash #=> {:cv=>["is too large (max is 5 MB)"]}
 ```
 
-See the [File Validation] guide and `validation_helpers` plugin documentation
-for more details.
+See the [File Validation] guide and [`validation_helpers`][validation_helpers
+plugin] plugin documentation for more details.
 
 ## Location
 
@@ -699,8 +772,8 @@ server and attached to a record, just like with raw files. The only difference
 is that they won't be additionally uploaded to temporary storage on assignment,
 as they were already uploaded on the client side. Note that by default **Shrine
 won't extract metadata from directly uploaded files**, instead it will just copy
-metadata that was extacted on the client side; see [this section][metadata direct uploads]
-for the rationale and instructions on how to opt in.
+metadata that was extacted on the client side; see [this section][metadata
+direct uploads] for the rationale and instructions on how to opt in.
 
 For handling client side uploads it's recommended to use **[Uppy]**. Uppy is a
 very flexible modern JavaScript file upload library, which happens to integrate
@@ -711,22 +784,16 @@ nicely with Shrine.
 The simplest approach is creating an upload endpoint in your app that will
 receive uploads and forward them to the specified storage. You can use the
 `upload_endpoint` Shrine plugin to create a Rack app that handles uploads,
-and mount it inside your application.
+and mount it inside your application (for web frameworks other than Rails
+see [Mounting Endpoints] wiki).
 
 ```rb
 Shrine.plugin :upload_endpoint
 ```
 ```rb
-# config.ru (Rack)
-map "/images/upload" do
-  run ImageUploader.upload_endpoint(:cache)
-end
-
-# OR
-
 # config/routes.rb (Rails)
 Rails.application.routes.draw do
-  mount ImageUploader.upload_endpoint(:cache) => "/images/upload"
+  mount ImageUploader.upload_endpoint(:cache) => "images/upload"
 end
 ```
 
@@ -759,24 +826,20 @@ end
 If you want to free your app from receiving file uploads, you can also upload
 files directly to the cloud (AWS S3, Google Cloud etc). In this flow the client
 is required to first fetch upload parameters from the server, and then use these
-parameters to make the upload. The `presign_endpoint` Shrine plugin can be used
-to create a Rack app that generates these upload parameters (provided that the
-underlying storage implements `#presign`):
+parameters to make the upload.
+
+You can use the `presign_endpoint` Shrine plugin to create a Rack app that
+generates these upload parameters (provided that the underlying storage
+implements `#presign`), and mount it inside your application (for web
+frameworks other than Rails see [Mounting Endpoints] wiki):
 
 ```rb
 Shrine.plugin :presign_endpoint
 ```
 ```rb
-# config.ru (Rack)
-map "/s3/params" do
-  run Shrine.presign_endpoint(:cache)
-end
-
-# OR
-
 # config/routes.rb (Rails)
 Rails.application.routes.draw do
-  mount Shrine.presign_endpoint(:cache) => "/s3/params"
+  mount Shrine.presign_endpoint(:cache) => "s3/params"
 end
 ```
 
@@ -969,54 +1032,60 @@ The gem is available as open source under the terms of the [MIT License].
 
 [Shrine]: https://shrinerb.com
 [plugin system]: https://twin.github.io/the-plugin-system-of-sequel-and-roda/
-[FileSystem]: https://shrinerb.com/rdoc/classes/Shrine/Storage/FileSystem.html
-[S3]: https://shrinerb.com/rdoc/classes/Shrine/Storage/S3.html
+[FileSystem]: /doc/storage/file_system.md#readme
+[S3]: /doc/storage/s3.md#readme
 [GCS]: https://github.com/renchap/shrine-google_cloud_storage
 [Cloudinary]: https://github.com/shrinerb/shrine-cloudinary
 [Transloadit]: https://github.com/shrinerb/shrine-transloadit
-[activerecord plugin]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/Activerecord.html
-[sequel plugin]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/Sequel.html
+[activerecord plugin]: /doc/plugins/activerecord.md#readme
+[sequel plugin]: /doc/plugins/sequel.md#readme
 [hanami plugin]: https://github.com/katafrakt/hanami-shrine
 [mongoid plugin]: https://github.com/shrinerb/shrine-mongoid
-[image_processing]: https://github.com/janko-m/image_processing
-[ImageMagick]: https://www.imagemagick.org/script/index.php
+[ImageProcessing]: https://github.com/janko/image_processing
+[on upload]: #on-upload
+[on-the-fly]: #on-the-fly
+[ImageProcessing::MiniMagick]: https://github.com/janko/image_processing/blob/master/doc/minimagick.md#readme
+[ImageProcessing::Vips]: https://github.com/janko/image_processing/blob/master/doc/vips.md#readme
+[ImageMagick]: https://imagemagick.org/
 [libvips]: http://libvips.github.io/libvips/
-[validation_helpers plugin]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/ValidationHelpers.html
-[upload_endpoint plugin]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/UploadEndpoint.html
-[presign_endpoint plugin]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/PresignEndpoint.html
+[derivation_endpoint plugin]: /doc/plugins/derivation_endpoint.md#readme
+[validation_helpers plugin]: /doc/plugins/validation_helpers.md#readme
+[upload_endpoint plugin]: /doc/plugins/upload_endpoint.md#readme
+[presign_endpoint plugin]: /doc/plugins/presign_endpoint.md#readme
 [Uppy]: https://uppy.io
 [tus]: https://tus.io
 [uppy tus]: https://uppy.io/docs/tus/
-[tus-ruby-server]: https://github.com/janko-m/tus-ruby-server
-[backgrounding plugin]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/Backgrounding.html
-[Advantages of Shrine]: https://shrinerb.com/rdoc/files/doc/advantages_md.html
+[tus-ruby-server]: https://github.com/janko/tus-ruby-server
+[backgrounding plugin]: /doc/plugins/backgrounding.md#readme
+[Advantages of Shrine]: /doc/advantages.md#readme
 [external storages]: https://shrinerb.com/#external
-[creating storage]: https://shrinerb.com/rdoc/files/doc/creating_storages_md.html
-[creating plugin]: https://shrinerb.com/rdoc/files/doc/creating_plugins_md.html
-[Retrieving Uploads]: https://shrinerb.com/rdoc/files/doc/retrieving_uploads_md.html
-[Using Attacher]: https://shrinerb.com/rdoc/files/doc/attacher_md.html
+[creating storage]: /doc/creating_storages.md#readme
+[creating plugin]: /doc/creating_plugins.md#readme
+[Retrieving Uploads]: /doc/retrieving_uploads.md#readme
+[Using Attacher]: /doc/attacher.md#readme
 [plugins]: https://shrinerb.com/#plugins
 [`file`]: http://linux.die.net/man/1/file
-[Extracting Metadata]: https://shrinerb.com/rdoc/files/doc/metadata_md.html
-[File Processing]: https://shrinerb.com/rdoc/files/doc/processing_md.html
-[File Validation]: https://shrinerb.com/rdoc/files/doc/validation_md.html
-[metadata direct uploads]: https://github.com/shrinerb/shrine/blob/master/doc/metadata.md#direct-uploads
+[Extracting Metadata]: /doc/metadata.md#readme
+[File Processing]: /doc/processing.md#readme
+[File Validation]: /doc/validation.md#readme
+[metadata direct uploads]: /doc/metadata.md#direct-uploads
 [uppy xhr upload]: https://uppy.io/docs/xhr-upload/
 [direct uploads walkthrough]: https://github.com/shrinerb/shrine/wiki/Adding-Direct-App-Uploads
 [uppy aws s3]: https://uppy.io/docs/aws-s3/
 [direct S3 uploads walkthrough]: https://github.com/shrinerb/shrine/wiki/Adding-Direct-S3-Uploads<Paste>
-[direct S3 uploads guide]: https://shrinerb.com/rdoc/files/doc/direct_s3_md.html
+[direct S3 uploads guide]: /doc/direct_s3.md#readme
 [roda demo]: https://github.com/shrinerb/shrine/tree/master/demo
 [rails demo]: https://github.com/erikdahlstrand/shrine-rails-example
 [shrine-tus]: https://github.com/shrinerb/shrine-tus
 [uppy aws s3 multipart]: https://uppy.io/docs/aws-s3-multipart/
-[uppy-s3_multipart]: https://github.com/janko-m/uppy-s3_multipart
+[uppy-s3_multipart]: https://github.com/janko/uppy-s3_multipart
 [resumable uploads walkthrough]: https://github.com/shrinerb/shrine/wiki/Adding-Resumable-Uploads
 [resumable demo]: https://github.com/shrinerb/shrine-tus-demo
-[backgrounding libraries]: https://github.com/shrinerb/shrine/wiki/Backgrounding-libraries
+[backgrounding libraries]: https://github.com/shrinerb/shrine/wiki/Backgrounding-Libraries
+[Mounting Endpoints]: https://github.com/shrinerb/shrine/wiki/Mounting-Endpoints
 [S3 lifecycle Console]: http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
 [S3 lifecycle API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_lifecycle_configuration-instance_method
 [Roda]: https://github.com/jeremyevans/roda
 [Refile]: https://github.com/refile/refile
-[CoC]: https://github.com/shrinerb/shrine/blob/master/CODE_OF_CONDUCT.md
+[CoC]: CODE_OF_CONDUCT.md
 [MIT License]: http://opensource.org/licenses/MIT

data/doc/advantages.md

@@ -26,6 +26,7 @@ provided as plugins.
 Shrine.plugin :upload_endpoint
 Shrine.plugin :presign_endpoint
 Shrine.plugin :download_endpoint
+Shrine.plugin :derivation_endpoint
 Shrine.plugin :rack_response
 Shrine.plugin :rack_file
 
@@ -42,7 +43,7 @@ Shrine was designed with simplicity in mind. Where other solutions favour
 complex class-level DSLs, Shrine chooses simple instance-level interfaces where
 you can write regular Ruby code.
 
-There are no `CarrierWave::Uploader::Base` and `Paperclip::Attachment` [God
+There are no `CarrierWave::Uploader::Base` and `Paperclip::Attachment` [god
 objects], Shrine has several core classes each with clear responsibilities:
 
 * Storage classes encapsulate file operations for the underlying service
@@ -64,31 +65,53 @@ minimal amount of code.
 ## Modularity
 
 Shrine uses a [plugin system] that allows you to pick and choose the features
-that you want, which makes it very flexible. Moreover, you're only loading the
-code for features that you use, which means that Shrine will generally load
-very fast.
+that you want. Moreover, you're only loading the code for features that you
+use, which means that Shrine will generally load very fast.
 
 ```rb
-Shrine.plugin :logging # loads the logging feature
+Shrine.plugin :logging
+
+# translates to
+
+require "shrine/plugins/logging"
+Shrine.plugin Shrine::Plugins::Logging
 ```
 
 Shrine comes with a complete attachment functionality, but it also exposes many
 low level APIs that can be used for building your own customized attachment
-flow.
+flow. For example, if you prefer the `Attachment`/`Blob` architecture Active
+Storage provides, you can ditch the Shrine's attachment implementation and use
+uploaders and uploaded files that are decoupled from attachment:
+
+```rb
+uploader      = ImageUploader.new(:store)
+uploaded_file = uploader.upload(image) # metadata extraction, upload location generation
+
+uploaded_file.id       #=> "44ccafc10ce6a4ff22829e8f579ee6b9.jpg"
+uplaoded_file.metadata #=> { ... extracted metadata ... }
+
+data = uploaded_file.to_json # serialization
+# ...
+uploaded_file = ImageUploader.uploaded_file(data) # deserialization
+
+uploaded_file.url #=> "https://..."
+uploaded_file.download { |tempfile| ... } # streaming download
+uploaded_file.delete
+```
 
 ### Dependencies
 
-Shrine is very diligent when it comes to dependencies. It has only one
-mandatory dependency - [Down], a gem for streaming downloads from a URL. Some
-Shrine plugins require additional dependencies, but you only need to load them
-if you're using those plugins.
+Shrine is very diligent when it comes to dependencies. It has two mandatory
+dependencies – [Down] and [ContentDisposition] – which are loaded only by
+components that need them. Some Shrine plugins require additional dependencies,
+but you only need to load them if you're using those plugins.
 
-Moreover, Shrine often let you choose between multiple alternative dependencies
-for doing the same task. For example, the `determine_mime_type` plugin allows
-you to choose between the [`file`] command, [FileMagic], [FastImage],
-[MimeMagic], or [Marcel] gem for determining the MIME type, while the
-`store_dimensions` plugin can extract dimensions using [FastImage],
-[MiniMagick], or [ruby-vips] gem.
+Moreover, Shrine often gives you the ability choose between multiple
+alternative dependencies for doing the same task. For example, the
+`determine_mime_type` plugin allows you to choose between the [`file`] command,
+[FileMagic], [FastImage], [MimeMagic], or [Marcel] gem for determining the MIME
+type, while the `store_dimensions` plugin can extract dimensions using
+[FastImage], [MiniMagick], or [ruby-vips] gem.
 
 ```rb
 Shrine.plugin :determine_mime_type, analyzer: :marcel
@@ -104,9 +127,9 @@ Shrine is designed to handle any types of files. If you're accepting uploads of
 multiple types of files, such as videos and images, chances are that the logic
 for handling them will be very different:
 
-* images can be processed on-the-fly, while videos should be transcoded on upload
-* you might want to store images on one service and videos on another
-* tools for extracting image metadata are different than ones for video metadata
+* small images can be processed on-the-fly, but large files should be processed in a background job
+* which storage service is most suitable might depend on the filetype (images, documents, audios, videos)
+* different filetypes have different metadata to extract which require different tools
 
 With Shrine you can create isolated uploaders for each type of file. Plugins
 that you want to be applied to both uploaders can be applied globally, while
@@ -129,12 +152,63 @@ end
 
 ## Processing
 
-Instead of having yet another vendored solution for generating image
-thumbnails, Shrine chose to adopt a generic **[ImageProcessing]** gem. The
-ImageProcessing gem was created for Shrine, but it can be used in any other
-file upload library. It has a very flexible API and takes care of many details
-for you, such as [auto orienting] the input image and [sharpening] the
-thumbnails after they are resized.
+Most file attachment libraries give you the ability to process files either "on
+upload" (Paperclip, CarrierWave) or "on-the-fly" (Dragonfly, Refile, Active
+Storage). Having only one option is not ideal, because some type of files
+it's more suitable to process on-the-fly (image thumbnails, document previews),
+while other types of files should be processed in a background job (video
+transcoding, raw images)
+
+Shrine is the first file attachment library that has support for both
+processing on upload and on-the-fly. So, if you're handling image uploads, you
+can choose to either generate a set of pre-defined image thumbnails in a
+background job:
+
+```rb
+class ImageUploader < Shrine
+  process(:store) do |io|
+    versions = { original: io }
+
+    io.download do |original|
+      pipeline = ImageProcessing::MiniMagick.source(original)
+
+      versions[:large]  = pipeline.resize_to_limit!(800, 800)
+      versions[:medium] = pipeline.resize_to_limit!(500, 500)
+      versions[:small]  = pipeline.resize_to_limit!(300, 300)
+    end
+
+    versions
+  end
+end
+```
+
+or generate thumbnails on-demand:
+
+```rb
+class ImageUploader < Shrine
+  derivation :thumbnail do |file, width, height|
+    ImageProcessing::MiniMagick
+      .source(file)
+      .resize_to_limit!(width.to_i, height.to_i)
+  end
+end
+```
+```rb
+photo.image.derivation_url(:thumbnail, "600", "400")
+#=> ".../thumbnail/600/400/eyJpZCI6ImZvbyIsInN0b3JhZ2UiOiJzdG9yZSJ9?signature=..."
+```
+
+### Image processing
+
+Many file attachment libraries, such as CarrierWave, Paperclip, Dragonfly and
+Refile, implement their own image processing macros. Instead of creating
+yet another in-house implementation, the **[ImageProcessing]** gem was created.
+
+Even though the ImageProcessing gem was created for Shrine, it's completely
+generic and can be used standalone, or in any other file upload library (e.g.
+Active Storage uses it now as well). It takes care of many details for you,
+such as [auto orienting] the input image and [sharpening] the thumbnails after
+they are resized.
 
 ```rb
 require "image_processing"
@@ -147,14 +221,15 @@ thumbnail = ImageProcessing::MiniMagick
 thumbnail #=> #<Tempfile:/var/folders/.../image_processing20180316-18446-1j247h6.png>
 ```
 
-### libvips
+#### libvips
 
 Probably the biggest ImageProcessing feature is the support for **[libvips]**.
-libvips is also a full-featured image processing library, which can process
-images very rapidly – often multiple times faster than ImageMagick – with low
-memory usage (see [Why is libvips quick]). The `ImageProcessing::Vips` backend
-implements the same API as `ImageProcessing::MiniMagick`, so you can easily
-swap one for the other.
+libvips is a full-featured image processing library like ImageMagick, with
+impressive performance characteristics – it's often **multiple times faster**
+than ImageMagick and has low memory usage (see [Why is libvips quick]).
+
+The `ImageProcessing::Vips` backend implements the same API as
+`ImageProcessing::MiniMagick`, so you can easily swap one for the other.
 
 ```rb
 require "image_processing/mini_magick"
@@ -172,28 +247,35 @@ ImageProcessing::Vips.resize_to_fit(800, 800).call(original)
 
 ### Other processors
 
-Shrine's processing block simply executes the Ruby code inside it, so you can
-call there any other processor your want. The only thing that Shrine requires
-is that processed files are returned as the block result.
+Both processing "on upload" and "on-the-fly" work in a way that you define a
+Ruby block, which accepts a source file and is expected to return a processed
+file. How you're going to do the processing is entirely up to you.
+
+This allows you to use any tool you want. For example, you could use the
+[image_optim] gem to perform additional image optimizations:
 
 ```rb
 class VideoUploader < Shrine
-  plugin :processing
+  derivation :thumbnail do |file, width, height|
+    thumbnail = ImageProcessing::MiniMagick
+      .source(file)
+      .resize_to_limit!(width, height)
+
+    image_optim = ImageOptim.new
+    image_optim.optimize_image!(thumbnail.path)
 
-  process(:store) do |io, context|
-    # define your processing
+    thumbnail
   end
 end
 ```
 
-## Metadata
+## Metadata & Validation
 
 Shrine automatically [extracts metadata][metadata] from each uploaded file,
-including derivates like image thumbnails, and saves them into the database
+including derivatives like image thumbnails, and saves them into the database
 column. In addition to filename, filesize, and MIME type that are extracted by
 default, you can also extract [image dimensions][store_dimensions], or your own
-[custom metadata][add_metadata]. This metadata can additionally be
-[validated][validation].
+[custom metadata][add_metadata].
 
 ```rb
 photo.image.metadata #=>
@@ -207,61 +289,61 @@ photo.image.metadata #=>
 # }
 ```
 
-## Direct Uploads
+For common metadata there are already [validation macros][validation_helpers],
+but you can also [validate any custom metadata][custom validations].
 
-Instead of submitting selected files synchronously via the form, it's generally
-better to start uploading files asynchronously as soon as they're selected.
-Shrine streamlines this workflow, allowing you to upload directly [to your
-app][upload_endpoint] or [to the cloud][presign_endpoint].
-
-[Refile] and [Active Storage] provide this functionality as well, and they also
-ship with a custom plug-and-play JavaScript solution for integrating these
-endpoints. In contrast, Shrine doesn't ship with any custom JavaScript, but
-instead recommends using **[Uppy]**. Uppy is a flexible JavaScript file upload
-library that allows uploading to a [custom endpoint][XHRUpload], to [AWS
-S3][AwsS3], or even to a [resumable endpoint][Tus]. It comes with a set of UI
-components, ranging from a simple [status bar][StatusBar] to a full-featured
-[dashboard][Dashboard]. Since Uppy is maintained by the whole JavaScript
-community, it will generally be better than any homegrown solution.
+```rb
+class DocumentUploader < Shrine
+  Attacher.validate do
+    # validation macros
+    validate_max_size 10*1024*1024
+    validate_mime_type_inclusion %W[application/pdf]
+
+    # custom validations
+    if get.metadata["page_count"] > 30
+      errors << "has too many pages (max is 30)"
+    end
+  end
+end
+```
 
 ## Backgrounding
 
 In most file upload solutions background processing was an afterthought, which
 resulted in complex implementations. Shrine was designed with backgrounding
-feature in mind from day one. It is supported via the `backgrounding` plugin
-and can be used with [any backgrounding library][backgrounding libraries].
-
-## Large Files
+feature in mind from day one. It is supported via the
+[`backgrounding`][backgrounding] plugin and can be used with [any backgrounding
+library][backgrounding libraries].
 
-If your application needs to handle large files (such as videos), Shrine will
-go out of the way to make this as resilient and performant as possible.
-
-### Streaming
+## Direct Uploads
 
-Shrine uses and encourages streaming uploads and downloads, where only a small
-part of the file is loaded into memory at any given time. This means that
-Shrine will use very little memory regardless of the size of the files.
+Shrine doesn't come with a plug-and-play JavaScript solution for client-side
+uploads like Refile and Active Storage, but instead it adopts **[Uppy]**. Uppy
+is a modern JavaScript file upload library, which offers support for uploading
+to [AWS S3][Uppy AwsS3], to a [custom endpoint][Uppy XHRUpload], or even to a
+[resumable endpoint][Uppy Tus]. It comes with a set of UI components, ranging
+from a simple [status bar][Uppy StatusBar] to a full-featured [dashboard][Uppy
+Dashboard]. Since Uppy is maintained by the wide JavaScript community, it's
+generally a better choice than any homegrown solution.
 
-Shrine storages also automatically support [partial downloads][Down streaming]
-(provided by the [Down] gem), which allows you to read only a portion of the
-file. This can be useful for extracting metadata, because common information such
-as MIME type or image dimensions are typically written in the beginning of the
-file, so it's enough to download just the first few kilobytes of the file.
+Shrine provides Rack components for uploads that integrate nicely with Uppy.
+So, whether you want Uppy to upload directly [to your app][upload_endpoint], or
+you want to authorize direct uploads [to the cloud][presign_endpoint], Shrine
+has it streamlined.
 
 ### Resumable uploads
 
-Another challenge with large files is that it can be difficult for your users
-to upload them to your app, especially on flaky internet connections. Since by
-default an upload is made in a single long HTTP request, any connection
-failures will cause the upload to fail and have to be restarted from the
-beginning.
+If your users are uploading large files, flaky internet connections can cause
+uploads to fail halfway, which can be a frustrating user experience. To fix
+this problem, [Transloadit] company has created an open HTTP-based protocol for
+resumable uploads – **[tus]**. There are already countless client and server
+[implementations][tus implementations] of the protocol in various languages.
 
-To fix this problem, [Transloadit] company has created an open HTTP-based
-protocol for resumable uploads – **[tus]**. To use it, you can choose from
-numerous client and server [implementations][tus implementations] of the
-protocol. In a typical app you would have a [JavaScript client][tus-js-client]
-(via [Uppy][uppy tus]) upload to a [Ruby server][tus-ruby-server], and then
-attach uploaded files using the handy [Shrine integration][shrine-tus].
+So, if you're expecting large file uploads, you can use Uppy as a [JavaScript
+client][Uppy Tus] and have it upload to [Ruby server][tus-ruby-server], then
+attach uploaded files using the handy [Shrine integration][shrine-tus]. Shrine
+handles uploads and downloads in a streaming fashion, so you can expect low
+memory usage.
 
 Alternatively, you can have [resumable multipart uploads directly to
 S3][uppy-s3_multipart].
@@ -282,11 +364,8 @@ than relying on the `Content-Type` request header), preventing exploits like
 
 The `remote_url` plugin requires specifying a `:max_size` option, which limits
 the maximum allowed size of the remote file. The [Down] gem which the
-`remote_url` plugin uses will immediately terminate the download if it reads
-from the `Content-Length` response header that the file will be too large. For
-chunked responses (where `Content-Length` header is absent) the download will
-will be terminated as soon as the received content surpasses the specified
-limit.
+`remote_url` plugin uses will [terminate the download early][Down max size]
+when it realizes it's too large.
 
 [Paperclip]: https://github.com/thoughtbot/paperclip
 [CarrierWave]: https://github.com/carrierwaveuploader/carrierwave
@@ -303,7 +382,8 @@ limit.
 [ROM]: http://rom-rb.org
 [Hanami::Model]: https://github.com/hanami/model
 [plugin system]: https://twin.github.io/the-plugin-system-of-sequel-and-roda/
-[Down]: https://github.com/janko-m/down
+[Down]: https://github.com/janko/down
+[ContentDisposition]: https://github.com/shrinerb/content_disposition
 [`file`]: http://linux.die.net/man/1/file
 [FileMagic]: https://github.com/blackwinter/ruby-filemagic
 [FastImage]: https://github.com/sdsykes/fastimage
@@ -313,36 +393,38 @@ limit.
 [mini_mime]: https://github.com/discourse/mini_mime
 [MiniMagick]: https://github.com/minimagick/minimagick
 [ruby-vips]: https://github.com/libvips/ruby-vips
-[God objects]: https://en.wikipedia.org/wiki/God_object
+[god objects]: https://en.wikipedia.org/wiki/God_object
 [ImageMagick]: https://www.imagemagick.org
 [refile-mini_magick]: https://github.com/refile/refile-mini_magick
-[ImageProcessing]: https://github.com/janko-m/image_processing
+[ImageProcessing]: https://github.com/janko/image_processing
 [auto orienting]: https://www.imagemagick.org/script/command-line-options.php#auto-orient
 [sharpening]: https://photography.tutsplus.com/tutorials/what-is-image-sharpening--cms-26627
 [libvips]: http://libvips.github.io/libvips/
 [Why is libvips quick]: https://github.com/libvips/libvips/wiki/Why-is-libvips-quick
-[metadata]: https://shrinerb.com/rdoc/files/doc/metadata_md.html
-[store_dimensions]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/StoreDimensions.html
-[add_metadata]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/AddMetadata.html
-[validation]: https://shrinerb.com/rdoc/files/doc/validation_md.html
-[upload_endpoint]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/UploadEndpoint.html
-[presign_endpoint]: https://shrinerb.com/rdoc/classes/Shrine/Plugins/PresignEndpoint.html
+[metadata]: /doc/metadata.md#readme
+[store_dimensions]: /doc/plugins/store_dimensions.md#readme
+[add_metadata]: /doc/plugins/add_metadata.md#readme
+[validation]: /doc/validation.md#readme
+[upload_endpoint]: /doc/plugins/upload_endpoint.md#readme
+[presign_endpoint]: /doc/plugins/presign_endpoint.md#readme
 [Uppy]: https://uppy.io
-[XHRUpload]: https://uppy.io/docs/xhrupload/
-[AwsS3]: https://uppy.io/docs/aws-s3/
-[Tus]: https://uppy.io/docs/tus/
-[StatusBar]: https://uppy.io/examples/statusbar/
-[Dashboard]: https://uppy.io/examples/dashboard/
-[background job]: http://shrinerb.com/rdoc/classes/Shrine/Plugins/Backgrounding.html
-[backgrounding libraries]: https://github.com/shrinerb/shrine/wiki/Backgrounding-libraries
-[Down streaming]: https://github.com/janko-m/down#streaming
+[Uppy XHRUpload]: https://uppy.io/docs/xhrupload/
+[Uppy AwsS3]: https://uppy.io/docs/aws-s3/
+[Uppy Tus]: https://uppy.io/docs/tus/
+[Uppy StatusBar]: https://uppy.io/examples/statusbar/
+[Uppy Dashboard]: https://uppy.io/examples/dashboard/
+[backgrounding]: /doc/plugins/backgrounding.md#readme
+[backgrounding libraries]: https://github.com/shrinerb/shrine/wiki/Backgrounding-Libraries
+[Down streaming]: https://github.com/janko/down#streaming
 [Transloadit]: https://transloadit.com
 [tus]: https://tus.io
 [tus implementations]: https://tus.io/implementations.html
-[tus-js-client]: https://github.com/tus/tus-js-client
-[uppy tus]: https://uppy.io/docs/tus/
-[tus-ruby-server]: https://github.com/janko-m/tus-ruby-server
+[tus-ruby-server]: https://github.com/janko/tus-ruby-server
 [shrine-tus]: https://github.com/shrinerb/shrine-tus
 [ImageTragick]: https://imagetragick.com
-[uppy-s3_multipart]: https://github.com/janko-m/uppy-s3_multipart
+[uppy-s3_multipart]: https://github.com/janko/uppy-s3_multipart
 [OWASP]: https://www.owasp.org/index.php/Unrestricted_File_Upload
+[image_optim]: https://github.com/toy/image_optim
+[validation_helpers]: /doc/plugins/validation_helpers.md#readme
+[custom validations]: /doc/validation.md#custom-validations
+[Down max size]: https://github.com/janko/down#maximum-size