shrine 2.19.3 → 3.6.0

data/doc/metadata.md

@@ -1,4 +1,6 @@
-# Extracting Metadata
+---
+title: Extracting Metadata
+---
 
 Before a file is uploaded, Shrine automatically extracts metadata from it, and
 stores them in the `Shrine::UploadedFile` object.
@@ -13,6 +15,18 @@ uploaded_file.metadata #=>
 # }
 ```
 
+Under the hood, `Shrine#upload` calls `Shrine#extract_metadata`, which you can
+also use directly to extract metadata from any IO object:
+
+```rb
+uploader.extract_metadata(io) #=>
+# {
+#   "size" => 345993,
+#   "filename" => "matrix.mp4",
+#   "mime_type" => "video/mp4",
+# }
+```
+
 The following metadata is extracted by default:
 
 | Key         | Default source                                     |
@@ -21,7 +35,9 @@ The following metadata is extracted by default:
 | `mime_type` | extracted from `io.content_type`                   |
 | `size`      | extracted from `io.size`                           |
 
-You can access extracted metadata in three ways:
+## Accessing metadata
+
+You can access the stored metadata in three ways:
 
 ```rb
 # via methods (if they're defined)
@@ -40,17 +56,7 @@ uploaded_file["filename"]
 uploaded_file["mime_type"]
 ```
 
-Under the hood, `Shrine#upload` calls `Shrine#extract_metadata`, which you can
-also use directly to extract metadata from any IO object:
-
-```rb
-uploader.extract_metadata(io) #=>
-# {
-#   "size" => 345993,
-#   "filename" => "matrix.mp4",
-#   "mime_type" => "video/mp4",
-# }
-```
+## Controlling extraction
 
 `Shrine#upload` accepts a `:metadata` option which accepts the following values:
 
@@ -83,11 +89,11 @@ By default, the `mime_type` metadata will be copied over from the
 `#content_type` attribute of the input file (if present). However, since
 `#content_type` value comes from the `Content-Type` header of the upload
 request, it's *not guaranteed* to hold the actual MIME type of the file (browser
-determines this header based on file extension). Moreover, only
-`ActionDispatch::Http::UploadedFile`, `Shrine::Plugins::RackFile::UploadedFile`,
-and `Shrine::Plugins::DataUri::DataFile` objects have `#content_type` defined,
-so, when uploading simple file objects, `mime_type` will be nil. That makes
-relying on `#content_type` both a security risk and limiting.
+determines this header based on file extension).
+
+Moreover, only `ActionDispatch::Http::UploadedFile`, `Shrine::RackFile`, and
+`Shrine::DataFile` objects have `#content_type` defined, so when uploading
+objects such as `File`, the `mime_type` value will be nil by default.
 
 To remedy that, Shrine comes with a
 [`determine_mime_type`][determine_mime_type] plugin which is able to extract
@@ -110,14 +116,17 @@ You can choose different analyzers, and even mix-and-match them. See the
 
 ## Image Dimensions
 
-Shrine comes with a `store_dimensions` plugin for extracting image dimensions.
-It adds `width` and `height` metadata values, and also adds `#width`,
-`#height`, and `#dimensions` methods to the `Shrine::UploadedFile` object. By
-default, the plugin uses [FastImage] to analyze dimensions, but you can also
-have it use [MiniMagick] or [ruby-vips]:
+Shrine comes with a [`store_dimensions`][store_dimensions] plugin for
+extracting image dimensions. It adds `width` and `height` metadata values, and
+also adds `#width`, `#height`, and `#dimensions` methods to the
+`Shrine::UploadedFile` object.
 
 ```rb
-Shrine.plugin :store_dimensions, analyzer: :mini_magick
+# Gemfile
+gem "fastimage" # default analyzer
+```
+```rb
+Shrine.plugin :store_dimensions
 ```
 ```rb
 uploaded_file = uploader.upload(image)
@@ -130,26 +139,31 @@ uploaded_file.height     #=> 900
 uploaded_file.dimensions #=> [1600, 900]
 ```
 
+By default, the plugin uses [FastImage] to analyze dimensions, but you can also
+have it use [MiniMagick] or [ruby-vips]. See the
+[`store_dimensions`][store_dimensions] plugin docs for more details.
+
 ## Custom metadata
 
 In addition to the built-in metadata, Shrine allows you to extract and store
-any custom metadata, using the `add_metadata` plugin (which extends
-`Shrine#extract_metadata`). For example, you might want to extract EXIF data
-from images:
+any custom metadata, using the [`add_metadata`][add_metadata] plugin (which
+internally extends `Shrine#extract_metadata`).
+
+For example, you might want to extract EXIF data from images:
 
 ```rb
-require "mini_magick"
+# Gemfile
+gem "exiftool"
+```
+```rb
+require "exiftool"
 
 class ImageUploader < Shrine
   plugin :add_metadata
 
   add_metadata :exif do |io, context|
     Shrine.with_file(io) do |file|
-      begin
-        MiniMagick::Image.new(file.path).exif
-      rescue MiniMagick::Error
-        # not a valid image
-      end
+      Exiftool.new(file.path).to_hash
     end
   end
 end
@@ -161,8 +175,12 @@ uploaded_file.exif             #=> {...}
 ```
 
 Or, if you're uploading videos, you might want to extract some video-specific
-meatadata:
+metadata:
 
+```rb
+# Gemfile
+gem "streamio-ffmpeg"
+```
 ```rb
 require "streamio-ffmpeg"
 
@@ -192,11 +210,21 @@ uploaded_file.metadata #=>
 ```
 
 The yielded `io` object will not always be an object that responds to `#path`.
-If you're using the `data_uri` plugin, the `io` will be a `StringIO` wrapper.
-With `restore_cached_data` or `refresh_metadata` plugins, `io` might be a
-`Shrine::UploadedFile` object. If you're using a metadata analyzer that
-requires the source file to be on disk, you can use `Shrine.with_file` to
-ensure you have a file object.
+For example, with the `data_uri` plugin the `io` can be a `StringIO` wrapper,
+while with `restore_cached_data` or `refresh_metadata` plugins the `io` might
+be a `Shrine::UploadedFile` object. So, we're using `Shrine.with_file` to
+ensure we have a file object.
+
+### Adding metadata
+
+If you wish to add metadata to an already attached file, you can do it as
+follows:
+
+```rb
+photo.image_attacher.add_metadata("foo" => "bar")
+photo.image.metadata #=> { ..., "foo" => "bar" }
+photo.save # persist changes
+```
 
 ## Metadata columns
 
@@ -216,20 +244,23 @@ photo.image_type #=> "image/jpeg"
 When attaching files that were uploaded directly to the cloud or a [tus
 server], Shrine won't automatically extract metadata from them, instead it will
 copy any existing metadata that was set on the client side. The reason why this
-is the default behaviour is because extracting the metadata would require (at
-least partially) retrieving file content from the storage, which could
-potentially be expensive depending on the storage and the type of metadata
-being extracted.
+is the default behaviour is because metadata extraction requires (at least
+partially) retrieving file content from the storage, which could potentially be
+expensive depending on the storage and the type of metadata being extracted.
+
+```rb
+# no additional metadata will be extracted in this assignment by default
+photo.image = '{"id":"9e6581a4ea1.jpg","storage":"cache","metadata":{...}}'
+```
+
+### Extracting on attachment
 
-There are two ways of extracting metadata from directly uploaded files. If you
-want metadata to be automatically extracted on assignment (which is useful if
-you want to validate the extracted metadata or have it immediately available
-for any other reason), you can load the `restore_cached_data` plugin:
+If you want metadata to be automatically extracted on assignment (which is
+useful if you want to validate the extracted metadata or have it immediately
+available for any other reason), you can load the `restore_cached_data` plugin:
 
 ```rb
-class ImageUploader < Shrine
-  plugin :restore_cached_data # automatically extract metadata from cached files on assignment
-end
+Shrine.plugin :restore_cached_data # automatically extract metadata from cached files on assignment
 ```
 ```rb
 photo.image = '{"id":"ks9elsd.jpg","storage":"cache","metadata":{}}' # metadata is extracted
@@ -241,100 +272,147 @@ photo.image.metadata #=>
 # }
 ```
 
-On the other hand, if you're using backgrounding, you can extract metadata
-during background promotion using the `refresh_metadata` plugin (which the
-`restore_cached_data` plugin uses internally):
+### Extracting in the background
+
+#### A) Extracting with promotion
+
+If you're using [backgrounding], you can extract metadata during background
+promotion using the `refresh_metadata` plugin (which the `restore_cached_data`
+plugin uses internally):
 
 ```rb
-class ImageUploader < Shrine
-  plugin :refresh_metadata
-  plugin :processing
+Shrine.plugin :refresh_metadata # allow re-extracting metadata
+Shrine.plugin :backgrounding
+
+Shrine::Attacher.promote_block do
+  PromoteJob.perform_async(self.class.name, record.class.name, record.id, name, file_data)
+end
+```
+```rb
+class PromoteJob
+  include Sidekiq::Worker
 
-  # this will be called in the background if using backgrounding plugin
-  process(:store) do |io, context|
-    io.refresh_metadata!(context) # extracts metadata and updates `io.metadata`
-    io
+  def perform(attacher_class, record_class, record_id, name, file_data)
+    attacher_class = Object.const_get(attacher_class)
+    record         = Object.const_get(record_class).find(record_id) # if using Active Record
+
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.refresh_metadata! # extract metadata
+    attacher.atomic_promote
   end
 end
 ```
 
-If you have metadata that is cheap to extract in the foreground, but also have
-additional metadata that can be extracted asynchronously, you can combine the
-two approaches. For example, if you're attaching video files, you might want to
-extract MIME type upfront and video-specific metadata in a background job, which
-can be done as follows (provided that `backgrounding` plugin is used):
+#### B) Extracting separately from promotion
+
+You can also extract metadata in the background separately from promotion:
 
 ```rb
-class MyUploader < Shrine
-  plugin :determine_mime_type # this will be called in the foreground
-  plugin :restore_cached_data
-  plugin :refresh_metadata
-  plugin :add_metadata
-  plugin :processing
+MetadataJob.perform_async(
+  attacher.class.name,
+  attacher.record.class.name,
+  attacher.record.id,
+  attacher.name,
+  attacher.file_data,
+)
+```
+```rb
+class MetadataJob
+  include Sidekiq::Worker
+
+  def perform(attacher_class, record_class, record_id, name, file_data)
+    attacher_class = Object.const_get(attacher_class)
+    record         = Object.const_get(record_class).find(record_id) # if using Active Record
 
-  # this will be called in the background if using backgrounding plugin
-  process(:store) do |io, context|
-    io.refresh_metadata!(context)
-    io
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.refresh_metadata!
+    attacher.atomic_persist
   end
+end
+```
 
-  add_metadata do |io, context|
-    next unless context[:action] == :store # this will be the case during promotion
+### Combining foreground and background
 
-    Shrine.with_file(io) do |file|
-      # example of metadata extraction
-      movie = FFMPEG::Movie.new(file.path) # uses the streamio-ffmpeg gem
+If you have some metadata that you want to extract in the foreground and some
+that you want to extract in the background, you can use the uploader context:
 
-      { "duration"   => movie.duration,
-        "bitrate"    => movie.bitrate,
-        "resolution" => movie.resolution,
-        "frame_rate" => movie.frame_rate }
-    end
+```rb
+class VideoUploader < Shrine
+  plugin :add_metadata
+
+  add_metadata do |io, **options|
+    next unless options[:background] # proceed only when `background: true` was specified
+
+    # example of metadata extraction
+    movie = Shrine.with_file(io) { |file| FFMPEG::Movie.new(file.path) }
+
+    { "duration"   => movie.duration,
+      "bitrate"    => movie.bitrate,
+      "resolution" => movie.resolution,
+      "frame_rate" => movie.frame_rate }
+  end
+end
+```
+```rb
+class PromoteJob
+  include Sidekiq::Worker
+
+  def perform(attacher_class, record_class, record_id, name, file_data)
+    attacher_class = Object.const_get(attacher_class)
+    record         = Object.const_get(record_class).find(record_id) # if using Active Record
+
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.refresh_metadata!(background: true) # specify the flag
+    attacher.atomic_promote
   end
 end
 ```
 
+Now triggering metadata extraction in the controller on attachment (using
+`restore_cached_data` or `refresh_metadata` plugin) will skip the video
+metadata block, which will be triggered later in the background job.
+
+### Optimizations
+
 If you want to do both metadata extraction and file processing during
 promotion, you can wrap both in an `UploadedFile#open` block to make
 sure the file content is retrieved from the storage only once.
 
 ```rb
-class MyUploader < Shrine
-  plugin :refresh_metadata
-  plugin :processing
+class PromoteJob
+  include Sidekiq::Worker
+
+  def perform(attacher_class, record_class, record_id, name, file_data)
+    attacher_class = Object.const_get(attacher_class)
+    record         = Object.const_get(record_class).find(record_id) # if using Active Record
 
-  process(:store) do |io, context|
-    io.open do |io, context|
-      io.refresh_metadata!(context)
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
 
-      original = io.download # reuses already open uploaded file
-      # ... processing ...
+    attacher.file.open do
+      attacher.refresh_metadata!
+      attacher.create_derivatives
     end
+
+    attacher.atomic_promote
   end
 end
 ```
 
-If you're dealing with large files, it's recommended to also use the `tempfile`
-plugin to make sure the same copy of the uploaded file is used for metadata
-extraction (`Shrine.with_file`) and processing (`UploadedFile#tempfile`).
+If you're dealing with large files and have metadata extractors that use
+`Shrine.with_file`, you might want to use the `tempfile` plugin to make sure
+the same copy of the uploaded file is reused for both metadata extraction and
+file processing.
 
 ```rb
 Shrine.plugin :tempfile # load it globally so that it overrides `Shrine.with_file`
 ```
 ```rb
-class MyUploader < Shrine
-  plugin :refresh_metadata
-  plugin :processing
-
-  process(:store) do |io, context|
-    io.open do |io, context|
-      io.refresh_metadata!(context)
-
-      original = io.tempfile # used the cached tempfile
-      # ... processing ...
-    end
-  end
+# ...
+attacher.file.open do
+  attacher.refresh_metadata!
+  attacher.create_derivatives(attacher.file.tempfile)
 end
+# ...
 ```
 
 [`file`]: http://linux.die.net/man/1/file
@@ -344,4 +422,7 @@ end
 [MiniMagick]: https://github.com/minimagick/minimagick
 [ruby-vips]: https://github.com/libvips/ruby-vips
 [tus server]: https://github.com/janko/tus-ruby-server
-[determine_mime_type]: /doc/plugins/determine_mime_type.md#readme
+[determine_mime_type]: https://shrinerb.com/docs/plugins/determine_mime_type
+[store_dimensions]: https://shrinerb.com/docs/plugins/store_dimensions
+[add_metadata]: https://shrinerb.com/docs/plugins/add_metadata
+[backgrounding]: https://shrinerb.com/docs/plugins/backgrounding

data/doc/multiple_files.md

@@ -1,4 +1,10 @@
-# Multiple Files
+---
+id: multiple-files
+title: Multiple Files
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
 
 There are times when you want to allow users to attach multiple files to a
 single resource, like an album having many photos or a playlist having many
@@ -64,24 +70,32 @@ files (or attachments) table will be the photos table.
 Let's create a table for the main resource and attachments, and add a foreign
 key in the attachment table for the main table:
 
+<Tabs>
+<TabItem value="sequel" label="Sequel">
+
 ```rb
-# with Sequel:
 Sequel.migration do
   change do
     create_table :albums do
       primary_key :id
-      column      :title, :text
+
+      String :title
     end
 
     create_table :photos do
       primary_key :id
       foreign_key :album_id, :albums
-      column      :image_data, :text
+
+      String :image_data
     end
   end
 end
+```
+
+</TabItem>
+<TabItem value="activerecord" label="Active Record">
 
-# with Active Record:
+```rb
 class CreateAlbumsAndPhotos < ActiveRecord::Migration
   def change
     create_table :albums do |t|
@@ -98,21 +112,33 @@ class CreateAlbumsAndPhotos < ActiveRecord::Migration
 end
 ```
 
+</TabItem>
+</Tabs>
+
 In the Photo model, create a Shrine attachment attribute named `image`
 (`:image` matches the `_data` column prefix above):
 
+<Tabs>
+<TabItem value="sequel" label="Sequel">
+
 ```rb
-# with Sequel:
 class Photo < Sequel::Model
-  include ImageUploader::Attachment.new(:image)
+  include ImageUploader::Attachment(:image)
 end
+```
 
-# with Active Record:
+</TabItem>
+<TabItem value="activerecord" label="Active Record">
+
+```rb
 class Photo < ActiveRecord::Base
-  include ImageUploader::Attachment.new(:image)
+  include ImageUploader::Attachment(:image)
 end
 ```
 
+</TabItem>
+</Tabs>
+
 ### 2. Declare nested attributes
 
 Using nested attributes is the easiest way to implement any dynamic
@@ -120,8 +146,10 @@ Using nested attributes is the easiest way to implement any dynamic
 relationship to the photos table, and allow it to directly accept attributes
 for the associated photo records by enabling nested attributes:
 
+<Tabs>
+<TabItem value="sequel" label="Sequel">
+
 ```rb
-# with Sequel:
 class Album < Sequel::Model
   one_to_many :photos
   plugin :association_dependencies, photos: :destroy # destroy photos when album is destroyed
@@ -129,14 +157,32 @@ class Album < Sequel::Model
   plugin :nested_attributes
   nested_attributes :photos, destroy: true
 end
+```
 
-# with Active Record:
+</TabItem>
+<TabItem value="activerecord" label="Active Record">
+
+```rb
 class Album < ActiveRecord::Base
   has_many :photos, dependent: :destroy
   accepts_nested_attributes_for :photos, allow_destroy: true
 end
 ```
 
+</TabItem>
+<TabItem value="mongoid" label="Mongoid">
+
+```rb
+class Album
+  include Mongoid::Document
+  embeds_many :photos
+  accepts_nested_attributes_for :photos
+end
+```
+
+</TabItem>
+</Tabs>
+
 Documentation on nested attributes:
 
 * [`Sequel::Model.nested_attributes`]
@@ -152,8 +198,26 @@ already created photos, so that the same form can be used for updating the
 album/photos as well (they will be submitted under the
 `album[photos_attributes]` parameter).
 
+<Tabs>
+<TabItem value="rails" label="Rails form builder">
+
+```rb
+form_for @album, html: { enctype: "multipart/form-data" } do |f|
+  f.text_field :title
+  f.fields_for :photos do |p| # adds new `album[photos_attributes]` parameter
+    p.hidden_field :image, value: p.object.cached_image_data, id: nil
+    p.file_field   :image
+    p.check_box    :_destroy unless p.object.new_record?
+  end
+  file_field_tag "files[]", multiple: true
+  f.submit "Create"
+end
+```
+
+</TabItem>
+<TabItem value="forme" label="Forme">
+
 ```rb
-# with Forme:
 form @album, action: "/photos", enctype: "multipart/form-data" do |f|
   f.input :title
   f.subform :photos do # adds new `album[photos_attributes]` parameter
@@ -164,20 +228,11 @@ form @album, action: "/photos", enctype: "multipart/form-data" do |f|
   f.input "files[]", type: :file, attr: { multiple: true }, obj: nil
   f.button "Create"
 end
-
-# with Rails form builder:
-form_for @album, html: { enctype: "multipart/form-data" } do |f|
-  f.text_field :title
-  f.fields_for :photos do |p| # adds new `album[photos_attributes]` parameter
-    p.hidden_field :image, value: p.object.cached_image_data
-    p.file_field   :image
-    p.check_box    :_destroy unless p.object.new_record?
-  end
-  file_field_tag "files[]", multiple: true
-  f.submit "Create"
-end
 ```
 
+</TabItem>
+</Tabs>
+
 In your controller you should still be able to assign all the attributes to the
 album, just remember to whitelist the new parameter for the nested attributes,
 in this case `album[photos_attributes]`.
@@ -203,7 +258,7 @@ photos_attributes = album_params[:photos_attributes].to_h.merge(new_photos_attri
 album_attributes  = album_params.merge(photos_attributes: photos_attributes)
 
 # create the album with photos
-Album.create(album_params)
+Album.create(album_attributes)
 ```
 
 In this case you need to make sure your form tag has
@@ -257,22 +312,30 @@ class ImageUploader < Shrine
 
   Attacher.validate do
     validate_max_size 10*1024*1024
-    validate_mime_type_inclusion %w[image/jpeg image/png]
+    validate_mime_type %w[image/jpeg image/png image/webp]
   end
 end
 ```
+<Tabs>
+<TabItem value="sequel" label="Sequel">
+
 ```rb
-# with Sequel:
 class Album < Sequel::Model
   # ... (nested_attributes already enables validating associated photos) ...
 end
+```
+
+</TabItem>
+<TabItem value="activerecord" label="Active Record">
 
-# with ActiveRecord:
+```rb
 class Album < ActiveRecord::Base
   # ...
   validates_associated :photos
 end
 ```
+</TabItem>
+</Tabs>
 
 Note that by default only metadata set on the client side will be available for
 validations. Shrine will not automatically run metadata extraction for directly
@@ -294,8 +357,8 @@ attributes feature gives you for free.
 [`Sequel::Model.nested_attributes`]: http://sequel.jeremyevans.net/rdoc-plugins/classes/Sequel/Plugins/NestedAttributes.html
 [`ActiveRecord::Base.accepts_nested_attributes_for`]: http://api.rubyonrails.org/classes/ActiveRecord/NestedAttributes/ClassMethods.html
 [`Mongoid::Document.accepts_nested_attributes_for`]: https://docs.mongodb.com/mongoid/master/tutorials/mongoid-nested-attributes/
-[`upload_endpoint`]: /doc/plugins/upload_endpoint.md#readme
-[`presign_endpoint`]: /doc/plugins/presign_endpoint.md#readme
+[`upload_endpoint`]: https://shrinerb.com/docs/plugins/upload_endpoint
+[`presign_endpoint`]: https://shrinerb.com/docs/plugins/presign_endpoint
 [Uppy]: https://uppy.io
 [direct app uploads]: https://github.com/shrinerb/shrine/wiki/Adding-Direct-App-Uploads
 [direct S3 uploads]: https://github.com/shrinerb/shrine/wiki/Adding-Direct-S3-Uploads